Thursday, May 31, 2012

CentOS Serial Console Server with Digi AccelePort Xem Module

In order to effectively manage UNIX, Linux and Cisco machines from a remote location, one needs to redirect the console to the serial port and hook this to a serial console server. It is also good on a security stand point because all the messages sent to the console are logged on the console server, thus serving in forensic investigations. Ideally, to have even the BIOS at the serial ports, you need to run Oracle/Sun Microsystems or IBM pServers machines. Most newer x86 servers can redirect their BIOS to the serial port. Do it, it's great! With the console server, you have access to everything the server has to offer from the comfort of your office.


Server Setup

Server Hardware


To create a console server, you can either purchase one from Avocent, Cyclades or the like. Or you can use a standard Linux machine in which you install serial port cards. The Digi port serial cards called Digiboard PCI PC/Xem are great for this purpose. To use the Digi products, get these items from Digi :
  • Digi part number 70001757. This is the AccelePort Xem Universal PCI (3.3V & 5V) 16-port RS-232 RJ-45 1U 19" rack with a MSRP of 1249 US$. This product number is for a bundle which contains the PCI card, the cable and a 1U rackmount RJ45 RS-232 port concentrator.
  • Digi part number 76000617. This is the Digi PORTS/16em Module Rack Mount RJ-45 with a MSRP of 722$US. This product is just an extra 1U rackmount RJ45 RS-232 port concentrator with the cable.
The nice thing about these Digi cards and concentrators is that with a single card, you can daisy-chain up to 4 concentrators. Note that the third and fourth concentrators will need an extra power-supply unit (sold by Digi of course).

Server Software Installation


Start by installing a Minimal CentOS 6 machine. Once it's up, edit the blacklist.conf file to add the « blacklist epca » line to prevent some errors later.

sudo vi /etc/modprobe.d/blacklist.conf

Reboot the machine to make sure this new blacklisted module will not be loaded.

sudo shutdown -r now

Install some packages required to build software. Here we also install the screen package which is pivotal in the use of the console server.

sudo yum -y install rpm-build make gcc g++ kernel-devel ncurses-devel screen

Download the latest module code.

wget ftp://ftp1.digi.com/support/beta/linux/dgap/dgap-1.3-22.src.rpm

Install the source rpm. We can then delete the source rpm after since we won't be needing it anymore.

sudo rpm -Uvh dgap*.src.rpm
rm dgap-1.3-22.src.rpm

Build the module.

sudo rpmbuild -ba /root/rpmbuild/SPECS/dgap-1.3.spec

Install the new package. Once it's installed, it will prompt you to run the mpi command to configure the driver. Don't do this right now! We need to get the module loaded first.

sudo rpm -Uvh /root/rpmbuild/RPMS/i386/dgap-1.3-22.i386.rpm

Make sure the new module is loaded.

sudo modprobe dgap
lsmod | grep dgap

Now run the mpi Digi configuration tool. This tool has a ncurses GUI.

sudo mpi

Follow these steps :

Config
Are you sure you want to continue? Yes
Do you want ID letters picked for you? Yes
How many adapters do you want to install? <select the number of PCI cards you have, normally just one>
What type of adapter do you have? 2 AccelePort Xem PCI
How many modules are connected to the adapter? <choose how many you have>
How many ports are on the EBI module 1? 16 ports
Is this configuration acceptable? Yes
Altpin : Is this acceptable? Yes
Would you like to load the driver with this configuration now? Yes

At this point the dgap module will unload and reload. Each time you will be prompted to say « Ok ». Then simply « Exit » from the utility.

We should now see those lines in our /var/log/messages file :

May 31 13:43:00 hostname kernel: dgap: board 0: AccelePort XEM (rev 1), irq 20
May 31 13:51:51 hostname kernel: dgap: dgap-1.3-22, Digi International Part Number 40002347_C
May 31 13:51:51 hostname kernel: dgap: For the tools package or updated drivers please visit http://www.digi.com
May 31 13:51:51 hostname kernel: dgap: board 0: AccelePort XEM (rev 1), irq 20

And some new devices in our /dev directory :

ls /dev/ttya*
/dev/ttya01  /dev/ttya03  /dev/ttya05  /dev/ttya07  /dev/ttya09  /dev/ttya11  /dev/ttya13  /dev/ttya15
/dev/ttya02  /dev/ttya04  /dev/ttya06  /dev/ttya08  /dev/ttya10  /dev/ttya12  /dev/ttya14  /dev/ttya16

Server Software Configuration


Now that we have both the hardware and software installed in our new console server, we need to configure it. We will use the screen software to jump from one console to the next. It will also enable us to log everything that happens at the console even when we're not using it.

So let's first create some directories.

sudo mkdir -p /etc/console/RCS
sudo mkdir -p /etc/console/screenrc
sudo mkdir -p /var/log/console

Make sure our future console logs don't grow too big.

sudo vi /etc/logrotate.d/console
# /etc/logrotate.d/console
#
# Clean up serial console log files.
#
# David Robillard, May 31st, 2012.

/var/log/console/*.log {
weekly
missingok
rotate 7
compress
delaycompress
notifempty
create 640 root root
}
# EOF

Check to make sure this new configuration file is ok?

sudo logrotate -d /etc/logrotate.conf

Now create a global configuration file for the serial console server. An example ports.conf file can be found in my DropBox account as you click on the URL.


Create a startup script. Again, the content of the script can be found on my DropBox account as you click on the URL.


Server Kernel Upgrade


A minor annoyance with this setup is that when we upgrade the console server's kernel, we have to rebuild the module. Here's how to do so :

Recreate the module.

sudo rpmbuild -ba /root/rpmbuild/SPECS/dgap-1.3.spec

Install the new package using --force because it's actually already installed.

sudo rpm -Uvh --force /root/rpmbuild/RPMS/i386/dgap-1.3-22.i386.rpm

Fix udev configuration.

sudo cp /etc/dgap/10-dgap.rules /etc/udev/rules.d

Re-run the configuration utility (see above on how to do this).

sudo mpi

Restart the serial consoles.

sudo /etc/init.d/console restart

Client Configuration


Client Hardware


The serial port (i.e. COM1 in PC language) of each clients must be fitted with a DB9 to RJ45 adapter sometimes refered to as null-modem adapters. It's easy to buy adapters that don't have the DB9 connecter attached to the RJ45 cables. If that case, connect the DB9 connector like this :

  1. blue RJ45 cable to DB9 pin number 1.
  2. red RJ45 cable to DB9 pin number 2.
  3. green RJ45 cable to DB9 pin nunber 3.
  4. white RJ45 cable to DB9 pin number 4.
  5. yellow RJ45 cable to DB9 pin number 5.
  6. orange RJ45 cable to DB9 pin number 6.
  7. dark brown RJ45 cable to DB9 pin number 7.
  8. black RJ45 cable to DB9 pin number 8.

The DB9 pin number 9 is not connected to anything.

Close the DB9 connector and then place it on the client's serial 0 port (or COM1 in PC language). Connect the RJ45 cable into a port in the Digi PORTS/16em serial console module.

Important : take a note of which port you connect which server. You need this to configure the /etc/console/ports.conf file on the server.

Client OS Configuration


Make sure you modify the /boot/grub/grub.conf file to enable serial console output. The modified lines are listed in bold.

sudo vi /boot/grub/grub.conf

# /boot/grub/grub.conf
#
# Grub configuration file. See grub(8).
#
# David Robillard, April 20th, 2012.

serial -unit=1 --speed=9600
terminal --timeout=8 console serial
default=0
timeout=5

#splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-220.17.1.el6.i686)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-220.17.1.el6.i686 ro root=UUID=473422f2-caf3-4eed-bcb7-edbbab98a7b2 nomodeset rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_LVM rd_NO_DM console=tty console=ttyS0,9660n8
initrd /boot/initramfs-2.6.32-220.17.1.el6.i686.img
title CentOS (2.6.32-220.el6.i686)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-220.el6.i686 ro root=UUID=473422f2-caf3-4eed-bcb7-edbbab98a7b2 nomodeset rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_NO_LVM rd_NO_DM console=tty console=ttyS0,9660n8
initrd /boot/initramfs-2.6.32-220.el6.i686.img

# EOF

Don't worry, the console=tty console=ttyS0,9660n8 portion of the kernel line are installed when the kernel is upgraded.

Next make sure the ttyS0 console exists in the /etc/securetty file. Again, the change to this file is listed in bold.

sudo vi /etc/securetty
console
vc/1
vc/2
vc/3
vc/4
vc/5
vc/6
vc/7
vc/8
vc/9
vc/10
vc/11
tty1
tty2
tty3
tty4
tty5
tty6
tty7
tty8
tty9
tty10
tty11
ttyS0

Once this is done, simply reboot the client.

sudo shutdown -r now

Console Usage


Connect the RJ45 cables from the module, record that information in the /etc/console/ports.conf file on the server. 

sudo vi /etc/console/ports.conf

Start the serial consoles.

sudo /etc/init.d/console start

Check which consoles have been started? Here we only have a single one active.

sudo screen -ls
There is a screen on:
5619.client (Detached)
1 Socket in /var/run/screen/S-root.

If we need to grab the client's console, we simply go like this :

sudo screen -r client

That will dump us into the client's console. We can exit from the console by striking the escape sequence which is « Ctrl-X D » which means « Control-X Detach ». So we first hit « Ctrl-X » and then hit « D ».

For those of us familiar with screen, you will notice that the escape sequence was changed from the default « Ctrl-A ». The reason for this is to be able to work inside a screen and still grab a console. In effect using a screen inside another screen. We thus need a way to differenciate between the two screen escape sequences.

HTH,

DA+

2 comments:

  1. We're using this instead of the DIGI-Card:
    http://www.delock.de/produkte/G_61519/merkmale.html

    - saves a _lot_ of money!
    - no external kernel modules needed
    - no break signal when switching off the console server
    (the DIGI Xem sends a break if you swich off it's power
    and that makes all Ciscos and SPARCs fall into their
    firmware prompts....)

    Furthermore we're using http://packages.debian.org/squeeze/conserver-server instead of screen. Try it, it's a cool piece of software.

    ReplyDelete
    Replies
    1. Hi Anonymous and thanks for the info. I like the product you showed me, it's probably quite a lot cheaper than the Digi Xem cards. Plus you have quite a lot of consoles on a single USB port. I'll certainly do a test drive!

      I've used conserver in the past but I still prefer to run screen. I've been using screen for 14 years now, so I guess I'm just used to it :) I also like the fact that you can detach / attach to a running screen. When you turn on logging and detach from a screen, you can see everything that goes at the console while you're away. It's great for security.

      Cheers,

      DA+

      Delete