iDrac, RACADM, sshpass, and BASH

If it were up to me, I suppose that the only thing that I’d really be responsible for would be core networking infrastructure (and consumption of craft brews).  Unfortunately that is not the case.  These days (and since the inception of the NetEng), Network Engineers are presumably responsible for anything that even remotely network related.  Server needs and IP?  Bluetooth whatchmacalit isn’t pairing with the thingamadoober?  Toaster with an IP address is constantly burning toast?  While we have and provide tools so that most people can handle these things on their own, unfortunately we still end up with a lot of these tasks.

When we ultimately get saddled with these tasks, the best thing that we can do is find the quickest/most efficient way to handle them.

Enter Dell and the OpenStack project that my team is undertaking. My main responsibilities in this project are handling the build-out of the Underlay network (you know, what we used to call ‘The Network’) as well as SDN components of OpenStack.  Remember the first paragraph where I note that the network guys ultimately handle anything remotely network related?  Yeah.

Now first, let me state this: I work for a large org with a small team of what I would consider to be Special Forces of Infrastructure, unfortunately there just aren’t enough of us.  So my normal server guy didn’t have the cycles to spare when the vendor we chose to implement OpenStack required us to provide the MAC addresses for all of the network interfaces of the 100+ Dell servers we just got.

So, it fell on me to GSD.

After creating list of IP’s to assign to the myriad of servers, and working with remote hands to get IP connectivity working, I still had to set various other settings, as well as get the requested MAC address information.

Enter RACADM.

For those who don’t know, you can actually SSH into iDrac and be met with a bevvy of (RACADM) commands at your disposal for getting data, as well as setting it.

My tasks were set as follows:

  1. Set the hostnames for iDrac on all of the servers (remember remote-hands only did the base IP configuration)
  2. Set the DNS servers and enable dynamic registration of hostnames (granted we could create static A records with powershell, but that wasn’t what we wanted and would require a separate workflow)
  3. Set iDrac to use a Tagged VLAN and run off of LOM3 (this enabled us to utilize 3 cables instead of 4 [2x 10GBE for data, 1x 1GBE for PXE/admin/iDrac instead of dedicating a second 1GBE for iDrac])
  4. Enable PXE booting from LOM3
  5. Get the MAC addresses for LOM3

So in order to facilitate all of this I needed to be able to ssh into all of these machines in an automated fashion-  sshpass makes this possible (sorry they host the project on SourceForge).
_Security note, sshpass allows you to put the password into the SSH command/script which is inherently insecure.  Ideally you would use public key auth to facilitate this work, but we don’t spend much time in iDrac, and typically it is only used for remote console which is only available via the web interface.  iDrac does have provisions to allow for public key auth which you can read about here.  It won’t be covered in this post._

**Usability Note, sshpass is a utility for OS X/Linux, so you’ll need one of those to make this happen, although I understand you should be able to do something similar with putty, but the scripts we are using will be BASH, which requires OS X/Linux unless you want to run cygwin, which won’t be covered**

So, grab sshpass from sourceforge:

bud@black-box:~$ wget -O sshpass.tar.gz http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsshpass%2Ffiles%2F&ts=1423675742&use_mirror=iweb

Untar/gzip the file:

bud@black-box:~$ tar -zxvf sshpass.tar.gz

Move into the directory and run the installer:

bud@black-box:~$ cd sshpass-1.05/ 
./install.sh

This will put sshpass into your path for general consumption.

Test sshpass:

bud@black-box:~$ sshpass -p calvin ssh -n -o StrictHostKeyChecking=no -l root 10.1.1.101 racadm racdump

This should dump a ton of stuff from iDrac on that particular host.

Okay, so now that sshpass works, lets get some stuff done.

Lets create a helper script (we’ll call it idrac.sh) that we can call to execute log in and execute our commands:

#!/bin/bash
#first argument is host
HOST=$1
#there may be many arguments so we use shift to iterate through them until the end
shift
#we then use the '$*' to treat them all as one argument
CMD=$*
#login information for idrac
USER=root
PASS=calvin
#put it all together
sshpass -p $PASS ssh -n -o StrictHostKeyChecking=no -l $USER $HOST racadm $CMD`

Okay, now we’re ready to do some work.  So, lets set those hostnames and DNS settings.

First we need to create a .csv (called hostlist.csv) with the data we want, formatted as server IP,DNS Name.  It should look similar to:

10.1.1.101, remote-server-1
10.1.1.102, remote-server-2
10.1.1.103, remote-server-3

Now lets write another script (called set-dns-settings.sh) to set everything:

#!/bin/bash
RACADM="./idrac.sh"
csv_input=$1
while IFS=, read IP dns; do
$RACADM $IP set iDRAC.Nic.DNSDomainName foo.bar.org
echo "${IP} (${dns}) domain set"
$RACADM $IP set iDRAC.IPv4.DNS1 10.1.1.1
echo "$IP (${dns})DNS1 set"
$RACADM $IP set iDRAC.IPv4.DNS2 10.2.2.2
echo "$IP (${dns})DNS2 set"
$RACADM $IP set iDRAC.Nic.DNSRacName $dns
echo "$IP (${dns})hostname set"
$RACADM $IP set iDRAC.Nic.DNSRegister 1
echo "$IP (${dns}) DNS Auto Registration set"
done < $csv_input

Run the script like this:

./set-dns-settings.sh hostlist.csv

You should receive output similar to:

10.1.1.101 (remote-server-1) domain set
10.1.1.101 (remote-server-1) DNS1 set
10.1.1.101 (remote-server-1) DNS2 set
10.1.1.101 (remote-server-1) hostname set
10.1.1.101 (remote-server-1) DNS Auto Registration set
10.1.1.101 (remote-server-2) domain set
10.1.1.101 (remote-server-2) DNS1 set
10.1.1.101 (remote-server-2) DNS2 set
10.1.1.101 (remote-server-2) hostname set
10.1.1.101 (remote-server-2) DNS Auto Registration set
10.1.1.101 (remote-server-3) domain set
10.1.1.101 (remote-server-3) DNS1 set
10.1.1.101 (remote-server-3) DNS2 set
10.1.1.101 (remote-server-3) hostname set
10.1.1.101 (remote-server-3) DNS Auto Registration set

This script will read the IPs and hostnames from the CSV file, then do the following for each line:

  1. Set the domain
  2. Set DNS Server 1
  3. Set DNS Server 2
  4. Set the hostname for iDrac
  5. Set iDrac to register its hostname in DNS (this requires a DNS environment that supports this)

It will report back after each entry has been updated.
**Note, for a large number of servers this may take a while, the iDrac SSH isn’t very quick.  Also I understand that this could probably be done more efficiently by putting more commands into a single racadm session, but all I need to do is start this script and go do something else for a while so I’m not particularly concerned about duration.

Okay, so now I’ve completed steps 1 and 2.  Now we need to move iDrac to use a tagged VLAN on LOM3.

#!/bin/bash
csv_input=$1
VLAN=800
RACADM="./idrac.sh"
while IFS=, read IP dns; do
#set idrac to use LOM3
$RACADM set iDRAC.NIC.Selection LOM3
#allow idrac to auto-detect (this will help to prevent us from locking ourselves out)
$RACADM set iDRAC.NIC.AutoDetect Enabled
#enable VLAN tagging on the iDrac NIC (this applies to whatever interface is used for iDrac
$RACADM set iDRAC.NIC.VLanEnable Enabled
#set the VLAN that will be tagged
$RACADM set iDRAC.NIC.VLanID $VLAN
#set the boot process to boot from PXE for LOM3
$RACADM set NIC.NICConfig.3.LegacyBootProto PXE
#reset the NIC to take the new settings
$RACADM jobqueue create NIC.Integrated.1-3-1 -s TIME_NOW -r pwrcycle
done < $csv_input

For the script above we will call it using the same .CSV file that we created earlier (even though we don’t need the DNS portion anymore).  This will then iterate through the entries and set all of the required interface settings above.

At this point we will have to coordinate with the remote hands (or potentially your own hands) to move all of the 1GBE cables from the dedicated iDrac port to LOM3.

After that is finished we are now ready to create another script (we’ll call it test-dns-ping.sh) test and make sure that all of our servers have registered in DNS and are pingable as well.  Lets give it a shot with fping (if you don’t have it, install it).

}
#!/bin/bash
csv_input=$1
domain=".foo.bar.org"
while IFS=, read IP dns; do
fping $remote$dns$domain
done < $csv_input

Notice we’re calling our .CSV file again, and we are iterating through all of the DNS names, testing both DNS resolution as well as availability.

Run the ping script as follows:

bud@black-box:~$ ./test-dns-ping.sh host-list.csv

You should see something similar to:

remote-host-1.foo.bar.org is alive
remote-host-2.foo.bar.org is alive
remote-host-3.foo.bar.org is alive

Now for our final task.  Getting the MAC address of LOM3 for all of our servers.  Lets create another script (named get-macs.sh):

#!/bin/bash
csv_input=$1
while IFS=, read IP dns; do
mac=`./idrac.sh $dns racdump | egrep '^NIC.Integrated'`
echo "$mac" | while read line; do
echo "$dns $line"
done
done < $csv_input

This script will return the MAC addresses of all however all we want is the MAC address of LOM3.  So all we have to do is invoke it as follows:

./get-macs.sh host-list.csv | grep 1-3-1

This will iterate through each of the hosts, grab the racdump, then parse out 1-3-1 which is the entry for LOM3.

This will return something like:

remote-server-1 NIC.Integrated.1-3-1    Ethernet                = 01:23:45:67:89:0A
remote-server-2 NIC.Integrated.1-3-1    Ethernet                = 01:23:45:67:89:0B
remote-server-2 NIC.Integrated.1-3-1    Ethernet                = 01:23:45:67:89:0C

Now we can take that data and give it to the folks who requested it.  Doing this manually would have taken a couple of days, but thanks to RACADM and some good ol’ bash we were able to bust through it in a few hours, and any new servers that come in are able to be configured it just a few seconds apiece instead of minutes.

Also, this is probably where I mention I wish we had been allowed to choose UCS for this project.

Tagged #BASH, #Dell, #iDrac, #RACADM, #Scripting, #sshpass

2021

Back to top ↑

2020

Self Service with Satellite

less than 1 minute read

Recently I had a customer who was running into some legacy infrastructure challenges with VM consistency, meeting (internal) customer expectations, and gener...

Winning with Soft Tokens

2 minute read

Well, I’d call that a sabbatical, but I think that would mean I was relaxing. Time to revive this thing (again).

Back to top ↑

2018

Back to top ↑

2017

Using Powershell and Posh-SSH to GSD.

5 minute read

I’m a guy who thinks you should use the right tool for the job. For instance, if you’re in a Windows environment, and you need to script something, installi...

iDrac, RACADM, sshpass, and BASH

8 minute read

If it were up to me, I suppose that the only thing that I’d really be responsible for would be core networking infrastructure (and consumption of craft brews...

Dynamic DNS and you

3 minute read

Okay, so I know the popular thing with network engineers is to remember the IP of EVERYTHING.  I’m pretty good at it too.  But having a lab at home and needi...

Back to top ↑