In my day to day I have a need for use of several virtual machines.
iDrac, RACADM, sshpass, and BASH
If it were up to me, I suppose that the only thing that I’d really be responsible for would be core networking infrastructure (and consumption of craft brews). Unfortunately that is not the case. These days (and since the inception of the NetEng), Network Engineers are presumably responsible for anything that even remotely network related. Server needs and IP? Bluetooth whatchmacalit isn’t pairing with the thingamadoober? Toaster with an IP address is constantly burning toast? While we have and provide tools so that most people can handle these things on their own, unfortunately we still end up with a lot of these tasks.
When we ultimately get saddled with these tasks, the best thing that we can do is find the quickest/most efficient way to handle them.
Enter Dell and the OpenStack project that my team is undertaking. My main responsibilities in this project are handling the build-out of the Underlay network (you know, what we used to call ‘The Network’) as well as SDN components of OpenStack. Remember the first paragraph where I note that the network guys ultimately handle anything remotely network related? Yeah.
Now first, let me state this: I work for a large org with a small team of what I would consider to be Special Forces of Infrastructure, unfortunately there just aren’t enough of us. So my normal server guy didn’t have the cycles to spare when the vendor we chose to implement OpenStack required us to provide the MAC addresses for all of the network interfaces of the 100+ Dell servers we just got.
So, it fell on me to GSD.
After creating list of IP’s to assign to the myriad of servers, and working with remote hands to get IP connectivity working, I still had to set various other settings, as well as get the requested MAC address information.
For those who don’t know, you can actually SSH into iDrac and be met with a bevvy of (RACADM) commands at your disposal for getting data, as well as setting it.
My tasks were set as follows:
- Set the hostnames for iDrac on all of the servers (remember remote-hands only did the base IP configuration)
- Set the DNS servers and enable dynamic registration of hostnames (granted we could create static A records with powershell, but that wasn’t what we wanted and would require a separate workflow)
- Set iDrac to use a Tagged VLAN and run off of LOM3 (this enabled us to utilize 3 cables instead of 4 [2x 10GBE for data, 1x 1GBE for PXE/admin/iDrac instead of dedicating a second 1GBE for iDrac])
- Enable PXE booting from LOM3
- Get the MAC addresses for LOM3
So in order to facilitate all of this I needed to be able to ssh into all of these machines in an automated fashion- sshpass makes this possible (sorry they host the project on SourceForge).
_Security note, sshpass allows you to put the password into the SSH command/script which is inherently insecure. Ideally you would use public key auth to facilitate this work, but we don’t spend much time in iDrac, and typically it is only used for remote console which is only available via the web interface. iDrac does have provisions to allow for public key auth which you can read about here. It won’t be covered in this post._
**Usability Note, sshpass is a utility for OS X/Linux, so you’ll need one of those to make this happen, although I understand you should be able to do something similar with putty, but the scripts we are using will be BASH, which requires OS X/Linux unless you want to run cygwin, which won’t be covered**
So, grab sshpass from sourceforge:
bud@black-box:~$ wget -O sshpass.tar.gz http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsshpass%2Ffiles%2F&ts=1423675742&use_mirror=iweb
Untar/gzip the file:
bud@black-box:~$ tar -zxvf sshpass.tar.gz
Move into the directory and run the installer:
bud@black-box:~$ cd sshpass-1.05/ ./install.sh
This will put sshpass into your path for general consumption.
bud@black-box:~$ sshpass -p calvin ssh -n -o StrictHostKeyChecking=no -l root 10.1.1.101 racadm racdump
This should dump a ton of stuff from iDrac on that particular host.
Okay, so now that sshpass works, lets get some stuff done.
Lets create a helper script (we’ll call it idrac.sh) that we can call to execute log in and execute our commands:
Okay, now we’re ready to do some work. So, lets set those hostnames and DNS settings.
First we need to create a .csv (called hostlist.csv) with the data we want, formatted as server IP,DNS Name. It should look similar to:
10.1.1.101, remote-server-1 10.1.1.102, remote-server-2 10.1.1.103, remote-server-3
Now lets write another script (called set-dns-settings.sh) to set everything:
Run the script like this:
You should receive output similar to:
10.1.1.101 (remote-server-1) domain set 10.1.1.101 (remote-server-1) DNS1 set 10.1.1.101 (remote-server-1) DNS2 set 10.1.1.101 (remote-server-1) hostname set 10.1.1.101 (remote-server-1) DNS Auto Registration set 10.1.1.101 (remote-server-2) domain set 10.1.1.101 (remote-server-2) DNS1 set 10.1.1.101 (remote-server-2) DNS2 set 10.1.1.101 (remote-server-2) hostname set 10.1.1.101 (remote-server-2) DNS Auto Registration set 10.1.1.101 (remote-server-3) domain set 10.1.1.101 (remote-server-3) DNS1 set 10.1.1.101 (remote-server-3) DNS2 set 10.1.1.101 (remote-server-3) hostname set 10.1.1.101 (remote-server-3) DNS Auto Registration set
This script will read the IPs and hostnames from the CSV file, then do the following for each line:
- Set the domain
- Set DNS Server 1
- Set DNS Server 2
- Set the hostname for iDrac
- Set iDrac to register its hostname in DNS (this requires a DNS environment that supports this)
It will report back after each entry has been updated.
**Note, for a large number of servers this may take a while, the iDrac SSH isn’t very quick. Also I understand that this could probably be done more efficiently by putting more commands into a single racadm session, but all I need to do is start this script and go do something else for a while so I’m not particularly concerned about duration.
Okay, so now I’ve completed steps 1 and 2. Now we need to move iDrac to use a tagged VLAN on LOM3.
For the script above we will call it using the same .CSV file that we created earlier (even though we don’t need the DNS portion anymore). This will then iterate through the entries and set all of the required interface settings above.
At this point we will have to coordinate with the remote hands (or potentially your own hands) to move all of the 1GBE cables from the dedicated iDrac port to LOM3.
After that is finished we are now ready to create another script (we’ll call it test-dns-ping.sh) test and make sure that all of our servers have registered in DNS and are pingable as well. Lets give it a shot with fping (if you don’t have it, install it).
Notice we’re calling our .CSV file again, and we are iterating through all of the DNS names, testing both DNS resolution as well as availability.
Run the ping script as follows:
bud@black-box:~$ ./test-dns-ping.sh host-list.csv
You should see something similar to:
remote-host-1.foo.bar.org is alive remote-host-2.foo.bar.org is alive remote-host-3.foo.bar.org is alive
Now for our final task. Getting the MAC address of LOM3 for all of our servers. Lets create another script (named get-macs.sh):
This script will return the MAC addresses of all however all we want is the MAC address of LOM3. So all we have to do is invoke it as follows:
./get-macs.sh host-list.csv | grep 1-3-1
This will iterate through each of the hosts, grab the racdump, then parse out 1-3-1 which is the entry for LOM3.
This will return something like:
remote-server-1 NIC.Integrated.1-3-1 Ethernet = 01:23:45:67:89:0A remote-server-2 NIC.Integrated.1-3-1 Ethernet = 01:23:45:67:89:0B remote-server-2 NIC.Integrated.1-3-1 Ethernet = 01:23:45:67:89:0C
Now we can take that data and give it to the folks who requested it. Doing this manually would have taken a couple of days, but thanks to RACADM and some good ol’ bash we were able to bust through it in a few hours, and any new servers that come in are able to be configured it just a few seconds apiece instead of minutes.
Also, this is probably where I mention I wish we had been allowed to choose UCS for this project.
So in my recent foray into templating languages I ran across liquid.
Recently I had a customer who was running into some legacy infrastructure challenges with VM consistency, meeting (internal) customer expectations, and gener...
Well, I’d call that a sabbatical, but I think that would mean I was relaxing. Time to revive this thing (again).
I spend a fair amount of time building labs and then breaking them.
“Hey Brandon, how about something networky for a change?” First- Shut up, I don’t even like you (just kidding, you’re probably very pleasant). Second- I th...
I’m a guy who thinks you should use the right tool for the job. For instance, if you’re in a Windows environment, and you need to script something, installi...
I’m in an environment now where I have to proof-of-concept complicated/large ideas for environments to prove their feasibility. The latest project: High...
If it were up to me, I suppose that the only thing that I’d really be responsible for would be core networking infrastructure (and consumption of craft brews...
Okay, so I know the popular thing with network engineers is to remember the IP of EVERYTHING. I’m pretty good at it too. But having a lab at home and needi...
A penny for my thoughts?