What is Service Pack for ProLiant?

If you have HP servers and you don’t know about Service Pack for ProLiant (SPP), you need to take a few and read this.  I’m not sugar coating anything, this post is mostly because I always forget the name of this dang utility and is nothing special.  So here’s the official description of what SPP does (source):

The Service Pack for ProLiant (SPP) is a comprehensive systems software and firmware update solution, which is delivered as a single ISO image. This solution uses Smart Update Manager (SUM) as the deployment tool and is tested on all HPE ProLiant Gen9 and Gen10 servers as defined in the Service Pack for ProLiant Server Support Guide found at www.hpe.com/servers/spp/documentation.

All in all, this little utility will update all your drivers and firmware for you.  Even better, it’s just an ISO that you create a bootable USB out of.  One thing to note is that you do need an active support agreement with HP Enterprise to obtain the software through official means.  Assuming you have that, check the documentation to make sure your model is supported and you’re off to the races!

Create a Bootable USB for Dell Server Firmware Upgrades

I ran into an issue recently where I needed to upgrade some firmware on a Dell Server that I was unable to install an OS on, the circumstances surrounding that issue being a story for another time.  I did this by utilizing the Dell Repository Manager.

After downloading and installing the software, open the Data Center Version which contains all of the server models.  Start by Syncing the Database.  There should be a prompt on the top right of the application or select the option on the welcome page.

After that is complete, we need to create a new Repository for the model of server that we have.  Select the Create a New Repository option on the welcome page and give it a name (and description if you so desire).  Select the Dell Online Catalog as your base repository and hit next.  Select next keeping the defaults for the Select Brand page and then select the Linux (32-bit and 64-bit) option on the Select DUP Format page and select next.

Select your server model(s) from the list on the Select Models page and select next.  Select next on the Optional Components page and then select finish.  It will build the repository and give you the following message.

Select close and finish and it should open the repository for you.  If not select the My Repositories tab, check your new repository and select open.  Now select your components tab.  Find the firmware you wish to update (if you hover over a package you will get a description) and select all of the ones you need.  You can then select the Copy To option at the top.  This will give you a prompt where you should select the bottom option (Create a NEW Bundle in the same repository and add component(s) into it).

On the Name and version screen, be sure to select the Linux 32-bit and 64-bit option.  Then fill out all the other information appropriately as prompted and select finish.

Now that we have a bundle, we can make that into an ISO.  We’re getting close to the finish line!  Select the bundle and select Create Deployment Tools.  Select the Create Bootable ISO (Linux Only) option and select next.

Select next until prompted for a storage location.  Navigate to where you would like to store the ISO and select OK.  Select next a couple more times and then finish.  Once you see the job complete in the job queue you will have your ISO in your storage location.

Now that we have our ISO, you just need to make it into a bootable USB.  I personally use Rufus for this but it is dealers choice.  Then just boot to the USB on the server and you are off to the races.

NetGear ReadyNAS Inaccessible After Firmware Upgrade

Recently I logged into my NetGear ReadyNAS and it prompted me that there was a firmware update (version 6.7.2 to 6.7.3). I had some time to kill so I said “What the heck” and let it do its thing. After it restarted, it lost all network connection. When I went down to the rack where we have the unit, all the drive IO lights were on but there was a red light next to a caution symbol. Uh oh. I also noticed that the link lights on my network card were good and blinking, yet I could not find the dang thing on the network. After waiting for a couple hours (that hopeful thinking that it might resolve itself) I gave the unit the good ol’ reboot to no avail. I put the unit in Tech Support Mode and used NetGear’s RAIDar application (found here) to find the joker…Both NICs were assigned a 192.168.168.168 IP address. Yeah that wasn’t my network. I found this strange considering there was DHCP enabled on the network it was connected to, so even if the NICs lost their IP config they should have just grabbed one from the pool. So I disconnected one of the NICs so that it didn’t have a duplicate IP and rebooted it. I then added a second IP address to my machine on the 192.168.168.0/24 network so I could talk to the thing. I still could not hit the web GUI and I’ll be honest, I’m no Linux expert with these custom kernels so I wasn’t going to start fiddling with that. You know what time it is? Time to call NetGear support…

After what can only be described as a terrible experience making it through tier 1, 2, and 3 support I finally got in touch with an engineer after 4 hours of struggling. Long story short, this guy SSH’d into the unit and discovered that the boot partition was completely full, and therefore could not complete the upgrade. Makes sense, but you would think they would code some logic in there to recognize that prior to beginning the upgrade. Regardless, the engineer cleaned up a bunch of old image files on the boot partition so that it was only around 40% full and wah-lah we have an operational ReadNAS again after a reboot.

Naturally, I picked this engineers brain on how to avoid this from happening again and luckily he gave me more information than I even asked for! For starters, you can check the usage on any of your partitions without having to get the thing in Tech Support Mode. To do this, you have to enable SSH on the unit via the web GUI. Disclaimer: if you make any changes via SSH, you void all support until it is Factory Reset (which will wipe all of your data). Once you’ve made it in, login as root using your admin password and run the following command:

This will output all of your volumes and their usage statistics. Basically a quick way to check if you have space in your boot volume for an upgrade. He also shared the following wisdom on how your volumes are effected as they grow:
80% Full – Degraded performance | 90% Full – Stability issues begin to occur | 95% Full – The volume becomes RO
Another thing to note that he shared was that the Logs section of the web UI only shows part on one log. If you select the download option you get them all. Found in this download is a disk_info.log file that has a whole bunch of info on your installed disks. He told me the thresholds that are normal for disks, and if I were to see abnormalities that that disk is more than likely beginning to fail. I will list these thresholds below:

ATA Error Count: 1-2 | Current Pending Sector Count and Uncorrectable Sector Count: 50, but if they begin suddenly rising rapidly that drive is on its way out

All in all the engineer I got me up and running and gave me some good info on how to avoid issues with my unit. Moving forward, I’ll be checking my volume capacity before upgrade!!