HPE Server Alerts: Gen 10 changes

In an earlier post I explained how to set up alerts for pre-Gen 10 HPE servers so you will know when something bad happens to your server. Things have changed starting with Gen 10. This post will walk through the new process.

Why use HPE server alerts?

One of the most important tools you have as a system administrator is the simple email alert. No matter what the problem is, it’s great to have a head start on fixing it before the phone starts ringing. In the case of HP servers you can get an email alert so you will have a head start on any problems that arise. 

About The iLO

HPE servers up to Gen 9 used management agents on the operating system to monitor and send alerts, but going forward the alerts are sent by the iLO using an agentless service.

iLO is shorthand for “integrated lights-out”. Basically, this is an out of band management system that you can access over the network. It uses a dedicated Ethernet port so you can log in to the iLO from another computer on the network using a web browser. The iLO has its own IP address and you can set it to be static or check your DHCP for the IP it has been given.

Here are some documents from HPE that you may want to have a look at if you run into a problem not covered here:

The iLO User Guide

Configuring iLO Management Settings

If you like you can update the iLO firmware by downloading the installer.Click on Revision History and get newest version (should be the top one).

The iLO Advanced license

Alerts on Gen 10 servers require a special license, the iLO Advanced license. You can check your license by logging in to the iLO with your web browser and going to Information, then Overview. It will probably be the first page that comes up. Look at “License Type”. If yours says “iLO Advanced” you’re good to go. If it says” iLO Standard” you will need to purchase a license.

To quote HPE:

“With the release of Gen10 servers, iLO no longer supports OS-based SNMP agents. The System management Assistant (SMA) is an Agentless Management Service feature for users who want to run applications that obtain SNMP information from the OS. ”

Another advantage of the iLO Advanced license is that you can use the Remote Console in the iLO with no time limit. Without the license you only get like 3 minutes and then it locks you out. The Remote Console is basically Remote Desktop (RDP) through the iLO.

To check your license status log into the iLO by entering the iLO IP address into a web browser.

When you get in, right on the first page, next to “License Type” you should see if you have the license or not. If you already have the iLO Advanced license it will say “iLO Advanced”. Otherwise it would say “iLO Standard”.

You can obtain a license by registering the server with HPE, but you can also request a trial license.

Once you get your license log in to the iLO again and go to Administration, then Licensing.

You can just copy and paste your key into the first box and click the Install button. Easy.

Installing the SNMP Service Feature

To get the emails to us we need to install the System Network Management Protocol (SNMP) service. To do that, open Server Manager (it comes with Windows Server and usually starts automatically every time you log in) go to “Manage”, then “Add Roles and Features”. Choose “Role-based”, Next, select the local server, Next, Next, on the “Select Features” screen check the box next to “SNMP Service” and the one inside it, “SNMP WMI Provider”. You may be prompted to install “SNMP Tools”, too, if you don’t have it installed already. Get it, you’ll need it.

Configuring the SNMP Service

Now we need to configure SNMP to collect information from the server. Open Services.msc by searching in the Start menu or Windows key + R and entering Services.msc. Scroll down to “SNMP Services” and double-click on it to open the properties (or right-click and choose “properties”). In the security tab, click Add button under “Accepted community names”. There are two Add buttons on this tab so click the top one. Select Community rights as “READ ONLY”. Type “Public” for the Community Name and click Add.

If you don’t have the Security tab close Services.msc and restart it.

Click the same Add button again and create another Community Name called “Private”, but this time make the Rights “READ WRITE”.

Now we will tell the SNMP service who to accept packets from. With the “Accept SNMP packets from these hosts” radio button selected, click on the lower Add button now and enter 127.0.0.1 as the IP address. This is the localhost so the SNMP service will only be listening to this server we are working on. Click Add and then Apply.

With this done, go to the Traps tab in SNMP Service properties. You will have to type directly into the drop-down list because it doesn’t populate itself. Type “public” and now the “Add to list” button will be clickable. Click it. The drop-down list now has an entry for “public”. Click the “Add” button lower down the Traps tab and enter 127.0.0.1 and then click the Add button.

Click Apply and Ok.

Installing the Agentless Management Service

HPE’s pitch for this new way of managing server alerts seriously begins with “Tired of installing agents on your servers to monitor them?”, but in reality there is still one more service that you need to install: the “Agentless Management” service. Download it here. Make sure you get the new version under “Revision History”.

Download it and run it as administrator (from the right-click menu).

Click Install.


In this case I am upgrading because I already had an older version installed, but the process is almost identical except for the wording on this screen. Click Install.

Click Yes.


Close.

Make sure the “Agentless Management” Service is running in services.msc

Configuring the iLO Settings

Your settings may end up being different than mine after this point depending on your domain and network settings. Also some older versions of iLO 5 look different or have different names for some of the fields. In these examples I am using version 1.46 of iLO 5.

Log in to the iLO in your browser. Go to “iLO Dedicated Network Port” on the left, then the “General” tab. Fill in your “iLO Subsytem Name (Hostname)”. I use “ILO-<servername>”. Fill in the domain name and check the “Use iLO Dedicated Network Port” checkbox and the “Automatic” radio button.

Click the “Apply” button. You will see a message that tells you to reset the iLO to get the changed to take effect. Do so now. You will be logged out for a minute while the iLO reboots.

Log back in the iLO and go to “iLO Dedicated Network Port” on the left, then the “IPv4” tab.

Turn DHCP off and specify a static IP address, subnet mask, and gateway. The only swtich that can be green is the “Other” switch that says “Ping Gateway on Startup”.

Don’t forget to “Apply” and “Reset iLO”.

Once the iLO comes back up log back in and go to
“iLO Dedicated Network Port”, IPv6 tab. Turn everything off. “Apply” and “Reset”.

Log back in and go to “Management” on the left and then the “SNMP Settings” tab.

Select the radio button next to “iLO Hostname” and click the switches for “iLO SNMP Alerts” and “Cold Start Trap Broadcast”. Click “Apply”.

Lower down on the same page, under “SNMP Alert Destinations, click the “New” button. Specify your SNMP server. I leave the “Trap Community” blank and use the default for the protocol. Click “Add” and leave evertything else on the “SNMP Settings” tab at the default value.

While still in the “Management” area, go to the “AlertMail” tab. Click the “Enable iLO AlertMail” switch and enter the “Recipient” email address that will receive the alerts, the “Sender” email address that has permission to send emails out, and the SMTP Server that will process the email. The port is probably going to be 25, the default SMTP port. I leave everything else off.

Click “Apply”.

Sending a Test Alert

To send a generic test alert, log in to the iLO in your browser and go to “Management” on the left and then the “AlertMail” tab. Click the “Send Test AlertMail” button as seen in the previous image.

You should get an email like this:

If you got this email, then congratulations! If something goes wrong with the server you should get an email that tells you about it. It’s a good idea to log in to the iLO and send a test email once a month or so to make sure everything is still working. Sometimes a service will stop and you have to restart it manually.

If you didn’t get the email, double-check all the settings to make sure everything is correct and has been “applied”. If not, redo them and remember to apply and reset the iLO to save the changes. If that still doesn’t help make sure the SNMP and AMS services are running. It can’t hurt to go ahead and restart them even if they are. Your domain and network will be different than mine so there may also be additional DNS or Exchange settings for you to tweak.

There is a log in the iLO that might be able to tell you what went wrong. Go to “Information” on the left of the iLO browser interface and then the “iLO Event Log” tab.

Have fun and see you next time!

Enabling The Third Drive Box On An HPE DL380 Gen10 Server

The HPE DL380 Gen10, like any HPE server, can be purchased in multiple configurations, as seen here:
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00019687en_us

Because of this, you may unbox your server and find that not everything works “right out of the box”. In the case of our server project, we bought a DL380 Gen 10 that was going to be used as a replication target for Virtual Machine backups. We want to use all three “boxes” of hard drive slots. Pretty simple: just a server full of large form factor SAS drives.

The server came with cables from the SAS ports on the motherboard to all three “backplanes” of the drive boxes, but only the first two boxes recognized that they had drives in them.

Turns out that first of all we had a bad SAS cable on box three, but that wasn’t the only problem. According to HPE support the third box doesn’t work unless you use a separate cable that wasn’t included.

This does make sense considering all of the possible configurations for the server. Presumable the third SAS cable that was installed would work for another common layout. Not ours, though. We needed another cable. Fortunately new servers come with next day support so we were able to get another cable the next day. (Well, the “next day” after two days of arguing with HPE support.)

If you find yourself in this situation, you need the black cable in this picture:

It is included in this kit: https://partsurfer.hpe.com/Search.aspx?SearchText=826709-B21. It is the black one with the three different types of connectors on one end and one single SAS on the other.

FYI there is also a version of the black cable that has a 90 degree, L-shaped SAS connector on the backplane end. That one will not fit for our purpose. You need the one with the straight SAS end.

To open the server so you can get to the cables, first open the latch on the top of the server and remove the cover:

The SAS cables are here:

This is how the server came out of the box, three blue SAS cables. Here is the end where they plug into the motherboard:

This is the other end, where they meet the backplane:

We need to remove the blue cable from port 3 and replace it with the black one with the three connectors on one end. You can see in this next picture where they all go.

Once you do this, all three boxes of hard drives should work.

Project: Setting Up a Server From Scratch

Lately, I’ve been installing enough new servers (or inheriting newly installed servers) that now seems like a good time to show what it’s like to start from scratch with a brand new server. Obviously, no one will have exactly the same implementation as this, but the idea is to show the basics of what it’s like to set up a new server.

This will not cover everything you can do with a server. It’s basically just a way for me to organize my posts into something fun.

This page right here is going to host links that, if followed chronologically, will show what it looks like to build a brand new server straight out of the box. Basically all I’m doing is reordering some of my existing posts, but this page will corral them into a mighty Voltron of a larger project. I will probably go back and edit the old posts to make more sense in the context of this project (or not depending on how much time I have). Don’t worry, the original posts will work as standalones.

Anyway, here’s what it’s like to buy and implement a server!

(This is a work in progress. All of the steps may not be here yet. I’ll remove this disclaimer when I feel it is more complete)

Unboxing the HPE DL380 Gen10 server

Enabling The Third Drive Box On An HPE DL380 Gen10 Server

Installing Windows Server 2019

New Windows Server 2019 Server: Miscellaneous Setup

Automating Windows Server Updates Part 1: Sconfig

Automating Windows Server Updates Part 2: PowerShell

Automating Windows Server Updates Part 3: Scheduling the Task

HPE Server Hardware Alert Emails: Gen 10 Changes