linux bash – hp proliant P410i hardware monitoring monitor and mail script

08.Sep.2019

General / Allgemein

it’s far from perfect, but was able to:

requirements:

creat a mail account somewhere
yum install sendemail and hpacucli

features/functions:

mail the status of the raid
show colorful on console status of raid (rebuilding orange, failed drive red, everything ok or unkown status white)

whishlist:

smart data of all drives

one will have to make sure all the proper hw raid controller interaction software-packages are installed.

vim /scripts/raid_hp_rebuild_status.sh
#!/bin/bash

# tested with:
# hardware: hp proliant DL360 G6 with p410i hw raid controller)
# software: hpacucli
# Centos7
# hpacucli version
#   ACU CLI Version: 9.20.9.0
#   SoulAPI Version: 6.1.11.0

# this script shall monitor the status of the hp hardware raid 
# if a drive of the raid goes into failed or Rebuilding
# a mail shall be send to the administrator
# the mail will contain the status of the raid
# as well as since when the raid is rebuilding (so administrator knows aprox how long rebuild might take)

# packages required:
# yum install sendemail lspci

start_time=$(date '+DATE: %Y-%m-%d TIME: %H:%M:%S'); # start time of this script
start_rebuild=""; # start tim eof detected rebuild
status_raid=""; # the status of the raid
status_hpacucli=""; # status how hpacucli is printing it

# define colors
RED='\033[0;31m'
ORANGE='\033[0;33m'
WHITE='\033[1;37m'
NC='\033[0m' # No Color
echo "" > /var/log/raid.log; # reset raid.log

echo "=== hp hardware raid monitoring script started on $start_time ==="  |tee -a /var/log/raid.log;
echo "=== controller status: ==="
hpacucli controller all show status |tee -a /var/log/raid.log;

while true; do

	echo "============================================="|tee -a /var/log/raid.log;

	status_hpacucli=$(/sbin/hpacucli ctrl slot=0 pd all show);
	/sbin/hpacucli ctrl slot=0 pd all show|tee -a /var/log/raid.log; # output hpacucli output line by line

	status_raid=""; # the status of the raid
	CURRENT_DATE=$(date '+DATE: %Y-%m-%d TIME: %H:%M:%S');

	if [[ $status_hpacucli == *"uilding"* ]]; then

		if [ -z "$start_rebuild" ]; then
			start_rebuild=$(date '+DATE: %Y-%m-%d TIME: %H:%M:%S');
		fi

		echo -e "${ORANGE}*** $CURRENT_DATE WARNING raid is recovery from failure (rebuild since $start_rebuild) have good backups in place in case of secondary drive outage${NC}" |tee -a /var/log/raid.log;
		interval=600; # send mail every 10min about build process

	elif [[ $status_hpacucli == *"ailed"* ]]; then
		echo -e "${RED}***** $CURRENT_DATE WARNING FATAL: a drive went offline(failed) - replace immediately${NC}" |tee -a /var/log/raid.log;
		interval=60; # send mail about fail every 1min
	else
		echo -e "${WHITE}* $CURRENT_DATE WARNING GOOD OR UNKOWN: raid status is either good or not listed here${NC}" |tee -a /var/log/raid.log;
		start_rebuild=""; # reset
		status_raid="";
		interval=3600; # send mail about good raid  every 1hour
	fi

	echo "============================================="|tee -a /var/log/raid.log;


	echo "=== gathering general info about the sever ==="|tee -a /var/log/raid.log;

	mypublicip=$(dig TXT +short o-o.myaddr.l.google.com @ns1.google.com);
	echo " == server's public ip: $mypublicip ==" >> /var/log/raid.log
	mail_subject="RAID STATUS of $(hostname)@$mypublicip";

	hostnamectl >> /var/log/raid.log;
	# df -h >> /var/log/raid.log;
	# ip addr show >> /var/log/raid.log;

	echo "=== trying to find all installed raid controllers: ==="  |tee -a /var/log/raid.log;
	lspci -knn | grep 'RAID bus controller' |tee -a /var/log/raid.log;
	echo "=== saving raid config to /etc/raid.config.backup ==="  |tee -a /var/log/raid.log;
	/sbin/hpacuscripting -c /etc/raid.config.backup

	# mail the log
	sendemail -o tls=yes -s mail.domain.com -f server@domain.com -t server@domain.com -u "$mail_subject" -xu server@domain.com -xp "28dhf87fh23" -m "$mail_subject" -a /var/log/raid.log

	sleep 60;
	clear;
done;

liked this article?

only together we can create a truly free world
plz support dwaves to keep it up & running!
(yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
really really hate advertisement
contribute: whenever a solution was found, blog about it for others to find!
talk about, recommend & link to this blog and articles
thanks to all who contribute!

admin