Monitoring Performance of Linux Servers

04.Apr.2018

Administration / Server, Cybercrime, CyberSec / ITSec / Sicherheit / Security / SPAM, Hardware / Reviews, hardware fail, networking

monitor (andoird) devices with bpf: https://www.socallinuxexpo.org/sites/default/files/presentations/bcc-scale.pdf

so guess “monitoring” is about two things:

availability and performance:
- detect performance bottlenecks
- get informed if parts (harddisks) / servers have failed / are about to fail
security:
- detect, report and (if possible) automatically fence off “unusual” network activity such as hacking attacks (DDoS / bruteforce)

… anything i have missed? (scroll down to comment please!)

hardware / software raid monitoring:

https://dwaves.de/2019/09/06/linux-server-monitor-software-raid-mail-notification-on-failure/

linux bash – how to find hardware raid controller – get info about hardware hp server – get RAID and power watt usage status from command line

well hardware raid adapters actually should mail you / beep loud if a harddisk fails… unfortunately this worked for (got it working with LSI MegaRaid but not for Adaptec under Windows 2012 Server, untested for linux (yet))

wishlist: here needs to be a tested script that informs you when a harddisk goes down, that works with hardware-raid and software-raid:

software raid:

https://www.howtoforge.com/how-to-configure-software-raid-to-send-an-email-when-something-is-wrong-with-raid

hp hardware raid:

untested: https://networklessons.com/uncategorized/send-e-mail-when-raid-fails-on-hp-proliant-running-linux/

“If you’re running Linux and don’t want to install the full HP management suite, you can develop a script around the cciss_vol_status utility to query controller/disk status. Also see: Installing HP Agents on OpenFiler” (src)

self made:

this script will output and follow changes to all log files on your server / workstation… very basic but can be handy to “checkout what the blackbox is currently doing”.

cat /scripts/mon_all_logs.sh 
#!/bin/bash

find /var/log/ -type f \( -name "*" \) ! -path '*.gz*' -exec tail -n0 -f "$file" {} +

basic server monitoring:

http://jperrin.org/centos/monitoring/grafana/keeping-an-eye-on-centos-performance-with-grafana/

for webservers: https://vestacp.com/ comes with a nice basic dashboard:

collectd,

cacti

SHELL is using:

https://www.splunk.com/

Features	Splunk Free	Splunk Light	Splunk Enterprise	Splunk Cloud
Maximum Daily Indexing Volume	500MB	20GB	Unlimited	Unlimited
Maximum Users	1	5	Unlimited	Unlimited
Universal Data Collection/ Indexing
Metrics Store
Data Collection Add-Ons
Monitoring and Alerting
Dashboards and Reports
Search and Analysis
Event Annotation
Automatic Data Enrichment
Anomaly Detection
Tables, Data Models and Pivot
Splunkbase Apps		App for AWS (only)
Splunk Premium Solutions
High Availability
Disaster Recovery
Clustering
Distributed Search
Performance Acceleration
Access Control		User and Admin only	Granular and Customizable	Granular and Customizable
Single Sign-On/LDAP
Developer Environment			Full access to APIs and SDKs	Full access to APIs and SDKs
Support	Community	Standard	Enterprise/Global	Enterprise/Global

“The UBA anomalies can be used for multiple SIEM workflows to deter and resolve threats quicker and with greater precision. Hunters and analysts can now use the UBA detected anomalies as a source type within Splunk ES as a starting point of the investigation, do ad hoc searching and pivot for detailed Incident Review and Breach Analysis.”

src: https://www.splunk.com/en_us/products/features-comparison-chart.html

FireEye

https://www.fireeye.com/

https://en.wikipedia.org/wiki/FireEye

Unlike other vendors that provide raw threat data, FireEye delivers high-fidelity intelligence derived from numerous sources across the globe, including human intelligence, open sources, active community engagement, connections to the threat underground and criminal marketplaces, and real-time data collected from a variety of technical sources.

The following are some of the major IOCs we can download and scan/alert on: