TOS-based systems (Atari ST) used a row of bombs to indicate a critical system error.
Number of bombs displayed revealed information about the error.
The error (also called an exception) is reported by the Motorola 68000 microprocessor.
The very first version of TOS used mushroom clouds; this was quickly changed, as it was considered politically incorrect.
- bomb: Reset, Initial PC2
- bombs: Bus Error
- bombs: Address Error
- bombs: Illegal Instruction
- bombs: Division by zero … (src)
a bit of system logging history
“Syslog project was the very first project. It started in 1980. It is the root project to Syslog protocol. At this time Syslog is a very simple protocol. At the beginning it only supports UDP for transport, so that it does not guarantee the delivery of the messages.
Next came syslog-ng in 1998. It extends basic syslog protocol with new features like :
- content-based filtering
- Logging directly into a database
- TCP for transport
- TLS encryption
Next came Rsyslog in 2004. It extends syslog protocol with new features like :
- RELP Protocol support (“It is most often used in environments which do not tolerate message loss, such as the financial industry”) (src)
- Buffered opration support
Let’s say that today they are three concurent projects that have grown separately upon versions, but also grown in parallel regarding what the neighbors was doing.
I personally think that today syslog-ng is the reference in most cases, as it is the most mature project offering the main features you may need, in addition to an easy and comprehensive setup and configuration.”
as always – show me the src: https://serverfault.com/questions/692309/what-is-the-difference-between-syslog-rsyslog-and-syslog-ng
syslogd
You can install syslogd under debian8 with:
apt-get install inetutils-syslogd; # which will of course (can only have one logging system) remove rsyslogd
manpage for config file: syslog.conf.man.txt
syslogd and rsyslogd config file structure
facility indicates where the message originated from (e.g. kernel, mail subsystem) while the
severity provides a glimpse of how important the message might be (e.g. error or informational) (src)
priority (PRI) value is a combination of severity and facility
filter conditions – http://www.rsyslog.com/doc/v8-stable/configuration/filters.html
syslogd sample config file from debian8: syslog.conf.txt
rsyslogd uses the same config-language as syslogd:
it follows this methology:
facility.level file.log auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog
the – “hyphen” in front of a path-name means – do not use the RAM-buffer-queue (faster) – write directly to disk (slower, but survives crash)
facility can be
auth (former security), authpriv, cron, daemon, kern, lpr, mail, mark, news, syslog, user, uucp, local0…local7
* means “all facilities” / sources
| Numerical Code | Facility | Description |
|---|---|---|
| 0 | kern | kernel messages |
| 1 | user | user-level messages |
| 2 | mail system | |
| 3 | daemon | system daemons |
| 4 | auth | security/authorization messages |
| 5 | syslog | messages generated internally by syslogd |
| 6 | lpr | line printer subsystem |
| 7 | news | network news subsystem |
| 8 | uucp | UUCP subsystem |
| 9 | cron | clock daemon |
| 10 | security | security/authorization messages |
| 11 | ftp | FTP daemon |
| 12 | ntp | NTP subsystem |
| 13 | logaudit | log audit |
| 14 | logalert | log alert |
| 15 | clock | clock daemon (note 2) |
| 16 | local0 | local use 0 (local0) |
| 17 | local1 | local use 1 (local1) |
| 18 | local2 | local use 2 (local2) |
| 19 | local3 | local use 3 (local3) |
| 20 | local4 | local use 4 (local4) |
| 21 | local5 | local use 5 (local5) |
| 22 | local6 | local use 6 (local6) |
| 23 | local7 | local use 7 (local7) |
level / severity
what should be logged to /var/log/file
rsyslogd kind of took over syslogd levels but they were modified
in rsyslogd:
debug – debug-level messages
info – informational messages
notice – normal but significant condition
warning – warning conditions (warn is depreciated)
err – error conditions
crit – critical conditions
emerg – system became unusable, rsyslogd tries to write kernel crashed to log if possible (panic is depreciated)
| Numerical Code | Severity | Description |
|---|---|---|
| 0 | emerg | system is unusable |
| 1 | alert | action must be taken immediately |
| 2 | crit | critical conditions |
| 3 | error | error conditions |
| 4 | warning | warning conditions |
| 5 | notice | normal but significant condition |
| 6 | info | informational messages |
| 7 | debug | debug-level messages |
src: https://wiki.gentoo.org/wiki/Rsyslog#Severity
for syslogd there is a rfc:
numerical severity (src https://tools.ietf.org/html/rfc5424)
Code
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages
Rsyslog – what is it?
Rsyslog is a rocket-fast system for log processing.
It offers high-performance, great security features and a modular design.
While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations.
(Yeah if it’s not complicated it’s not “Made in Germany” :-D)
Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied (based on v7, December 2013).
Even with remote destinations and more elaborate processing the performance is usually considered “stunning“.
https://en.wikipedia.org/wiki/Rsyslog
concepts: http://www.rsyslog.com/doc/master/concepts/index.html
apt-cache show rsyslog
Package: rsyslog
Version: 8.4.2-1+deb8u2
Installed-Size: 1808
Debian Package Maintainer: Michael Biebl
Author: https://twitter.com/rgerhards
Architecture: i386
Provides: linux-kernel-log-daemon, system-log-daemon
Depends: libc6 (>= 2.17), libestr0 (>= 0.1.4), libjson-c2 (>= 0.10), liblogging-stdlog0 (>= 1.0.2), liblognorm1 (>= 0.3.0), libuuid1 (>= 2.16), zlib1g (>= 1:1.1.4), init-system-helpers (>= 1.18~), lsb-base (>= 3.2-14), initscripts (>= 2.88dsf-13.3)
Recommends: logrotate
Suggests: rsyslog-mysql | rsyslog-pgsql, rsyslog-mongodb, rsyslog-doc, rsyslog-gnutls, rsyslog-gssapi, rsyslog-relp
Conflicts: linux-kernel-log-daemon, system-log-daemon
Description-en: reliable system and kernel logging daemon
Rsyslog is a multi-threaded implementation of syslogd (a system utility
providing support for message logging), with features that include:
* reliable syslog over TCP, SSL/TLS and RELP
* on-demand disk buffering
* email alerting
* writing to MySQL or PostgreSQL databases (via separate output plugins)
* permitted sender lists
* filtering on any part of the syslog message
* on-the-wire message compression
* fine-grained output format control
* failover to backup destinations
* enterprise-class encrypted syslog relaying
It is the default syslogd on Debian systems.
Description-md5: c25fbd9bcc88ab95bcc33eea4228fd42
Homepage: http://www.rsyslog.com/
Tag: admin::logging, implemented-in::c, interface::daemon, role::program,
works-with::db, works-with::logfile
Section: admin
Priority: important
Filename: pool/main/r/rsyslog/rsyslog_8.4.2-1+deb8u2_i386.deb
Size: 576504
MD5sum: db3b87ab08e79cce55ad0a6ce9b35f17
SHA1: 19f998a3864de9acdf702ab6ef52acc82ba59183
SHA256: ea16447fc40e40312dc6dd258fc8a110024770e225e091ecab70607495c52ffa
rsyslog config files:
the main config file should be located under: /etc/rsyslog.conf
here are samples from:
creating your own logging entry
rsyslog.conf has it’s own scripting language! holy cow! compelx complex!
vim /etc/rsyslog.conf; # open it up and add those green lines
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
*.info; auth,authprive.none; mail,news.none -/var/log/messages
local1.info -/var/log/user
ESC :wq # save and quit in vim
rsyslogd -N1; # test config (gave me errors… but it works never the less)
service rsyslog restart; # restart rsyslog service
logger -p local1.warn “Hello World”; # trigger a test warning
tail /var/log/user; # check what happened… you should see your entry
Jun 12 11:09:17 CentOS7 user: Hello World
logger
“a shell command interface to the syslog(3) system log module”
logger -t Backup “backup was run”; # make rsyslogd write this to /var/log/messages
logrotate
in order to compress and store away old log files.
“logrotate ‐ rotates, compresses, and mails system logs… It allows automatic rotation, compression, removal, and
mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.
Normally, logrotate is run as a daily cron job.”
example config files:
systemd journald
/lib/systemd/systemd-journald
“systemd-journald is a system service that collects and stores logging data.
It creates and maintains structured, indexed journals based on logging information that is received from the kernel, from user processes via the libc syslog(3) call, from standard input and standard error of system services or via its native API.
It will implicitly collect numerous metadata fields for each log messages in a secure and unfakeable way.
See systemd.journal-fields(7) for more information about the collected metadata.
Log data collected by the journal is primarily text-based but can also include binary data where necessary. All objects stored in the journal can be up to 2^64-1 bytes in size.
By default, the journal stores log data in /run/log/journal/.
Since /run/ is volatile, log data is lost at reboot. To make the data persistent, it is sufficient to create /var/log/journal/ where systemd-journald will then store the data.
systemd-journald will forward all received log messages to the AF_UNIXSOCK_DGRAM socket /run/systemd/journal/syslog, if it exists, which may be used by Unix syslog daemons to process the data further.
/etc/systemd/journald.conf (from debian8)
how to cause a kernel panic
fork bomb, open up a terminal, become root and:
:(){ :|:& };:
# or
echo c > /proc/sysrq-trigger
checkout ulimit to prevent such fork bombs – https://dwaves.de/2017/06/09/lpic-1-102-110-1-perform-security-administration-tasks/
klogd
klogd is a system daemon which intercepts and logs Linux kernel messages.
https://packages.debian.org/en/jessie/system-log-daemon
setup under debian: (rsyslog is used for this per default)
apt-get install system-log-daemon
Package system-log-daemon is a virtual package provided by:
syslog-ng-core 3.5.6-2+b1
socklog-run 2.1.0-8
rsyslog 8.4.2-1+deb8u2
inetutils-syslogd 2:1.9.2.39.3a460-3
dsyslog 0.6.0+nmu1
busybox-syslogd 1:1.22.0-9+deb8u1
You should explicitly select one to install.
E: Package ‘system-log-daemon’ has no installation candidate
web based log anlysis tools: incomplete setup tutorial.
maybe better but UNTESTED (!!!): http://terraltech.com/syslog-server-with-rsyslog-and-loganalyzer/
web interface for log files – http://loganalyzer.adiscon.com/ (former PHPLogcon)
apt-get install loganalyzer; # install setup under debian8
cd /var/www/html;
wget http://download.adiscon.com/loganalyzer/loganalyzer-4.1.5.tar.gz;
tar fxvz loganalyzer-4.1.5.tar.gz;
chown -R www-data:www-data /var/www/html;
“Syslog file is not readable, read access may be denied”
“unless the webserver user www-data is not in the adm group, phpLogCon will not be able to read the syslog file.” (src, wiki)
1. First add a new group by using this command, we call the group logadmin in this sample.
groupadd logadmin
2. Now we add the www-data user to the new created logadmin group!
usermod -a -G logadmin www-data
Verify the success by using this command:
groups www-data
3. Now it is time to edit the /etc/logrotate.conf configuration file. Search for the default “create” command, and change it to the following:
vim /etc/logrotate.conf; # open and edit conf
# create new (empty) log files after rotating old ones
create -> create 640 root logadmin
The next time logrotate will run, the logfiles will become readable to the webserver by default.
more or less relevant videos:
a video from the debian maintainer of rsyslog – Michael Biebl: systemd integration and CI in Debian/Ubuntu – also talks about Debian
https://www.youtube.com/watch?v=6F1mrT5qSWI
open source chicks are sexy:
Links:
http://loganalyzer.adiscon.com/doc/install.html
https://www.laub-home.de/wiki/Syslog_Server_unter_Debian_Linux_mit_Rsyslog
http://www.rsyslog.com/doc/master/index.html
RFCs:
The Syslog Protocol – https://tools.ietf.org/html/rfc5424
liked this article?
- only together we can create a truly free world
- plz support dwaves to keep it up & running!
- (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
- really really hate advertisement
- contribute: whenever a solution was found, blog about it for others to find!
- talk about, recommend & link to this blog and articles
- thanks to all who contribute!
