TOS-based systems (Atari ST) used a row of bombs to indicate a critical system error.

Number of bombs displayed revealed information about the error.

The error (also called an exception) is reported by the Motorola 68000 microprocessor.

The very first version of TOS used mushroom clouds; this was quickly changed, as it was considered politically incorrect.

  1. bomb: Reset, Initial PC2
  2. bombs: Bus Error
  3. bombs: Address Error
  4. bombs: Illegal Instruction
  5. bombs: Division by zero … (src)

a bit of system logging history

„Syslog project was the very first project. It started in 1980. It is the root project to Syslog protocol. At this time Syslog is a very simple protocol. At the beginning it only supports UDP for transport, so that it does not guarantee the delivery of the messages.

Next came syslog-ng in 1998. It extends basic syslog protocol with new features like :

  • content-based filtering
  • Logging directly into a database
  • TCP for transport
  • TLS encryption

Next came Rsyslog in 2004. It extends syslog protocol with new features like :

Let’s say that today they are three concurent projects that have grown separately upon versions, but also grown in parallel regarding what the neighbors was doing.

I personally think that today syslog-ng is the reference in most cases, as it is the most mature project offering the main features you may need, in addition to an easy and comprehensive setup and configuration.“

as always – show me the src: https://serverfault.com/questions/692309/what-is-the-difference-between-syslog-rsyslog-and-syslog-ng

syslogd

You can install syslogd under debian8 with:

apt-get install inetutils-syslogd; # which will of course (can only have one logging system) remove rsyslogd

syslogd.man.txt

manpage for config file: syslog.conf.man.txt

syslogd and rsyslogd config file structure

facility indicates where the message originated from (e.g. kernel, mail subsystem) while the

severity provides a glimpse of how important the message might be (e.g. error or informational) (src)

priority (PRI) value is a combination of severity and facility

filter conditions – http://www.rsyslog.com/doc/v8-stable/configuration/filters.html

syslogd sample config file from debian8: syslog.conf.txt

rsyslogd uses the same config-language as syslogd:

it follows this methology:

facility.level file.log

auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog

the hyphen“ in front of a path-name means – do not use the RAM-buffer-queue (faster) – write directly to disk (slower, but survives crash)

facility can be

auth (former security), authpriv, cron, daemon, kern, lpr, mail, mark, news, syslog, user, uucp, local0…local7

*  means „all facilities“ / sources

Numerical Code Facility Description
0 kern kernel messages
1 user user-level messages
2 mail mail system
3 daemon system daemons
4 auth security/authorization messages
5 syslog messages generated internally by syslogd
6 lpr line printer subsystem
7 news network news subsystem
8 uucp UUCP subsystem
9 cron clock daemon
10 security security/authorization messages
11 ftp FTP daemon
12 ntp NTP subsystem
13 logaudit log audit
14 logalert log alert
15 clock clock daemon (note 2)
16 local0 local use 0 (local0)
17 local1 local use 1 (local1)
18 local2 local use 2 (local2)
19 local3 local use 3 (local3)
20 local4 local use 4 (local4)
21 local5 local use 5 (local5)
22 local6 local use 6 (local6)
23 local7 local use 7 (local7)

level / severity

what should be logged to /var/log/file

rsyslogd kind of took over syslogd levels but they were modified

in rsyslogd:

debug – debug-level messages

info – informational messages

notice – normal but significant condition

warning – warning conditions (warn is depreciated)

err – error conditions

crit – critical conditions

emerg – system became unusable, rsyslogd tries to write kernel crashed to log if possible (panic is depreciated)

Numerical Code Severity Description
0 emerg system is unusable
1 alert action must be taken immediately
2 crit critical conditions
3 error error conditions
4 warning warning conditions
5 notice normal but significant condition
6 info informational messages
7 debug debug-level messages

src: https://wiki.gentoo.org/wiki/Rsyslog#Severity

for syslogd there is a rfc:

numerical severity (src https://tools.ietf.org/html/rfc5424)

Code

0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages

Rsyslog – what is it?

rsyslogd.man.txt

rsyslog.conf.man.txt

Rsyslog is a rocket-fast system for log processing.

It offers high-performance, great security features and a modular design.

While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations.

(Yeah if it’s not complicated it’s not „Made in Germany“ :-D)

Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied (based on v7, December 2013).

Even with remote destinations and more elaborate processing the performance is usually considered „stunning„.

https://en.wikipedia.org/wiki/Rsyslog

concepts: http://www.rsyslog.com/doc/master/concepts/index.html

apt-cache show rsyslog
Package: rsyslog
Version: 8.4.2-1+deb8u2
Installed-Size: 1808
Debian Package Maintainer: Michael Biebl

Author: https://twitter.com/rgerhards

http://blog.gerhards.net/

Architecture: i386
Provides: linux-kernel-log-daemon, system-log-daemon
Depends: libc6 (>= 2.17), libestr0 (>= 0.1.4), libjson-c2 (>= 0.10), liblogging-stdlog0 (>= 1.0.2), liblognorm1 (>= 0.3.0), libuuid1 (>= 2.16), zlib1g (>= 1:1.1.4), init-system-helpers (>= 1.18~), lsb-base (>= 3.2-14), initscripts (>= 2.88dsf-13.3)
Recommends: logrotate
Suggests: rsyslog-mysql | rsyslog-pgsql, rsyslog-mongodb, rsyslog-doc, rsyslog-gnutls, rsyslog-gssapi, rsyslog-relp
Conflicts: linux-kernel-log-daemon, system-log-daemon
Description-en: reliable system and kernel logging daemon
Rsyslog is a multi-threaded implementation of syslogd (a system utility
providing support for message logging), with features that include:
* reliable syslog over TCP, SSL/TLS and RELP
* on-demand disk buffering
* email alerting
* writing to MySQL or PostgreSQL databases (via separate output plugins)
* permitted sender lists
* filtering on any part of the syslog message
* on-the-wire message compression
* fine-grained output format control
* failover to backup destinations
* enterprise-class encrypted syslog relaying

It is the default syslogd on Debian systems.
Description-md5: c25fbd9bcc88ab95bcc33eea4228fd42
Homepage: http://www.rsyslog.com/
Tag: admin::logging, implemented-in::c, interface::daemon, role::program,
works-with::db, works-with::logfile
Section: admin
Priority: important
Filename: pool/main/r/rsyslog/rsyslog_8.4.2-1+deb8u2_i386.deb
Size: 576504
MD5sum: db3b87ab08e79cce55ad0a6ce9b35f17
SHA1: 19f998a3864de9acdf702ab6ef52acc82ba59183
SHA256: ea16447fc40e40312dc6dd258fc8a110024770e225e091ecab70607495c52ffa

rsyslog config files:

the main config file should be located under: /etc/rsyslog.conf

here are samples from:

rsyslog.conf.debian8.txt

rsyslog.conf.suse12.txt

rsyslog.conf.centos7.txt

creating your own logging entry

rsyslog.conf has it’s own scripting language! holy cow! compelx complex!

vim /etc/rsyslog.conf; # open it up and add those green lines

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

*.info; auth,authprive.none; mail,news.none -/var/log/messages
local1.info -/var/log/user

ESC :wq # save and quit in vim

rsyslogd -N1; # test config (gave me errors… but it works never the less)

service rsyslog restart; # restart rsyslog service

logger -p local1.warn „Hello World“; # trigger a test warning

tail /var/log/user; # check what happened… you should see your entry

Jun 12 11:09:17 CentOS7 user: Hello World

logger

„a shell command interface to the syslog(3) system log module“

logger.man.txt

logger -t Backup „backup was run“; # make rsyslogd write this to /var/log/messages

logrotate

in order to compress and store away old log files.

„logrotate ‐ rotates, compresses, and mails system logs… It allows automatic rotation, compression, removal, and
mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

Normally, logrotate is run as a daily cron job.“

logrotate.conf.man.txt

example config files:

logrotate.conf.debian8.txt

systemd journald

/lib/systemd/systemd-journald

„systemd-journald is a system service that collects and stores logging data.

It creates and maintains structured, indexed journals based on logging information that is received from the kernel, from user processes via the libc syslog(3) call, from standard input and standard error of system services or via its native API.

It will implicitly collect numerous metadata fields for each log messages in a secure and unfakeable way.

See systemd.journal-fields(7) for more information about the collected metadata.

Log data collected by the journal is primarily text-based but can also include binary data where necessary. All objects stored in the journal can be up to 2^64-1 bytes in size.

By default, the journal stores log data in /run/log/journal/.

Since /run/ is volatile, log data is lost at reboot. To make the data persistent, it is sufficient to create /var/log/journal/ where systemd-journald will then store the data.

systemd-journald will forward all received log messages to the AF_UNIXSOCK_DGRAM socket /run/systemd/journal/syslog, if it exists, which may be used by Unix syslog daemons to process the data further.

systemd-journald.man.txt

journalctl.man.txt

journald.conf.man.txt

/etc/systemd/journald.conf (from debian8)

how to cause a kernel panic

fork bomb, open up a terminal, become root and:

:(){ :|:& };:

# or
echo c > /proc/sysrq-trigger

(src)

checkout ulimit to prevent such fork bombs – http://dwaves.de/2017/06/09/lpic-1-102-110-1-perform-security-administration-tasks/

klogd

klogd is a system daemon which intercepts and logs Linux kernel messages.

(src)

https://packages.debian.org/en/jessie/system-log-daemon

setup under debian: (rsyslog is used for this per default)

apt-get install system-log-daemon

Package system-log-daemon is a virtual package provided by:
syslog-ng-core 3.5.6-2+b1
socklog-run 2.1.0-8
rsyslog 8.4.2-1+deb8u2
inetutils-syslogd 2:1.9.2.39.3a460-3
dsyslog 0.6.0+nmu1
busybox-syslogd 1:1.22.0-9+deb8u1
You should explicitly select one to install.

E: Package ’system-log-daemon‘ has no installation candidate

web based log anlysis tools: incomplete setup tutorial.

maybe better but UNTESTED (!!!): http://terraltech.com/syslog-server-with-rsyslog-and-loganalyzer/

web interface for log files – http://loganalyzer.adiscon.com/ (former PHPLogcon)

apt-get install loganalyzer; # install setup under debian8

cd /var/www/html;

wget http://download.adiscon.com/loganalyzer/loganalyzer-4.1.5.tar.gz;

tar fxvz loganalyzer-4.1.5.tar.gz;

chown -R www-data:www-data /var/www/html;

„Syslog file is not readable, read access may be denied“

„unless the webserver user www-data is not in the adm group, phpLogCon will not be able to read the syslog file.“ (src, wiki)

1. First add a new group by using this command, we call the group logadmin in this sample.

groupadd logadmin

2. Now we add the www-data user to the new created logadmin group!

usermod -a -G logadmin www-data

Verify the success by using this command:

groups www-data

3. Now it is time to edit the /etc/logrotate.conf configuration file. Search for the default „create“ command, and change it to the following:

vim /etc/logrotate.conf; # open and edit conf

# create new (empty) log files after rotating old ones
create -> create 640 root logadmin

The next time logrotate will run, the logfiles will become readable to the webserver by default.

more or less relevant videos:

a video from the debian maintainer of rsyslog – Michael Biebl: systemd integration and CI in Debian/Ubuntu – also talks about Debian

open source chicks are sexy:

Links:

http://kb.monitorware.com/

http://loganalyzer.adiscon.com/doc/install.html

https://www.laub-home.de/wiki/Syslog_Server_unter_Debian_Linux_mit_Rsyslog

http://www.rsyslog.com/doc/master/index.html

RFCs:

The Syslog Protocol – https://tools.ietf.org/html/rfc5424

admin