did you know that you can pause and resume programs in linux?

show user’s limits

you could limit the amount of files a user may open at the same time.

you could limit the amount of RAM a user is allowed to use.

users may reduce their limit themselves but not set them to unlimited (only root can do that).

see also: https://dwaves.de/2017/06/09/lpic-1-102-110-1-perform-security-administration-tasks/

user@debian:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7924
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7924
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

basic job control and niceness (priority of processes)

while physically at the machine…


vim /scripts/stress_loop.sh;

#!/bin/bash
# test-script that does useless work but consumes a lot of CPU ressources
while true; do
echo -n "+.";
done;

chmod u+x /scripts/stress_loop.sh; # mark it runnable

/scripts/stress_loop.sh; # run it
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+...
# you could now press Ctrg+Z - to pause it and put process into background
[1]+  Stopped                 /scripts/stress_loop.sh

fg; # resume it in foreground <- do this
bg; # resume it in background <- if you do that - you will have to blindly type fg 1; # to bring job 1 into foreground

# hit Ctrl+C - to cancel/abort foreground process

/scripts/stress_loop.sh > /dev/null & # start script as background-job without output

jobs; # list all running jobs
[1]+  Running                 /scripts/stress_loop.sh > /dev/null &

fg 1; # bring it to foreground
/scripts/stress_loop.sh > /dev/null
Ctrl+C; # abort cancel process

# now let's start it multiple times from different consoles

# Ctrl+Alt+F1 - go to terminal one and login
nice /scripts/stress_loop.sh; # run with default priority

# Ctrl+Alt+F2 - go to terminal two and login
nice -10 /scripts/stress_loop.sh; # run with niceness of 10

# Ctrl+Alt+F3 - go to terminal three and login
nice --20 /scripts/stress_loop.sh; # with niceness of -20

# Ctrl+Alt+F4 - go to terminal three and login
# check out what is going on
htop


you will see that the process with the smaller nice-level – gets more CPU ressources.

go back to terminal1: Ctrl+Alt+F1 … and you will see the script still at work… but way slower as if it was running alone.

go to terminal2: Ctrl+Alt+F2 … and you will see – this process with niceness of +10 is running even slower.

while process on terminal3: Ctrl+Alt+F3 with niceness of -20 is running at full speed.

you can re-adjust priority – also after process was started:

root@debian:~# renice 0 2546
2546 (process ID) old priority -20, new priority 0

you can terminate all 3 scripts with:

killall stress_loop.sh

default nice-level is zero (0) – nice-values below zero are only allowed to be started by root.

non-root users are allowed to re-nice their processes but only to larger (lower priority) levels.

keep processes running after logout

two possibilities: nohup or screen

nohup – will write all output to a file called nohup – you will have to append & in order to make it run in the background – before logging out.

screen – is starting a new bash – even interactive processes that are running in the foreground continue running when you detach (Ctrg+A then D) and logout (Ctrl+D).


nohup /scripts/stress_loop.sh & # start script as background process
nohup: ignoring input and appending output to ‘nohup.out’

jobs
[1]+  Running                 nohup /scripts/stress_loop.sh &

# if you logout now and re-login you will realize
jobs
# no jobs are displayed
# but
ps uax|grep loop; # it is still running
root      2986 97.1  0.2   5080  2752 ?        R    13:21   1:00 /bin/bash /scripts/stress_loop.sh

tail -f nohup; # all output gets written to this file

killall stress_loop.sh

apt-get install screen; # screen need to be installed

screen; # start screen
/scripts/stress_loop.sh; # start the script

# detach (Ctrg+A then D) and logout (Ctrl+D) and re-login

screen -ls; # list all running screens
There is a screen on:
        3028.pts-0.debian       (05/16/2017 01:24:19 PM)        (Detached)
1 Socket in /var/run/screen/S-root.

screen -R 3028.pts-0.debian; # resume this screen
Ctr+C - cancel foreground job
exit; # exit screen

# cool eh?
# you can name your screens

screen -S loop; # start screen with name 3067.loop

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would.

Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill existing windows, view a list of windows, turn output logging on and off, copy-and-paste text between windows, view the scrollback history, switch between windows in whatever manner you wish, etc.

All windows run their programs completely indepen‐ dent of each other.

Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user’s terminal.

When a program terminates, screen (per default) kills the window that contained it.

If this window was in the foreground, the display switches to the previ‐ ous window; if none are left, screen exits. Shells usually distinguish between running as login-shell or sub-shell.

Screen runs them as sub-shells, unless told otherwise (See „shell“ .screenrc command).

Screen version 4.02.01 (GNU) 28-Apr-14

Copyright (c) 2010 Juergen Weigert, Sadrul Habib Chowdhury
Copyright (c) 2008, 2009 Juergen Weigert, Michael Schroeder, Micah Cowan, Sadrul Habib Chowdhury
Copyright (c) 1993-2002, 2003, 2005, 2006, 2007 Juergen Weigert, Michael Schroeder
Copyright (c) 1987 Oliver Laumann

This program is free software;

Send bugreports, fixes, enhancements, t-shirts, money, beer & pizza to screen-devel@gnu.org
Capabilities: +copy +remote-detach +power-detach +multi-attach +multi-user +font +color-256 +utf8 +rxvt +builtin-telnet
[Press Space or Return to end.]

Example: Process/Job control

sleep 600 &; # start a job and send to background
[1] 4206 # process1 is running with PID 4206 asigned

user@suse:~> echo $!; # return the PID of the last run job
4206

user@suse:~> grep ^State /proc/4206/status; # grep the status of that process
State:  S (sleeping)

user@suse:~> fg; # bring last started job to foreground
sleep 600
^Z; # send to background and stop
[1]+  Stopped                 sleep 600

user@suse:~> grep ^State /proc/4206/status; # grep the status of that process
State:  T (stopped)

user@suse:~> kill -CONT 4206; # resume the job
user@suse:~> jobs
[1]+  Running                 sleep 600 &

kill -STOP 4206; # pause process

user@suse:~> jobs
[1]+  Stopped                 sleep 600

jobs
[1]+  Getötet(killed)                sleep 600

kill -l; # list of all available signals
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

kill -SIGHUP 4206; # "hangup" force shell to quit all child processes

# there are three different possibilities to send the same signal to the process
kill -SIGINT 4206; # "interrupt" like Ctrl+C "cancel"
kill -INT    4206; # "interrupt" short version
kill -2      4206; # "interrupt" invoked by numeric id

kill -TERM 4206; # default for "kill" and "killall", ask the process to quit gracefully (e.g. database shutdown)
kill -15   4206; #  invoked by numeric id

kill -KILL 4206; # "kill" if process is not reacting to quit gracefully via SIGINT, you can force quit with SIGKILL
kill -9    4206; # "kill" invoked by numeric id

what processes are currently running?

listing all running processes programs finding infos about an process.


ps aux |less; # what processes are currently running?
#            processID       virtual memory reserved
#           /                /      /"resident set size" - actual memory taken/reserved by process
#           |                |      |
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.5 185256  5964 ?        Ss   09:22   0:02 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
root         2  0.0  0.0      0     0 ?        S    09:22   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    09:22   0:00 [ksoftirqd/0]
root         7  0.0  0.0      0     0 ?        S    09:22   0:00 [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    09:22   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    09:22   0:00 [migration/0]
root        10  0.0  0.0      0     0 ?        S    09:22   0:00 [watchdog/0]
root        11  0.0  0.0      0     0 ?        S    09:22   0:00 [watchdog/1]
root        12  0.0  0.0      0     0 ?        S    09:22   0:00 [migration/1]
root        13  0.0  0.0      0     0 ?        S    09:22   0:00 [ksoftirqd/1]
root        15  0.0  0.0      0     0 ?        S<   09:22   0:00 [kworker/1:0H]
root        17  0.0  0.0      0     0 ?        S    09:22   0:00 [kdevtmpfs]
root        18  0.0  0.0      0     0 ?        S<   09:22   0:00 [netns]
root        19  0.0  0.0      0     0 ?        S<   09:22   0:00 [perf]
root        20  0.0  0.0      0     0 ?        S    09:22   0:00 [khungtaskd]
root        21  0.0  0.0      0     0 ?        S<   09:22   0:00 [writeback]
...                                            |              \_cpu time used
                                                \_status of process

top; # like windows task manager but on console, hit Shift+M to sort for MEMORY usage
# RES = currently used RAM by process
# example: gdm (gnome3) uses 156812KByte of 156MByte RAM
top - 12:10:47 up  2:48,  2 users,  load average: 0,10, 0,03, 0,01
Tasks: 149 total,   1 running, 148 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,0 us,  0,0 sy,  0,0 ni, 99,8 id,  0,2 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem:   1007380 total,   791520 used,   215860 free,     2076 buffers
KiB Swap:  2103292 total,        0 used,  2103292 free.   531884 cached Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 1643 gdm       20   0 1532396 156812  77164 S 0,000 15,57   0:05.59 gnome-shell
 1241 root      20   0  303928  37072  21236 S 0,000 3,680   0:00.29 Xorg
 1629 gdm       20   0  887972  32888  24700 S 0,000 3,265   0:00.44 gnome-settings-
 1286 polkitd   20   0  528168  27784   9468 S 0,000 2,758   0:00.38 polkitd
 1600 gdm       20   0  558216  13080  11356 S 0,000 1,298   0:00.06 gnome-session-b
 1243 root      20   0  280604   8780   5940 S 0,000 0,872   0:00.29 accounts-daemon
 1637 root      20   0  287564   8696   7692 S 0,000 0,863   0:00.05 upowerd
 1590 root      20   0  242652   8540   7372 S 0,000 0,848   0:00.02 gdm-session-wor
  543 root      20   0   33308   8524   8176 S 0,000 0,846   0:01.55 systemd-journal
 2164 root      20   0  102732   7828   6816 S 0,000 0,777   0:00.04 sshd
...

htop; # might not be preinstalled in your system but is even more intuitive and colorful than top
# it has a "ascii-grafical" display of RAM and CPU usage, hit F6 and select M_RESIDENT to sort for currently reserved RAM 
  2  [                                                                         0.0%]     Load average: 0.09 0.03 0.01
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||     285/983MB]     Uptime: 02:50:22
  Swp[                                                                     0/2053MB]
Sort by          PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
PID             1664 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.00 gnome-shell --mode=gdm
USER            1665 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.32 gnome-shell --mode=gdm
PRIORITY        1666 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.28 gnome-shell --mode=gdm
NICE            1667 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.07 gnome-shell --mode=gdm
M_SIZE          1669 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.00 gnome-shell --mode=gdm
M_RESIDENT      1670 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.00 gnome-shell --mode=gdm
M_SHARE         1671 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.00 gnome-shell --mode=gdm
STATE           1672 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:00.00 gnome-shell --mode=gdm
PERCENT_CPU     1643 gdm        20   0 1496M  153M 77164 S  0.0 15.6  0:05.62 gnome-shell --mode=gdm
PERCENT_MEM     1576 root       20   0  296M 37072 21236 S  0.0  3.7  0:00.00 /usr/bin/Xorg :0 -background none -verbose -auth /run/gdm/auth-for-gdm-eJrfFw/database -seat seat
TIME            1577 root       20   0  296M 37072 21236 S  0.0  3.7  0:00.00 /usr/bin/Xorg :0 -background none -verbose -auth /run/gdm/auth-for-gdm-eJrfFw/database -seat seat
Command         1241 root       20   0  296M 37072 21236 S  0.0  3.7  0:00.28 /usr/bin/Xorg :0 -background none -verbose -auth /run/gdm/auth-for-gdm-eJrfFw/database -seat seat
                1633 gdm        20   0  867M 32888 24700 S  0.0  3.3  0:00.17 /usr/lib/gnome-settings-daemon-3.0/gnome-settings-daemon
                1634 gdm        20   0  867M 32888 24700 S  0.0  3.3  0:00.00 /usr/lib/gnome-settings-daemon-3.0/gnome-settings-daemon
                1635 gdm        20   0  867M 32888 24700 S  0.0  3.3  0:00.03 /usr/lib/gnome-settings-daemon-3.0/gnome-settings-daemon
                1636 gdm        20   0  867M 32888 24700 S  0.0  3.3  0:00.00 /usr/lib/gnome-settings-daemon-3.0/gnome-settings-daemon
                1629 gdm        20   0  867M 32888 24700 S  0.0  3.3  0:00.45 /usr/lib/gnome-settings-daemon-3.0/gnome-settings-daemon
                1291 polkitd    20   0  515M 27784  9468 S  0.0  2.8  0:00.00 /usr/lib/polkit-1/polkitd --no-debug
                1292 polkitd    20   0  515M 27784  9468 S  0.0  2.8  0:00.07 /usr/lib/polkit-1/polkitd --no-debug
                1293 polkitd    20   0  515M 27784  9468 S  0.0  2.8  0:00.00 /usr/lib/polkit-1/polkitd --no-debug
...

So how much RAM is a process actually taken?

-> check out the RSS field, ignore the VSZ

The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct thread_info, and struct task_struct.

This is usually at least 20 KiB of memory that is always resident.

SIZE is the virtual size of the process (code+data+stack).

RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM.

- It does not include memory that is swapped out.
+ It does include memory from shared libraries as long as the pages from those libraries are actually in memory.
+ It does include all stack and heap memory.

VSZ is the Virtual Memory Size.
It includes all memory that the process can access, including memory that is swapped out and memory that is from shared libraries.

So if process A has a 500K binary
+ and is linked to 2500K of shared libraries
+ has 200K of stack/heap allocations of which 100K is actually in memory (rest is swapped), and has only actually loaded 1000K of the shared libraries and 400K of its own binary then:

VSZ: 500K + 2500K + 200K = 3200K (this is what the process would consume - if it fully loads itself to RAM including all of it's shared libraries)

RSS: 400K + 1000K + 100K = 1500K (this is how much RAM is actually taken, because a process might not need to load 100% of itself and the shared libraries to RAM)

Since part of the memory is shared, many processes may use it, so if you add up all of the RSS values you can easily end up with more space than your system has.

PROCESS STATE CODES

Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a
D uninterruptible sleep (usually IO)
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped, either by a job control signal or because it is being traced
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent

Zombie process:

usually a child process returns an result like 0 = everything ok, 1 = some error

a parent process hangs or quits before child-process... and one child process terminates with some result, it can not return the results to it's parent and turn into zombie 😀 (terminated but result not fetched by parent... so it also hangs?)

what shared libraries is the process with PID X using?

cat /proc/2176/maps; # what shared libraries is the process with PID 2176 (bash) using?
00400000-004a6000 r-xp 00000000 00:25 30785                              /bin/bash
006a5000-006a6000 r--p 000a5000 00:25 30785                              /bin/bash
006a6000-006aa000 rw-p 000a6000 00:25 30785                              /bin/bash
006aa000-006b5000 rw-p 00000000 00:00 0
01394000-015c6000 rw-p 00000000 00:00 0                                  [heap]
7f4df4252000-7f4df43ec000 r-xp 00000000 00:25 23161                      /lib64/libc-2.22.so
7f4df43ec000-7f4df45eb000 ---p 0019a000 00:25 23161                      /lib64/libc-2.22.so
7f4df45eb000-7f4df45ef000 r--p 00199000 00:25 23161                      /lib64/libc-2.22.so
7f4df45ef000-7f4df45f1000 rw-p 0019d000 00:25 23161                      /lib64/libc-2.22.so
7f4df45f1000-7f4df45f5000 rw-p 00000000 00:00 0
7f4df45f5000-7f4df45f7000 r-xp 00000000 00:25 23167                      /lib64/libdl-2.22.so
7f4df45f7000-7f4df47f7000 ---p 00002000 00:25 23167                      /lib64/libdl-2.22.so
7f4df47f7000-7f4df47f8000 r--p 00002000 00:25 23167                      /lib64/libdl-2.22.so
7f4df47f8000-7f4df47f9000 rw-p 00003000 00:25 23167                      /lib64/libdl-2.22.so
7f4df47f9000-7f4df4824000 r-xp 00000000 00:25 25974                      /lib64/libtinfo.so.5.9
7f4df4824000-7f4df4a23000 ---p 0002b000 00:25 25974                      /lib64/libtinfo.so.5.9
7f4df4a23000-7f4df4a27000 r--p 0002a000 00:25 25974                      /lib64/libtinfo.so.5.9
7f4df4a27000-7f4df4a2c000 rw-p 0002e000 00:25 25974                      /lib64/libtinfo.so.5.9
7f4df4a2c000-7f4df4a2d000 rw-p 00000000 00:00 0
7f4df4a2d000-7f4df4a6e000 r-xp 00000000 00:25 26789                      /lib64/libreadline.so.6.3

admin