intro:

it seems very relevant to have the late as possible kernel up and running: https://googleprojectzero.blogspot.com/2018/09/a-cache-invalidation-bug-in-linux.html

first: backup your system!

on boot time under “Advanced” you can chose to boot the old kernel – never the less – backup your system completely before progressing.

in general

latest in debian security problems and in what version they have been fixed: https://security-tracker.debian.org/tracker/source-package/linux

https://www.linux.com/news/event/elce/2017/hardening-kernel-protect-against-attackers

compiling kernel takes a lot of CPU processing power and time. cputime so to speak – but ALSO harddisk space. 14GByte in my case.

root@debian9:~#  du -hs /usr/src/*
1.3M    /usr/src/linux-config-4.9
160K    /usr/src/linux-patch-4.9-rt.patch.xz
14G     /usr/src/linux-source-4.9
91M     /usr/src/linux-source-4.9.tar.xz

it (per default) compiles ALL the drivers and modules that it could possibly need… (default compile target all)

So if you do your compilation inside a VM give it atleast access to 2 cores… to speed things up a little 😉

per default make will consume 100% of one core – but you can start it multi-threaded…

On distributed-memory systems, you can use distcc to farm out compile jobs to other machines. This takes a little bit of setup, but it can really speed up your build if you happen to have some extra machines around.

On shared-memory multicore systems, you can just use make -j, which will try to spawn build jobs based on the dependencies in your makefiles. You can run like this:

make -j

which will impose no limit on the number of jobs spawned, or you can run with an integer parameter:

make -j2

which will limit the number of concurrent build jobs. Here, the limit is 2 concurrent jobs. Usually you want this to be something close to the number of cores on your system – but at the same time – not make it overheat 😀

(src)

further more you could meassure the time it takes to compile with

time make -j2

“If you have an Athlon XP CPU instead of an Intel 386 family processor, or lots of RAM, or a certain graphics card, you can tweak the kernel and compile it for optimum performance on your specific hardware. In the process, you can also probably free some memory by getting rid of unneeded options, shorten boot time, and increase responsiveness.” (src)

if you need NFS support – it is said to be better (performance wise) to be compiled into the kernel than loaded as a module… can’t proof that.

But as always – keep it small and beautiful (Unix Philosophy) – the less software you need – the better for security and stability.

“if you were to compile all possible functions as modules, the kernel would be as small as possible, and you would only suffer a small performance hit the first time a module was loaded.

However, you don’t want to make everything a module, because some functions need be available at boot: for instance, you couldn’t read the partition that holds the modules if you didn’t already have the needed code in the kernel.

But whenever it’s reasonable, set things up to be modules.”

kernel map

video

versions

sources and headers are stored under /usr/src

it used to be:

even version numbers = stable for production use

odd version numbers = testing/development/experimental

but not anymore since kernel version 2.6


major release

minor release

patch level

root@debian9:/usr/src# hostnamectl
 Static hostname: debian9
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 532eabca552b4075a8679094397c8dba
 Boot ID: 17db2c126e76421591e9b43e012201f2
 Virtualization: microsoft
 Operating System: Debian GNU/Linux 9 (stretch)
 Kernel: Linux 4.9.0-3-amd64
 Architecture: x86-64

download-install linux kernel headers

headers are enough to compile kernel-modules but not enough to compile the actual kernel.

apt-get install linux-headers-$(uname -r); # install linux header files

three tasks:

  1. compile from latest sources available in debian repository (currently: 4.9.30)
  2. compile very latest development kernel “unstable” debian (src) (4.11.6-1)
  3. compile very latest kernel.org sources of the lastest kernel version

… so let’s get started.

1. compile from latest sources available in debian repository (currently: 4.9.30)

uname -a; # show currently running kernel
Linux debian9 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
current
# add backports repo to sources.list
echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/apt/sources.list
aptitude update;
aptitude search linux-source
i linux-source - Linux kernel source (meta-package)
i A linux-source-4.9 - Linux kernel source for version 4.9 with Debian patches
aptitude install linux-source; # will install software needed to compile as well as the sources of the kernel you are currently using
aptitude install linux-source-4.9; # you probably want the latest kernel
aptitude install screen; # you might want to use screen to prevent abort by logout timeout during compilation
screen -S kernel; # start a screen session
cd /usr/src/;

tar Jfxv linux-source-4.9.tar.xz; # uncompress

root@debian9:/usr/src# ll
total 91M
drwxr-xr-x  4 root root 4.0K Jun 28 15:32 .
drwxr-xr-x 10 root root 4.0K Jun 27 11:06 ..
drwxr-xr-x  2 root root 4.0K Jun 28 15:31 linux-config-4.9
-rw-r--r--  1 root root 157K Jun 26 17:27 linux-patch-4.9-rt.patch.xz
drwxr-xr-x 23 root root 4.0K Jun 26 17:27 linux-source-4.9
-rw-r--r--  1 root root  91M Jun 26 17:27 linux-source-4.9.tar.xz

root@debian9:/usr/src# cd linux-source-4.9
# create simbolic link /usr/scr/linux
root@debian9:/usr/src# ln -sv linux-source-4.9 linux
root@debian9:/usr/src# cd linux
# reuse the last used config
root@debian9:/usr/src/linux# cp /boot/config-$(uname -r) .config
make clean; # do some magic
root@debian9:/usr/src/linux-source-4.9# make menuconfig; # let's you select/deselect wanted features
# if that fails with
HOSTCC scripts/kconfig/mconf.o
In file included from scripts/kconfig/mconf.c:23:0:
scripts/kconfig/lxdialog/dialog.h:38:20: fatal error: curses.h: No such file or directory
 #include CURSES_LOC
 ^
compilation terminated.
scripts/Makefile.host:124: recipe for target 'scripts/kconfig/mconf.o' failed
make[1]: *** [scripts/kconfig/mconf.o] Error 1
Makefile:548: recipe for target 'menuconfig' failed
make: *** [menuconfig] Error 2

# go
apt search libncurses
libncurses5-dev/oldstable,oldstable 5.9+20140913-1+deb8u2 amd64
developer's libraries for ncurses

apt install libncurses5-dev

# building 4.15 kernel failed me with some bio.h ssl stuff missing
apt install libssl-dev

# now rerun
make menuconfig; # let's you select/deselect wanted features
# or (debian specific?)
make nconfig

i just leave everything like it is and exit and save .config

if you select a entry and type ? you will get a short explanation of that element /searchterm

here you could append a custom string to your kernel version… kind of *brand* it.

time make -j2; # start compilation magic target "all" ... please stand by ...
make -j2 deb-pkg; # to build packages like

# if you want to download them:
# /usr/src/linux-image-4.15.0-rc5_4.15.0-rc5-1_i386.deb
# linux-image-4.15.0-rc5_4.15.0-rc5-1_i386.deb.sha512sum
# that can be installed later with dpkg -i linux-image.deb

# because one is operating inside a screen-session you can now detach
# and still have the process continue in the background all night long :-D
# hit:

Ctrl+A then D

# shows you the currently running sessions
screen -ls

There is a screen on:
 33795.kernel (06/28/2017 04:13:57 PM) (Detached)
1 Socket in /run/screen/S-root.

screen -R 33795.kernel; # to re-attach to that screen session

# next step will install binaries to their places in your system
make modules_install
make install
reboot

# if everything went good you should be able to verify that you installed a new kernel by
uname -a; # show currently running kernel
Linux debian9 4.9.30 #1 SMP Wed Jun 28 17:20:31 CEST 2017 x86_64 GNU/Linux
hostnamectl
 Static hostname: debian9
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 532eabca552b4075a8679094397c8dba
 Boot ID: 4d60c6fd21334858b2569177cfbfdd91
 Virtualization: microsoft
 Operating System: Debian GNU/Linux 9 (stretch)
 Kernel: Linux 4.9.30
 Architecture: x86-64

YES WE DID IT! 🙂

so now you have the most recent kernel compiled that is available through debian repositories.

2. compile very latest development kernel “unstable” debian (src) (4.11.6-1)

view latest changes:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

https://git.kernel.org/?s=idle

https://www.kernel.org/feeds/kdist.xml

To build a kernel image based on the kernel team’s unreleased development version:

apt-get install build-essential fakeroot rsync git
apt-get build-dep linux

The last two commands will install the build dependencies required by the kernel build process.

the sources take about 650MByte of disk space:

cd /usr/src/; # this is essential :-D all the next steps will be relative to this directory
git clone -b sid --single-branch https://anonscm.debian.org/git/kernel/linux.git

# sid currently means kernel version 4.11.7-1

git clone -b master --single-branch https://anonscm.debian.org/git/kernel/linux.git

# will currently download version 4.11.3-1~exp2

# 2017.01 will download kernel version 4.14.12-2
du -hs linux; # creates a new directory called linux with 719MBytes of sources
719M linux

so maybe one should work from lower-version to higher-version… so let’s start with 4.11.3…

This will check out the Debian packaging.

“dist” is normally the distribution codename such as “wheezy” or “sid” (unstable).

For the very latest version, usually based on an upstream release candidate, use “master”.

Note that this will download several hundred megabytes of data.

apt-get source -d linux

This will download the linux upstream source (and the last released Debian patches).

Depending on which version you are trying to build, you might need to override APT’s version selection or download a tarball from people.debian.org instead.

cd linux
debian/rules orig
debian/rules:78: *** Cannot find orig tarball linux_4.11.3.orig.tar.xz. Stop.

but google can… https://www.google.de/?gws_rd=ssl#q=”linux_4.11.3.orig.tar.xz”

mkdir ../orig;
cd ../orig;
wget https://launchpad.net/debian/+archive/primary/+files/linux_4.11.3.orig.tar.xz
# depending on what version you are compiling
wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.11.3.tar.xz
mv /usr/src/orig/linux-4.11.3.tar.xz /usr/src/orig/linux_4.11.3.tar.xz; # rename

wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.14.12.tar.xz
mv /usr/src/orig/linux-4.14.12.tar.xz /usr/src/orig/linux_4.14.12.orig.tar.xz; # rename

cd /usr/src/linux;
debian/rules orig; # rerun, should now work
mkdir -p ../orig
tar -C ../orig -xaf ../orig/linux_4.14.12.orig.tar.xz
# for 4.14.12 gave me some quilt error
# rules:68 Patch af9005 does not apply (enforce with -f)

This unpacks the upstream source and merges it with the Debian packaging.

debian/rules debian/control

This generates a Debian package control file based on the current definitions of the various kernel flavours which can be built.

debian rules debian control errors – but it is not bug – it’s a feature 😀

make nconfig; # nice gui

# all i changed here is under General -> set a custom string of cuztom
# save and quit

root@debian9:/usr/src/linux# scripts/config --disable DEBUG_INFO; # uses less disk space
root@debian9:/usr/src/linux# make clean
root@debian9:/usr/src/linux# screen -S kernel; # start new screen session
root@debian9:/usr/src/linux# time make -j4 deb-pkg

will meassure the time it takes to compile and start the compilation process with two processes running at the same time using two cpu cores.

As a result of the build, a custom kernel package linux-image-3.2.19_3.2.19-1_i386.deb (name will reflect the version of the kernel and build number) will be created in the directory one level above the top of the tree.

...
INSTALL debian/headertmp/usr/include/linux/wimax/ (1 file)
INSTALL debian/headertmp/usr/include/linux/ (453 files)
INSTALL debian/headertmp/usr/include/asm/ (62 files)
Using default distribution of 'unstable' in the changelog
Install lsb-release or set $KDEB_CHANGELOG_DIST explicitly
dpkg-deb: building package 'linux-headers-4.11.3cuztom+' in '../linux-headers-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb'.
dpkg-deb: building package 'linux-libc-dev' in '../linux-libc-dev_4.11.3cuztom+-1_amd64.deb'.
dpkg-deb: building package 'linux-image-4.11.3cuztom+' in '../linux-image-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb'.
dpkg-source: info: using source format '3.0 (custom)'
dpkg-source: info: building linux-4.11.3cuztom+ in linux-4.11.3cuztom+_4.11.3cuztom+-1.dsc
dpkg-genchanges: info: including full source code in upload

real    60m35.682s
user    110m24.744s
sys     6m31.984s
root@debian9:/usr/src/linux#

The newly created kernel image may be installed using dpkg just as any other package:

dpkg -i ../linux-image-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb;
reboot;

HURRAY! IT WORKS 🙂

root@debian9:/home/user# hostnamectl
   Static hostname: debian9
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 532eabca552b4075a8679094397c8dba
           Boot ID: b9f02930b8e14b39a56ce400384d2037
    Virtualization: microsoft
  Operating System: Debian GNU/Linux 9 (stretch)
            Kernel: Linux 4.11.3cuztom+
      Architecture: x86-64
fakeroot debian/rules target
make: *** No rule to make target 'target'. Stop.

optional: To build all possible packages for this architecture, run:

fakeroot debian/rules binary

To build all architecture-dependent packages, run:

fakeroot debian/rules binary-arch

To build all architecture-independent packages, run:

fakeroot debian/rules binary-indep

src: https://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official-vcs

3. compile very latest kernel.org sources of the latest kernel version

so the next challenge is to get the very-latest kernel sources straight from the Torvalds… and try to compile those.

latest kernel sources from https://www.kernel.org/

cd /usr/src/; # move to right dir
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.8.tar.xz; # download
tar Jfxv linux-4.11.8.tar.xz; # unpack

ln -sv linux-4.11.8 linux; # creat new symlink
cd linux;
root@debian9:/usr/src/linux# cp /boot/config-$(uname -r) .config;
make clean; # do some magic
make menuconfig; # let's you select/deselect wanted features

time make -j8 all; # start compilation magic ... please stand by ...
....
  IHEX2FW firmware/whiteheat_loader.fw
  IHEX2FW firmware/whiteheat.fw
  IHEX2FW firmware/keyspan_pda/keyspan_pda.fw
  IHEX2FW firmware/keyspan_pda/xircom_pgs.fw

real    32m17.975s
user    116m22.318s
sys     8m12.533s
...

# next step will install binaries to their places in your system
make install modules_install;
make install; # ?
reboot;

# if everything went good you should be able to verify that you installed a new kernel by
root@debian9:/home/user# hostnamectl
Static hostname: debian9
Icon name: computer-vm
Chassis: vm
Machine ID: 532eabca552b4075a8679094397c8dba
Boot ID: 253fa0fb912c4587850be9344da839fb
Virtualization: microsoft
Operating System: Debian GNU/Linux 9 (stretch)
Kernel: Linux 4.11.8cuztom
Architecture: x86-64
uname -a; # show currently running kernel
Linux debian9 4.11.8cuztom #1 SMP Thu Jun 29 16:01:42 CEST 2017 x86_64 GNU/Linux

holy moly! IT WORKS 🙂

misc:

kernel size

the kernel itself is a little larger than 4MByte…

root@debian9:~# ll /boot/vmlinuz-4.*
-rw-r--r-- 1 root root 4.2M Jun 29 15:07 /boot/vmlinuz-4.11.3cuztom+
-rw-r--r-- 1 root root 4.2M Jun 29 16:15 /boot/vmlinuz-4.11.8cuztom
-rw-r--r-- 1 root root 4.2M Jun 29 16:08 /boot/vmlinuz-4.11.8cuztom.old
-rw-r--r-- 1 root root 4.1M Jun 26 17:27 /boot/vmlinuz-4.9.0-3-amd64
-rw-r--r-- 1 root root 4.1M Jun 29 10:03 /boot/vmlinuz-4.9.30

pretty nice actually compared to the 7MBytes of Windows 8.1

dracut

This package builds a bootable initramfs for Linux kernel packages. The initramfs is loaded along with the kernel and is responsible for mounting the root filesystem and starting the main init system.

https://packages.debian.org/sid/utils/dracut

Links:

https://kernel-handbook.alioth.debian.org/

you can browser kernel sources online: https://sources.debian.net/src/linux/

watch Linus Torvalds at work at kernel.org: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

https://www.heise.de/ct/artikel/Die-Neuerungen-von-Linux-4-12-3712705.html

Among the material likely coming for Linux 4.13 that we have already covered on Phoronix includes:

– Initial AMD Raven Ridge graphics support (sans no display due to no DC/DAL yet), Vega fixes, and other updates.

Many Allwinner DRM changes.

DRM sync objects are landing.

Raspberry Pi / VC4 improvements.

Various updates to the Intel DRM driver.

Large directory support for EXT4.

XPad updates and Google Rose Touchpad support.

AES-128-CBC support in Fscrypt, the file-system generic crypto code currently utilized by EXT4 and F2FS.

– Possibly the AMD SME/SEV security features supported by new EPYC CPUs.

– Continued push for more HDMI CEC drivers.

Stay tuned for thorough Linux 4.13 kernel feature coverage once the merge window opens following the 4.12 debut.

src: http://www.phoronix.com/scan.php?page=news_item&px=Linux-4.13-Early-Look

https://wiki.archlinux.org/index.php/Kernels/Traditional_compilation#Compile_the_kernel

https://stackoverflow.com/questions/23050188/cant-make-menuconfig

https://www.linux.com/learn/recompile-your-kernel-perfect-fit

while compiling … you might come accross…

https://www.kernel.org/doc/Documentation/vm/hwpoison.txt

What is hwpoison?

Upcoming Intel CPUs have support for recovering from some memory errors
(“MCA recovery”). This requires the OS to declare a page “poisoned”,
kill the processes associated with it and avoid using it in the future.

This patchkit implements the necessary infrastructure in the VM.

To quote the overview comment:

  • High level machine check handler.
  • Handles pages reported by the hardware as being corrupted usually due to a 2bit ECC memory or cache failure.
  • this focusses on pages detected as corrupted in the background.
  • When the current CPU tries to consume corruption the currently running process can just be killed directly instead.
  • This implies that if the error cannot be handled for some reason it’s safe to just ignore it because no corruption has been consumed yet.
  • Instead when that happens another machine check will happen.
  • Handles page cache pages in various states.
    • The tricky part here is that we can access any page asynchronous to other VM users, because memory failures could happen anytime and anywhere, possibly violating some of their assumptions.
  • This is why this code has to be extremely careful. Generally it tries to use normal locking rules, as in get the standard locks, even if that means the error handling takes potentially a long time.
  • Some of the operations here are somewhat inefficient and have non linear algorithmic complexity, because the data structures have not been optimized for this case.
  • This is in particular the case for the mapping from a vma to a process. Since this case is expected to be rare we hope we can get away with this.
  • Limitations:
    • Not all page types are supported and never will. Most kernel internal objects cannot be recovered, only LRU pages for now.
    • Right now hugepage support is missing.

Andi Kleen, Oct 2009 (src)

liked this article?

  • only together we can create a truly free world
  • plz support dwaves to keep it up & running!
  • (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
  • really really hate advertisement
  • contribute: whenever a solution was found, blog about it for others to find!
  • talk about, recommend & link to this blog and articles
  • thanks to all who contribute!
admin