intro:
it seems very relevant to have the late as possible kernel up and running: https://googleprojectzero.blogspot.com/2018/09/a-cache-invalidation-bug-in-linux.html
first: backup your system!
on boot time under “Advanced” you can chose to boot the old kernel – never the less – backup your system completely before progressing.
in general
https://www.linux.com/news/event/elce/2017/hardening-kernel-protect-against-attackers
compiling kernel takes a lot of CPU processing power and time. cputime so to speak – but ALSO harddisk space. 14GByte in my case.
root@debian9:~# du -hs /usr/src/* 1.3M /usr/src/linux-config-4.9 160K /usr/src/linux-patch-4.9-rt.patch.xz 14G /usr/src/linux-source-4.9 91M /usr/src/linux-source-4.9.tar.xz
it (per default) compiles ALL the drivers and modules that it could possibly need… (default compile target all)
So if you do your compilation inside a VM give it atleast access to 2 cores… to speed things up a little 😉
per default make will consume 100% of one core – but you can start it multi-threaded…
On distributed-memory systems, you can use distcc to farm out compile jobs to other machines. This takes a little bit of setup, but it can really speed up your build if you happen to have some extra machines around.
On shared-memory multicore systems, you can just use make -j, which will try to spawn build jobs based on the dependencies in your makefiles. You can run like this:
make -j
which will impose no limit on the number of jobs spawned, or you can run with an integer parameter:
make -j2
which will limit the number of concurrent build jobs. Here, the limit is 2 concurrent jobs. Usually you want this to be something close to the number of cores on your system – but at the same time – not make it overheat 😀
(src)
further more you could meassure the time it takes to compile with
time make -j2
“If you have an Athlon XP CPU instead of an Intel 386 family processor, or lots of RAM, or a certain graphics card, you can tweak the kernel and compile it for optimum performance on your specific hardware. In the process, you can also probably free some memory by getting rid of unneeded options, shorten boot time, and increase responsiveness.” (src)
if you need NFS support – it is said to be better (performance wise) to be compiled into the kernel than loaded as a module… can’t proof that.
But as always – keep it small and beautiful (Unix Philosophy) – the less software you need – the better for security and stability.
“if you were to compile all possible functions as modules, the kernel would be as small as possible, and you would only suffer a small performance hit the first time a module was loaded.
However, you don’t want to make everything a module, because some functions need be available at boot: for instance, you couldn’t read the partition that holds the modules if you didn’t already have the needed code in the kernel.
But whenever it’s reasonable, set things up to be modules.”
kernel map
video
versions
sources and headers are stored under /usr/src
it used to be:
even version numbers = stable for production use
odd version numbers = testing/development/experimental
but not anymore since kernel version 2.6
major release
minor release
patch level
root@debian9:/usr/src# hostnamectl Static hostname: debian9 Icon name: computer-vm Chassis: vm Machine ID: 532eabca552b4075a8679094397c8dba Boot ID: 17db2c126e76421591e9b43e012201f2 Virtualization: microsoft Operating System: Debian GNU/Linux 9 (stretch) Kernel: Linux 4.9.0-3-amd64 Architecture: x86-64
download-install linux kernel headers
headers are enough to compile kernel-modules but not enough to compile the actual kernel.
apt-get install linux-headers-$(uname -r); # install linux header files
three tasks:
- compile from latest sources available in debian repository (currently: 4.9.30)
- compile very latest development kernel “unstable” debian (src) (4.11.6-1)
- compile very latest kernel.org sources of the lastest kernel version
… so let’s get started.
1. compile from latest sources available in debian repository (currently: 4.9.30)
uname -a; # show currently running kernel Linux debian9 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux current # add backports repo to sources.list echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/apt/sources.list
aptitude update; aptitude search linux-source i linux-source - Linux kernel source (meta-package) i A linux-source-4.9 - Linux kernel source for version 4.9 with Debian patches aptitude install linux-source; # will install software needed to compile as well as the sources of the kernel you are currently using aptitude install linux-source-4.9; # you probably want the latest kernel aptitude install screen; # you might want to use screen to prevent abort by logout timeout during compilation screen -S kernel; # start a screen session cd /usr/src/; tar Jfxv linux-source-4.9.tar.xz; # uncompress root@debian9:/usr/src# ll total 91M drwxr-xr-x 4 root root 4.0K Jun 28 15:32 . drwxr-xr-x 10 root root 4.0K Jun 27 11:06 .. drwxr-xr-x 2 root root 4.0K Jun 28 15:31 linux-config-4.9 -rw-r--r-- 1 root root 157K Jun 26 17:27 linux-patch-4.9-rt.patch.xz drwxr-xr-x 23 root root 4.0K Jun 26 17:27 linux-source-4.9 -rw-r--r-- 1 root root 91M Jun 26 17:27 linux-source-4.9.tar.xz root@debian9:/usr/src# cd linux-source-4.9 # create simbolic link /usr/scr/linux root@debian9:/usr/src# ln -sv linux-source-4.9 linux root@debian9:/usr/src# cd linux # reuse the last used config root@debian9:/usr/src/linux# cp /boot/config-$(uname -r) .config make clean; # do some magic root@debian9:/usr/src/linux-source-4.9# make menuconfig; # let's you select/deselect wanted features # if that fails with HOSTCC scripts/kconfig/mconf.o In file included from scripts/kconfig/mconf.c:23:0: scripts/kconfig/lxdialog/dialog.h:38:20: fatal error: curses.h: No such file or directory #include CURSES_LOC ^ compilation terminated. scripts/Makefile.host:124: recipe for target 'scripts/kconfig/mconf.o' failed make[1]: *** [scripts/kconfig/mconf.o] Error 1 Makefile:548: recipe for target 'menuconfig' failed make: *** [menuconfig] Error 2 # go apt search libncurses libncurses5-dev/oldstable,oldstable 5.9+20140913-1+deb8u2 amd64 developer's libraries for ncurses apt install libncurses5-dev # building 4.15 kernel failed me with some bio.h ssl stuff missing apt install libssl-dev # now rerun make menuconfig; # let's you select/deselect wanted features # or (debian specific?) make nconfig
i just leave everything like it is and exit and save .config
if you select a entry and type ? you will get a short explanation of that element /searchterm
here you could append a custom string to your kernel version… kind of *brand* it.
time make -j2; # start compilation magic target "all" ... please stand by ... make -j2 deb-pkg; # to build packages like # if you want to download them: # /usr/src/linux-image-4.15.0-rc5_4.15.0-rc5-1_i386.deb # linux-image-4.15.0-rc5_4.15.0-rc5-1_i386.deb.sha512sum # that can be installed later with dpkg -i linux-image.deb# because one is operating inside a screen-session you can now detach # and still have the process continue in the background all night long :-D # hit: Ctrl+A then D # shows you the currently running sessions screen -ls There is a screen on: 33795.kernel (06/28/2017 04:13:57 PM) (Detached) 1 Socket in /run/screen/S-root. screen -R 33795.kernel; # to re-attach to that screen session # next step will install binaries to their places in your system make modules_install make install reboot # if everything went good you should be able to verify that you installed a new kernel by uname -a; # show currently running kernel Linux debian9 4.9.30 #1 SMP Wed Jun 28 17:20:31 CEST 2017 x86_64 GNU/Linux hostnamectl Static hostname: debian9 Icon name: computer-vm Chassis: vm Machine ID: 532eabca552b4075a8679094397c8dba Boot ID: 4d60c6fd21334858b2569177cfbfdd91 Virtualization: microsoft Operating System: Debian GNU/Linux 9 (stretch) Kernel: Linux 4.9.30 Architecture: x86-64
YES WE DID IT! 🙂
so now you have the most recent kernel compiled that is available through debian repositories.
2. compile very latest development kernel “unstable” debian (src) (4.11.6-1)
view latest changes:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
https://git.kernel.org/?s=idle
https://www.kernel.org/feeds/kdist.xml
To build a kernel image based on the kernel team’s unreleased development version:
apt-get install build-essential fakeroot rsync git apt-get build-dep linux
The last two commands will install the build dependencies required by the kernel build process.
the sources take about 650MByte of disk space:
cd /usr/src/; # this is essential :-D all the next steps will be relative to this directory git clone -b sid --single-branch https://anonscm.debian.org/git/kernel/linux.git # sid currently means kernel version 4.11.7-1 git clone -b master --single-branch https://anonscm.debian.org/git/kernel/linux.git # will currently download version 4.11.3-1~exp2 # 2017.01 will download kernel version 4.14.12-2 du -hs linux; # creates a new directory called linux with 719MBytes of sources 719M linux
so maybe one should work from lower-version to higher-version… so let’s start with 4.11.3…
This will check out the Debian packaging.
“dist” is normally the distribution codename such as “wheezy” or “sid” (unstable).
For the very latest version, usually based on an upstream release candidate, use “master”.
Note that this will download several hundred megabytes of data.
apt-get source -d linux
This will download the linux upstream source (and the last released Debian patches).
Depending on which version you are trying to build, you might need to override APT’s version selection or download a tarball from people.debian.org instead.
cd linux debian/rules orig debian/rules:78: *** Cannot find orig tarball linux_4.11.3.orig.tar.xz. Stop.
but google can… https://www.google.de/?gws_rd=ssl#q=”linux_4.11.3.orig.tar.xz”
mkdir ../orig; cd ../orig; wget https://launchpad.net/debian/+archive/primary/+files/linux_4.11.3.orig.tar.xz # depending on what version you are compiling wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.11.3.tar.xz mv /usr/src/orig/linux-4.11.3.tar.xz /usr/src/orig/linux_4.11.3.tar.xz; # rename wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.14.12.tar.xz mv /usr/src/orig/linux-4.14.12.tar.xz /usr/src/orig/linux_4.14.12.orig.tar.xz; # rename cd /usr/src/linux; debian/rules orig; # rerun, should now work mkdir -p ../orig tar -C ../orig -xaf ../orig/linux_4.14.12.orig.tar.xz # for 4.14.12 gave me some quilt error # rules:68 Patch af9005 does not apply (enforce with -f)
This unpacks the upstream source and merges it with the Debian packaging.
debian/rules debian/control
This generates a Debian package control file based on the current definitions of the various kernel flavours which can be built.
debian rules debian control errors – but it is not bug – it’s a feature 😀
make nconfig; # nice gui# all i changed here is under General -> set a custom string of cuztom # save and quit root@debian9:/usr/src/linux# scripts/config --disable DEBUG_INFO; # uses less disk space
root@debian9:/usr/src/linux# make clean root@debian9:/usr/src/linux# screen -S kernel; # start new screen session root@debian9:/usr/src/linux# time make -j4 deb-pkg
will meassure the time it takes to compile and start the compilation process with two processes running at the same time using two cpu cores.
As a result of the build, a custom kernel package linux-image-3.2.19_3.2.19-1_i386.deb (name will reflect the version of the kernel and build number) will be created in the directory one level above the top of the tree.
...
INSTALL debian/headertmp/usr/include/linux/wimax/ (1 file)
INSTALL debian/headertmp/usr/include/linux/ (453 files)
INSTALL debian/headertmp/usr/include/asm/ (62 files)
Using default distribution of 'unstable' in the changelog
Install lsb-release or set $KDEB_CHANGELOG_DIST explicitly
dpkg-deb: building package 'linux-headers-4.11.3cuztom+' in '../linux-headers-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb'.
dpkg-deb: building package 'linux-libc-dev' in '../linux-libc-dev_4.11.3cuztom+-1_amd64.deb'.
dpkg-deb: building package 'linux-image-4.11.3cuztom+' in '../linux-image-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb'.
dpkg-source: info: using source format '3.0 (custom)'
dpkg-source: info: building linux-4.11.3cuztom+ in linux-4.11.3cuztom+_4.11.3cuztom+-1.dsc
dpkg-genchanges: info: including full source code in upload
real 60m35.682s
user 110m24.744s
sys 6m31.984s
root@debian9:/usr/src/linux#
The newly created kernel image may be installed using dpkg just as any other package:
dpkg -i ../linux-image-4.11.3cuztom+_4.11.3cuztom+-1_amd64.deb;
reboot;
HURRAY! IT WORKS 🙂
root@debian9:/home/user# hostnamectl Static hostname: debian9 Icon name: computer-vm Chassis: vm Machine ID: 532eabca552b4075a8679094397c8dba Boot ID: b9f02930b8e14b39a56ce400384d2037 Virtualization: microsoft Operating System: Debian GNU/Linux 9 (stretch) Kernel: Linux 4.11.3cuztom+ Architecture: x86-64
fakeroot debian/rules target
make: *** No rule to make target 'target'. Stop.
optional: To build all possible packages for this architecture, run:
fakeroot debian/rules binary
To build all architecture-dependent packages, run:
fakeroot debian/rules binary-arch
To build all architecture-independent packages, run:
fakeroot debian/rules binary-indep
src: https://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official-vcs
3. compile very latest kernel.org sources of the latest kernel version
so the next challenge is to get the very-latest kernel sources straight from the Torvalds… and try to compile those.
latest kernel sources from https://www.kernel.org/
cd /usr/src/; # move to right dir wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.8.tar.xz; # download tar Jfxv linux-4.11.8.tar.xz; # unpack ln -sv linux-4.11.8 linux; # creat new symlink cd linux; root@debian9:/usr/src/linux# cp /boot/config-$(uname -r) .config; make clean; # do some magic make menuconfig; # let's you select/deselect wanted features
time make -j8 all; # start compilation magic ... please stand by ... .... IHEX2FW firmware/whiteheat_loader.fw IHEX2FW firmware/whiteheat.fw IHEX2FW firmware/keyspan_pda/keyspan_pda.fw IHEX2FW firmware/keyspan_pda/xircom_pgs.fw real 32m17.975s user 116m22.318s sys 8m12.533s ... # next step will install binaries to their places in your system make install modules_install; make install; # ? reboot; # if everything went good you should be able to verify that you installed a new kernel by root@debian9:/home/user# hostnamectl Static hostname: debian9 Icon name: computer-vm Chassis: vm Machine ID: 532eabca552b4075a8679094397c8dba Boot ID: 253fa0fb912c4587850be9344da839fb Virtualization: microsoft Operating System: Debian GNU/Linux 9 (stretch) Kernel: Linux 4.11.8cuztom Architecture: x86-64 uname -a; # show currently running kernel Linux debian9 4.11.8cuztom #1 SMP Thu Jun 29 16:01:42 CEST 2017 x86_64 GNU/Linux
holy moly! IT WORKS 🙂
misc:
kernel size
the kernel itself is a little larger than 4MByte…
root@debian9:~# ll /boot/vmlinuz-4.* -rw-r--r-- 1 root root 4.2M Jun 29 15:07 /boot/vmlinuz-4.11.3cuztom+ -rw-r--r-- 1 root root 4.2M Jun 29 16:15 /boot/vmlinuz-4.11.8cuztom -rw-r--r-- 1 root root 4.2M Jun 29 16:08 /boot/vmlinuz-4.11.8cuztom.old -rw-r--r-- 1 root root 4.1M Jun 26 17:27 /boot/vmlinuz-4.9.0-3-amd64 -rw-r--r-- 1 root root 4.1M Jun 29 10:03 /boot/vmlinuz-4.9.30
pretty nice actually compared to the 7MBytes of Windows 8.1
dracut
This package builds a bootable initramfs for Linux kernel packages. The initramfs is loaded along with the kernel and is responsible for mounting the root filesystem and starting the main init system.
https://packages.debian.org/sid/utils/dracut
Links:
https://kernel-handbook.alioth.debian.org/
you can browser kernel sources online: https://sources.debian.net/src/linux/
watch Linus Torvalds at work at kernel.org: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
https://www.heise.de/ct/artikel/Die-Neuerungen-von-Linux-4-12-3712705.html
Among the material likely coming for Linux 4.13 that we have already covered on Phoronix includes:
– Initial AMD Raven Ridge graphics support (sans no display due to no DC/DAL yet), Vega fixes, and other updates.
– DRM sync objects are landing.
– Raspberry Pi / VC4 improvements.
– Various updates to the Intel DRM driver.
– Large directory support for EXT4.
– XPad updates and Google Rose Touchpad support.
– AES-128-CBC support in Fscrypt, the file-system generic crypto code currently utilized by EXT4 and F2FS.
– Possibly the AMD SME/SEV security features supported by new EPYC CPUs.
– Continued push for more HDMI CEC drivers.
Stay tuned for thorough Linux 4.13 kernel feature coverage once the merge window opens following the 4.12 debut.
src: http://www.phoronix.com/scan.php?page=news_item&px=Linux-4.13-Early-Look
https://wiki.archlinux.org/index.php/Kernels/Traditional_compilation#Compile_the_kernel
https://stackoverflow.com/questions/23050188/cant-make-menuconfig
https://www.linux.com/learn/recompile-your-kernel-perfect-fit
while compiling … you might come accross…
https://www.kernel.org/doc/Documentation/vm/hwpoison.txt
What is hwpoison?
Upcoming Intel CPUs have support for recovering from some memory errors
(“MCA recovery”). This requires the OS to declare a page “poisoned”,
kill the processes associated with it and avoid using it in the future.
This patchkit implements the necessary infrastructure in the VM.
To quote the overview comment:
- High level machine check handler.
- Handles pages reported by the hardware as being corrupted usually due to a 2bit ECC memory or cache failure.
- this focusses on pages detected as corrupted in the background.
- When the current CPU tries to consume corruption the currently running process can just be killed directly instead.
- This implies that if the error cannot be handled for some reason it’s safe to just ignore it because no corruption has been consumed yet.
- Instead when that happens another machine check will happen.
- Handles page cache pages in various states.
- The tricky part here is that we can access any page asynchronous to other VM users, because memory failures could happen anytime and anywhere, possibly violating some of their assumptions.
- This is why this code has to be extremely careful. Generally it tries to use normal locking rules, as in get the standard locks, even if that means the error handling takes potentially a long time.
- Some of the operations here are somewhat inefficient and have non linear algorithmic complexity, because the data structures have not been optimized for this case.
- This is in particular the case for the mapping from a vma to a process. Since this case is expected to be rare we hope we can get away with this.
- Limitations:
- Not all page types are supported and never will. Most kernel internal objects cannot be recovered, only LRU pages for now.
- Right now hugepage support is missing.
Andi Kleen, Oct 2009 (src)
liked this article?
- only together we can create a truly free world
- plz support dwaves to keep it up & running!
- (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
- really really hate advertisement
- contribute: whenever a solution was found, blog about it for others to find!
- talk about, recommend & link to this blog and articles
- thanks to all who contribute!
