vmware esxi 6.7 and 7.0 nested within kvm-qemu – tinkering with challenges – sb.v00 failed to decompress (need more RAM) and admission check failed for memory resource

04.Oct.2021

General / Allgemein, Virtualization / KVM / VirtualBox / xenserver

let’s say: in theory – it works – in practice vmware and kvm (RedHat) would REALLY have to work together to make this work for real.

in reality – there are (currently) massive sata harddisk driver problems between esxi 6.7 and kvm-qemu’s virtual harddisk

admin-user know: esxi is hardware picky (Dell bought VMWare in 2004 so, “officially” only Dell servers are 100% supported, but many servers of other vendors (SuperMicro, HP) work too) and yes with a little tinkering, it esxi 6.7 can “work” nested within kvm-qemu (right now (2021-10 latest Debian, latest kvm-qemu, (will retest that on older hp machine with CentOS7)) only with Xeon CPUs and massive harddisk (driver?) problems, but on the net, someone made it work with AMD too), nested esxi 7.0 is work in progress…

here is a config that worked: esxi-6.7-nested-within-kvm-qemu-tested-config-vmname.xml.txt

basic setup of kvm-qemu on Debian:

https://dwaves.de/2020/12/05/gnu-linux-debian-10-how-to-install-kvm-virtualization-qemu-basic-virsh-commands/

esxi 6.7 setup:

vmware esxi within kvm-qemu – sb.v00 failed to decompress and admission check failed for memory resource – errors mean: need more RAM?

this is for homeLab testing, not recommended for production use, because: vendor will not give support.

ok it is experimental, but those strange error messages above could mean two things:

general help during setup: (what is missing here: Alt+F12 = the verbose debug log screen)

0. the right cpu for the job

SetupA) (works) SuperMicro + Xeon E3 (Ivy Bridge) + Debian 11 + esxi 6.7 (the software is the problem, not the hardware, aka storage controller (SATA) problems)

# with this hardware it runs just fine within esxi

hostnamectl; # tested on host
Operating System: Debian GNU/Linux 11 (bullseye)
Kernel: Linux 5.10.0-8-amd64
Architecture: x86-64

head /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 58
model name	: Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz
stepping	: 9
microcode	: 0x21
cpu MHz		: 1672.704
cache size	: 8192 KB
physical id	: 0

dmidecode |less
Manufacturer: Supermicro
Product Name: X9SCI/X9SCA

# run some checks
# (not all are required to pass)
# (those results are actual results for this setup)

su - root
get install cpu-checker
kvm-ok
INFO: /dev/kvm exists

virt-host-validate

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

egrep -c '(vmx|svm|ept)' /proc/cpuinfo
# should be more than 0
24

# want Y to be returned
cat /sys/module/kvm/parameters/ignore_msrs
N

# want N to be returned
cat /sys/module/kvm_intel/parameters/enable_apicv
N
# want Y to be returned
cat /sys/module/kvm_intel/parameters/nested
Y

# want Y to be returned 
cat /sys/module/kvm_intel/parameters/ept
Y

# (creditz: https://fabianlee.org/2018/09/19/kvm-deploying-a-nested-version-of-vmware-esxi-6-7-inside-kvm/)

1. not enough RAM! X-D

even when the checkbox “copy host cpu configuration” is set, this checkbox will automatically be disabled and “Enable available CPU security flaw mitigations” will automatically be enabled when vm is started. no worries.

SetupB) (would be super fast, but no functional (yet)) Asus + Ryzen 5 + NVMe

dmidecode | less
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: Pro WS 565-ACE
Version: Rev X.0x

# would be cool if it worked with that NVMe enabled hardware
head /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 113
model name	: AMD Ryzen 5 3600 6-Core Processor (which translates for kmv-qemu to EPYC-IBPB)
stepping	: 0
microcode	: 0x8701021
cpu MHz		: 2566.794
cache size	: 512 KB
physical id	: 0

For x86 guests there are 2 additional CPU flags associated with Spectre/Meltdown mitigation: spec-ctrl, and ibpb:

spec-ctrl: exposes Indirect Branch Restricted Speculation (IBRS)
ibpb: exposes Indirect Branch Prediction Barriers

src: https://www.qemu.org/2018/02/14/qemu-2-11-1-and-spectre-update/

# os used
hostnamectl 
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-17-amd64
      Architecture: x86-64

but then…. esxi 6.7 on kvm-qemu (Debian 10) OS: “Generic default” Copy CPU: “EPYC-IBPB” and 8GB of RAM

testing:

# if server is intel
cat /sys/module/kvm_intel/parameters/nested
1
# if server is amd
cat /sys/module/kvm_amd/parameters/nested
1

2. the picky harddisk controller problem

no harddisk found:

try this: stay at this screen for at least 10min… (Alt+F12 = watch the verbose debug output change as it detects)

1. actually starts detecting hardware BEFORE pressing enter

2. needs more time to detect hardware

wait for it…

dada!

hewego… X-D

thin-provision (bit slower but way smaller (dynamic growing) harddisk)

cat /scripts/kvm-qemu/harddisk_new_thin.sh
#!/bin/bash
PATH_VM="/where/vms/are/stored/$1.qcow2"
if test -f "$PATH_VM"; then
echo "... $PATH_VM hd file already exists, cancel process (otherwise will be overwritten)"
else
echo "... creating new thin provisioned virtual qcow2 harddisk with /path/name $PATH_VM and size $2GB ==="
qemu-img

3. virtual network adapter e1000e works for exsi 6.7

4. a working vmname.xml config example:

sometimes it is easier to copy and paste a working config via xml editor which can be done inside virt-manager (click on xml) or directly from bash-terminal:

here is a config that worked: esxi-6.7-nested-within-kvm-qemu-tested-config-vmname.xml.txt

virsh edit vmname

the end result should be…

esxi 7.0: problems with virtual e1000

the newer version of esxi does not recognize the e1000 anymore 🙁

enable libvirt kvm qemu logging:

# to get a more verbose output combine setting this
vim /etc/libvirt/libvirtd.conf
# find and unquote those lines
log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util"
log_outputs="3:syslog:libvirtd"

# no service needs to be restarted (?) "it just works" :)
# fire up this one-liner (which works if tail does not have to process too much log files X-D)

# without color but: you can scroll it :)
find /var/log/* -type f \( -name "*" \) ! -path '*.gz*' -exec tail -n0 -f "$file" {} +

# with color but no scroll :( (ccze is not available on all distros, but it is on CentOS7, Debian 9 to 11)
find /var/log/* -type f \( -name "*" \) ! -path '*.gz*' -exec tail -n0 -f "$file" {} + | ccze

https://dwaves.de/2017/06/15/linux-monitor-all-logs-in-real-time-d-follow-all-show-changes-to-log-files-under-varlog/

# sample output

==> /var/log/debug <== 
Oct 17 18:13:26 SuperStorageSrv1 libvirtd[572]: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

==> /var/log/syslog <== 
Oct 17 18:13:26 SuperStorageSrv1 libvirtd[572]: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

can it work on AMD CPUs: Ryzen?

theoretically yes

AMD‘s naming (other than intel i3, i5, i7, i9…) is a bit confusing, but basically there is currently the Ryzen 5, 7 and 9

a nice benchmark online comparison site: https://cpu.userbenchmark.com/

currently leading the charts: Core i9-11900K, +5% faster than AMD Ryzen 9 5900X (src)

Zen core architecture (2017–present)

Zen-based CPUs and some APUs use the “Ryzen”-brand: List of AMD Ryzen microprocessors, while some APUs use the brand “Athlon”: List of AMD accelerated processing unit microprocessors.

An AMD Ryzen 5 2600

Ryzen 1200 die

Zen series CPUs and APUs (released 2017)

Summit Ridge Ryzen 1000 series (desktop)
Whitehaven Ryzen Threadripper 1000 series (desktop)
Raven Ridge Ryzen 2000 APU series with RX Vega (desktop & laptop)
Naples Epyc (server)

Zen+ series CPUs and APUs (released 2018)

Pinnacle Ridge Ryzen 2000 series (desktop)
Colfax Ryzen Threadripper 2000 series (desktop)
Picasso Ryzen 3000 APU series with RX Vega (desktop & laptop)

Zen 2 series CPUs and APUs (released 2019)

Matisse Ryzen 3000 series (desktop)
Castle Peak Ryzen Threadripper 3000 series (desktop)
Renoir Ryzen 4000 APU series with RX Vega (desktop & laptop)
Lucienne Ryzen 5000 APU series (laptop)
Rome Epyc (server)

Zen 3 series CPUs and APUs (released 2020)

Vermeer Ryzen 5000 series (desktop)
Cezanne Ryzen 5000 series (laptop)
Milan Epyc (server)

src: https://en.wikipedia.org/wiki/List_of_AMD_processors#K8_core_architecture_(2003%E2%80%932014)