cilium/little-vm-helper

little-vm-helper

little-vm-helper (lvh) is a VM management tool, aimed for testing and development of features that depend on the kernel, such as BPF. It is used in cilium, tetragon, and pwru. It can also be used for kernel development. It is not meant, and should not be used for running production VMs. Fast booting and image building, as well as being storage efficient are the main goals.

It uses qemu and libguestfs tools. See dependencies.

Configurations for specific images used in the Cilium project can be found in: https://github.com/cilium/little-vm-helper-images.

Usage

For an example script, see scripts/example.sh.

LVH can be used to:

• build root images for VMs
• download images
• build kernels
• download kernels
• boot VMs using above

Build root images

Build example images:

$ mkdir _data
$ go run ./cmd/lvh images example-config > _data/images.json
$ go run ./cmd/lvh images build --dir _data # this may require sudo as relies on /dev/kvm

The first command will create a configuration file:

jq . < _data/images.json
[
  {
    "name": "base.img",
    "packages": [
      "less",
      "vim",
      "sudo",
      "openssh-server",
      "curl"
    ],
    "actions": [
      {
        "comment": "disable password for root",
        "op": {
          "Cmd": "passwd -d root"
        },
        "type": "run-command"
      }
    ]
  },
  {
    "name": "k8s.qcow2",
    "parent": "base.img",
    "packages": [
      "docker.io"
    ]
  }
]

The configuration file includes:

• a set of packages for the image
• an optional parent image
• a set of actions to be performed after the installation of the packages. There are multiple actions supported, see pkg/images/actions.go.

Once the images build command completes, the two images described in the configuration file will be present in the images directory. Note that the images are stored as sparse files so they take less space:

$ ls -lh _data/images/
856M _data/images/base.img
1.7G _data/images/k8s.qcow2

Download images

Instead of building your own images, you can use lvh to download images built from our cilium/little-vm-helper-images repository. For that use the lvh images pull command.

You can find a list of the various images built under https://quay.io/organization/lvh-images, see more information the project README.md

If you don't know which image to choose, the most complete image to date is the kind type. You can find the various tags for each kernel versions under https://quay.io/repository/lvh-images/kind?tab=tags.

For example, pulling an image for testing the 5.15 kernel built by lvh image CI on 2026/01/06 would be:

lvh images pull quay.io/lvh-images/kind:5.15-20260106.135428

Build kernels

$ mkdir -p _data/kernels
$ go run ./cmd/lvh kernels --dir _data init
$ go run ./cmd/lvh kernels --dir _data add bpf-next git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git --fetch
$ go run ./cmd/lvh kernels --dir _data build bpf-next

Please note, to cross-build for a different architecture, you can use the --arch=arm64 or --arch=amd64 flag.

The configuration file keeps the url for a kernel, together with its configuration options:

$ jq . < _data/kernels.json
{
  "kernels": [
    {
      "name": "bpf-next",
      "url": "git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git"
    }
  ],
  "common_opts": [
    [
      "--enable",
      "CONFIG_LOCALVERSION_AUTO"
    ],
     ... more options ...
  ]
}

There are options that are applied to all kernels (common_opts) as well as kernel-specific options.

The kernels are kept in worktrees. Specifically, there is a git bare directory (git) that holds all the objects, and one worktree per kernel. This allows efficient fetching and, also, having each kernel on its own separate directory.

For example:

$ ls -1 _data/kernels
5.18/
bpf-next/
git/

Currently, kernels are built using the bzImage for x86_64 or Image.gz for arm64, and tar-pkg targets (see pkg/kernels/conf.go).

Download kernels

List the available versions

$ lvh kernels catalog
bpf-next
rhel8
4.9
4.19
[...]
6.3
6.6

Retrieve the tags for a given version:

$ lvh kernels catalog 6.6
6.6-20240123.120815
6.6-20240123.175813
[...]
6.6-20240404.144247
6.6-20240408.100959
6.6-main

See lvh kernels catalog --help for more details.

Download the kernel and related artifacts (BTF, modules, etc.)

$ lvh kernels pull 6.6-main
$ find 6.6-main/ -maxdepth 3
6.6-main/
6.6-main/boot
6.6-main/boot/vmlinuz-6.6.25
6.6-main/boot/btf-6.6.25
6.6-main/boot/System.map-6.6.25
6.6-main/boot/vmlinux-6.6.25
6.6-main/boot/config-6.6.25
6.6-main/lib
6.6-main/lib/modules
6.6-main/lib/modules/6.6.25

See lvh kernels pull --help for more details.

Booting images

You can use the run subcommand to start images.

For example:

go run ./cmd/lvh run --image _data/images/base.img --kernel _data/kernels/bpf-next/arch/x86_64/boot/bzImage

Or, with the kernel installed in the image, using the image bootloader:

go run ./cmd/lvh run --image _data/images/base.img

[!IMPORTANT] Only amd64 images contain a compatible bootloader. So even though kernels are present in the arm64 images, you'll need to supply it to QEMU through the --kernel lvh option flag from your host. See the above section "Download Kernels" on how to download kernels from these images.

[!TIP] As booting images goal is often to test BPF program on various kernel versions, here are some useful lvh options:

• --port to bind the VM ssh port to a custom port, letting you ssh -p 2222 root@localhost to have a fully capable shell.
• --host-mount to mount your working directory, making applying changes and testing them quicker (be aware that it requires kernel modules to load correctly).
• --cpu and --mem to adjust the specs for the VM, making building and running inside the VM quicker.

Overall, a typical command would look like this on x86_64 (for arm64, you would need to add --kernel for booting):

lvh run --image ~/_data/images/kind_5.15.qcow2 --port 2222:22 --host-mount ~/tetragon --cpu 4 --mem 8G

OCI images are also supported:

go run ./cmd/lvh run --image quay.io/lvh-images/root-images:main

[!NOTE] Building images and kernels is only supported on Linux. On the other hand, images and kernels already build on Linux can be booted in macOS (both x86 and arm64). The only requirement is qemu-system-x86_64. As macOS does not support KVM, the commands to boot images are:

go run ./cmd/lvh run --image _data/images/base.img --qemu-disable-kvm

FAQ

Why not use packer to build images?

Existing packer builders (e.g., https://github.com/cilium/packer-ci-build/blob/710ad61e7d5b0b6872770729a30bcdade2ee1acb/cilium-ubuntu.json#L19, https://www.packer.io/plugins/builders/qemu) are meant to manage VMs with longer lifetimes than a single use, and use facilities that introduce unnecessary overhead for our use-case.

Also, packer does not seem to have a way to provision images without booting a machine. There is an outdated chroot package https://github.com/summerwind/packer-builder-qemu-chroot, and cloud chroot builders (e.g., https://www.packer.io/plugins/builders/amazon/chroot that uses https://github.com/hashicorp/packer-plugin-sdk/tree/main/chroot).

That being said, if we need packer functionality we can create a packer plugin (https://www.packer.io/docs/plugins/creation#developing-plugins).

Why not use vagrant (or libvirt-based tools)?

These tools also target production VMs with lifetime stretching beyond a single use. As a result, they introduce overhead in booting time, provisioning time, and storage.

What are the dependencies of LVH?

On debian distribution, here is a list of packages needed for LVH to work.

TODO

• development workflow for MacOS X
• images: configuration option for using different deb distros (hardcoded to sid now)
• images: build tetragon images
  • unit tests
  • e2e tests (kind)
• images: docker image with required binaries (libguestfs, mmdebstrap, etc.) to run the tool - [x] is that possible? libguestfs needs to boot a mini-VM
• kernels: add support for building kernels
• runner: qemu runner wrapper
• images bootable VMs: running qemu with --kernel is convenient for development. If we want to store images externally (e.g., AWS), it might make sense to support bootable VMs.
• improve boot time: minimal init, use qemu microvm (https://qemu.readthedocs.io/en/latest/system/i386/microvm.html, https://mergeboard.com/blog/2-qemu-microvm-docker/)
• images: on a failed run, save everything in a image-failed-$(date) directory
• use guestfish --listen (see https://github.com/libbpf/ci/blob/cbb3b92facbad705bbb619b496d0debb4b3d806f/prepare-rootfs/run.sh#L345)

Notes

• earlier attempt: https://github.com/kkourt/kvm-dev-scripts