I’ve been studying for the Cloud Native Computing Foundation’s CKA and CKAD certification exams for a while now. Anyone familiar with these exams knows that in order to practice, you need a cluster. Hands-on experience with Kubernetes clusters, of the kubeadm variety, is key to passing these exams.

For the past year and a half, I’ve had access to an AWS environment, with a GitLab repository, flashy CI/CD and Terraform/Packer to deliver one or more pre-provisioned Kubernetes environments for me to teach Kubernetes to students, or to practice it on my own.

Now that I’m out on my own, and that lab environment is a thing of the past, I’ve been imagining a replacement of my own.

I had a few requirements:

  • Keep the code in GitLab for development and personal use
  • Use CI/CD
  • Periodically push the updated code to GitHub to share with the world
  • Use Terraform for automation
  • Flexibility in configuration of cluster with variables
  • Leverage KVM and libvirt
  • Execute workloads locally
  • Avoid Cloud expenses
  • Keep the project simple and straightforward
  • Iterate as needed
  • Build something I can use with the CNCF CKA/D courses
  • Test and evaluate cluster build process and scripts
    • LFS258 (CKA)
    • LFD259 (CKAD)

So, how do we get from here to there?

Terraform module for KVM/Libvirt virtual machines

During my research for this project, I stumbled across the Libvirt VM Terraform module.

The Hashicorp Terraform Registry describes it:

  • Creates one or more VMs
  • One NIC for each domain, connected to the network using the bridge interface
  • Setup network interface using DHCP or static configuration
  • cloud_init VM configuration (Ubuntu and Netplan compliance)
  • Optionally add multiple extra disks
  • Tests the ssh connection

Sounds good to me! By using one invocation of the module for Control Plane hosts, and one for Worker nodes, you can build a flexible cluster configuration that you can configure for clusters of varying sizes and node counts.

Hardware details for a lab in a box

I have a Dell Precision T5500 workstation sitting under my desk, which I recently upgraded:

  • Ubuntu 22.04
  • Dual six-core Xeons
  • Plenty of storage
    • RAID-1 SSD for system: 250GB
    • RAID-1 HDD for data: 6TB
  • Dual 1GB network connections
    • Public/General network
    • Private/Data/Application Network
  • 72GB of RAM
  • Decent GPU, possible future use?

The workstation was already up and running with Ubuntu 22.04, serving as my “Lab in a Box” server:

  • GitLab: Keep it local!
    • Place to do primary development
      • Release projects/content on GitHub
    • Local GitLab instance
    • Publically accessible (by me)
    • CI/CD
      • KVM/libvirt runner
  • Terraform
    • Write automation for labs and projects
    • Execute workloads on KVM/libvirt
  • Packer
    • Create images for projects
    • Also use virt-builder and Image Builder as needed
  • Ansible
    • Might use in future projects, or for lab infrastructure
  • Cockpit
    • Web-based server information and management
    • For when you want to treat your “lab in the box” as an appliance and skip the CLI

Terraform and KVM/libvirt are already installed and configured on my server. All I needed to do was to enable dnsmasq and change the subnet on the default KVM network.

Adding the CKA/D Cluster Builder functionality should be rather straightforward.

Install Terraform

I’ve written quite a bit of infrastructure-as-code in Terraform, using AWS, for lab environments and projects in the past. I decided to try my hand with Terraform with KVM/libvirt to provide Infrastructure-as-Code (IaC) automation for my projects.

To install, add the repositories for Hashicorp. Once the repositories have been added, install Terraform.

Verify your installation by checking the version of Terraform:

Assuming you see a version number in response, Terraform is good to go. Next is KVM.

KVM and Libvirt

The foundation of the “Lab in a Box” stack is libvirt. That’s what executes the Terraform workloads, so you must install KVM and libvirt on your computer. The exact set of steps vary based on your Linux distribution. Here are some relevant links for Ubuntu and RHEL:

Once you have KVM/libvirt installed, verify it:

Normally, you’ll be able to list virtual machines:

Unless you happen to have virtual machines running, this command currently returns nothing. That’s to be expected. Now you can move on to configuring the default libvirt network.

Configure the default libvirt network

For your cluster to work, you need proper name resolution on your default bridge network. You can accomplish this by configuring dnsmasq on your default bridge network. Take a look at your default bridge network:

This is my network configuration:

My network has the following changes from a stock implementation with libvirt:

  • I’ve changed the network from 192.168.122.0 to 10.0.1.0, because the CNCF CKA/D courses specify to NOT use the 192.168 network for nodes
  • I’ve enabled dnsmasq, with the line <domain name='k8s.local' localOnly='yes'/>
  • Local domain is k8s.local

If you need to edit your network, using my configuration as a guide, the procedure is:

Take down the default network:

Edit the default network:

Make your edits as needed.

Deploy the new default network configuration:

Do this without any virtual machines deployed to the default network. Do one final check of your default network configuration:

Configure a Storage Pool

You need a storage pool for your virtual machines. The Terraform file is configured to use the default storage pool by default. You can change it in the variables if you want to use a different pool.

If you need to create a storage pool, reference Creating a Directory-based Storage Pool with virsh. It’ll get you up and running in no time.

Check for the default storage pool:

As long as you have one or more storage pools configured, you see them in the output.

Verify the default storage pool configuration:

I’ve configured my default storage pool as a directory-based pool, backed by an LVM volume mounted at /media/virtual-machines:

Now that you have everything installed and configured, let’s get the code!

Git clone

Head on over to the cka-d-cluster-builder-lab Git repository. Go ahead and clone the repository.

Change directory into the repository directory and take a look at the code:

By default, a cluster will be created with a single control plane node and two worker nodes. Feel free to adjust the values as you see fit. You can change all kinds of things in the variables at the top of the file. For instance, the node count, by node type:

Node name prefix, by node type:

Node sizing, by node type:

Disk pool, by node type:

User/Key information, all nodes:

Other information, all nodes:

Feel free to customize as needed. The virtual machine sizing is based on the requirements for the CNCF LFD259 course, but you might be able to tighten things up on disk and memory and run an acceptable cluster. Alternatively, if you want to build clusters with more nodes and you have the resources—go for it!

How does it work? The rest is the stuff you should never have to touch. Just use the variables to change the configuration.

Configuring the Terraform provider

I use libvirt as a provider:

Build everything and report the details:

Deploy a Kubernetes cluster using Terraform and KVM virtualization

OK, time to deploy a cluster! First, initialize your Terraform environment:

Next, generate a Terraform plan:

This shows you what Terraform will build, and validates the code for errors. There are no errors, so proceed with a terraform apply. Terraform handles all the heavy lifting, and builds the environment as requested:

Look at the virtual machines that have been created:

To see the IP addresses for the nodes:

The three nodes are ready! For now, leave your cluster up. Time to build a Kubernetes cluster!

Deploy a Kubernetes cluster nodes on KVM Virtual Machines

To get started, log into each node with a separate connection (use a different window or tab or a multiplexer), and become the root user. When deploying a Kubernetes cluster, you have choices of a container runtime. I’m writing up two options, but feel free to experiment!

Option 1: Docker

Perform these steps on each node.

First, uninstall existing versions of Docker containerd:

Install software prerequisites, if needed:

Add the Docker repository GPG keys:

Add Docker repository:

Install containerd:

Add the ubuntu user to the docker group, and then enable the Docker daemon:

Use systemctl status docker command to confirm that it’s running, and then press Q.

There’s an issue with the stock /etc/containerd/config.toml file and Kubernetes 1.26 and above. Set the configuration file aside, and restart the containerd service.

Backup and disable existing /etc/containerd/config.toml:

Configure /etc/containerd/config.toml:

Container runtime installed!

Docker configuration

Perform the following steps on all nodes.

Configure the systemd cgroup driver:

Load the overlay and br_netfilter modules:

Configure kernel parameters for bridging and IPv4 forwarding:

Apply kernel parameters:

Node configuration is complete. Proceed to Deploy the Kubernetes Cluster using kubeadm now.

Option 2: CRI-O

Add CRI-O Repositories:

Add CRI-O Repository GPG Keys:

Install CRI-O Packages:

Set the cgroup driver:

Override the Pause Container Image:

Sync CRI-O and the distribution’s runc versions:

Enable and start the CRI-O service:

Now you have a container runtime installed, time to do some configuration.

CRI-O configuration

Perform the following steps on all nodes.

Load the overlay and br_netfilter modules:

Configure kernel parameters for bridging and IPv4 forwarding:

Apply kernel parameters:

Node configuration is complete.

Deploy the Kubernetes cluster using kubeadm

Now that all nodes are prepared and have a container runtime installed and running, you can deploy a Kubernetes cluster using kubeadm. Start by installing the kubeadm packages. Perform the following steps on all nodes.

Update apt index and install prerequisites:

Download Kubernetes Repository GPG Key:

Add the Kubernetes repository:

See which versions of Kubernetes are available, looking for the latest available (1.26 at the time of this writing):

Install the release just before the lastest version, so you can practice upgrading to the latest.

Refresh and install Kubernetes packages:

Lock down Kubernetes package versions:

Pull Kubernetes container images on the Control Plane

Perform the following step on the Control Plane node. Pull the Kubernetes container images:

All the Linux work is done! We can proceed to doing the Kubernetes things!

Deploy a Kubernetes cluster using kubeadm

kubernetes.io: Creating a cluster with kubeadm

Now, you can use kubeadm to configure and deploy Kubernetes on the nodes you’ve set up. First, inititalize the Control Plane Node:

Next, configure kubectl:

Verify your work:

Now install a networking provider to get things working. Deploy Calico Networking CNI:

Perform the following steps on the Control Plane node. Download the Calico CNI manifest:

If you want to customize your Calico deployment, just edit the calico.yaml manifest. To deploy it:

Verify your work:

The status of the control plane changes to Ready after a minute or so. Type CTRL-C to exit the watch command.

Now you have a functional Control Plane!

Join the Worker nodes to the cluster

Now that you have a functional Control Plane, you can join the worker nodes to the cluster. Use the join command that you saved when you initialized the Control Plane. You must use sudo to run the command as root.

Perform the following steps on all Worker nodes (this is an example, your command will be different):

If you can’t remember the join command, use the following command to retrieve it:

Join each worker node to the cluster, using sudo:

Confirm Worker nodes

Perform the following steps on the Control Plane node.

Verify that the Worker nodes have joined the cluster:

Once all nodes are in the Ready state, exit the watch command with CRTL-C.

Take a look at all the resources in our cluster:

There you have it! A Kubernetes cluster, assembled with kubeadm.

OPTIONAL: Configure kubectl on your virtualization host

You might want to manage your cluster using kubectl on your virtualization host. This is a common use case scenario that’s easy to implement. If you don’t have kubectl installed on your virtualization host, install it before continuing.

Copy KUBECONFIG to KVM host:

You can get fancy and create a context for it, but I’m not covering that here. Check your work:

When you’re done with the cluster, you can unset KUBECONFIG to stop using the configuration.

Tear down the KVM virtual machines using Terraform

When you’re done, you use the terraform destroy command to tear down the cluster:

Verify your work:

All cluster resources have been destroyed. All cleaned up!

Summary

By leveraging the Terraform Module for KVM/Libvirt Virtual Machines, you can build a set of nodes on a KVM hypervisor, in a quick and repeatable way. It’s perfect for getting hands-on with Kubernetes clusters. Using the materials in this repository, you can customize the Terraform for your needs and use cases.

References

Author


Tom Dean

Just a guy, learning all the Kubernetes things and automating the world!

0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *