Introduction

Having recently upgraded from a Macbook Pro with an intel chip, to a M1 Macbook, one of the biggest challenges was getting minikube to run without jumping through a load of hurdles or having to port-forward just to get something similar to ingress working.

After trying multiple "solutions" (podman, docker etc) this post will describe how to use multipass to run minikube and then configuring your local machine to use it.

Updates and Edits

2022-08-02

After getting everything set-up as per the original article, I kept encountering weird edge cases where some amd64 images just wouldn't start; there were usually no errors, but the containers wouldn't output any logs when I knew there should have been some.

I tried a lot of different things, but what eventually solved my issue was to NOT install most of the packages needed to get this working (e.g. qemu-user-static, gcc-10-base) and instead use the tonistiigi/binfmt docker image.

My cloud-init.yaml file became the following:

---
package_update: true

packages:
  - curl
  - conntrack

## ...

# runcmd runs after the start-up process has completed
runcmd:
  # Install minikube (note: because I'm using a M1, I use the ARM64 architecture)
  - curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64
  - sudo install minikube-linux-arm64 /usr/local/bin/minikube
  # Install Docker
  - sudo mkdir -p /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  - sudo apt-get update
  - sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  # Add ubuntu user to the docker group (so you don't need to use sudo with docker commands)
  - sudo usermod -aG docker ubuntu
  # Ensure ubuntu owns the home directory
  - chown -R ubuntu:ubuntu /home/ubuntu
  # Install binfmt abstraction layer
  - docker run --privileged --rm tonistiigi/binfmt --install all

Multipass VM Setup

The rest of this tutorial assumes that you have installed multipass.

Firstly, we need to spin up a multipass VM. We'll create it with six CPUs, 10GB memory and 100GB disk. Feel free to adjust these as your use case requires.

multipass launch --name remote-minikube --cpus=6 --mem=10G --disk=100G Launched: remote-minikube

Next, we'll exec into the VM to install/configure what's needed

multipass exec remote-minikube -- /bin/bash</code>

The following commands are run while on the VM:

# Install minikube (note: because I'm using a M1, I use the ARM64 architecture)
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64
sudo install minikube-linux-arm64 /usr/local/bin/minikube

# Install Docker
sudo apt-get update

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

sudo mkdir -p /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Add your user to the docker group (so you don't need to use sudo with docker commands)
# Note: You'll need to log out/in for the changes to take effect.
sudo usermod -aG docker ${USER}

# Install conntrack
sudo apt-get install conntrack

# Install qemu-user-static to allow amd64 images to be used
sudo apt-get install qemu-user-static

# (Optional) install additional packages that may be needed for running amd64 images that use the C runtime
sudo apt-get install gcc-10-base libc6 libcrypt1 libgcc-s1 libidn2-0 libunistring2 zlib1g

# Run minikube with the bare-metal drive
minikube start  --extra-config=apiserver.service-node-port-range=30000-39999 --kubernetes-version 1.18.20 --driver=none

Confirm everything is working by running minikube status

minikube status minikube type: Control Plane host: Running kubelet: Running apiserver: Running
kubeconfig: Configured

Set-up local environment to use the VM minikube instance

Now our VM is running minikube, we need to configure our local environment to be able to talk to it.

While on the VM, grab the contents of ~/.kube/config and add it to your local ~/.kube/config. Note: If you're adding to an already existing config file, ensure you copy over the cluster, context and user. Ensure you change the folder path (/home/ubuntu) if appropriate.

Then, grab the contents of /home/ubuntu/.minikube/ca.crt, /home/ubuntu/.minikube/profiles/minikube/client.crt and /home/ubuntu/.minikube/profiles/minikube/client.key and add them to your local machine at the path that you've defined in ~/.kube/config.

Now, set your context to minikube using kubectx minikube.

You should now be able to query the pods, use kubectl apply etc.

kubectl get pods No resources found in default namespace

For extra convenience you can get the IP of the VM (by running ip a on the VM, or getting it from multipass list) and adding it to /etc/hosts under a name of your choice.

[vm ip]   minikube

Automating all the steps

Luckily for us, multipass supports cloud-init out of the box, so we can automate a lot of the initial set-up with a configuration file.

First, create a cloud-init.yaml file:

---
package_update: true

packages:
  - ca-certificates
  - curl
  - conntrack
  - gnupg
  - lsb-release
  - qemu-user-static
  - gcc-10-base
  - libc6
  - libcrypt1
  - libgcc-s1
  - libidn2-0
  - libunistring2
  - zlib1g

## Note: this isn't required but is provided as a useful reference
## for what cloud-init can do
write_files:
  - content: |-
      my dummy content

    owner: ubuntu:ubuntu
    path: /home/ubuntu/file.txt
    permissions: '0644'

## bootcmd happens at the beginning of the start-up process
## Note: this isn't required but is provided as a useful reference
## for what cloud-init can do
bootcmd:
  - echo $(whoami) > /root/boot.txt

# runcmd runs after the start-up process has completed
runcmd:
  # Install minikube (note: because I'm using a M1, I use the ARM64 architecture)
  - curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64
  - sudo install minikube-linux-arm64 /usr/local/bin/minikube
  # Install Docker
  - sudo mkdir -p /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  - sudo apt-get update
  - sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  # Add ubuntu user to the docker group (so you don't need to use sudo with docker commands)
  - sudo usermod -aG docker ubuntu
  # Ensure ubuntu owns the home directory
  - chown -R ubuntu:ubuntu /home/ubuntu

Now, coupling this configuration file with a couple make recipes we can automate almost everything:

## Makefile
SHELL:=/bin/bash -o pipefail

VM_EXISTS:=$(shell multipass list | grep '^minikube\s' | awk '{print $$1}')
.PHONY: multipass-minikube-start
multipass-minikube-start:
ifeq ("$(VM_EXISTS)", "")
	multipass launch --name minikube --cpus=6 --mem=10G --disk=100G --cloud-init ./cloud-init.yaml
	multipass exec minikube -- minikube start --extra-config=apiserver.service-node-port-range=30000-39999 --kubernetes-version 1.18.20 --driver=none
else
	@echo "minikube vm is already running"
endif

.PHONY: multipass-configure
multipass-configure: multipass-minikube-start
	# Create the minikube directory if it doesn't exist
	mkdir -p ~/.minikube/profiles/minikube
	# Ensure the correct files are copied over from the VM
	multipass transfer minikube:/home/ubuntu/.minikube/ca.crt /Users/${USER}/.minikube/ca.crt
	multipass transfer minikube:/home/ubuntu/.minikube/profiles/minikube/client.crt /Users/${USER}/.minikube/profiles/minikube/client.crt
	multipass transfer minikube:/home/ubuntu/.minikube/profiles/minikube/client.key /Users/${USER}/.minikube/profiles/minikube/client.key
	# Copy over the VM kube config and place it in a new file
	multipass transfer minikube:/home/ubuntu/.kube/config ~/.kube/remote-config
	# Update the contents for the current user
	sudo sed -i "" 's/home\/ubuntu/Users\/${USER}/g' ~/.kube/remote-config
	# Change the name to remote-minikube to avoid conflicts
	sudo sed -i "" "s/name: minikube/name: remote-minikube/g" ~/.kube/remote-config
	sudo sed -i "" "s/cluster: minikube/cluster: remote-minikube/g" ~/.kube/remote-config
	sudo sed -i "" "s/user: minikube/user: remote-minikube/g" ~/.kube/remote-config
	# Remove current-context to avoid conflicts
	sudo sed -i "" '/^current-context/d' ~/.kube/remote-config
	# Use the new file
	export KUBECONFIG="${HOME}/.kube/config:${HOME}/.kube/remote-config"

Then, we can use kubectl --context=remote-minikube to interact with the remote cluster or we can change our context permanently with kubectx remote-minikube.

Tip: if needed, the default location for cloud-init logs will be /var/log/cloud-init-output.log, on the VM.