A k3s cluster can be running within minutes. A repeatable k3s cluster—one that survives rebuilds, node replacement, and the inevitable “what changed?” investigation—requires more than a short install script.
The most useful way to automate k3s provisioning is to treat Terraform and Ansible as two specialists sharing a carefully designed handoff:
- Terraform creates durable infrastructure: virtual machines, networks, disks, firewall rules, addresses, and metadata.
- Ansible converges those machines into a Kubernetes control plane and worker fleet.
This boundary sounds obvious, but it is where many homelab and small-platform automation projects become fragile. Terraform knows a VM exists; it does not know whether SSH is ready, whether a package lock is active, or whether a k3s server has completed datastore initialization. Ansible can configure a machine, but it should not be responsible for discovering which VM Terraform meant to create.
A dependable cluster emerges when that handoff becomes an explicit interface rather than an informal sequence of commands.
Start with a cluster contract
Before writing resources or playbooks, define what a cluster is in machine-readable terms. A useful contract answers four questions:
- What nodes exist? Define stable names, roles, addresses, CPU, memory, storage, and placement.
- How are nodes reached? Specify SSH user, private network, bastion requirements, and host key strategy.
- What is the control-plane topology? Decide whether there is one server, embedded etcd high availability, or an external datastore.
- What should exist after convergence? Include k3s version, disabled bundled components, CNI assumptions, labels, taints, and kubeconfig delivery.
For a compact environment, a data structure such as this is often enough:
The important detail is not the syntax. It is that the same identity flows through every layer: VM name, DNS record, inventory hostname, Kubernetes node name, and monitoring label. Stable identity dramatically reduces operational ambiguity. k3s-agent-01 is easier to replace, investigate, and rejoin than a hostname assembled from ephemeral IDs.
Terraform should describe the floor, not the furniture
Terraform is at its best when declaring infrastructure that is expensive or undesirable to create manually. In a Proxmox-backed environment, that might include cloning a cloud image, attaching storage, setting CPU and memory, assigning network interfaces, and reserving IP addresses through a DHCP or DNS workflow.
It is tempting to use Terraform provisioners to install k3s immediately after VM creation. Resist that impulse except for narrow bootstrap tasks. Provisioners blur ownership and make failure recovery awkward: a resource may be successfully created while provisioning fails halfway through, leaving Terraform state accurate but the machine only partially configured.
Instead, Terraform should produce structured outputs that Ansible can consume:
This output is the handoff artifact. A pipeline can transform it into an Ansible inventory, or Ansible can use a dynamic inventory plugin or a small inventory-generation script. The implementation matters less than the principle: avoid manually maintaining a second list of host addresses in inventory.ini.
Terraform should also establish prerequisites that make configuration reliable:
- an SSH key injected through cloud-init or the VM template;
- DNS or predictable private IP allocation;
- a non-root automation user with passwordless sudo;
- firewall rules for SSH, Kubernetes API access, Flannel or alternative CNI traffic, and kubelet communication;
- sufficient disk capacity for container images and persistent volumes;
- VM guest-agent support, if the virtualization platform uses it to report addresses and readiness.
Cloud-init deserves special attention. A VM can be “created” while its cloud-init process is still creating users, expanding disks, or applying network configuration. Terraform completion is not operating-system readiness. Build that expectation into the next stage.
Make Ansible wait, then converge
The first responsibility of the Ansible workflow is not installing k3s. It is proving that each host is ready to receive configuration.
A practical playbook begins with wait_for_connection, gathers facts, and validates assumptions such as supported architecture, available disk, and operating system family. This turns hidden assumptions into visible failures. It is better to fail because /var/lib/rancher has too little free space than to discover ImagePullBackOff errors after deployment.
Next, apply a base role shared by servers and agents. Keep this role small and intentional:
- set hostnames and
/etc/hostsentries only when DNS is unavailable; - install required packages such as
curl,ca-certificates, andnfs-commonwhen needed; - disable swap where the chosen distribution and Kubernetes configuration require it;
- load required kernel modules and apply sysctl settings;
- configure time synchronization;
- create any storage mount points or dedicated data disks.
Avoid copying a large generic “Kubernetes hardening” role without understanding each setting. k3s deliberately bundles and configures much of the Kubernetes stack. Add only the operating-system controls your environment requires.
Bootstrap the first server as a special event
A control plane is not merely another host in an inventory group. The first server creates the cluster’s initial identity, token, certificates, and datastore state. Model it separately.
Install the initial server with a version pinned through a variable, not an unbounded curl | sh invocation. Pinning allows deliberate upgrades and makes a rebuild reproduce the same platform version.
For example, Ansible can render /etc/rancher/k3s/config.yaml before enabling the service:
Whether to disable the bundled ServiceLB or Traefik depends on the platform design. Do not disable components simply because an online guide does. Disable them when another component has a clearly defined responsibility for that function.
After starting k3s, wait for more than a listening port. Query kubectl get nodes, verify the server becomes Ready, and confirm the expected system pods settle. This creates a meaningful readiness checkpoint before agents attempt to join.
Treat the join token as a secret and a dependency
Agents need the server URL and join token. The token is not ordinary inventory data; it is cluster credential material.
Retrieve it from the initial server after k3s is ready, then keep it in Ansible memory for the current run or store it in an encrypted secret backend. Do not commit it to Git, print it in verbose logs, or place it in plain-text Terraform outputs.
Agent configuration can then be rendered consistently:
The agent role should be idempotent. If k3s is already installed and joined to the intended server, a second Ansible run should report little or no change. If the server address changes, make that a deliberate migration procedure rather than an accidental overwrite of a live node configuration.
Verify the platform, not just the installation
A green Ansible recap means tasks completed. It does not prove the cluster is usable. Add post-provision checks that exercise the platform from an operator’s perspective:
- all expected nodes are
Ready; - CoreDNS and networking components are healthy;
- a test workload receives an address and becomes ready;
- a service is reachable through the expected ingress or load-balancing path;
- the kubeconfig works from the administrative workstation or CI runner;
- node labels and taints match the declared topology.
These checks can be implemented with Ansible kubernetes.core modules, kubectl commands with explicit assertions, or a small smoke-test manifest applied and removed during provisioning.
For a cluster intended to host stateful workloads, test storage too. A successful pod is not evidence that a PVC can bind, mount, and retain data across a pod restart.
Design for reruns and replacement
The strongest automation is not judged by its first run. It is judged by its second run and by its behavior after a node fails.
Terraform should be able to replace a VM without forcing operators to rediscover its intended role. Ansible should be able to prepare the replacement, join it to the cluster, and validate it. The Kubernetes layer should then allow workloads to move according to their availability rules.
Document the destructive paths as carefully as the creation path:
- how to drain and remove an agent;
- how to rotate a join token;
- how to upgrade k3s in a controlled order;
- how to restore a server from snapshots or etcd backups;
- how to rebuild the cluster from Terraform state and Ansible configuration.
Terraform and Ansible are not competing tools in this workflow. Terraform establishes the terrain. Ansible gives the terrain a role. k3s becomes reliable when the boundary between those jobs is explicit, observable, and safe to cross again and again.
No comments yet. Be the first to share your thoughts!