A k3s cluster is deceptively easy to create. One command can turn a Linux machine into a Kubernetes server; another can attach an agent. That convenience is valuable, but it can also hide the engineering work required to make a cluster reproducible.
The difficult question is not, “How do I install k3s?” It is, “How can I rebuild this environment next month, on different hardware, without relying on memory, SSH history, or a collection of half-documented manual steps?”
Terraform and Ansible answer that question well when each tool is given a clear boundary. Terraform creates the machines and the surrounding infrastructure. Ansible configures those machines and installs k3s. Kubernetes then manages the workloads running inside the cluster.
Think of the process as an assembly line:
- Terraform prepares the factory floor.
- Ansible equips and configures each station.
- k3s becomes the runtime where applications are assembled and operated.
This separation is more than an organizational preference. It is the foundation for predictable rebuilds, safer changes, and less configuration drift.
Start with ownership boundaries
The most common source of automation pain is asking one tool to own too much.
Terraform is strongest when it describes durable infrastructure resources: virtual machines, networks, disks, firewall rules, cloud-init metadata, DNS records, and IP allocations. In a Proxmox environment, that often means cloning a VM template, assigning CPU and memory, attaching storage, and configuring a network interface for every planned control-plane and worker node.
Ansible is strongest after those machines are reachable. It can install packages, manage system configuration files, configure kernel modules, apply sysctl settings, create users, distribute SSH keys, and install k3s in the required order.
k3s itself should own Kubernetes concerns: namespaces, Deployments, Services, Ingress resources, Helm releases, storage classes, and application configuration.
A healthy rule is this: Terraform should not SSH into a machine to perform long-lived server configuration, and Ansible should not become the source of truth for VM creation. Terraform provisioners can be useful as an escape hatch, but using them for the entire bootstrap process blurs state management and makes failures harder to reason about.
Design the Terraform output as an automation contract
Terraform does more than create VMs. It produces the information Ansible needs to configure them.
That output is a contract between the infrastructure layer and the configuration layer. Make it explicit.
For example, Terraform can output a structured list of nodes containing names, addresses, roles, and optional labels:
The exact resource names depend on the provider, but the principle remains: publish machine facts in a form another system can consume reliably.
Avoid building an inventory from assumptions such as sequential IP addresses or manually maintained host files. A static inventory works for a small lab until a node is replaced, renamed, or assigned a new address. A generated inventory makes the relationship visible and repeatable.
One practical pattern is to have a small script transform terraform output -json into an Ansible inventory file. Another is to use an inventory plugin or dynamically inject host variables into Ansible. The important part is that Terraform remains the authority for infrastructure identity.
Build a base image that is boring on purpose
A reusable Linux template dramatically improves provisioning speed and consistency. It should not contain k3s itself, cluster tokens, or environment-specific secrets. Those belong to Ansible or a secret-management system.
A good base image usually includes:
- A supported Linux distribution and current package updates
qemu-guest-agentfor VM IP discovery and lifecycle visibility- A dedicated automation user with SSH key access
- Python, because most Ansible modules require it
- Cloud-init support when the virtualization platform uses it
- Sensible baseline packages such as
curl,sudo, and CA certificates
Keep the template intentionally unremarkable. The template is the raw material, not the final product. If it accumulates cluster-specific configuration, every new environment inherits hidden assumptions.
Cloud-init can complement Terraform nicely. Terraform supplies machine-specific details such as hostname, SSH keys, static networking, and initial user data. Ansible then waits for SSH and applies the desired server state. This creates a clean sequence: clone, boot, discover, configure.
Make node preparation idempotent
Before k3s is installed, every node needs a small amount of operating-system preparation. This is where Ansible earns its place.
A typical preparation role might:
- Disable swap, if the selected configuration requires it
- Load required kernel modules such as
br_netfilterandoverlay - Set forwarding and bridge-network sysctl values
- Install utilities used for troubleshooting and storage integration
- Configure time synchronization
- Add internal certificate authorities, mirrors, or proxy settings
- Ensure hostnames and
/etc/hostsentries match the inventory
The goal is idempotence: running the playbook twice should produce the same result and should not perform unnecessary changes on the second run.
For example, use Ansible modules such as ansible.builtin.package, ansible.builtin.systemd, ansible.posix.sysctl, and ansible.builtin.copy rather than long shell commands whenever possible. Shell commands are occasionally necessary, especially around installation scripts, but they should be guarded with creates, changed_when, or version checks.
Idempotence is what turns automation from a launch sequence into an operational tool. When a worker is replaced or a setting changes, you should be able to rerun the same roles with confidence.
Bootstrap the first server deliberately
The first k3s server is a special machine. It creates the initial cluster state and generates the token that joining nodes need.
Install it separately from the remaining nodes. An Ansible inventory can represent this cleanly with a k3s_server_init group containing exactly one host, a k3s_servers group for all control-plane nodes, and a k3s_agents group for workers.
The initial server should receive explicit configuration rather than relying entirely on defaults. Depending on the environment, that may include:
- A stable API endpoint or virtual IP
- Disabled bundled components, such as Traefik, when another ingress strategy is planned
- A chosen CNI configuration
- Embedded etcd for a highly available server topology
- Node taints and labels
- A dedicated data directory or storage location
After installation, retrieve the join token from the initial server using slurp or a protected fact. Mark token-handling tasks with no_log: true. Tokens should never appear in terminal logs, CI output, or a committed inventory file.
For a single-server homelab, the server token is still sensitive. For a high-availability control plane, it becomes even more important because every additional server needs it to join the embedded etcd cluster.
Join servers before agents
Once the first server is healthy, add any additional server nodes. They should point at the stable API endpoint and use the server join token. Only after the control plane is established should worker nodes join using the agent installation path.
This order matters operationally. If agents arrive before the API endpoint is consistently available, you create noisy failures that look like networking issues but are really sequencing issues.
Use Ansible retries to wait for the API server, then validate from the initial server:
A provisioner should not declare success merely because the installation command exited with code zero. It should verify that nodes reach Ready, core system pods are healthy, and the expected labels or taints are present.
Treat secrets and state as production assets
Terraform state can contain sensitive values, including VM metadata, generated passwords, and IP addresses. Store remote state in an appropriate backend, protect access, and avoid committing local state files.
Likewise, use Ansible Vault, SOPS, or an external secret manager for secrets that cannot be generated dynamically. Examples include API credentials, private registry authentication, DNS provider tokens, and encrypted backup credentials.
The k3s token deserves special handling, but it should not be the only secret considered. The moment the cluster receives an ingress controller, external DNS integration, or persistent backup tooling, its automation becomes connected to more sensitive systems.
Make rebuilds a first-class test
The real test of this design is replacement.
Destroy one worker VM. Apply Terraform. Regenerate inventory. Run the Ansible playbook. Confirm that the new node joins with the correct name, labels, and configuration. Then test a more meaningful scenario: rebuild the entire cluster from code and restore application data from backups.
A cluster that only works when preserved is not fully provisioned. It is merely running.
Terraform, Ansible, and k3s form a strong pipeline because each layer can be inspected independently. Terraform shows what infrastructure should exist. Ansible shows how each host should behave. Kubernetes shows what workloads should run.
That clarity is the real outcome. The cluster stops being a handcrafted machine and becomes a repeatable system: one that can grow, fail, recover, and be rebuilt without turning its history into a dependency.
No comments yet. Be the first to share your thoughts!