The Control Plane Relay: Provisioning k3s with Terraform and Ansible

A reliable k3s cluster is not created by one tool doing everything. It is created by a clean handoff between tools with different responsibilities.

Terraform is excellent at describing what infrastructure must exist: virtual machines, network interfaces, disks, IP assignments, DNS records, and firewall rules. Ansible is built for describing how those machines must be configured: operating-system packages, kernel settings, users, services, files, and the k3s installation itself.

Treating Terraform and Ansible as one indistinguishable provisioning layer is tempting. Terraform can run remote provisioners. Ansible can call virtualization APIs. Both approaches work for a first experiment. Over time, though, the boundary becomes blurred: infrastructure changes trigger imperative shell scripts, configuration state disappears into Terraform resources, and diagnosing failures becomes unnecessarily difficult.

A better model is a relay race. Terraform carries the infrastructure baton to a defined exchange point. Ansible takes it from there and converges the hosts into a Kubernetes cluster. k3s is the workload platform that emerges after the handoff.

Define the infrastructure contract first#

Before writing an Ansible playbook, define what Terraform promises to provide. This contract should be explicit enough that configuration automation does not need to guess about the environment.

For a small virtualized k3s cluster, Terraform commonly owns:

• Virtual machine creation and lifecycle
• CPU, memory, disk, and firmware configuration
• Network attachments and VLAN selection
• Static DHCP reservations or predictable IP addresses
• Cloud-init user data and SSH keys
• Security groups or virtual-network firewall rules
• DNS records, if the environment exposes an API for them
• Metadata that identifies server and agent nodes

For example, a Terraform module might create three control-plane nodes and two worker nodes. It should expose their addresses as structured outputs rather than burying them in resource names:

The exact provider is less important than the outcome: Terraform produces a machine-readable inventory boundary. That boundary is the baton.

Avoid making Terraform responsible for installing k3s through remote-exec. Provisioners are intentionally a last-resort feature in Terraform because they have weak lifecycle semantics. If SSH fails halfway through a shell script, Terraform knows a command failed, but it cannot model the detailed configuration state that was partially applied. The next apply may recreate infrastructure, retry an opaque script, or leave an operator to investigate manually.

Terraform should finish when the machines are reachable and satisfy a minimal bootstrap baseline. It does not need to know whether a Kubernetes node has joined the cluster.

Make the handoff deterministic#

Ansible needs an inventory. A manually maintained inventory.ini works initially, but it becomes a source of drift when VM addresses or node counts change. Generate the inventory from Terraform outputs instead.

One practical pattern is to serialize outputs after terraform apply:

The rendering script can assign hosts to meaningful groups:

A dynamic inventory plugin can also work, especially when infrastructure metadata is available from a cloud or virtualization API. The important design decision is not the format. It is that inventory is derived from the same source of truth that created the hosts.

At this point, add a readiness gate. A VM resource reported as created is not always a host ready for SSH. Cloud-init may still be configuring networking, package managers may hold locks, and DNS may not yet be visible. Ansible's wait_for_connection is a simple and useful first task:

This separates an ordinary startup delay from a genuine installation failure.

Build every node for Kubernetes before building the cluster#

k3s reduces operational complexity, but it does not remove host-level requirements. Use a common Ansible role for every node before differentiating servers from agents.

That role should handle the durable operating-system baseline:

• Install required packages such as curl, iptables, and NFS utilities when needed
• Disable swap, both immediately and persistently
• Load required kernel modules such as br_netfilter and overlay
• Apply Kubernetes networking sysctls
• Configure time synchronization and hostname resolution
• Create the administrative user and SSH policy
• Ensure a host firewall permits the intended cluster traffic

For example, Kubernetes packet forwarding requires these settings on each node:

This phase should be safe to rerun. Idempotence is more than an aesthetic preference: it makes repair possible. A node rebuilt from a template, a changed sysctl, or a failed package update should not require a new sequence of one-off commands.

Bootstrap the first server deliberately#

The first k3s server is special. It establishes the initial control-plane endpoint and creates the node token needed by joining nodes. Install it before attempting to configure additional servers or agents.

Use a configuration file rather than an oversized installation command. /etc/rancher/k3s/config.yaml makes the cluster's intent reviewable and version-controlled:

Whether to disable bundled components depends on the platform design. k3s includes Traefik and a service load balancer by default. Keeping them is sensible for many compact clusters. Disabling them is appropriate when an alternative ingress controller or load-balancer strategy is intentionally managed elsewhere. The key is to decide once and encode the decision consistently.

After installation, retrieve the token securely from the bootstrap server. Mark tasks that handle it with no_log: true, and do not print it in pipeline logs. Ansible can expose it as a delegated fact for the current run, while a secret manager is preferable for longer-lived automation.

Do not continue merely because the installation script exited successfully. Verify that the API server is responding and that the bootstrap node is registered:

Join servers, then agents, with stable endpoints#

High availability needs more than multiple server VMs. Joining nodes should use a stable control-plane address, such as a virtual IP, a TCP load balancer, or a DNS name backed by multiple healthy endpoints. Pointing every node at the first server's individual IP creates a quiet single point of failure.

Additional servers join with the server URL and token. Agents do the same with their role-specific configuration. Keep these roles separate even if their installation commands look similar. Servers need embedded datastore participation and control-plane ports; agents need only the cluster endpoint, token, and node-specific settings.

After each phase, validate from the cluster rather than only from Ansible task status:

These checks reveal problems that SSH-based automation cannot see: CNI failures, certificate mismatches, unready nodes, or system pods stuck in image pull loops.

Keep day-two operations in the design#

The most valuable provisioning workflow is one that remains understandable six months later. Store Terraform state remotely with locking. Pin Terraform providers, VM template versions, k3s versions, and Ansible collection versions. Keep generated inventory out of source control, but retain the code that generates it. Run terraform plan and Ansible syntax or lint checks in CI/CD before applying changes.

Most importantly, resist the urge to make every configuration concern part of cluster bootstrap. Terraform creates the terrain. Ansible prepares the hosts and forms the cluster. Kubernetes manifests, Helm releases, and application deployment should follow as a separate layer.

That separation gives each tool a clear story during an incident. If a VM is missing, inspect Terraform. If a node has the wrong kernel setting or k3s service state, inspect Ansible. If an ingress route or deployment is broken, inspect Kubernetes. The relay is complete when those boundaries are not only automated, but obvious.