The integration of private container registries within a K3s environment is a critical architectural requirement for production-grade deployments. In enterprise settings, the reliance on public repositories like Docker Hub is often untenable due to security constraints, such as the need to keep proprietary images off public repositories, or performance bottlenecks where wide-area network latency increases image pull times. K3s addresses these challenges by providing a mechanism to configure containerd—its embedded container runtime—via a specific configuration file located at /etc/rancher/k3s/registries.yaml. This file allows operators to define mirrors, manage authentication credentials, and configure Transport Layer Security (TLS) settings. Because K3s utilizes an embedded containerd instance, the registries.yaml file serves as the primary interface for translating high-level registry requirements into the low-level configuration that containerd requires to pull images.

The Registry Configuration Architecture

K3s implements its registry management by checking for the existence of the /etc/rancher/k3s/registries.yaml file upon startup. If this file is present, K3s uses the definitions contained within to generate the corresponding containerd configuration. This architecture ensures that registry settings are decoupled from the main K3s service configuration, allowing for easier updates and node-specific adjustments.

The deployment of this file is not centralized; it must exist on every single node in the cluster that is required to pull images from a private registry. This includes both server nodes and agent nodes. A common oversight in cluster administration is neglecting the server nodes. Since server nodes are schedulable by default, if they have not been explicitly tainted to prevent workloads from running on them, they will attempt to pull images for any pods scheduled to them. Consequently, the absence of registries.yaml on a server node will lead to image pull failures if the requested image resides in a private repository.

Implementation of the Registries.yaml File

The /etc/rancher/k3s/registries.yaml file is the central point of configuration for all image retrieval logic. It supports several key sections: mirrors, configs, and authentication.

To create or update this configuration, administrators typically use a command that writes directly to the file system. An example of a basic configuration involves mirroring docker.io to a private endpoint while providing authentication and TLS certificates.

```bash
sudo tee /etc/rancher/k3s/registries.yaml > /dev/null <

registries.yaml - K3s private registry configuration

mirrors:

Try your private mirror first for docker.io pulls

"docker.io":
endpoint:
- "https://registry.example.com"
configs:

Authentication for the private registry

"registry.example.com":
auth:
username: "admin"
password: "RegistryPassword"
tls:

CA certificate for the registry's TLS certificate

ca_file: "/etc/rancher/k3s/registry-ca.crt"
EOF
```

Once this file is placed on the node, the K3s service must be restarted to apply the changes. The specific command depends on the role of the node:

• For server nodes: sudo systemctl restart k3s
• For agent nodes: sudo systemctl restart k3s-agent

Understanding Mirror Configuration and Fallback Logic

Mirrors are used to redirect requests for images from a public registry to a private endpoint. When a mirror is configured for docker.io, containerd will attempt to pull the image from the specified endpoint first.

Containerd employs an implicit "default endpoint" for all registries. This default endpoint acts as a safety net and is always tried as a last resort, regardless of whether other endpoints are listed in the registries.yaml file. For example, the default endpoint for docker.io is https://index.docker.io/v2. If a pull request is made for registry.example.com:5000/rancher/mirrored-pause:3.6, containerd will eventually fall back to https://registry.example.com:5000/v2.

It is important to note that rewrites are not applied to pulls against the default endpoint. Furthermore, if an administrator wishes to completely prevent containerd from attempting to reach the default registry endpoint, the node must be started with the --disable-default-registry-endpoint flag.

Advanced TLS and Authentication Strategies

Security for private registries is managed through the configs section of the registries.yaml file. Depending on the security requirements of the environment, several levels of TLS configuration are available.

Certificate Authority (CA) Integration

For registries using custom CA certificates, the certificate must be present on the K3s node. The certificate can be transferred from the registry server using secure copy:

bash scp registry-server:/etc/ssl/certs/registry-ca.crt /etc/rancher/k3s/registry-ca.crt

The configuration in registries.yaml then points to this file:

yaml configs: "registry.example.com:5000": tls: ca_file: "/etc/rancher/k3s/registry-ca.crt"

Mutual TLS (mTLS) Configuration

In high-security environments, Mutual TLS (mTLS) may be required. This involves providing not only the CA certificate but also a client certificate and a private key to authenticate the K3s node to the registry.

yaml configs: "registry.example.com": tls: ca_file: "/etc/rancher/k3s/registry-ca.crt" cert_file: "/etc/rancher/k3s/registry-client.crt" key_file: "/etc/rancher/k3s/registry-client.key"

Insecure Registry Handling

For development environments where TLS is not implemented, K3s allows the skipping of TLS verification. This is achieved by setting the insecure_skip_verify flag to true.

yaml configs: "registry.example.com:5000": tls: insecure_skip_verify: true

Pull-Through Cache Implementation

A pull-through cache is a specialized registry that caches images from an upstream source (like Docker Hub) to reduce bandwidth and speed up subsequent pulls.

To deploy a simple pull-through cache, a Docker Compose file can be used to run the registry image with specific environment variables for proxying.

```yaml

docker-compose.yml for a pull-through cache

services:
docker-cache:
image: registry:3
ports:
- "5000:5000"
environment:
REGISTRYPROXYREMOTEURL: https://registry-1.docker.io
REGISTRYPROXYUSERNAME: your-docker-hub-username
REGISTRYPROXYPASSWORD: your-docker-hub-pat
volumes:
- /opt/docker-cache:/var/lib/registry
restart: always
```

After deploying the cache, K3s must be configured to use this internal endpoint as a mirror for docker.io:

```yaml

registries.yaml

mirrors:
"docker.io":
endpoint:
- "http://docker-cache.internal:5000"
```

Private Registry Image Deployment Workflow

When deploying applications that use images from a private registry, the standard Kubernetes Deployment manifest is used. Because the authentication is handled at the containerd level via registries.yaml, there is no need to define imagePullSecrets within the pod specification.

```yaml

deployment with private image

apiVersion: apps/v1
kind: Deployment
metadata:
name: private-app
spec:
replicas: 2
selector:
matchLabels:
app: private-app
template:
metadata:
labels:
app: private-app
spec:
containers:
- name: app
# This image will be pulled from the private registry
image: registry.internal.example.com/myapp:1.0
ports:
- containerPort: 8080
```

Mirroring K3s System Images for Air-Gapped Environments

In air-gapped environments, system images (such as the pause image) must be manually mirrored to a private registry. This process requires a host with Docker or similar tooling.

The workflow for mirroring is as follows:

• Obtain the k3s-images.txt file from the official GitHub repository for the specific release version.
• Pull the listed images from the public registry:
  docker pull docker.io/rancher/mirrored-pause:3.6
• Retag the image for the private registry:
  docker tag docker.io/rancher/mirrored-pause:3.6 registry.example.com:5000/rancher/mirrored-pause:3.6
• Push the retagged image to the private registry:
  docker push registry.example.com:5000/rancher/mirrored-pause:3.6

Local Registry Installation within K3s

Since K3s does not include a built-in local registry plugin, a Docker registry can be installed as a service within the cluster and exposed via the default Traefik ingress on ports 80 and 443.

The setup involves creating a dedicated namespace and applying a registry manifest:

bash kubectl create namespace docker-registry kubectl apply -f docker-registry.yaml -n docker-registry

To ensure that tools like Docker or Podman can push to this local insecure registry, specific configurations must be applied:

• For Docker: Configure http://registry.localhost as an insecure registry in the Docker daemon settings.
• For Podman: Use the --tls-verify=false flag during the push:
  podman push --tls-verify=false registry.localhost/test:latest

Troubleshooting Registry Failures

When an image pull fails with a "no basic auth credentials" error, the investigation must focus on the node where the pod was scheduled. This can be identified using:

bash kubectl get pod -o wide -n NAMESPACE POD

Once the node is identified, the following troubleshooting steps should be performed:

• Verify the presence and correctness of the configuration file:
  sudo cat /etc/rancher/k3s/registries.yaml
• Inspect the containerd logs for specific error details:
  sudo tail -n 100 /var/lib/rancher/k3s/agent/containerd/containerd.log
• Ensure the K3s service is restarted to reload the configuration:
  sudo systemctl restart k3s

Runtime Integration and Constraints

A critical point of failure occurs when users attempt to use registries.yaml while utilizing a container runtime provided by the host Linux distribution rather than the embedded containerd. The registries.yaml support is exclusive to the embedded containerd.

If a system is configured with the --container-runtime-endpoint flag to use an external containerd, the registries.yaml file will be ignored. To resolve this, the external containerd must be uninstalled, the flag removed from the K3s startup command, and the embedded containerd restored.

Furthermore, it is important to distinguish between a container runtime and a registry. Containerd possesses an image store, but it does not function as an image registry that other nodes can pull from. For nodes to share images, a separate local image registry must be established, or the environment must wait for the implementation of peer-to-peer mesh image sharing.

Comparison of Registry Configuration Methods

Production-Grade Configuration Summary

A complete production configuration for registries.yaml often combines internal mirrors for public images and direct connections to company-specific registries.

```yaml

/etc/rancher/k3s/registries.yaml - Production configuration

mirrors:

Try the internal mirror first for docker.io pulls

"docker.io":
endpoint:
- "https://docker-mirror.internal.example.com"

Internal registry for company images

"registry.internal.example.com":
endpoint:
- "https://registry.internal.example.com"
configs:

Internal mirror authentication

"docker-mirror.internal.example.com":
auth:
username: "k3s-puller"
password: "SecurePullPassword"
tls:
ca_file: "/etc/rancher/k3s/internal-ca.crt"

Private company registry

"registry.internal.example.com":
auth:
username: "k3s-puller"
password: "SecurePullPassword"
tls:
cafile: "/etc/rancher/k3s/internal-ca.crt"
certfile: "/etc/rancher/k3s/k3s-client.crt"
key_file: "/etc/rancher/k3s/k3s-client.key"
```

Conclusion

The management of private registries in K3s is a multifaceted process that hinges on the correct implementation of the /etc/rancher/k3s/registries.yaml file. By leveraging mirrors, administrators can optimize the retrieval of public images, while the configs section allows for the secure integration of proprietary image repositories through Basic Auth and mTLS. The architecture emphasizes the importance of node-level consistency, as every schedulable node must possess the configuration to ensure successful pod deployment. While the embedded containerd simplifies this process, the operator must remain vigilant about the distinction between image stores and registries, and ensure that the container runtime is correctly aligned with K3s's embedded logic. Ultimately, the ability to transition from insecure development registries to hardened, production-grade mTLS configurations allows K3s to scale from small lab environments to secure, air-gapped enterprise infrastructures.

Sources

1. OneUptime
2. K3s Documentation
3. GitHub Gist - gdamjan
4. K3s GitHub Discussions