Resolving K3s Configuration File Permission Denied Errors

The interaction between the Kubernetes command-line tool, kubectl, and the K3s lightweight Kubernetes distribution often results in a specific, recurring authorization failure. Users attempting to query cluster resources—such as pods, services, or nodes—frequently encounter a warning and a subsequent fatal error stating that the system is unable to read the configuration file located at /etc/rancher/k3s/k3s.yaml. This phenomenon occurs because the K3s installation process, by default, creates a kubeconfig file owned by the root user with highly restrictive permissions. Specifically, the file is set to mode 0600, meaning only the root user has read and write access, while all other users on the system are completely barred from accessing its contents.

This architectural decision is rooted in security. The /etc/rancher/k3s/k3s.yaml file contains sensitive administrative credentials, including plain text admin passwords in some versions, which could lead to a catastrophic security breach if the file were world-readable. However, for developers and system administrators who wish to run kubectl commands without prefixing every single operation with sudo, this restriction becomes a significant operational hurdle. The error manifests as a warning suggesting the server be started with --write-kubeconfig-mode to modify permissions, followed by a "permission denied" error. This indicates that the current user session lacks the necessary privileges to open the file, thereby preventing kubectl from authenticating with the K3s API server.

The Mechanics of the Permission Denied Error

The error is triggered when the kubectl binary attempts to locate and load the cluster configuration. In a standard K3s environment, the tool is configured to look for the configuration at /etc/rancher/k3s/k3s.yaml. When a non-root user executes a command, the operating system's file permission system blocks the read request.

The following table details the specific error markers associated with this failure:

This failure is particularly prevalent after updating K3s. In older versions, K3s may have created world-readable configuration files. As security standards evolved, later versions shifted to the 0600 restriction to mitigate the risk of exposing admin passwords. This change often catches users by surprise when they update their environment, as commands that previously worked without sudo suddenly fail.

Strategic Remediation: User-Specific Configuration

One of the most secure ways to resolve this issue is to avoid modifying the permissions of the system-wide configuration file. Instead of making the root-owned file accessible to everyone, a user-specific copy of the configuration can be created. This approach maintains the integrity of the primary file while granting the user the necessary access.

The process involves utilizing the KUBECONFIG environment variable. This variable tells kubectl exactly where to look for the configuration file, bypassing the default search path of /etc/rancher/k3s/k3s.yaml.

To implement this strategy, the following steps are required:

• First, define the location for the user-specific config by exporting the environment variable:
  export KUBECONFIG=~/.kube/config

• Second, create a local directory and copy the restricted file into it while changing ownership to the current user:
  sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && chown $USER ~/.kube/config && chmod 600 ~/.kube/config && export KUBECONFIG=~/.kube/config

By copying the file to ~/.kube/config and changing the ownership to the current user (chown $USER), the user can now read the configuration without needing root privileges. Setting the permissions to 600 on the local copy ensures that the security of the credentials is maintained, as only the specific user can access their own copy. To make this change permanent, the export KUBECONFIG=~/.kube/config command should be added to the user's shell configuration file, such as .bashrc or .bash_profile.

This method provides significant advantages for interoperability. Many other Kubernetes ecosystem tools, such as Helm, expect to find a configuration file at ${HOME}/.kube/config. By establishing this file, the user ensures that kubectl and other tools function according to standard Kubernetes assumptions.

Modifying K3s Server Write Permissions

For environments where a shared configuration file is preferred, or for those who wish to automate the permission settings during installation, K3s provides a mechanism to change the permissions of the generated k3s.yaml file. This is achieved through the --write-kubeconfig-mode flag.

Setting the mode to 644 allows the root user to read and write the file, while allowing all other users to read it. This eliminates the need for sudo for every kubectl command.

There are several ways to apply this configuration:

• During Installation:
  When installing K3s using the official shell script, the flag can be passed as an argument:
  curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

• Via Systemd Environment Configuration:
  To ensure the setting persists across reboots without reinstalling, the value can be added to the systemd environment file:
  echo K3S_KUBECONFIG_MODE=\"644\" >> /etc/systemd/system/k3s.service.env

• Via the K3s Configuration File:
  K3s supports a YAML-based configuration file located at /etc/rancher/k3s/config.yaml. Adding the following line to this file achieves the same result as the CLI flag:
  write-kubeconfig-mode: "0644"

The use of a configuration file is often preferred in production or complex environments. K3s loads configuration from /etc/rancher/k3s/config.yaml and any drop-in files in /etc/rancher/k3s/config.yaml.d/*.yaml. These files are processed in alphabetical order. If a value is defined in both the YAML file and via CLI arguments, the CLI arguments take precedence.

Troubleshooting Common Implementation Failures

Attempting to fix the permission issue manually can sometimes lead to other technical conflicts. A common scenario occurs when a user tries to run the K3s server manually with the --write-kubeconfig-mode flag while the K3s service is already running in the background.

The following sequence of events often occurs:

• The user executes:
  sudo k3s server --write-kubeconfig-mode 644

• The system returns a fatal error:
  FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :6443: bind: address already in use

This error occurs because the K3s server is already bound to port 6443. To resolve this and apply the permission change, the user must stop the existing service first:

• Stop the K3s service:
  sudo systemctl stop k3s

• Manually change the file permissions:
  sudo chmod 644 /etc/rancher/k3s/k3s.yaml

• Restart the service or run the server command.

Once these steps are completed, the cluster should be accessible. The user can verify the fix by running a basic query without sudo:
kubectl get pods -n azure-iot-operations

Comparison of Resolution Methods

Depending on the environment—whether it is a local development box, an IoT edge device, or a professional production cluster—the choice of resolution method varies based on the balance between convenience and security.

For professional environments, such as Industrial IoT or Edge computing, it is strongly advised to continue using the sudo command. Making the k3s.yaml file world-readable introduces a vulnerability where any user with shell access to the machine can potentially gain administrative control over the entire Kubernetes cluster.

Detailed Analysis of Configuration Merging and Precedence

The K3s configuration system is designed to be flexible, allowing for a mix of static files and dynamic CLI arguments. Understanding how these values merge is critical when attempting to resolve permission errors through configuration files.

Configuration is loaded from the primary /etc/rancher/k3s/config.yaml and drop-in files in /etc/rancher/k3s/config.yaml.d/*.yaml. The merging logic follows these rules:

• Value Override: If the same key exists in multiple files, the last value found is the one used.
• Value Append: A + can be appended to a key to append the value to an existing string or slice rather than replacing it.
• CLI Precedence: CLI arguments (e.g., --write-kubeconfig-mode 644) always take precedence over values defined in the YAML configuration files.

This means that if a user has write-kubeconfig-mode: "0600" in their config.yaml but starts the server with --write-kubeconfig-mode 644, the final permission set will be 644. This hierarchy allows administrators to maintain a base configuration in a file while overriding specific settings for different nodes or environments using CLI flags.

Conclusion

The "permission denied" error when accessing /etc/rancher/k3s/k3s.yaml is a direct result of K3s's security-first approach to kubeconfig management. By restricting the primary configuration file to root ownership and 0600 permissions, K3s prevents unauthorized access to cluster administrative credentials. While this creates a hurdle for users who wish to avoid sudo, several robust solutions exist.

For users prioritizing security and interoperability, the creation of a user-specific configuration file in ~/.kube/config combined with the KUBECONFIG environment variable is the optimal path. This preserves the system's security posture while enabling seamless integration with other Kubernetes tools. For those managing home labs or development environments where convenience is paramount, utilizing the --write-kubeconfig-mode 644 flag—either during installation, via systemd, or through the config.yaml file—effectively removes the restriction.

Ultimately, the choice of remediation depends on the deployment context. In high-security edge or IoT environments, the restriction should remain in place, and sudo should be used. In all other cases, the provided methods allow for a balance of functionality and security, ensuring that kubectl can operate efficiently without compromising the underlying system's integrity.

Sources

1. Resolving the K3s Config File Permission Denied Error
2. Note to self: Error loading config file k3s.yaml
3. K3s Kubectl Permission
4. Unable to read etc/rancher/k3s/k3s.yaml
5. K3s GitHub Issue 1541
6. K3s Installation Configuration Documentation