K3s is a lightweight Kubernetes distribution specifically engineered to pack the full Kubernetes API into a single binary of less than 100MB. This streamlined design makes it an ideal candidate for edge computing, Internet of Things (IoT) deployments, and rapid development environments. However, the inherent simplicity of K3s comes with a default trade-off: it utilizes an embedded SQLite database. While SQLite is efficient for single-node deployments and zero-configuration startups, it represents a single point of failure and lacks the concurrency and durability requirements for production-grade clusters.

For production environments requiring high availability (HA) and stringent data durability, transitioning to an external datastore is not merely an option but a necessity. The choice of datastore directly impacts the cluster's overall resilience, its ability to scale under load, and the operational complexity the engineering team must manage. MySQL and MariaDB serve as critical alternatives for organizations that already possess established MySQL infrastructure or prefer the specific operational characteristics and tooling associated with the MySQL ecosystem.

Comprehensive Datastore Comparison Matrix

Selecting the appropriate backend requires a nuanced understanding of the trade-offs between operational overhead and system robustness. The following table delineates the differences between the available K3s datastore options.

High Availability Architecture Design

Transitioning K3s to a MySQL backend transforms the cluster architecture from a standalone instance into a distributed system. In a production-ready HA setup, the architecture is divided into three distinct layers: the load balancing layer, the server (control plane) layer, and the worker (agent) layer, all centering around the external database.

The flow of traffic begins at the Load Balancer, which typically exposes the K3s API on port 6443. This load balancer distributes incoming requests across multiple K3s server nodes. These server nodes do not store the cluster state locally; instead, they all connect to a centralized external MySQL database. This ensures that if one server node fails, the remaining nodes possess the same state, allowing the cluster to maintain continuity. The agent nodes, which run the actual workloads, connect to the load balancer to communicate with the available API servers.

MySQL Database Preparation and Provisioning

Before the K3s server can be initialized, the MySQL backend must be meticulously prepared to handle Kubernetes object data. The configuration requires a specific database, a dedicated user with precise privileges, and specific encoding settings to ensure data integrity.

The first critical step is establishing a connection to the MySQL server using the command line:

mysql -h mysql.example.com -u admin -p

Once connected, the database must be created with UTF8 encoding. Specifically, the use of utf8mb4 is mandatory because Kubernetes object names and labels can contain characters that exceed the capabilities of standard UTF8.

CREATE DATABASE k3s CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Following the database creation, a dedicated service account must be provisioned. To ensure broader compatibility across various MySQL versions and connection clients, the mysql_native_password authentication plugin is utilized.

CREATE USER 'k3s_user'@'%' IDENTIFIED WITH mysql_native_password BY 'your-secure-password';

K3s requires full administrative control over its own schema to manage the lifecycle of Kubernetes objects, perform migrations, and maintain state. Therefore, all privileges on the k3s database must be granted to the service user.

GRANT ALL PRIVILEGES ON k3s.* TO 'k3s_user'@'%';

To ensure that the permission changes are registered by the MySQL server immediately, the privileges must be flushed.

FLUSH PRIVILEGES;

Finally, the session is closed to return to the system shell.

EXIT;

K3s Server Installation and Configuration

With the database prepared, the K3s server can be installed. There are two primary methods for providing the MySQL connection string: via direct installation flags or through a persistent configuration file.

The direct installation method uses a curl command to pipe the K3s installer into the shell, passing the --datastore-endpoint flag. The connection string format is strictly defined as mysql://username:password@tcp(hostname:port)/database.

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s_user:your-secure-password@tcp(mysql.example.com:3306)/k3s"

For production environments, relying on command-line flags is suboptimal as it exposes passwords in the process list and makes updates cumbersome. Instead, a configuration file should be implemented at /etc/rancher/k3s/config.yaml. This method allows for the implementation of critical security features, such as Transport Layer Security (TLS).

To implement this, the directory must first be created:

sudo mkdir -p /etc/rancher/k3s

Then, the configuration is written to the file. The inclusion of the tls=true parameter in the connection string is vital for production, as it ensures that all communication between the K3s server nodes and the MySQL database is encrypted, preventing man-in-the-middle attacks and data eavesdropping.

```yaml

K3s Server Configuration for MySQL Backend

MySQL connection string with TLS enabled

The tls=true parameter enables encrypted connections

datastore-endpoint: "mysql://k3s_user:your-secure-password@tcp(mysql.example.com:3306)/k3s?tls=true"
```

MySQL Server Optimization and Network Configuration

The MySQL server itself may require adjustments to allow the K3s nodes to connect. In many default MySQL installations, the server binds only to localhost (127.0.0.1), which prevents external K3s server nodes from reaching the datastore.

To resolve this, the MySQL option file (typically .cnf) must be modified. Setting the bind-address to 0.0.0.0 instructs the MySQL server to listen on all available network interfaces.

ini [mysqld] bind-address = 0.0.0.0 port = 3306

In environments where the cluster is not exposed to the public internet and resides within a trusted private network, this configuration allows any local IP within the defined subnet to initiate a connection. However, this should always be paired with strict firewall rules or MySQL host-based access controls.

Load Balancer Implementation with HAProxy

To achieve true high availability, the K3s API servers must be placed behind a load balancer. This prevents the load balancer's IP from becoming a single point of failure for the agent nodes. HAProxy is a recommended tool for this purpose due to its robust TCP health checking capabilities.

The configuration file /etc/haproxy/haproxy.cfg must be configured to handle TCP traffic on port 6443. The frontend section defines where clients connect, and the backend section defines the pool of K3s server nodes.

```haproxy
global
log /dev/log local0
maxconn 4096
defaults
log global
mode tcp
option tcplog
timeout connect 10s
timeout client 1m
timeout server 1m

Frontend for K3s API server

Clients connect here to access the Kubernetes API

frontend k3sapi
bind *:6443
defaultbackend k3s_servers

Backend pool of K3s server nodes

Health checks ensure traffic only goes to healthy nodes

backend k3s_servers
balance roundrobin
option tcp-check

K3s server nodes with health checks

The 'check' keyword enables active health monitoring

server k3s-server-1 10.0.0.11:6443 check inter 5s fall 3 rise 2
server k3s-server-2 10.0.0.12:6443 check inter 5s fall 3 rise 2
server k3s-server-3 10.0.0.13:6443 check inter 5s fall 3 rise 2
```

The health check parameters (inter 5s fall 3 rise 2) are critical. They ensure that if a K3s server node becomes unresponsive, it is removed from the rotation within 15 seconds, and is only reintegrated after two consecutive successful health checks.

Automated Backup and Recovery Strategies

Data loss in the datastore is catastrophic, as it results in the total loss of the cluster's state, including all deployment definitions, secrets, and configmaps. Implementing an automated backup strategy for the MySQL datastore is mandatory.

The recommended approach is a bash-based automation script utilizing mysqldump. This utility allows for the creation of logically consistent snapshots of the database.

The following script implements a professional backup routine:

```bash

!/bin/bash

k3s-mysql-backup.sh

Automated backup script for K3s MySQL datastore

Configuration

DBHOST="mysql.example.com"
DBPORT="3306"
DBNAME="k3s"
DBUSER="k3suser"
BACKUPDIR="/var/backups/k3s"
RETENTION_DAYS=30

Create backup directory

mkdir -p "${BACKUP_DIR}"

Generate timestamp

TIMESTAMP=$(date +%Y%m%d%H%M%S)
BACKUPFILE="${BACKUPDIR}/k3sbackup_${TIMESTAMP}.sql.gz"

Perform the backup

--single-transaction: Consistent snapshot without locking tables

--routines: Include stored procedures and functions

--triggers: Include triggers

--quick: Retrieve rows one at a time (memory efficient for large databases)

mysqldump \
-h "${DBHOST}" \
-P "${DBPORT}" \
-u "${DBUSER}" \
-p"${DBPASSWORD}" \
--single-transaction \
--routines \
--triggers \
--quick \
"${DBNAME}" \
| gzip > "${BACKUPFILE}"

Verify backup

if [ -f "${BACKUPFILE}" ] && [ -s "${BACKUPFILE}" ]; then
echo "Backup successful: ${BACKUPFILE}"
sha256sum "${BACKUPFILE}" > "${BACKUP_FILE}.sha256"
else
echo "Backup failed!"
exit 1
fi

Clean up old backups

find "${BACKUPDIR}" -name "k3sbackup*.sql.gz" -mtime +${RETENTIONDAYS} -delete
find "${BACKUPDIR}" -name "k3sbackup*.sha256" -mtime +${RETENTIONDAYS} -delete
```

Deep Dive into Backup Flags

The mysqldump flags used in the script are chosen to ensure minimum impact on cluster performance and maximum data integrity:

• --single-transaction: This is essential for InnoDB tables. It allows the backup to happen without locking the entire database, meaning the K3s cluster can continue to operate while the backup is running.
• --routines and --triggers: These ensure that any stored logic within the database is captured, preventing schema mismatches during a restore.
• --quick: This tells mysqldump to retrieve rows one at a time rather than buffering the entire result set in memory, which is critical when the K3s datastore grows to several gigabytes.

The script also implements a checksumming mechanism using sha256sum. This provides a way to verify that the backup file has not been corrupted during storage or transit. Finally, a retention policy of 30 days is enforced using the find command to prevent the backup disk from reaching capacity.

Alternative External Datastores: PostgreSQL and etcd

While MySQL is a powerful option, certain environments may necessitate the use of PostgreSQL or a dedicated etcd cluster.

PostgreSQL Implementation

PostgreSQL is frequently chosen for cloud-native environments because managed services like AWS RDS, Google Cloud SQL, and Azure Database provide built-in failover and automated snapshots. The backup process for PostgreSQL differs from MySQL, requiring the use of pg_dump.

A typical PostgreSQL backup script would look like this:

```bash

!/bin/bash

Configuration

DBHOST="postgres.example.com"
DBPORT="5432"
DBNAME="k3s"
DBUSER="k3suser"
BACKUPDIR="/var/backups/k3s"
RETENTION_DAYS=30

Create backup directory if it doesn't exist

mkdir -p "${BACKUP_DIR}"

Generate timestamp for backup filename

TIMESTAMP=$(date +%Y%m%d%H%M%S)
BACKUPFILE="${BACKUPDIR}/k3sbackup_${TIMESTAMP}.sql.gz"

Perform the backup

--no-owner: Don't output commands to set object ownership

--no-acl: Don't output commands to set access privileges

Compress with gzip to save storage space

PGPASSWORD="${DBPASSWORD}" pgdump \
-h "${DBHOST}" \
-p "${DBPORT}" \
-U "${DBUSER}" \
-d "${DBNAME}" \
--no-owner \
--no-acl \
--format=plain \
| gzip > "${BACKUP_FILE}"

Verify backup was created successfully

if [ -f "${BACKUPFILE}" ] && [ -s "${BACKUPFILE}" ]; then
echo "Backup successful: ${BACKUP_FILE}"

Calculate and store checksum for integrity verification

sha256sum "${BACKUPFILE}" > "${BACKUPFILE}.sha256"
else
echo "Backup failed!"
exit 1
fi

Clean up old backups beyond retention period

find "${BACKUPDIR}" -name "k3sbackup*.sql.gz" -mtime +${RETENTIONDAYS} -delete
find "${BACKUPDIR}" -name "k3sbackup*.sha256" -mtime +${RETENTIONDAYS} -delete
echo "Backup complete"
```

External etcd

For massive clusters with strict performance requirements, an external etcd cluster is the gold standard. While this introduces the highest level of operational complexity—requiring the management of a separate quorum-based cluster—it provides the lowest latency for state updates and the highest level of consistency.

Storage Considerations for K3s

When implementing MySQL as a backend, it is important to remember that K3s is a streamlined version of Kubernetes. This means that some of the expansive storage options found in full Kubernetes (K8s) may not be available or may behave differently. Engineers must familiarize themselves with the specific storage classes and CSI (Container Storage Interface) drivers supported by the K3s distribution to ensure that the applications running on top of the MySQL-backed cluster have the necessary persistence layers.

Analysis of Datastore Selection

The transition from the default SQLite to an external MySQL datastore represents a fundamental shift in the operational posture of a K3s cluster. By decoupling the state (the database) from the control plane (the K3s servers), the system evolves from a "pet" (where a single node is irreplaceable) to a "cattle" architecture (where any server node can be destroyed and recreated without data loss).

The use of MySQL specifically is a strategic decision. For organizations with existing DBA (Database Administration) teams, leveraging MySQL allows them to apply existing expertise in query optimization, backup rotation, and hardware tuning to the Kubernetes infrastructure. The implementation of utf8mb4 encoding and mysql_native_password ensures that the technical constraints of the Kubernetes API are met while maintaining compatibility with a wide array of client libraries.

From a security perspective, the movement toward a config.yaml file with tls=true addresses the critical vulnerability of plaintext credentials and unencrypted traffic. In a distributed environment where K3s servers and the MySQL database may reside on different physical or virtual machines, encryption in transit is non-negotiable.

Ultimately, the reliability of a K3s MySQL deployment is not determined by the installation of the software, but by the rigor of the surrounding ecosystem: the precision of the HAProxy load balancing, the frequency and verification of mysqldump backups, and the strictness of the network access controls.

Sources

1. OneUptime
2. Zaher.dev