The Petabyte-Scale Migration of Reddit's Kafka Infrastructure from EC2 to Kubernetes

The architecture of modern data-driven platforms relies heavily on the ability to ingest, process, and distribute massive streams of information in real-time. At the heart of Reddit's operational capabilities lies Apache Kafka, an open-source message streaming platform that serves as the central nervous system for the entire ecosystem. In a high-traffic environment like Reddit, Kafka does not merely act as a peripheral utility; it is a mission-critical backbone supporting hundreds of business-critical services. It handles the asynchronous communication between producers—applications that write messages into Kafka partitions—and consumers—applications that read those messages out. Because Kafka sits in the middle of this pipeline, it ensures that messages are stored reliably, decoupling the timing of producers and consumers so that data integrity is maintained even if the two entities operate at completely different intervals.

The scale of this infrastructure is immense. Reddit manages a fleet comprising over 500 brokers, which are individual Kafka servers that work together in a cluster to manage data distribution. These brokers handle tens of millions of messages every second. The sheer volume of data being moved and processed is measured in petabytes, making the infrastructure's stability a non-negotiable requirement. If the Kafka layer were to experience a significant outage, large portions of Reddit's core functionality would effectively break, leading to massive service disruptions across the platform. This critical dependency necessitates a level of operational precision that leaves no room for error, especially when considering the complexity of migrating such a massive, live dataset without interrupting service.

The Architecture of Legacy Kafka Management on Amazon EC2

Before the transition to a containerized environment, Reddit’s Kafka infrastructure was hosted on Amazon EC2 virtual machines. This traditional deployment model utilized a heterogeneous mix of orchestration and configuration management tools to maintain the fleet.

The reliance on manual intervention was a significant operational bottleneck. Engineers and operators were responsible for handling critical tasks such as software upgrades, configuration changes, and the replacement of faulty or aging machines by running commands directly from their individual laptops. While this approach may have been functional during the initial growth phases of the platform, it became unsustainable as the fleet expanded. The complexity of managing hundreds of EC2 instances manually led to several critical issues:

• Increased operational latency: The time required to execute changes across the fleet grew disproportionately to the number of brokers.
• Error-prone workflows: Manual commands issued from disparate machines increased the likelihood of configuration drift and human error.
• Escalating costs: The overhead of managing unoptimized EC2 instances and the manual labor required for maintenance drove up operational expenditures.

The Complexity of Metadata and Client Coupling

A fundamental challenge in Reddit's legacy environment was the management of Kafka metadata and the tight coupling of client applications to the broker infrastructure.

Kafka maintains a detailed internal state known as metadata, which is essential for the cluster's self-awareness. This metadata contains vital information, including:
- The identity and status of all existing brokers.
- The mapping of which specific broker holds which segments of data.
- The location of data replicas across the cluster.

In the legacy setup, this metadata was managed by ZooKeeper, an external service. The reliance on ZooKeeper introduced a layer of complexity regarding cluster lifecycle management. Specifically, there is no supported method to recreate this metadata on a completely fresh cluster while maintaining system availability. This constraint meant that new brokers could only join an existing, running cluster; they could not be used to replace the cluster entirely.

Furthermore, the method by which client applications interacted with Kafka presented a significant risk to stability. Rather than using a single, load-balanced endpoint to discover the cluster, many applications across the Reddit ecosystem were configured to connect directly to specific broker hostnames. Typically, these applications targeted the first few brokers in a cluster. This configuration created a dangerous dependency where the decommissioning or failure of a single specific broker would immediately break hundreds of dependent services. Because Reddit did not control the layer through which clients discovered and connected to the Kafka brokers, they lacked the necessary abstraction to perform seamless infrastructure shifts.

The Four Hard Constraints of the Migration Strategy

The decision to migrate a petabyte-scale, real-time data stream from EC2 to Kubernetes was governed by four non-negotiable constraints. These constraints dictated the architecture of the migration and eliminated several common migration patterns.

• Zero Downtime Requirement: Kafka had to remain online throughout the entire process. There was no acceptable window for maintenance. This requirement ruled out scheduled cutovers, dual-write strategies (where producers write to both old and new systems), and replay-based migrations (where data is re-read from the beginning).
• Metadata Integrity Constraint: Because Kafka's metadata could not be rebuilt from scratch without downtime, the migration had to occur within the context of the existing metadata state.
• No Client Configuration Changes: The migration had to be transparent to the application layer. Forcing hundreds of service teams to update their connection strings or endpoints was deemed impossible.
• Reversibility: Every single step of the migration had to be reversible. No action could be taken that would leave the system in a state where a rollback to the EC2 environment was impossible.

Orchestrating the Data Plane Migration with Cruise Control and Strimzi

The migration strategy focused on moving the data plane—the actual movement of messages and partitions—before addressing the control plane. To achieve this, Reddit utilized a combination of the Strimzi operator and Cruise Control.

Strimzi is an open-source Kubernetes operator that provides a declarative way to manage Kafka clusters. Instead of manual configuration, developers can describe their desired Kafka state in a YAML configuration file, and Strimzi handles the deployment, upgrades, and maintenance. This approach promised more predictable operations and fewer manual interventions. However, because Reddit needed to perform highly customized operations during the transition, they utilized a forked version of the Strimzi operator to allow for granular control over the migration logic.

Cruise Control served as the engine for the actual data movement. Cruise Control is a specialized Kafka tool designed to automate the rebalancing of data across brokers in a controlled and measured manner. The migration utilized Cruise Control to incrementally shift the following elements from EC2 brokers to the Kubernetes-based Strimzi brokers:

• Partition Leadership: This is the process of reassigning which broker is responsible for serving reads and writes for a specific partition.
• Replicated Data: Kafka uses a replication factor to ensure redundancy, meaning every partition is stored on multiple brokers. The migration required moving these replicas to the new Kubernetes brokers one partition at a time.

The execution was a gradual process of "shifting" rather than a "cutover." Reddit monitored the transition through a dedicated dashboard, observing a steady decline in partition leadership on the EC2 brokers while seeing a parallel rise in leadership on the Strimzi brokers on Kubernetes. Network traffic patterns mirrored this shift, allowing the engineering team to pause or reverse the movement at any point if performance metrics deviated from the expected baseline.

Decoupling the Control Plane and the Move to KRaft

A critical design decision in this migration was the separation of concerns between the data plane and the control plane. The control plane involves the management of the cluster's metadata.

Reddit deliberately chose to keep the ZooKeeper-based control plane intact while the data plane was being moved to Kubernetes. By ensuring the data plane was fully stable on Kubernetes before touching the metadata management system, Reddit avoided the risk of compounding failures. This approach ensured that if the data movement encountered issues, the underlying metadata management remained in a known, stable state.

Once the data plane was successfully transitioned—meaning all EC2 brokers were decommissioned and all data and traffic were successfully running on Kubernetes—Reddit moved to the final architectural evolution: the migration from ZooKeeper to KRaft.

KRaft (Kafka Raft) is Kafka's built-in metadata management system. The primary advantage of KRaft is that it eliminates the need for an external service like ZooKeeper by integrating metadata management directly into the Kafka protocol. Because the data plane was already settled on Kubernetes and the environment was stable, this final transition was significantly less risky. Using the documented steps provided by both Strimzi and the Kafka community, the team successfully completed the migration to a unified, self-managed metadata architecture.

Finalization and Decommissioning of EC2 Infrastructure

The final stages of the migration focused on returning the system to a standard, supported operational state. After the successful migration of both the data plane and the control plane to the Kubernetes environment, the following steps were completed:

• Removal of Configuration Overrides: The custom, forked version of the Strimzi operator used during the migration was no longer necessary. All custom logic and overrides were stripped away.
• Hand-off to Standard Strimzi: Control of the Kafka clusters was handed back to the unmodified, standard Strimzi operator, ensuring that the production environment remained aligned with official support paths and standard community practices.
• EC2 Decommissioning: Once the Kubernetes environment was confirmed to be fully operational and handling all production traffic, the legacy Amazon EC2 infrastructure was decommissioned.

The result was a massive, petabyte-scale infrastructure shift that achieved the goal of moving from a manual, expensive, and rigid EC2-based model to a modern, automated, and scalable Kubernetes-based model, all while maintaining continuous service for Reddit's millions of users.

Analysis of Large-Scale Infrastructure Migrations

The Reddit Kafka migration serves as a high-level case study in risk mitigation for distributed systems. By breaking the migration into discrete phases—moving the data plane via Cruise Control, stabilizing it on Kubernetes, and then migrating the control plane to KRaft—the engineering team effectively managed the "blast radius" of potential failures.

The decision to avoid a "big bang" migration in favor of a granular, partition-by-partition leadership shift is the most significant takeaway for any DevOps or Infrastructure engineer. This methodology, combined with the use of declarative tools like Strimzi, transformed a potentially catastrophic event into a controlled operational procedure. Furthermore, the move from a manual, laptop-driven configuration model to a declarative, operator-based model represents a fundamental shift in how large-scale stateful applications should be managed in the cloud-native era. The ability to achieve zero downtime while changing the underlying orchestration layer, the hosting provider (in a sense, moving from raw EC2 to managed K8s via operators), and the metadata management system (ZooKeeper to KRaft) simultaneously is a landmark achievement in site reliability engineering.

Sources

1. Reddit Engineering via ByteByteGo