Hadoop Kubernetes Architecture and Deployment

The integration of Apache Hadoop into Kubernetes environments represents a pivotal shift in the management of big data workloads, transitioning from static, mutable infrastructure to dynamic, immutable container-native deployments. Apache Hadoop is an open-source framework specifically engineered for the reliable, scalable, and distributed computing of massive datasets. By utilizing simple programming models, the library allows for distributed processing across clusters of computers, scaling from a single server to thousands of individual machines. Each machine in such a cluster provides local computation and storage. A defining characteristic of the Hadoop library is its design to detect and handle failures at the application layer rather than relying on the underlying hardware to provide high availability. This ensures that a highly available service is maintained even when individual computers in the cluster are prone to failure.

Historically, Hadoop was designed to address the specific need of handling large datasets efficiently using commodity hardware. This approach allows users to store and process massive amounts of data cheaply, enabling the easy reference of older data while keeping storage costs minimal. However, as technology evolves, new requirements and use cases have emerged, challenging the traditional Hadoop model. The rise of serverless computing—where cloud platforms automatically manage and scale hardware resources based on application needs—has introduced new alternatives. Container-native, open-source, and function-as-a-service (FaaS) platforms, such as fn, Apache OpenWhisk, and nuclio, can be integrated with Kubernetes to run serverless applications, potentially eliminating the need for traditional Hadoop deployments. Specifically, frameworks like nuclio are aimed at automating data science pipelines using serverless functions. Despite these emerging trends, Kubernetes is increasingly becoming the optimal choice for managing big data workloads due to its orchestration capabilities and the ability to provide an immutable deployment environment.

Hadoop Core Modules and Functional Architecture

The functionality of Apache Hadoop is powered by four primary modules, each serving a distinct role in the distribution of storage and analytics workloads. By distributing these tasks across multiple nodes, Hadoop enables parallel processing, which results in faster, more efficient, and lower-cost data analytics.

• HDFS (Hadoop Distributed File System). This is a specialized file system designed to run on low-end hardware. It provides superior throughput compared to traditional file systems and incorporates built-in fault tolerance, making it capable of handling massive datasets across a distributed environment.

• YARN (Yet Another Resource Negotiator). YARN acts as the operational brain of the cluster. It is responsible for task management, the scheduling of jobs, and the overall resource management of the cluster to ensure optimal utilization of hardware.

• MapReduce. This is the default big data processing engine for Hadoop. It supports the parallel computation of large datasets. While MapReduce is the standard, Hadoop also supports other processing engines, including Apache Spark and Apache Tez.

• Hadoop Common. This module provides the essential set of libraries and utilities that are utilized across all other Hadoop modules, ensuring consistency and interoperability.

Comparative Analysis of Deployment Paradigms

The transition from traditional data center deployments to Kubernetes-based public cloud environments marks a fundamental change in infrastructure management, as evidenced by the operational shift at organizations like Salesforce.

In traditional environments, operating system updates and deployments are often managed by tools like Puppet and Ambari, which are geared toward mutable infrastructure. In these scenarios, binaries and configurations are modified "in place." This approach creates a risk where failures during the update process result in partial changes across hosts. Furthermore, the temptation for engineers to apply manual fixes for urgent issues leads to lingering configuration drifts, where the actual state of the server deviates from the documented configuration.

Conversely, the use of Kubernetes and public cloud infrastructure allows for an immutable form of deployment. In this model, the specific set of binaries and the configuration are packaged directly into the VM or container image. This ensures that every instance of a service is identical, eliminating configuration drift and simplifying the recovery process, as a failed node can be replaced by a fresh instance of the same image.

Kubernetes Deployment Technical Implementation

Deploying Hadoop on Kubernetes requires a precise configuration of Pods, Services, and Persistent Volumes to replicate the distributed nature of the HDFS architecture. The deployment typically utilizes the apache/hadoop:3.4.1 Docker image.

NameNode Configuration and Orchestration

The NameNode serves as the master node in HDFS. Its deployment requires a combination of a Pod for the process and Services for networking.

The NameNode Pod is configured with specific resource limits to ensure stability, typically limited to 1G of memory and 500m of CPU. The container uses a shell command to check for the existence of the directory /hadoop/nn/current; if it does not exist, the command hdfs namenode -format -force is executed to format the file system before starting the NameNode with hdfs namenode.

For persistence, the NameNode utilizes a hostPath volume mapping the local path /hadoop/nn to the container's /hadoop/nn. Configuration is managed via a ConfigMap named hadoop-config, which mounts core-site.xml and hdfs-site.xml into /opt/hadoop/etc/hadoop/.

Networking for the NameNode is handled by two distinct services:

1. Headless Service. A service named company is created with clusterIP: None. This service uses the selector dns: hdfs-subdomain, allowing Kubernetes to generate A/AAAA records. This is critical for the internal communication of the HDFS cluster.

2. NodePort Service. A service named namenode-np is created to provide external access to the Web UI. It maps port 50470 (targetPort) to nodePort: 30570. This allows administrators to access the NameNode Web UI at http://dns_or_ip_of_any_k8s_node:30570.

The YAML configuration for the NameNode and its services is as follows:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: namenode
namespace: bigdata
labels:
app: namenode
dns: hdfs-subdomain
spec:
nodeSelector:
kubernetes.io/hostname: "kube1"
hostname: namenode
subdomain: company
containers:
- name: namenode
image: apache/hadoop:3.4.1
command: ["/bin/bash", "-c"]
args:
- |
if [ ! -d "/hadoop/nn/current" ]; then
hdfs namenode -format -force
fi
hdfs namenode
resources:
limits:
memory: "1G"
cpu: "500m"
volumeMounts:
- name: hadoop-config
mountPath: /opt/hadoop/etc/hadoop/core-site.xml
subPath: core-site.xml
- name: hadoop-config
mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml
subPath: hdfs-site.xml
- name: namenode-path
mountPath: /hadoop/nn
volumes:
- name: namenode-path
hostPath:
path: /hadoop/nn
type: Directory
- name: hadoop-config
configMap:

name: hadoop-config

apiVersion: v1
kind: Service
metadata:
name: company
namespace: bigdata
spec:
selector:
dns: hdfs-subdomain
clusterIP: None
ports:
- name: rpc

port: 9000

apiVersion: v1
kind: Service
metadata:
name: namenode-np
namespace: bigdata
spec:
type: NodePort
selector:
app: namenode
ports:
- name: namenode-ui
port: 50470
targetPort: 50470
nodePort: 30570
```

DataNode Configuration and Storage

DataNodes are the worker nodes responsible for storing the actual data. Unlike the NameNode, DataNodes require a distributed storage strategy.

The DataNode Pod is configured with the apache/hadoop:3.4.1 image and utilizes the command hdfs datanode. Resource limits are set to 512M of memory and 500m of CPU. A nodeSelector is used to bind the pod to a specific host, such as kubernetes.io/hostname: "kube1".

Storage for DataNodes is implemented via hostPath volumes. The local path /hadoop/disk1 is mapped to the container's /hadoop/disk1. Before deploying, it is necessary to create the path on the data nodes (e.g., kube1, kube2, kube3) and set the appropriate ownership permissions.

The YAML configuration for a DataNode is as follows:

yaml apiVersion: v1 kind: Pod metadata: name: datanode01 namespace: bigdata labels: app: datanode01 dns: hdfs-subdomain spec: nodeSelector: kubernetes.io/hostname: "kube1" hostname: datanode01 subdomain: company containers: - name: datanode01 image: apache/hadoop:3.4.1 command: ["/bin/bash", "-c"] args: - | hdfs datanode resources: limits: memory: "512M" cpu: "500m" volumeMounts: - name: hadoop-config mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml - name: datanode-path mountPath: /hadoop/disk1 volumes: - name: datanode-path hostPath: path: /hadoop/disk1 type: Directory - name: hadoop-config configMap: name: hadoop-config

Software Versioning and Release Lifecycle

The Apache Hadoop project continuously evolves, with recent releases focusing on bug fixes and stability.

• Hadoop 3.5 Line. This is the most recent stable release. It introduces 485 bug fixes, improvements, and enhancements compared to the 3.4 line.

• Hadoop 3.4.3 Line. This release is recommended for users of version 3.4.2 and earlier. It is important to note that this specific release does not include the bundle.jar containing the AWS SDK, which is used by the s3a connector in the hadoop-aws module.

• Hadoop 3.4.1. This version is utilized in current Kubernetes container-native deployment examples to establish the baseline for HDFS on K8s.

Analysis of Hadoop Limitations and Modern Alternatives

Despite its strengths in processing massive datasets on inexpensive hardware, Hadoop faces significant challenges in the modern data landscape.

Operational and Performance Constraints

Hadoop is fundamentally inefficient when dealing with small datasets. Because it is designed for massive scale, the overhead of distributing tasks across a cluster makes it cost-prohibitive and slow for quick analytics of smaller data volumes. Furthermore, while Hadoop excels at combining, processing, and transforming data, it lacks an integrated, easy-to-use method for outputting the final data. This creates a bottleneck for business intelligence teams who require streamlined paths for visualizing and reporting on processed data.

Security and Integrity Gaps

Security is a notable weakness in default Hadoop installations. The framework includes lax security enforcement by default and fails to implement encryption or decryption at the network or storage levels. This makes Hadoop clusters vulnerable if not augmented with third-party security tools or extensive manual configuration.

The Shift Toward Serverless and Kubernetes

The aforementioned drawbacks, combined with the emergence of serverless computing, have led to a decline in Hadoop's market lead. The primary advantage of moving to Kubernetes is the ability to integrate serverless functions for data science pipelines via tools like nuclio. This shift allows for:

• Automated scaling of hardware resources.
• Elimination of the complex management associated with HDFS and YARN.
• Integration of function-as-a-service (FaaS) patterns into big data workflows.

Conclusion

The transition of Apache Hadoop to Kubernetes marks the convergence of big data processing and modern cloud-native orchestration. By moving from the mutable, manual management of bare-metal clusters—characterized by config drift and complex recovery—to an immutable, containerized approach, organizations can achieve higher reliability and on-demand scalability. The technical implementation requires a rigorous application of Kubernetes primitives, including Headless Services for HDFS subdomain DNS resolution and hostPath volumes for persistent data storage.

However, the architectural superiority of Kubernetes also exposes the inherent limitations of Hadoop. The inefficiency of Hadoop with small datasets and its default lack of encryption highlight why serverless alternatives like Apache OpenWhisk and nuclio are gaining traction. While Hadoop remains a powerful tool for massive, low-cost data storage and parallel processing, its future lies in its integration with orchestration layers that can mitigate its security flaws and operational rigidity. The shift toward immutable infrastructure ensures that as Hadoop evolves—seen in the progression from 3.4 to 3.5—the deployment process remains consistent, predictable, and scalable.

Sources

1. BMC Blogs
2. Fatih Mete GitHub Pages
3. Apache Hadoop
4. Salesforce Engineering