Pulumi Infrastructure Drift Detection and Remediation

Infrastructure drift represents a critical point of failure in the modern DevOps lifecycle, occurring when the actual state of cloud resources deviates from the desired state defined within an organization's Infrastructure as Code (IaC) configuration. In a complex cloud environment, this divergence typically manifests through "click-ops"—manual, out-of-band changes made directly via a cloud provider's console—or as a result of changes a cloud resource makes independently over its own operational lifecycle. When the recorded state in a Pulumi Stack no longer mirrors the actual reality of the cloud deployment, the integrity of the deployment pipeline is compromised, leading to unpredictable behavior during subsequent updates.

To combat this, Pulumi has introduced comprehensive drift detection and remediation capabilities integrated into Pulumi Deployments and the Pulumi Kubernetes Operator. These features allow engineering teams to continuously monitor their infrastructure, receive notifications when deviations occur, and optionally automate the reconciliation process to ensure the cloud environment remains synchronized with the source of truth. This ecosystem transforms IaC from a static deployment tool into a continuous lifecycle management system.

The Mechanics of Infrastructure Drift

Drift is defined as the gap between the actual state of cloud resources and the state Pulumi has recorded for them. This recorded state serves as the anchor for all subsequent operations. When drift occurs, the Pulumi state file becomes an inaccurate representation of the environment.

The impact of drift is multifaceted. For a DevOps engineer, undetected drift means that a pulumi up command might produce unexpected results, as the engine may attempt to modify resources based on outdated assumptions. For a security officer, drift often signifies unauthorized changes to security groups or IAM roles, potentially opening vulnerabilities that are not documented in the version-controlled configuration.

Pulumi Cloud addresses this by utilizing a preview-only refresh. A refresh operation queries the cloud provider to determine the current state of all managed resources. By running this in a preview-only mode, Pulumi can detect differences without immediately altering the actual infrastructure. This allows the system to identify "drift" as a specific state—a delta between the desired configuration and the actual cloud reality.

Pulumi Cloud Drift Management

Pulumi Cloud provides a centralized management layer for drift detection and remediation, moving the process from a manual CLI task to an automated service.

The Drift Tab and Observability

The Drift tab on the stack page serves as the primary observability hub for infrastructure health. This interface aggregates every drift run, regardless of its origin. Whether a run was triggered via the Command Line Interface (CLI), the Click to Deploy menu, or an automated schedule, the results are surfaced here.

When a stack is in a drifted state, the Pulumi Cloud UI displays a warning bell icon on the Drift tab, providing an immediate visual indicator to the operator that the environment has diverged. If a run detects drift, the UI generates an information card containing a diff summary of the changes, detailing exactly what has changed in the cloud.

Ad Hoc Drift Detection

Users can trigger drift detection and remediation manually through several vectors:

• Click to Deploy: Users can navigate to the Click to Deploy menu for a specific stack and select "Detect drift" to initiate a preview-only refresh. Alternatively, they can select "Remediate drift" to reconcile the infrastructure.
• CLI Operations: Running pulumi refresh --preview-only executes a drift detection run. Running pulumi refresh also initiates this process, as it performs a preview before executing the actual refresh.

Automated Drift Detection and Remediation

To eliminate the need for manual checks, Pulumi allows the configuration of scheduled drift detection and automatic remediation.

Scheduling Logic

Scheduled drift runs ensure that infrastructure is audited at regular intervals. This is configured through the Pulumi Cloud console by navigating to Stack > Settings > Schedules and selecting Drift. This converts drift detection from a reactive process into a proactive maintenance task.

Automatic Remediation

Automatic remediation is the process of re-applying the Pulumi program to force the cloud resources to match the desired state defined in the IaC configuration. When autoRemediate is enabled, Pulumi does not simply notify the user of drift; it actively corrects it by re-applying the last known good state from prior deployments. This ensures that "click-ops" changes are overwritten and the environment is restored to its compliant state.

Programmatic Configuration

Configuration of these features can be handled through the Pulumi Deployments REST API or the Pulumi Service Provider.

To create a drift detection and remediation schedule via the REST API, the following curl command is used:

bash curl -H "Accept: application/vnd.pulumi+json" \ -H "Content-Type: application/json" \ -H "Authorization: token $PULUMI_ACCESS_TOKEN" \ --request POST \ --data '{"scheduleCron":"0 0 * * *","autoRemediate":true}' \ https://api.pulumi.com/api/stacks/{organization}/{project}/{stack}/deployments/drift/schedules

For those preferring to manage their deployment settings as code, the Pulumi Service Provider allows the definition of drift schedules in TypeScript or Python.

Example in TypeScript:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const organizationName = "my-org";
const projectName = "my-project";
const stackName = "production";

// Creating a DriftSchedule for automatically running drift detection
const driftDetectionSchedule = new pulumiservice.DriftSchedule("driftDetectionSchedule", {
organization: organizationName,
project: projectName,
stack: stackName,
scheduleCron: "0 0 * * *", // Run drift detection daily at midnight
autoRemediate: true, // Automatically remediate any drift detected
});

export const driftScheduleId = driftDetectionSchedule.scheduleId;
```

Pulumi Kubernetes Operator Integration

The Pulumi Kubernetes Operator extends these capabilities into the Kubernetes ecosystem, allowing the management of Pulumi Stacks via Kubernetes Custom Resource Definitions (CRDs).

Operator Drift Implementation

The operator is designed to mirror the drift detection features of Pulumi Cloud. Traditionally, the operator applies changes immediately using pulumi up -y. The implementation of drift detection introduces a non-destructive state check based on pulumi refresh --preview-only.

This allows Kubernetes users to detect when cloud resources diverge from the desired state defined in their Stack without forcing an immediate reconciliation.

Configuration via Stack CRs

Users can define drift detection schedules directly within the Stack CR specification. This allows the infrastructure lifecycle to be managed via standard Kubernetes YAML manifests.

Example configuration:

yaml apiVersion: pulumi.com/v1 kind: Stack metadata: name: my-stack spec: stack: org/project/stack projectRepo: https://github.com/example/repo driftDetection: schedules: - cron: "*/15 * * * *" # Check every 15 minutes autoRemediate: false

Operator Technical Architecture

The implementation of drift detection in the operator requires changes across the API, the agent, and the protocol buffers.

API Changes

The operator's API is expanded to support drift detection settings and status tracking.

• operator/api/pulumi/shared/stack_types.go: This file is updated to include a DriftDetection spec, which contains Schedules (a list of DriftSchedule objects consisting of a cron string and an autoRemediate boolean). Additionally, a DriftDetectionStatus is added, featuring LastCheck as a metav1.Time object.
• operator/api/pulumi/v1/stack_types.go: A new DriftDetected condition constant is introduced to track the state of drift.

Protocol Buffer and Agent Changes

To support the required functionality, the communication layer between the operator and the agent is modified.

• agent/pkg/proto/agent.proto: A preview_only field is added to the RefreshRequest. After this modification, the proto code is regenerated using the following command:

bash cd agent && make protoc

• agent/pkg/server/server.go: The Refresh() function is updated to handle the preview_only flag. It utilizes Stack.PreviewRefresh() for non-destructive drift detection. It is noted that the Pulumi Automation API already supports this functionality via Stack.PreviewRefresh().

Observability in Kubernetes

The operator provides visibility into drift through Kubernetes-native mechanisms.

Status Conditions

The status of a Stack CR reflects whether drift has been detected.

Example when drift is detected:

yaml status: conditions: - type: DriftDetected status: "True" reason: Changes message: "2 additions, 0 deletions, and 1 change" lastTransitionTime: "2025-01-23T10:30:00Z" - type: Ready status: "True" reason: ProcessingCompleted message: "the stack has been processed and is up to date" driftDetection: lastCheck: "2025-01-23T10:30:00Z" lastUpdate: type: refresh state: succeeded permalink: "https://app.pulumi.com/org/project/stack/updates/42"

Example when no drift is detected:

yaml status: conditions: - type: DriftDetected status: "False" reason: NoChanges message: "No changes detected" lastTransitionTime: "2025-01-23T10:30:00Z" - type: Ready status: "True" reason: ProcessingCompleted message: "the stack has been processed and is up to date" driftDetection: lastCheck: "2025-01-23T10:30:00Z" lastUpdate: type: refresh state: succeeded permalink: "https://app.pulumi.com/org/project/stack/updates/42"

Kubernetes Events

For integration with external notification systems (such as Slack or PagerDuty), the operator emits Kubernetes events when drift is identified.

Example event:

Summary of Drift Detection Features

The following table provides a detailed comparison of the drift detection and remediation mechanisms across different Pulumi interfaces.

Lifecycle Management and TTL Stacks

Beyond drift detection, Pulumi has expanded its infrastructure lifecycle capabilities with the introduction of Time-to-Live (TTL) Stacks. While drift detection focuses on the accuracy of the state, TTL Stacks focus on the duration of the infrastructure's existence.

TTL Stacks automatically clean up infrastructure based on flexible policies after a specified amount of time has elapsed. This is particularly critical for self-service environments where developers may spin up temporary stacks for testing. Without TTL policies, these stacks often become stale, leading to "ghost" infrastructure and unnecessary cloud costs.

Joe Duffy, co-founder and CEO of Pulumi, has noted that while drift detection is the more immediately recognizable and critical feature for production stability, TTL stacks are a significant advancement for organizational cost control and environment hygiene.

Analysis of the Drift Reconciliation Workflow

The lifecycle of drift management follows a structured logical path: Detection, Notification, and Resolution.

The detection phase relies on the contrast between the "Desired State" (defined in code), the "Recorded State" (stored in the Pulumi state file), and the "Actual State" (the real-time configuration in the cloud). Drift occurs when the Actual State diverges from the Recorded State.

Once drift is detected, the notification phase begins. In Pulumi Cloud, this is visualized through the Drift Tab and the warning bell. In the Kubernetes Operator, this is communicated via status conditions and events. This ensures that the human operator or the automated system is aware of the discrepancy.

The final phase is resolution, which can take two paths: remediation or adoption.
- Remediation: The system re-applies the desired state, effectively overwriting the manual changes. This is the preferred path for maintaining strict compliance and security.
- Adoption: If the "drift" was actually an intentional manual change that should be kept, the user can adopt the drift, updating the IaC code to match the current actual state.

The automation of this workflow through scheduled runs and autoRemediate flags reduces the cognitive load on engineers and minimizes the "mean time to recovery" (MTTR) for infrastructure inconsistencies.

Sources

1. Implement Drift Detection with Scheduled Refresh #1037
2. Drift detection and remediation
3. Pulumi Launches New Infrastructure Lifecycle Features