The execution of infrastructure as code (IaC) often involves a complex orchestration between a local CLI, a central engine, and various cloud provider resource providers. When a deployment fails or behaves unexpectedly, the default output of a command like pulumi up may be insufficient to diagnose the root cause, particularly when dealing with asynchronous cloud API calls or internal engine state transitions. To resolve these issues, Pulumi implements a sophisticated, multi-layered logging and debugging architecture that allows operators to peel back the layers of abstraction and inspect the raw communication between the Pulumi engine and the underlying cloud APIs. Understanding the nuances of verbose logging is critical for maintaining production-grade infrastructure, as it transforms a generic "Internal Server Error" or "Access Denied" message into a traceable sequence of HTTP requests and responses.
The Pulumi Logging Ecosystem
Pulumi bifurcates its diagnostic output into two distinct categories to separate the concerns of the tool's internal operations from the logic of the user-defined infrastructure program. This separation ensures that developers can isolate whether a failure is occurring within the Pulumi engine (the coordinator) or within the program logic (the definition of the resources).
CLI Verbose Logging
This layer focuses on the internals of the Pulumi engine and the resource providers. When a user executes a command, the engine coordinates the resource graph, calculates diffs, and communicates with providers to effect change. Verbose logging captures these internal transitions. This is essential for diagnosing issues such as provider crashes, authentication failures with the cloud backend, or unexpected state transitions that do not bubble up to the main console output.
Program Logging
Program logging is used for emitting custom diagnostics directly from the Pulumi code. Since Pulumi programs are written in general-purpose languages like Python, TypeScript, or Go, standard print statements often fail to behave as expected due to the asynchronous nature of Pulumi Outputs. Program logging provides a structured way to emit information that is synchronized with the deployment lifecycle.
Mastering the CLI Verbosity Scale
The Pulumi CLI uses a numeric scale to control the granularity of the logs it produces. This allows users to increase the detail level incrementally as they narrow down the source of a problem.
Log Level Range
Pulumi emits logs at levels ranging from 1 to 11. As the number increases, the amount of data written to the log stream increases.
- Levels 1 through 9: These levels provide progressively more detailed information about engine operations and resource provider activities. Level 9 is frequently cited as the standard for comprehensive debug information submitted for support cases.
- Level 10: This is a high-verbosity threshold. Up to and including level 10, Pulumi is designed to avoid intentionally exposing known credentials to prevent sensitive data from leaking into log files.
- Level 11: This is the maximum verbosity level. At this stage, Pulumi will intentionally expose some known credentials. This is a high-risk mode that should be used only in isolated environments or during critical debugging sessions where credential verification is the primary goal.
Advanced Log Control Flags
To make the massive volume of data generated by verbose logging manageable, the CLI provides specific flags to control where the logs go and how they are propagated.
The --logtostderr Flag
By default, Pulumi writes its internal logs to the top-level temporary directory of the operating system, which is typically /tmp on Linux and macOS, or the path defined by the $TMPDIR environment variable. While this keeps the console clean, it makes real-time monitoring difficult. Using --logtostderr forces Pulumi to write these logs to the standard error stream instead. This is highly beneficial for users who want to monitor the engine's progress in real-time or redirect the output to a specific file for later analysis.
The --logflow Flag
The Pulumi architecture involves a hand-off between the engine and the resource providers (the plugins that actually speak to AWS, Azure, GCP, etc.). By default, the -v flag only affects the engine. If the engine is set to verbose but the resource provider is not, the logs will show the engine asking the provider to do something, but they will not show the provider's internal logic or the actual HTTP response from the cloud API.
The --logflow flag solves this by passing the specified log level from the CLI down to the resource providers. This is the only way to see the detailed HTTP requests and responses being sent to cloud provider APIs. For example, if a user is receiving a 404 Not Found or a 403 Forbidden error from a cloud provider, combining -v=9 with --logflow will reveal the exact API endpoint being called and the raw response body from the cloud service.
Strategic Debugging Command Combinations
For professionals tasked with resolving catastrophic deployment failures, combining multiple flags is the most efficient path to resolution.
Comprehensive Diagnostic Execution
To obtain a complete view of both the engine internals and the custom program diagnostics, the following command pattern is recommended:
pulumi up --logtostderr --logflow -v=9 -d 2> out.txt
Breakdown of this command's impact:
pulumi up: Initiates the deployment process.--logtostderr: Ensures all logs are streamed to the console/stderr rather than hidden in/tmp.--logflow: Forces resource providers to adopt the high verbosity level.-v=9: Sets the engine and provider verbosity to a high level, capturing detailed API interactions.-d: Enables program debug logging.2> out.txt: Redirects the stderr stream (where the logs are now flowing) into a file namedout.txtfor persistent storage and analysis.
Alternative Logging Scenarios
Depending on the goal, different combinations may be more appropriate:
- For general engine debugging without provider noise:
pulumi up --logtostderr -v=9 - For capturing specific provider API calls to a log file:
pulumi up --logflow --logtostderr -v=5 2> pulumi-debug.log
Programmatic Debugging in Python
Debugging Pulumi programs in Python presents a unique challenge because of the pulumi.Output type. Because Pulumi handles resources asynchronously, the value of a resource property (like an IP address or a Bucket ID) is not available during the initial execution of the program; it is only available once the resource is actually created in the cloud.
The Output Problem
If a developer attempts to use a standard Python print() statement on a Pulumi Output, the result will be a representation of the object, such as <pulumi.output.Output>, rather than the actual value.
bucket = aws.s3.Bucket("data")
print(bucket.id) # This will not print the actual ID.
Solutions for Printing Outputs
To extract and print the actual value of an output for debugging purposes, developers must use one of two methods:
- The .apply() Method: This method allows the developer to pass a callback function that will be executed once the value is known.
bucket.id.apply(lambda id: print(f"Bucket ID: {id}"))
- The pulumi.log Module: Pulumi provides a built-in logging module that is designed to handle the lifecycle of the program.
pulumi.log.info(f"Creating bucket in stack: {pulumi.get_stack()}")
Attaching Interactive Debuggers
When logging is insufficient and the developer needs to inspect the call stack or change variable values on the fly, an interactive debugger is required. This is particularly useful for complex logic within the __main__.py file.
Implementing debugpy for Python
To attach a debugger to a Pulumi Python program, the debugpy library must be integrated into the entry point of the application.
- Add the following code to
__main__.py:
python
import debugpy
debugpy.listen(5678)
debugpy.wait_for_client()
- Execution Flow: When
pulumi upis run, the program will hitdebugpy.wait_for_client()and pause execution. - IDE Configuration: In Visual Studio Code, the user must create a "Python: Remote Attach" launch configuration pointing to port 5678. Once the debugger attaches, the program resumes, allowing the developer to step through the code line by line.
For users of the CLI in other contexts, the --attach-debugger flag can be used to enable the ability to attach a debugger to the program and any source-based plugins being executed.
Common Pulumi Error Patterns and Resolutions
Through the use of verbose logging and debugging, several recurring patterns of failure have been identified.
Output Type Mismatch
A frequent error occurs when a developer treats a pulumi.Output object as a standard Python string. This results in a TypeError: unsupported operand type(s) for +: 'Output' and 'str'.
Resolution: Instead of using the + operator, developers should use pulumi.concat() to combine outputs and strings, or wrap the logic inside an .apply() block.
Missing Configuration Variables
When a program expects a configuration value that has not been set for the current stack, Pulumi throws an error: error: Missing required configuration variable 'my-infra:instance_type'.
Resolution: The missing variable must be set via the CLI using the command:
pulumi config set instance_type t3.micro
Resource Existence Conflicts
When Pulumi attempts to create a resource that already exists in the cloud provider but is not tracked in the Pulumi state file, an error occurs: error: importing arn:aws:s3:::my-bucket - already exists.
Resolution: This is resolved by either using the pulumi import command to bring the existing resource under Pulumi's management or by changing the physical name of the resource in the code to avoid the naming collision.
The Pulumi Up Command Specification
The pulumi up command is the primary mechanism for deploying and updating infrastructure. It operates by computing a goal state from the program and comparing it against the current state.
Command Syntax and Options
The general syntax for the command is:
pulumi up [template|url] [flags]
The following table details the available options for the pulumi up command:
| Flag | Full Name | Description |
|---|---|---|
-s |
--stack |
Specifies the target stack for deployment. |
-y |
--yes |
Skips the confirmation prompt and auto-approves the update. |
--diff |
--diff |
Displays a detailed difference between the current and desired state. |
--target |
--target |
Allows updating specific resources by providing their URN. |
--refresh |
--refresh |
Refreshes the state against the cloud provider before applying updates. |
-p |
--parallel |
Sets the level of parallelism for resource operations. |
--skip-preview |
--skip-preview |
Bypasses the preview step and proceeds directly to deployment. |
-f |
--skip-preview |
An alias for skipping the preview step. |
--replace |
--replace |
Forces the replacement of a specific resource identified by URN. |
--cwd |
-C |
Specifies a different working directory to load the project from. |
--attach-debugger |
N/A | Enables the attachment of a debugger to the program and plugins. |
Analysis of Automation API and Edge Case Failures
In complex environments, particularly when using the Node.js Automation API, developers may encounter "empty errors" where pulumi up or pulumi preview fails without providing a descriptive error message.
The Automation API Challenge
The Automation API allows Pulumi to be embedded within other applications, effectively treating the Pulumi CLI as a library. In some versions (e.g., Pulumi CLI 3.14.0 running on Node 16.9.1), users have reported that compiling TypeScript to Node.js and disabling TS usage in pulumi.yaml can lead to failures where no error is returned.
The limitation here is that the Automation API may swallow certain engine-level exceptions that would otherwise be visible in a standard CLI execution. In these scenarios, the absence of a detailed error makes the use of verbose logging (-v=9) and stderr redirection absolutely critical, as it allows the developer to see if the failure is happening at the process level rather than the application level.
Furthermore, community feedback indicates that while -v=9 is powerful, it is occasionally difficult to read and may omit critical environmental data such as the list of installed plugins, packages, and their specific versions. This suggests that for highly complex debugging, the verbose logs should be supplemented with a manual dump of the environment using commands like pulumi plugin ls.
Conclusion: Integrated Diagnostic Strategy
Effective troubleshooting in Pulumi requires a hierarchical approach to observability. One should not start with maximum verbosity, as the sheer volume of data can obscure the actual problem. Instead, a strategic escalation is recommended: start with the standard pulumi up output, move to pulumi.log or .apply() for program logic verification, and finally escalate to pulumi up --logtostderr --logflow -v=9 for deep engine and API inspection.
The integration of debugpy for Python developers and the --attach-debugger flag for those needing step-through execution provides a complete toolkit for solving the most opaque infrastructure bugs. By understanding the relationship between the engine, the resource providers, and the cloud APIs, an engineer can transform the deployment process from a "black box" into a transparent sequence of operations, ensuring that infrastructure remains stable, reproducible, and maintainable across all environments.