Pulumi Console and Logging Architecture

The Pulumi ecosystem employs a sophisticated, multi-tiered approach to visibility, state management, and diagnostic output. At the core of this architecture is the Pulumi Console, a web-based application designed to automate the management of deployment state and facilitate high-level collaboration between developers and operators. This application serves as the central nervous system for infrastructure as code (IaC) operations, integrating seamlessly with the Pulumi CLI. By default, the CLI utilizes the Console unless an explicit opt-out is performed by the user. The Console provides a collaborative environment where teams can coordinate updates and implement rigorous security measures via Role-based Access Control (RBAC), ensuring that access to specific stacks is managed with fine-grained precision.

Beyond the administrative interface, Pulumi implements a dual-pronged logging strategy: CLI logging and program logging. CLI logging is designed for the diagnosis of the Pulumi engine's internal operations and the behavior of resource providers. Conversely, program logging allows developers to emit custom diagnostics directly from within their infrastructure code. These two systems, while distinct in purpose, are designed to be used in tandem to provide a comprehensive view of the deployment lifecycle. When a developer executes a command, the resulting logs are not only displayed in the CLI but are also streamed to Pulumi Cloud, where they are archived for historical auditing and troubleshooting.

The Pulumi Console Interface and CLI Integration

The Pulumi Console is not merely a viewer but a management layer that abstracts the complexities of state tracking. By integrating the web application with the command-line interface, Pulumi ensures that deployment state is handled automatically, reducing the risk of state corruption or loss that can occur with manual state file management.

The primary mechanism for accessing this environment via the command line is the pulumi console command. This command opens the current stack within the Pulumi Console web interface, allowing the operator to transition from a terminal-based workflow to a visual management environment.

The pulumi console command supports several flags to refine the experience:

• -h, --help: Provides the help documentation for the console command.
• -s, --stack string: Specifies the name of the stack to be viewed, allowing the user to bypass the current active stack and target a specific environment.

In addition to these specific flags, the command inherits several global options from the parent Pulumi CLI:

• --color string: Controls the colorization of the output. Available choices include always, never, raw, and auto (which is the default).
• -C, --cwd string: Allows the user to run Pulumi as if it had been started in a different directory, which is critical for managing projects with complex folder structures.
• --disable-integrity-checking: Disables the integrity check of checkpoint files.
• -e, --emoji: Enables the use of emojis in the output for better visual scanning.
• -Q, --fully-qualify-stack-names: Ensures that stack names are shown in their fully qualified form.
• --logflow: Determines if log settings are flowed down to child processes, such as plugins.
• --logtostderr: Redirects logs to the standard error stream instead of writing them to temporary files.
• --memprofilerate int: Enables more precise and computationally expensive memory allocation profiles by setting the runtime.MemProfileRate.
• --non-interactive: Disables the interactive mode for all commands, which is essential for automated scripts.
• --otel-traces string: Exports OpenTelemetry traces to a specified endpoint. The system supports file:// for local JSON files and grpc:// for remote collectors.
• --profiling string: Emits CPU and memory profiles and an execution trace to files named [filename].[pid].{cpu,mem,trace}.
• --tracing file: Emits tracing to a specified endpoint using the file: scheme to write data to a local file.
• -v, --verbose int: Enables verbose logging. For example, -v=3 enables basic verbosity, while any value greater than 3 is considered very verbose.

CLI Verbose Logging and Engine Diagnostics

CLI logging focuses on the internal mechanics of the Pulumi engine and the resource providers. This is the primary tool for diagnosing issues related to the deployment process itself rather than the logic of the program.

The verbosity of these logs is controlled via the -v flag. Pulumi utilizes a scale of log levels ranging from 1 to 11. The higher the number, the more detailed the output.

The impact of these levels is significant regarding security:

• Log levels 1 through 10: Pulumi avoids intentionally exposing known credentials.
• Log level 11: Pulumi will intentionally expose some known credentials. This level is high-risk and should be utilized only in extreme debugging scenarios where the credentials are necessary to diagnose the failure.

By default, these logs are stored in the top-level temporary directory, typically /tmp or the directory defined by the $TMPDIR environment variable. To modify this behavior, Pulumi provides specific flags:

• --logtostderr: This flag forces the logs to be written to stderr. This is particularly useful when the user wants to see the logs in real-time in the console or redirect them to a file using shell redirection.
• --logflow: This flag is critical for debugging cloud provider API interactions. Without --logflow, only the Pulumi engine logs at the specified verbosity level, while the resource providers continue to use their default log levels. When --logflow is enabled, the verbosity level is passed down to the providers.

At high verbosity levels, such as -v=9, the use of --logflow allows resource providers to log raw HTTP requests and responses to their cloud APIs. This capability is indispensable for diagnosing incorrect resource references, unexpected 404 errors, or problematic API parameters.

Program Logging and SDK Integration

While CLI logging tracks the engine, program logging allows developers to embed diagnostics directly into their code. This allows for context-aware logging based on the logic of the infrastructure.

These messages appear in the CLI output and are stored in Pulumi Cloud for historical reference. The Pulumi SDK provides several methods for emitting these logs across different languages.

The following table outlines the primary logging functions and their purposes:

In Go, the SDK allows for the use of LogArgs to provide additional context:

go args := &pulumi.LogArgs{ Resource: resource, StreamID: 0, Ephemeral: false, } ctx.Log.Info("message", nil) ctx.Log.Info("message", args)

In Java, the syntax follows a similar pattern:

java public static void stack(Context ctx) { ctx.log().info("message"); ctx.log().info("message", resource); ctx.log().debug("hidden by default"); ctx.log().warn("warning"); ctx.log().error("fatal error"); }

For general program debug logging, the -d or --debug flag can be used when running key commands such as pulumi up, pulumi preview, pulumi destroy, pulumi import, pulumi refresh, or pulumi watch.

Advanced Configuration and Integration

Combining CLI and Program Logging

For comprehensive troubleshooting, developers often need to combine both engine-level verbosity and program-level debug flags. This provides a complete picture of what the program is requesting and how the engine and providers are responding.

An example of this combined execution is:

bash $ pulumi up --logtostderr --logflow -v=9 -d 2> out.txt

In this command, the --logtostderr and --logflow flags ensure that provider-level HTTP traffic is captured, -v=9 sets the engine verbosity, and -d enables the program-level debug logs. The output is then redirected to out.txt for later analysis.

Pipeline Logging and Environment Variables

In CI/CD pipelines, altering the command string may not always be possible. Pulumi provides environment variables that map directly to CLI flags to enable logging.

The following environment variables are equivalent to the aforementioned CLI flags:

• PULUMIOPTIONLOGFLOW=true
• PULUMIOPTIONLOGTOSTDERR=true
• PULUMIOPTIONVERBOSE=9

Provider Diagnostic Logging

Beyond the Pulumi engine, some resource providers offer their own diagnostic logging. This is especially true for providers that use a bridged Terraform provider. These providers can be controlled using the TF_LOG environment variable.

The available levels for TF_LOG are:

• TRACE
• DEBUG
• INFO
• WARN
• ERROR

An example of executing a deployment with both Pulumi engine and Terraform provider tracing is:

bash $ TF_LOG=TRACE pulumi up --logtostderr --logflow -v=10 2> out.txt

Integration with Serilog in .NET

For .NET developers, integrating professional logging frameworks like Serilog into Pulumi projects allows for greater flexibility and the ability to persist logs to files.

To implement this, the Serilog NuGet package must be added. A configuration method is typically created to define the logger behavior, as standard config file approaches may not align with the Pulumi execution model.

A typical configuration method looks like this:

csharp public static void ConfigureLogger() { Log.Logger = new LoggerConfiguration() .Enrich.FromLogContext() .Enrich.WithExceptionDetails(new DestructuringOptionsBuilder() .WithDefaultDestructurers() .WithDestructuringDepth(8)) .WriteTo.PulumiLogSink() .WriteTo.File(path: $"{Environment.CurrentDirectory}\\log\\log.txt") .CreateLogger(); Log.Information("Logging configured"); }

Key components of this integration include:

• WithExceptionDetails: This requires a specific package and is used to capture detailed exception data.
• WithDestructuringDepth(8): This is essential to limit the depth of the object graph being logged to prevent infinite loops or excessive memory use.
• WriteTo.PulumiLogSink(): This custom sink implementation ensures that Serilog logs are forwarded to the Pulumi output system, preventing the loss of standard Pulumi output.
• WriteTo.File: This creates a local history of logs in a text file.

Handling Command Output in .NET

When utilizing Run.Invoke() in C#, the console may not automatically display the output of executed commands. While errors are shown, normal stdout must be accessed explicitly.

This is achieved using the .Apply() method on the command output. For example:

csharp publishCommand.Apply(x => { Console.WriteLine(x.Stdout); return x; });

Since Apply<T>() requires a return value, the original object x is returned. This allows the output to be visible during the pulumi preview or pulumi up steps, specifically during the preview phase when Run.Invoke() is utilized.

Technical Analysis of Pulumi Data Flow

The interaction between the Pulumi Console, the CLI, and the logging systems creates a robust feedback loop. The data flow begins at the resource provider level, where HTTP interactions are captured via --logflow. This data is passed to the Pulumi engine, which filters it based on the -v verbosity level.

Simultaneously, the program execution emits logs via the SDK (e.g., ctx.log().info). These logs are interleaved with the engine logs in the CLI output. The engine then forwards this combined stream to the Pulumi Console/Cloud.

The use of Inputs and Outputs further complicates this flow. Outputs are values not known until the cloud provider API returns them at runtime. Consequently, these values must be handled as Output<T> and accessed via .Apply(). Inputs are properties that configure resources, such as ResourceGroupName, and can accept either a raw value of type T or an Output<T>.

This architecture ensures that:

• Visibility is scalable: From basic info to raw HTTP trace data.
• Security is maintainable: Credentials are protected until log level 11.
• Collaboration is centralized: State and logs are managed in the Pulumi Console.
• Debugging is precise: Integration with tools like Serilog and TF_LOG allows for deep-dive forensics.

Sources

1. pulumi console | CLI commands
2. Logging
3. pulumi-with-serilog
4. Pulumi Console
5. how-to-use-pulumi-with-c-inputs-and-outputs