Orchestrating Incident Response: A Comprehensive Guide to Opsgenie and Atlassian Operations Terraform Providers

The convergence of Infrastructure as Code (IaC) principles with incident management has fundamentally altered how Site Reliability Engineering (SRE) and DevOps teams manage on-call rotations, alert routing, and escalation policies. Traditionally, configuring an incident response tool like Opsgenie involved navigating a complex graphical user interface, where changes were subject to manual error, lack of version control, and opaque audit trails. The introduction of Terraform providers for Opsgenie and, more recently, the broader Atlassian Operations ecosystem, has shifted this paradigm. By codifying API interactions into declarative configuration files, organizations can treat their incident management infrastructure with the same rigor and reproducibility as their cloud infrastructure. This approach allows teams to code, edit, review, and version-control their IT operations configurations, ensuring that the setup for handling critical production incidents is deterministic, auditable, and reproducible across environments.

The Evolution from Opsgenie to Atlassian Operations

The landscape of Terraform-based incident management has undergone a significant structural shift. Historically, the dedicated Opsgenie Terraform Provider served as the primary interface for managing resources within the Opsgenie platform. This provider allowed users to interact with a specific subset of resources, including users, teams, escalations, schedules, and various integrations. The core value proposition was the ability to replace manual API calls or UI clicks with declarative HCL (HashiCorp Configuration Language) code. However, the strategic direction of the Atlassian ecosystem has evolved. In December of the previous year, the Atlassian Operations Terraform Provider was announced, effectively replacing Opsgenie's existing standalone provider within the context of Jira Service Management. This new provider is not merely a rebranding; it represents a functional replication and expansion of the capabilities previously housed in the Opsgenie Provider.

The transition is driven by the integration of Opsgenie into the broader Atlassian Operations suite, which encompasses Jira Service Management and Compass. The Atlassian Operations Terraform Provider enables users to manipulate resources across this unified ecosystem. For teams that are transitioning from standalone Opsgenie to Jira Service Management, or those who have recently completed such a migration, the technical implication is a migration of Terraform configurations. Existing HCL definitions that target the opsgenie provider source must be updated to target the atlassian_operations provider. This shift is not only about naming conventions but also about access to a broader range of resources. The new provider now supports ten additional resources compared to the legacy offering, making it significantly easier to get started with Operations in Jira Service Management. This evolution ensures that the incident management layer remains tightly coupled with the service management and observability layers, providing a cohesive view of operational health.

Provider Configuration and Authentication

To effectively manage incident response resources via Terraform, the provider block must be correctly configured with appropriate credentials. The authentication mechanism relies on an API Key, which acts as the bearer token for all subsequent API interactions. This key is not generated by the Terraform provider itself but must be obtained from the platform's security settings. Specifically, within the Opsgenie or Atlassian Operations interface, an administrator must navigate to Settings and then to API Key Management. Here, a new API Integration can be created with Read/Write permissions. It is critical to note that the API Key is sensitive data. In Terraform configurations, this key should never be hardcoded in plain text within version-controlled repositories. Instead, it should be injected via environment variables or a secure vault solution.

The configuration differs slightly depending on the geographic instance being targeted. For users operating on the standard global instance, the default API URL is typically sufficient. However, for organizations operating within the European Union (EU) instance, compliance with data residency requirements mandates the use of a specific endpoint. The configuration must explicitly set the api_url to api.eu.opsgenie.com to ensure that all data transmission and storage remains within the EU. Failure to configure this correctly may result in data sovereignty violations or connectivity issues. Below is a standard configuration block for the provider, illustrating the required arguments and the use of environment variables for sensitive data.

```hcl
terraform {
required_providers {
opsgenie = {
source = "opsgenie/opsgenie"
version = "~> 0.6"
}
}
}

provider "opsgenie" {
apikey = var.opsgenieapikey
# api
url = "api.eu.opsgenie.com" # Uncomment for EU instance
}

variable "opsgenieapikey" {
type = string
sensitive = true
description = "The API Key for the OpsGenie Integration"
}
```

For the newer Atlassian Operations provider, the configuration structure remains conceptually similar, requiring the API key and potentially the specific base URL depending on the Atlassian subscription type (cloud or data center, though the provider is primarily targeted at Cloud instances). The environment variable OPSGENIE_API_KEY is commonly used for local development and acceptance testing, allowing developers to run tests without committing credentials to the repository.

Core Resources: Teams, Schedules, and Rotations

The foundational elements of any incident management strategy are the teams that own the workload and the schedules that determine who is paged when. The Terraform providers for both Opsgenie and Atlassian Operations expose these concepts as first-class resources.

Managing Teams

Teams represent the logical grouping of users who share on-call responsibilities. In the Terraform model, a team resource is defined by its name, description, and its members. The member block is critical as it establishes the relationship between users and the team, including their specific role within the team. Roles typically include admin and user. An admin has full control over the team's configuration, including the ability to manage members and modify the team's settings. A user is a standard member who can be part of schedules and receive alerts but cannot modify the team structure.

When defining teams, it is essential to reference user IDs. Since user IDs are often not known at the time of writing the initial configuration, Terraform data sources are used to look up existing users. This ensures that the configuration remains idempotent and does not fail due to missing ID references.

```hcl

Data sources to lookup existing users

data "opsgenie_user" "alice" {
username = "[email protected]"
}

data "opsgenie_user" "bob" {
username = "[email protected]"
}

Creating a team with members

resource "opsgenieteam" "platformengineering" {
name = "Platform Engineering"
description = "Responsible for core infrastructure and platform services"

member {
id = data.opsgenie_user.alice.id
role = "admin"
}

member {
id = data.opsgenie_user.bob.id
role = "user"
}
}
```

Configuring Schedules and Rotations

A schedule defines the time window during which on-call personnel are expected to respond to alerts. The schedule resource requires a name, a timezone, and an enabled status. The timezone is crucial for ensuring that on-call personnel are paged at reasonable hours relative to their local context. For example, a team distributed across North America and Europe might use a schedule that aligns with UTC or a specific regional timezone to balance the burden.

Within a schedule, rotation resources define the recurring pattern of on-call assignment. Rotations can be weekly, daily, or custom intervals. A weekly rotation is the most common pattern, where one person is the primary on-call contact for a week, and then the responsibility passes to the next member in the sequence. The rotation resource is linked to the schedule via the schedule_id. It specifies the frequency, the participants, and the order in which they serve.

```hcl

Creating a weekly on-call schedule

resource "opsgenieschedule" "platformoncall" {
name = "Platform On-Call"
description = "Weekly on-call rotation for platform team"
timezone = "America/NewYork"
owner
teamid = opsgenieteam.platform_engineering.id
enabled = true
}

Defining the weekly rotation

resource "opsgenieschedulerotation" "platformweekly" {
schedule
id = opsgenieschedule.platformoncall.id
name = "Weekly Rotation"
timezone = "America/NewYork"
type = "weekly"
frequency = 1
start
date = "2023-01-01T09:00:00Z"
end_date = "2099-12-31T09:00:00Z"

participants {
userid = data.opsgenieuser.alice.id
order = 1
type = "primary"
start_time = "09:00:00"
duration = "168:00:00"
}

participants {
userid = data.opsgenieuser.bob.id
order = 2
type = "primary"
start_time = "09:00:00"
duration = "168:00:00"
}
}
```

Advanced Routing and Integration Resources

Beyond basic team and schedule management, the power of Terraform in this context lies in the ability to automate the complex logic of alert routing and escalation. The Atlassian Operations Terraform Provider, in particular, has expanded its scope to include advanced routing and notification policies.

Escalations and Routing Rules

Escalation policies define what happens if an incident is not acknowledged or resolved within a specified timeframe. These policies can be chained to ensure that critical issues receive immediate attention from higher-level leadership if the first responder is unavailable. In the Terraform model, an escalation resource is configured with a name, a description, and a series of steps. Each step defines a participant (team or user), a wait time, and a reminder frequency.

Routing rules are a more granular mechanism that allows organizations to direct alerts based on dynamic conditions. For instance, alerts tagged with severity: critical might be routed to the core-infra team, while alerts tagged with severity: low might be routed to a general platform team. The Atlassian Operations provider supports routing_rule resources, enabling this level of dynamic traffic management. This is particularly useful for large organizations with many microservices, where static routing would be unmanageable.

Integrations and Alert Policies

Integrations are the bridge between external monitoring systems (such as Prometheus, Datadog, or CloudWatch) and the incident management platform. The Terraform providers support the creation of both api_based_integration and email_integration resources. An API-based integration generates a unique webhook URL that can be configured in the monitoring system to push alerts to the platform. Email integrations allow alerts to be sent via SMTP, which is useful for legacy systems that only support email-based notifications.

The Atlassian Operations provider further enhances this capability with alert_policy and notification_policy resources. These policies allow for fine-grained control over how alerts are processed and who is notified. Both Alert Policies and Notification Policies support an optional order attribute, which is critical in complex environments where multiple policies might apply to the same alert. The order attribute controls the execution sequence, ensuring that specific logic is evaluated before others. For example, a suppression policy might need to run before a notification policy to prevent unnecessary pages.

Resource Comparison: Legacy Opsgenie vs. Atlassian Operations

To assist DevOps teams in planning their migration or new implementations, the following table compares the resource support between the legacy Opsgenie Terraform Provider and the newer Atlassian Operations Terraform Provider.

Resource Category Legacy Opsgenie Provider Atlassian Operations Provider Notes
User Resource Data Source Only In Atlassian Operations, users are managed via the directory; Terraform can only read user data.
Team Resource Resource Supports members and roles in both.
Schedule Resource Resource Supports rotations in both.
Schedule Rotation Resource Resource Part of the schedule configuration.
Escalation Resource Resource Core component of incident flow.
Email Integration Resource Resource For email-based alert ingestion.
API Integration Resource Resource For webhook-based alert ingestion.
Notification Rule - Resource New capability for specific notification logic.
Routing Rule - Resource New capability for dynamic alert routing.
Custom Role - Resource Allows for granular permission management.
Alert Policy - Resource Controls alert processing logic.
Notification Policy - Resource Controls who is notified.
User Contact - Resource Manages contact preferences for users.

Development, Testing, and Migration Strategies

For DevOps and SRE teams, the migration from a legacy, ticketing-heavy structure to a modern, developer-first IaC workflow offers the opportunity to strip away administrative bloat. When migrating from Opsgenie to a new platform like All Quiet or Jira Service Management, the "checklist" is not a spreadsheet but a mapping of Terraform resources. Teams should begin by auditing their current HCL files to identify which resources are stateful and which can be refactored.

During the development and testing phase, it is essential to run acceptance tests to verify that the Terraform configurations interact correctly with the API. This requires setting the OPSGENIE_API_KEY environment variable and ensuring that the test environment has a clean state. The provider's documentation for testing and development highlights the necessity of these environment variables for automated test execution.

Furthermore, teams should leverage the order attribute in Alert and Notification Policies to model complex business logic. For example, a team might define a policy that suppresses alerts for a specific service during maintenance windows. This policy should have a higher priority (lower order number) than the general notification policy to ensure it takes effect first. By codifying these rules, teams gain deeper control over how incidents are handled and can reduce noise fatigue, which is a common pain point in on-call rotations.

Conclusion

The integration of Terraform with Opsgenie and the broader Atlassian Operations ecosystem represents a significant maturation in incident management practices. By moving from manual configuration to code-based management, teams gain the benefits of version control, peer review, and automated deployment. The transition from the legacy Opsgenie Provider to the Atlassian Operations Provider is not just a technical upgrade but a strategic alignment with a unified service management platform.

The depth of resource support in the newer provider, including routing rules, alert policies, and notification policies, allows for sophisticated automation that was previously impossible or cumbersome to achieve. The ability to define teams, schedules, and rotations in HCL ensures that on-call rotations are reproducible and error-free. Moreover, the explicit handling of EU data residency and the robust authentication model via API keys address key compliance and security concerns.

For SRE teams, this shift enables a developer-first approach to operations. The incident response infrastructure becomes a first-class citizen in the codebase, subject to the same CI/CD pipelines and quality gates as the production applications. As the Atlassian Operations Terraform Provider continues to evolve, with its status as "under development" implying ongoing additions, the scope for IaC-based incident management will only expand. Teams that adopt this approach early will be better positioned to handle the increasing complexity of distributed systems, ensuring that their incident response is as scalable, reliable, and efficient as the systems they protect.

Sources

  1. Opsgenie Terraform Provider Documentation
  2. How to Create Opsgenie Teams and Schedules with Terraform
  3. Terraform OpsGenie Provider Documentation
  4. Announcing more resources for Atlassian Operations Terraform
  5. Migrating from Opsgenie to All Quiet: The Terraform (IaC) Transformation Guide - Part I
  6. Atlassian Operations Terraform Provider GitHub Repository

Related Posts