Avoid State Update from Unmerged PRs

Date: May 29, 2025

A common challenge is managing state updates from unmerged pull requests (PRs). This post explores these pitfalls and best practices.

1. The Criticality of Terraform State

Terraform uses a state file to map your configurations to actual cloud resources, manage dependencies, and optimize performance. Any inconsistency in this state can lead to operational issues, from failed deployments to accidental resource changes.

2. Pitfalls of Applying from Unmerged Pull Requests

Applying terraform apply directly from feature branches before PRs are merged, while seemingly quick for testing, is risky:

• State Corruption and Conflicts: If multiple developers apply changes from unmerged PRs to a shared state, chaos can ensue. This can manifest as race conditions and overwrites, where concurrent applies lead to unpredictable modifications of shared resources, or an inconsistent state, where the state file becomes a hybrid, unintended configuration, making future terraform plan operations unreliable.
• Infrastructure Drift: The main branch should be your single source of truth. Applying from unmerged PRs undermines this. This leads to a divergence from main, where if a feature branch's changes are applied but the PR is later abandoned or significantly altered, the live infrastructure drifts from what main defines. Consequently, you face difficult reconciliation, as subsequent applies from main may encounter complex plans when Terraform tries to reconcile these discrepancies, risking unexpected outcomes. You might also end up with "shadow" infrastructure", meaning resources exist in your cloud environment that are unmanaged by your primary codebase.
• Compromised Main Branch Integrity: Even if a PR with pre-applied changes is merged, main branch integrity can suffer. There might be hidden issues in applied changes, as an apply might encounter provider issues or API limits not visible in the plan, causing the actual infrastructure to differ from the intended state. This can lead to merging a "liar", where the code in main doesn't accurately reflect the true (and potentially flawed) pre-applied state, resulting in misleading future plans. It also risks undermining CI/CD trust, as pipelines applying from main might behave unexpectedly if the state was already altered by an out-of-band PR apply.
• Traceability Challenges: A clear audit trail is crucial. Applying from unmerged PRs obscures this. The "why" becomes obscured, making it harder to trace infrastructure changes back to a specific, approved PR in main. This can lead to circumventing review, where the formal PR review might be seen as less critical if changes are already live. Furthermore, it results in difficult rollbacks, as reverting problematic changes applied from unmerged PRs is more complex than reverting a merge commit.

3. Strategies for Managing Terraform State Changes

To mitigate these risks, teams adopt specific workflows.

"Apply After Merge": The Gold Standard

This is the most recommended and safest approach. First, developers create feature branches and submit PRs. Then, automated checks and terraform plan are run, and peers review the plan. Finally, and only after PR approval and merge into main, terraform apply is executed, typically via CI/CD. This keeps main as the source of truth and changes linear.

"Apply Before Merge": Benefits and Tooling (including Atlantis)

Some teams terraform apply before merging to verify changes in a real environment. This critically depends on sophisticated tooling like Atlantis. Such tools are needed to manage PR-level locking, which prevents concurrent applies to the same state/directory, and to facilitate automated plan and apply commands via PR comments (e.g., atlantis plan, atlantis apply). Without such tools, this approach is risky.

Essential Supporting Practices

Two essential supporting practices are crucial. Firstly, using remote state backends with locking is vital for preventing concurrent writes. Secondly, implementing automated terraform plan in PRs provides vital visibility for reviewers.

4. How Modern Platforms Like Scalr Streamline State Management

Specialized platforms like Scalr enhance and automate Terraform workflows, addressing the pitfalls of unmanaged PR applies. While tools like Atlantis offer excellent PR-based automation, platforms like Scalr provide broader governance.

Automated Best Practice Enforcement

These platforms programmatically enforce best practices, ensuring apply operations occur under controlled conditions, aligning with "Apply After Merge" or a safe "Apply Before Merge" strategy.

Scalr's Branch Awareness

A key Scalr feature is branch awareness. It understands the implications of code branches for your state. This includes providing warnings for risky operations, flagging attempts to change state from branches with unmerged PRs. It also involves automatic prevention of unintended auto-applies, stopping auto-applies if the state-generating branch (e.g., a feature branch) differs from the workspace's main configuration branch. Furthermore, Scalr ensures controlled state modifications by making sure these checks lead to state changes primarily originating from the main branch or verified PR workflows, preventing accidental overwrites. This built-in intelligence directly addresses the avoid-state-update-by-unmerged-prs concern.

Controlled Pre-Merge Workflows

For "apply-before-merge" workflows, Scalr provides guardrails. This includes VCS integration for PR comment-triggered applies (e.g., /scalr apply), similar to Atlantis but within Scalr's ecosystem. It also means the subjection of pre-merge applies to workspace policies and permissions, and maintaining clear audit trails linking changes to PRs and users. This structured approach is safer than ad-hoc local applies.

5. Illustrative Workflow Snippets

CI/CD: terraform plan on Pull Request

name: 'Terraform Plan on PR'
on: pull_request
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: hashicorp/setup-terraform@v3
    - run: terraform init
      working-directory: ./terraform
    - run: terraform validate
      working-directory: ./terraform
    - run: terraform plan -no-color -input=false -out=tfplan
      working-directory: ./terraform

CI/CD: terraform apply on Merge to Main

name: 'Terraform Apply on Merge'
on:
  push:
    branches: [ main ]
jobs:
  apply:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: hashicorp/setup-terraform@v3
    - run: terraform init
      working-directory: ./terraform
    - run: terraform plan -input=false -out=tfplan # Re-plan for safety
      working-directory: ./terraform
    - run: terraform apply -auto-approve tfplan
      working-directory: ./terraform

Orchestrated Applies via Specialized Platforms

Platforms like Scalr (or tools like Atlantis) manage apply triggers based on workspace settings and PR interactions. Typically, on PR creation, a plan is generated and posted. After approvals, the workflow diverges: the "Apply After Merge" approach (Scalr default) means the merge triggers a platform apply from the target branch. Alternatively, the "Apply Before Merge" approach (common in Atlantis, configurable in Scalr) involves a PR comment (e.g., /atlantis apply, /scalr apply) triggering a controlled apply, with the platform managing locks and state.

6. Summary Table: Comparing Approaches

7. Conclusion: Prioritizing State Integrity

Effective Terraform state management is non-negotiable. Applying changes from unmerged PRs directly is risky without a robust system. "Apply After Merge" is a strong default. For pre-merge applies, tools like Atlantis offer PR-driven automation. For comprehensive governance and an integrated experience, platforms like Scalr are highly recommended. Their features like branch awareness and policy enforcement provide essential guardrails, ensuring state integrity and enabling teams to move faster and safer.