Migrating Off Terraform Cloud/Enterprise: Part 2

Part 2: The "How": Core Migration Steps for Decoupling

With a clear understanding of why a migration is necessary and a thorough inventory of your current setup (as detailed in Part 1), it's time to delve into the how. This second part focuses on the core technical procedures for decoupling your Infrastructure as Code (IaC) workflows from Terraform Cloud (TFC) or Terraform Enterprise (TFE). The primary objective is to regain control over your Terraform state and execution environment, paving the way for your chosen alternative.

1. The Primary Goal: Decoupling from TFC/TFE

The essence of this phase is to carefully extract two critical components from TFC/TFE:

• Terraform State Files: These files are the definitive record of your managed infrastructure. Migrating them correctly is paramount to avoid disruptions or accidental resource modifications.
• Terraform Configurations & Workflow Logic: While your HCL code likely resides in VCS, TFC/TFE might hold specific workspace settings, variable configurations, and run trigger logic that needs to be replicated or re-evaluated in your new setup.

Successfully decoupling means your local Terraform CLI (or your new automation platform) can manage your infrastructure without relying on TFC/TFE for state storage or run execution.

2. Key Migration Steps and Considerations

The exact steps can vary slightly based on your current TFC/TFE configuration (e.g., whether you're using the cloud {} block or a standard remote backend configuration for TFE).

• Step 1: Final State Backup and Freeze Changes (If Possible)
  • Before initiating the active migration of any workspace, perform one last backup of its state file (as covered in Part 1).
  • Ideally, implement a temporary freeze on infrastructure changes for the specific workspace(s) being migrated. This minimizes the risk of state drift during the transition. Communicate this clearly to the relevant teams.
• Step 2: Prepare Your New Backend
  • You should have already decided on your new backend type (e.g., AWS S3 with DynamoDB, Azure Blob Storage, Google Cloud Storage, or a TACO-managed backend).
  • Set up the storage:
    • AWS S3: Create an S3 bucket. Enable versioning (critical for rollback capability) and server-side encryption. Create a DynamoDB table for state locking. Ensure appropriate IAM permissions for access.
    • Azure Blob Storage: Create an Azure Storage Account and a Blob Container. Enable versioning and configure access controls. Azure Blob Storage provides native state locking.
    • Google Cloud Storage (GCS): Create a GCS bucket. Enable versioning and encryption. GCS also offers native state locking.
  • Note down the necessary configuration details for your chosen backend (bucket names, table names, keys, regions, etc.).
• Step 3: Update Terraform Backend Configuration in Your Code
  • This is where you tell your Terraform code where its state will now live.
  • In your Terraform configuration files (typically backend.tf or main.tf), modify the terraform {} block.
  • If migrating from TFC using the cloud {} block:
    • You will remove the cloud {} block entirely.
    • You will add a new backend "<your_chosen_backend_type>" block.
  • If migrating from TFE or TFC using a remote backend:
    • You will modify the existing backend "remote" block to backend "<your_chosen_backend_type>".
  • Important: The key (for S3/Azure) or prefix (for GCS) determines the path within the bucket where the state file for this specific configuration will be stored. Plan your naming convention carefully to organize state files logically (e.g., by project, environment, component).
• Step 4: Initialize the New Backend and Migrate State
  • Navigate to the directory containing your Terraform configuration with the updated backend block.
  • Run terraform init.
  • Terraform will detect that you're changing the backend configuration.
  • If you were previously using TFC/TFE as a remote backend and have local credentials configured for it, Terraform might offer to automatically migrate the state. It will prompt: "Do you want to copy the existing state to the new backend?"
    • Carefully review the prompt. If you are confident in your downloaded backup and the new backend configuration, you can type yes. Terraform will then attempt to pull the latest state from TFC/TFE and push it to your newly configured backend.
  • If automatic migration is not offered or if you prefer manual control (recommended for critical workloads):
    1. Ensure you have the state file you backed up in Part 1 (e.g., downloaded_workspace.tfstate).
    2. Run terraform init -migrate-state. Terraform will initialize the new backend.
    3. Then, push your backed-up state file to the new backend: terraform state push downloaded_workspace.tfstate (Ensure downloaded_workspace.tfstate is the correct, most recent state file for this configuration.)
  • Verification: After terraform init (and potentially state push), run terraform plan. If the backend migration was successful and the state is correctly recognized, the plan should show "No changes. Your infrastructure matches the configuration." or only reflect intentional changes you've made. Any unexpected proposed changes (especially destructions) indicate a potential issue with the state migration that needs immediate investigation.
• Step 5: Manage Variables and Environment Configuration
  • Variables previously managed within the TFC/TFE workspace UI (Terraform variables, environment variables) now need to be managed according to your new setup.
  • Options include:
    • .tfvars files (for non-sensitive variables, committed to VCS or managed securely).
    • Environment variables in your CI/CD system.
    • Secrets management solutions (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) for sensitive data.
  • Replicate the necessary variable configurations for your new execution environment.
• Step 6: Test Thoroughly
  • Perform a terraform plan to ensure Terraform can read the state from the new backend and correctly interprets your infrastructure.
  • If possible, apply a small, non-critical change to confirm the terraform apply process works as expected with the new backend and variable setup.
• Step 7: Update CI/CD Pipelines
  • If you have CI/CD pipelines that interacted with TFC/TFE (e.g., triggering runs via API), these will need to be updated.
  • They should now execute Terraform CLI commands directly, configured to use your new backend and variable management strategy.
• Step 8: Decommission Old TFC/TFE Workspaces (Cautiously)
  • Once you are fully confident that a workspace is successfully migrated and operating correctly with the new backend, you can consider decommissioning the old workspace in TFC/TFE.
  • Proceed with caution. It's often wise to leave the old workspace in a read-only or non-operational state for a period before full deletion, just in case a rollback is unexpectedly needed. Ensure all state data is securely and redundantly backed up from TFC/TFE before deletion.

Example: Migrating to Google Cloud Storage

terraform {
  backend "gcs" {
    bucket  = "your-new-gcs-bucket-name" # Replace
    prefix  = "path/to/your/terraform.tfstate" # e.g., networking/prod
  }
}

Example: Migrating to Azure Blob Storage

terraform {
  backend "azurerm" {
    resource_group_name  = "your-resource-group" # Replace
    storage_account_name = "yourstorageaccountname" # Replace
    container_name       = "your-container-name" # Replace
    key                  = "path/to/your/terraform.tfstate" # e.g., networking/prod/terraform.tfstate
  }
}

Example: Migrating to AWS S3

# Remove this if it exists:
# terraform {
#   cloud {
#     organization = "your-tfc-org"
#     workspaces {
#       name = "your-workspace-name"
#     }
#   }
# }

# Add or modify to this:
terraform {
  backend "s3" {
    bucket         = "your-new-s3-bucket-name" # Replace
    key            = "path/to/your/terraform.tfstate" # e.g., networking/prod/terraform.tfstate
    region         = "your-aws-region" # Replace
    dynamodb_table = "your-dynamodb-lock-table-name" # Replace
    encrypt        = true
  }
}

Choosing Your Target Backend: Key Considerations Revisited

As mentioned in Part 1, the choice of your new backend is crucial.

• Cloud Object Storage (AWS S3, Azure Blob, GCS):
  • Pros: Highly available, durable, cost-effective, native versioning and encryption, often integrates well with other cloud services and CI/CD tools. State locking is well-supported.
  • Cons: Requires you to manage access permissions (IAM roles/policies) and potentially the setup of locking mechanisms (though often native now).
• TACO-Managed Backend: If you're moving to a Terraform Automation and Collaboration Software (TACO), it will typically provide its own managed state backend.
  • Pros: Simplifies backend setup and management as it's handled by the TACO platform. Often comes with integrated UI, RBAC, and other collaboration features.
  • Cons: You are reliant on the TACO vendor for state management; less direct control than managing your own object storage.

Successfully navigating these core migration steps will free your IaC workflows from TFC/TFE. This newfound independence allows you to select an operational model—be it a full open-source stack, a TACO, or a GitHub Actions-centric approach—that best aligns with your team's needs and strategic goals.

Next in the Series (Part 3): Exploring Alternative Paths: The Open-Source Stack vs. TACOs.