Using the AWS S3 Backend Block in Terraform

Using the AWS S3 backend makes it much easier to scale your Terraform usage.

Ryan Fee

• 6 min read

If you're using Terraform to manage your infrastructure on Amazon Web Services, you'll inevitably need to configure a remote backend. The s3 backend block is the standard solution for storing your Terraform state files in an Amazon S3 bucket. This is an essential practice for team collaboration, state locking, and maintaining the integrity of your infrastructure as code.

The Terraform state file is a JSON file that acts as a record of your deployed resources. It maps your Terraform configuration to the actual resources in your AWS account. Storing this file remotely with the s3 backend provides several key benefits:

  • Collaboration: It allows multiple team members to work on the same infrastructure without overwriting each other's changes.
  • State Locking: It prevents concurrent terraform apply operations, which can lead to state file corruption and resource conflicts.
  • Security & Durability: State files can contain sensitive data. Storing them in a secure, encrypted, and highly available S3 bucket is a core best practice.

Basic Usage and Configuration

To use the s3 backend, you need a pre-existing Amazon S3 bucket. For state locking, it's also a best practice to use a DynamoDB table.

Here's a basic s3 backend block configuration:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "my-app.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}
  • bucket: The globally unique name of your S3 bucket.
  • key: The name of the object (the state file itself) within the bucket. You should use a unique key for each separate Terraform configuration.
  • region: The AWS region where your S3 bucket is located.
  • dynamodb_table: The name of the DynamoDB table used for state locking. This prevents concurrent writes.
  • encrypt: A boolean value that enables server-side encryption for the state file at rest.

After adding this block to your main Terraform configuration file, you must run terraform init. This command initializes the backend and prompts you to migrate any existing local state to the remote S3 bucket.

Use Cases and Best Practices

The s3 backend is essential for any production Terraform projects.

  • Team Projects: When multiple developers are working on a single infrastructure, the s3 backend ensures everyone is using the same, up-to-date state file.
  • CI/CD Pipelines: In a continuous integration and continuous deployment (CI/CD) workflow, the pipeline needs a consistent place to read and write the state file. The s3 backend provides a reliable and secure endpoint for tools like AWS CodePipeline or GitHub Actions to execute Terraform.
  • Production Environments: For production infrastructure, the s3 backend is non-negotiable. Its built-in state locking and data durability features are critical for preventing downtime and ensuring the integrity of your production environment.

Best Practices:

  • Separate State: Use a separate S3 bucket and DynamoDB table for each environment (e.g., dev, stage, prod). This isolates state files and prevents accidental cross-environment changes.
  • Permission Scoping: Use IAM policies to grant the minimum required permissions to your users or roles. They should only have permissions to read from and write to their specific state bucket and locking table.
  • Versioning: Enable versioning on your S3 bucket. This gives you a history of your state files and allows you to revert to a previous version if something goes wrong.

Authentication and Examples

You should never hardcode credentials like access keys directly in your configuration. Instead, use one of the supported authentication methods:

Environment Variables: The most common approach. Terraform can automatically use credentials set in environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

export AWS_ACCESS_KEY_ID="<your-access-key-id>"
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
export AWS_DEFAULT_REGION="us-east-1"

terraform init

AWS CLI: If you're logged in via aws configure, Terraform can leverage those credentials automatically. This is a great option for local development.

This method is great for local development. When you configure the AWS CLI, it caches a token and credentials that Terraform can automatically use.

Configure the AWS CLI on your machine.

aws configure
AWS Access Key ID [None]: <your-access-key-id>
AWS Secret Access Key [None]: <your-secret-access-key>
Default region name [None]: us-east-1
Default output format [None]: json

After this is complete, you can run Terraform commands without any extra authentication configuration.

terraform init

Terraform will automatically find the cached credentials and use them.

IAM Roles: The most secure method for resources running in AWS. An EC2 instance, Lambda function, or CodeBuild job can assume an IAM role with the necessary permissions to access the S3 bucket.

You attach an IAM role to the resource, and Terraform assumes that role's permissions automatically.

  1. Create an IAM Role with a trust policy that allows a specific service (e.g., ec2.amazonaws.com or codebuild.amazonaws.com) to assume it.
  2. Attach a policy to the role that grants permissions to access the S3 bucket and DynamoDB table. For example, a policy might look like this:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-terraform-state-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-terraform-state-bucket"
    },
    {
      "Effect": "Allow",
      "Action": "dynamodb:*",
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/terraform-locks"
    }
  ]
}
  1. Attach the IAM Role to your EC2 instance profile, Lambda function, or CodeBuild project. Terraform, when run on that resource, will automatically authenticate using the temporary credentials provided by the role. No secrets are stored on the resource itself.

Service Principal: For CI/CD pipelines running on platforms outside of AWS, a dedicated IAM user with limited permissions is the standard approach. Store the access key and secret in your CI/CD platform's secrets manager.

  1. Create a dedicated IAM User for your CI/CD pipeline.
  2. Generate an Access Key and Secret Key for that user. Note: This is the only time you will see the secret key, so you must save it immediately.
  3. Attach an IAM Policy to the user with the required permissions to access your S3 bucket and DynamoDB table, just as in the IAM Roles example above.
  4. Store the Access Key and Secret Key as secure variables or secrets in your CI/CD platform's secrets manager. You should never commit these credentials to your code repository.
  5. In your pipeline configuration, set these secrets as environment variables before running Terraform. This ensures your credentials are never exposed in logs or source code.

Advanced Concepts

Partial Backend Configuration

Terraform's design prevents you from using variables directly inside the backend block. However, you can leave out sensitive or environment-specific information and supply it at runtime using a backend configuration file or command-line flags with terraform init.

Example using a file:

Run terraform init:

terraform init -backend-config="backend.conf"

backend.conf:

region = "us-east-1"
dynamodb_table = "terraform-locks"

main.tf (partial config):

terraform {
  backend "s3" {
    bucket = "my-terraform-state-bucket"
    key    = "my-app.tfstate"
  }
}

This method prevents sensitive information from being committed to source control.

Managing Multiple Environments with Workspaces

For a single configuration that deploys to multiple environments, Terraform workspaces can be used to manage different state files within the same bucket.

Example:

# Create and switch to a development workspace
terraform workspace new dev

# Create and switch to a production workspace
terraform workspace new prod

When you switch between workspaces, Terraform automatically changes the key to include the workspace name (e.g., env:/dev/my-app.tfstate), ensuring each environment has its own isolated state file.

A Note on OpenTofu and Dynamic Backends

While the classic Terraform CLI maintains its strict rule against dynamic backend blocks, the OpenTofu project, a fork of Terraform, has broken with this convention.

OpenTofu, starting with version 1.8, introduced the ability to use variables and local values within the backend block. This is a significant feature that addresses one of the most long-standing user requests in the Terraform community.

With OpenTofu, you can now write a more flexible and DRY backend configuration, particularly useful for multi-environment setups.

Here's how a dynamic s3 backend block could look in OpenTofu:

variable "env" {
  type    = string
  default = "dev"
}

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-${var.env}-bucket"
    key            = "my-app.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks-${var.env}"
    encrypt        = true
  }
}

This simple configuration allows you to switch between environments by simply changing the env variable. You can pass the variable at the command line:

tofu init -backend-config="env=prod"

This removes the need for separate backend configuration files or complex scripts to handle different environments, making your codebase cleaner and less prone to manual errors. For teams managing many similar environments, or for those who simply prefer a more concise and variable-driven configuration, OpenTofu's support for dynamic backend blocks is a game-changer.