Terraform in a team, without a US cloud platform

The default answer, and what it actually sells

The default answer to the question of how a team runs Terraform together has become HCP Terraform (née Terraform Cloud). It is a decent product. It is also a US SaaS sitting in the critical path of every apply, and, in the honest accounting, teams buy it for three things: a shared home for the state, a lock on that state, and somewhere to keep the credentials. Three things. All three of them are solvable with commodity EU infrastructure, and OpenTofu quietly removed the last hard part. This repository is the working proof of that, and I want to walk through it, including the parts that annoyed me.

The lock was the excuse, and it is gone

Start with the lock, as that was the excuse for the longest time. The old S3 backend needed a DynamoDB table beside the bucket to serialise applies, which meant a second piece of infrastructure, in a second service, with a second set of permissions, existing purely to hold one row. From OpenTofu 1.10 onward the backend takes use_lockfile = true and locks through S3 conditional writes instead. No table. No separate lock infrastructure at all. Any S3-compatible object store with conditional-write support qualifies; we use Scaleway Object Storage, one bucket called eux-tofu-state, with a per-repository prefix (website/ for this one). Concurrent applies are blocked by the object store itself. There is genuinely nothing left to coordinate by hand, which is the sentence I kept wanting to write for years.

        before                          now
┌───────────────────┐         ┌───────────────────┐
│   S3 bucket       │         │   S3 bucket       │
│   terraform.state │         │   terraform.state │
└─────────┬─────────┘         │   .tflock         │
          │ lock in           └───────────────────┘
          ▼                    conditional write
┌───────────────────┐          locks in place
│  DynamoDB table   │
│  one row          │
└───────────────────┘

The backend block in infra/main.tf is roughly ten lines, plus four skip_* flags that exist for one reason: Scaleway is S3-compatible, not AWS.

backend "s3" {
  bucket                      = "eux-tofu-state"
  key                         = "website/terraform.tfstate"
  region                      = "fr-par"
  endpoints                   = { s3 = "https://s3.fr-par.scw.cloud" }
  use_lockfile                = true
  skip_credentials_validation = true
  skip_region_validation      = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
}

Each of the four does something specific. skip_credentials_validation stops the provider calling AWS STS to check the key, which of course fails, as the key is a Scaleway key. skip_region_validation stops it rejecting fr-par, a region name that is not in the AWS list. skip_requesting_account_id stops the lookup of an AWS account number that does not exist. And skip_metadata_api_check stops the instance-metadata probe, the one that hangs for a while on a laptop before giving up. None of them are dangerous. They are the price of speaking S3 to something that is not Amazon.

The bucket policy that locks you out of your own state

Now the bucket policy, which is where I lost an afternoon. Scaleway's policy version 2023-04-17 cannot grant to project_id principals, although you would reasonably expect it to, given the whole platform is organised by project. The policy therefore pins the IAM application id instead. Fine. The trap sits one step further: a bucket policy denies everything not explicitly allowed. Which means the key that Terraform itself uses has to grant itself s3:* in the very policy it is applying, or the apply succeeds and then locks the tool out of the bucket holding its own state. There are comments about this in infra/site.tf, written in the tone of somebody who had done it once already.

Credentials, and the bootstrap question

Credentials next. The team story is a password manager, 1Password in our case, though any EU alternative slots into the same shape. A per-project item is rendered into .env by op inject from a .env.tpl that is committed to the repository (see .mise-tasks/dev/init.sh). Onboarding a colleague is two commands. Rotation is editing one item. No credentials in the repository, no credentials sitting in a US SaaS, and the template being committed means the shape of the environment is reviewable even though the values never are.

There is a bootstrap question hiding under all of this, and it deserves an answer rather than a shrug: who creates the state bucket? A root repository does, one level up, and it also provisions the projects themselves and the scoped API keys. We call it the project registry pattern. The point is that the bootstrap problem gets solved once, centrally, instead of once per repository by whoever is unlucky enough to start the next one.

• eux/

  • root
    • projects
    • state bucket
    • scoped API keys
  • website
    • infra
    • site

What you give up

Honestly: the web UI, run history, the policy engines (Sentinel and friends), drift detection, and plan-on-PR comments. Plan-on-PR is the one worth having back, and it costs a CI job. The rest cost nothing to replicate for a small team, mostly by not needing them. Run history is the shell history of whoever ran the thing. Drift detection matters more the larger the estate. Policy engines matter when the number of people who can apply exceeds the number of people you can talk to.

Which teams genuinely miss these? Not many small ones, in my experience. But that is exactly the honest line: when the team is large, when compliance requires a policy engine with an audit trail, when applies must run somewhere nobody has a laptop key for, then Spacelift or env0 or HCP earns the money. Do not pretend otherwise.

One more operational note, on locks, as the failure mode changed. In the DynamoDB era a stale lock was a row somebody had to go and delete, sometimes through the AWS console, always with a small feeling of trespass. With conditional writes the stale lock is an object in the bucket, and the recovery is tofu force-unlock with the id printed in the error. Same amount of care required, fewer moving parts, and the thing you are unlocking lives beside the thing it protects.

Why it is worth the afternoon

The jurisdiction argument is the one I would leave you with, though it should be obvious by now. A state file contains secrets and the complete topology of the infrastructure — every resource, every relationship, occasionally a password that a provider decided to persist. It is arguably the most sensitive artefact a team produces, more sensitive than the source, as it describes the running system rather than the intent. And that is precisely the artefact HCP stores in the United States. Ten lines of backend configuration, a bucket policy, and a password manager, and the artefact stays here. Sovereign by construction, down through the tooling, not bolted on afterwards.