Terraform Drift Detection: When Auto-Sync Is the Wrong Fix

At 2:14 AM, a critical application server is thrashing under a sudden traffic spike. The instance was launched with the standard credit specification rather than T3's default unlimited, so once it exhausted its accrued CPU credits, it came down to its baseline—10% per vCPU for a t3.micro—and requests started timing out.

You log into the AWS Management Console, switch the credit mode to unlimited, and then resize the instance from t3.micro to t3.medium because the sustained load profile requires higher baseline performance. Changing the type of an EBS-backed instance requires a stop/modify/start cycle—not a simple reboot—so the fix costs minutes of deliberate downtime, everything held in memory, and—because the address was auto-assigned rather than an Elastic IP—a new public IPv4 that half your monitoring config still points at. Service stabilizes. The incident resolves. Terraform drift detection will notice all of it, and that will turn out not to be the useful part.

Nine days later, a colleague updates an unrelated security group rule in the same environment and executes terraform apply. Terraform refreshes state against the live cloud API, sees t3.medium in reality and t3.micro in the HCL configuration, and marks an in-place update (~) for instance_type into the execution plan right beneath an informational note reading Note: Objects have changed outside of Terraform. Though presented as an in-place modification, the AWS provider executes this change via a Stop → Modify → Start sequence under the hood. The diff is displayed in full, but it gets approved anyway—buried beneath a routine change everyone agreed to. Applying the plan takes the instance offline during peak business hours and assigns it yet another IP. Every control worked exactly as designed, yet the incident returned.

The advice that follows naturally is to close the feedback loop automatically: monitor the cloud environment, diff it against Git, and generate a pull request that updates the HCL code to match live infrastructure. It is a reasonable instinct, and—as the current vendor landscape described below illustrates—it is also the direction the tooling ecosystem is moving by default. The gap between code and reality is the problem, so closing it mechanically looks like the fix.

That advice is incomplete in a critical way. Auto-syncing drift is not merely a passive safety net; it is an automated write path into your infrastructure's source of truth. Without a rigorous governance framework, automated PR generation launders unreviewed emergency hacks and security regressions directly into version control. Drift detection answers what changed—it cannot answer whether that change was worth keeping.

What Terraform Drift Detection Actually Sees

terraform plan refreshes by default, querying cloud APIs to discover live resource attributes before proposing changes—and holds that refreshed data in memory for comparison. Two exceptions matter for automated pipelines: -refresh=false skips the phase entirely, and applying a saved plan file (terraform apply tfplan) does not execute a refresh at apply time, since it applies the prior state recorded in the plan artifact. A pipeline that plans on a schedule and applies from an artifact can therefore act on a drift signal that is already stale.

Native Terraform capabilities—such as terraform plan informational notes and HCP Terraform's health assessments (a scheduled, tier-gated feature)—alert teams when state diverges. Tooling across the ecosystem has evolved to streamline this visibility: as of mid-2026, commercial platforms like Firefly focus on codifying unmanaged infrastructure into HCL diffs, while platforms like Spacelift and env0 offer scheduled drift detection with opt-in reconciliation features to re-apply defined code over unmanaged changes—reconciliation-by-default is increasingly the path of least resistance these tools present. Feature sets in this space move quickly; verify current capabilities against each vendor's own documentation before depending on a specific behavior.

Terraform's CLI also offers -generate-config-out, but it is worth being precise about what it does, because the distinction matters for drift management: it emits HCL configuration only for addresses named by import blocks that have no existing resource block in your code. For a resource Terraform already manages, there is nothing for -generate-config-out to generate. Closing the loop on managed drift is a code-authoring problem, not an import problem.

Furthermore, accepting every observed change as the new baseline misinterprets the primary purpose of Infrastructure as Code. HCL configuration represents human intent, whereas live infrastructure represents current reality. Equating the two automatically strips away intent.

To gate CI pipelines explicitly on detected divergence, teams often rely on terraform plan -detailed-exitcode, which returns 2 when a plan contains proposed actions or pending changes. However, note its boundary: if an out-of-band change affects an unmanaged attribute or one suppressed by ignore_changes, Terraform proposes no action, and the command exits 0.

A Triage Framework: Codify, Revert, or Ignore

When automated systems detect a divergence between code and live infrastructure, the correct response is rarely a blind "sync." Organizations managing production infrastructure at scale need a triage framework that routes detected drift into one of three dispositions:

1. Codify (The Happy Path)

This applies to valid, deliberate emergency fixes—like our 2 AM EC2 instance right-sizing.

• Selection Criteria: The manual edit addresses an architectural bottleneck, survives load testing, complies with security boundaries, and represents the intended ongoing baseline.
• Action: Write the corresponding HCL, open a peer-reviewed PR, and merge it to align Git with reality before the next automated deployment runs.

2. Revert (The Default Safety Net)

Emergency changes made under pressure are frequently sloppy or insecure. If an engineer opens an AWS Security Group to 0.0.0.0/0 to debug a network routing issue at 3 AM, what happens next depends entirely on how that group's rules are declared—and the more current the codebase, the worse the outcome. With rules inline in aws_security_group, the provider treats the block as authoritative: the drift is visible, an auto-sync system will happily generate a PR codifying that wide-open CIDR into Git, and a re-apply removes it. With rules managed as standalone aws_vpc_security_group_ingress_rule resources—the pattern HashiCorp now recommends—the added rule is bound to no address in state at all. Terraform reports no drift, no PR is generated, and terraform apply will not remove it. The safety net has a hole in precisely the shape of the change you most want it to catch.

• Selection Criteria: The edit violates security policy, was temporary troubleshooting scratchpad, or represents an unauthorized deviation from agreed standards.
• Action: Re-apply the canonical configuration to overwrite live state—but only where the drifted object is actually represented in your configuration. Where child objects are managed as separate rule resources, reverting is a detection problem before it is an apply problem, and enforcement has to come from outside Terraform (AWS Config rules, SCPs, CSPM) because Terraform cannot revert what it never sees.

3. Ignore (Delegated Ownership)

Infrastructure elements frequently change due to legitimate non-human actors—such as Auto Scaling Groups adjusting instance counts, Kubernetes operators mutating node pools, or cloud provider controllers injecting runtime tags.

• Selection Criteria: The attribute is actively mutated by an authorized external controller or runtime engine with a dedicated feedback loop outside Terraform.
• Action: Explicitly exclude the attribute from drift tracking using Terraform lifecycle rules or provider-level ignore parameters.

The Revert vs. Ignore Tension: Be cautious when using ignore_changes. While it successfully suppresses drift noise, it also disables Terraform's ability to enforce baselines on those attributes. If an engineer manually widens an ingress rule on a security group whose rules are scoped inside an ignored attribute block, Terraform will never revert that change. ignore_changes should be restricted to operational, non-security attributes with designated external owners.

Where Terraform Drift Detection Breaks Down in Production

Placing automated PR generation on top of drift detection without a triage model creates severe operational failure modes.

Controller Conflicts and Notification Exhaustion

When external controllers such as AWS Auto Scaling manage specific resource attributes, automated drift tools detect constant divergence. If every capacity adjustment or autoscaled instance launch generates an automated PR, team channels are flooded with noise. Within weeks, engineers experience alert fatigue and rubber-stamp every machine-generated PR—defeating the purpose of code review.

The fix here is explicit declaration using Terraform's lifecycle { ignore_changes = [...] } block to delegate attribute management to the appropriate external controller.

Operational Caveat: The configuration above has a sharp edge on resource replacement, and it is worth being precise about where the edge comes from. desired_capacity is Optional+Computed, so omitting it already produces no diff — the ignore_changes entry is documentation of intent here, not the mechanism, and it only does real work if you also declare the attribute. The hazard is the omission itself. ignore_changes governs updates, not creation, and neither does anything to help on a create: any change that forces a new Auto Scaling Group — a name_prefix change, for instance — calls CreateAutoScalingGroup with no DesiredCapacity, and the API defaults it to MinSize. A fleet scaled out to 9 instances comes back at 2. With create_before_destroy = true, this capacity collapse happens silently: the replacement group reports healthy at minimum capacity while the scaled-out group is torn down, leaving latency graphs as your first alert.

Computed and Provider-Defaulted Noise

Provider-defaulted noise arrives in two distinct shapes, and they require different remedies.

The first is computed attribute noise: a provider upgrade suddenly starts reporting aws_instance root block device defaults or newly surfaced optional metadata fields across existing resources. There is no global provider switch for non-tag computed drift. The levers here are procedural: staging provider upgrades in non-production environments first, reading the initial plan diff as a provider changelog rather than live infrastructure drift, and applying targeted ignore_changes only where an attribute is genuinely mutated by an external system.

The second is tag noise injected by external controllers onto resources Terraform does manage — Karpenter and the AWS Load Balancer Controller stamping kubernetes.io/* and karpenter.sh/* discovery tags onto Terraform-managed subnets and security groups, or a CSPM remediation bot writing compliance tags across an account. Two adjacent cases look like the same problem and are not: tags AWS reserves under the aws: prefix never surface as drift because the provider filters them out of tags/tags_all on read, and tags on resources Terraform never created — the EKS-managed cluster security group, for instance — produce no diff because no address tracks them. ignore_tags is the wrong tool for both. Scattering ignore_changes across hundreds of resource blocks for the tags that do need suppressing creates unmaintainable HCL. Here, provider-level controls offer a clean solution. The AWS provider exposes ignore_tags as a sibling of default_tags, scoping suppression once at the provider block:

Two limits worth knowing before relying on this: ignore_tags suppresses tags injected by external actors—controllers, agents, and deployment tooling—onto resources Terraform manages. It does not govern tags managed by dedicated resources like aws_ec2_tag, and declaring the same key in both default_tags and ignore_tags puts the provider in contradiction with itself — one block asserts the tag, the other suppresses it. Rather than relying on a specific error message, treat the overlap as a configuration bug: audit that your default_tags keys and your ignore_tags keys and prefixes are disjoint before rollout. Verify tag suppression against an actual plan diff rather than assuming a clean sweep.

Governing Autonomous Write Paths

Auto-generating code from live environments effectively creates an automated committer agent with direct write influence over your production source of truth. The blast-radius controls that apply to any autonomous agent with write access to production apply here for the same reason: the write path, not the writer's intent, defines the risk. The same discipline that makes machine-authored commits reviewable — an explicit, documented commit contract — is what makes a machine-authored drift PR auditable rather than merely plausible. Automated PR generation is useful for drafting code, but auto-merging drift PRs is architectural negligence.

Reconciling Drift: Commands, Dispositions, and Misconceptions

When explaining manual drift resolution, tutorials frequently suggest running terraform import. This is technically incorrect for managed infrastructure.

terraform import brings unmanaged cloud resources into the Terraform state file. If a resource was already provisioned by Terraform, running terraform import against it will fail with an error stating that the resource is already managed. The state file already knows about the resource; what is out of sync is the HCL configuration file. Note also that this failure is scoped to the address, not the object. terraform import writes a state entry; it never authors HCL, and it rejects any address that does not already exist in your configuration. So the trap requires a human step: hand-write a second resource block at a fresh address, import the same live object into it, and the import succeeds. You now have two state entries bound to one cloud object. Terraform expects each remote object to map to exactly one address, and this is sharper than an outright error precisely because nothing complains — the duplication stays invisible while both blocks agree, then surfaces as fighting plans once their configurations diverge, or as an orphaned state entry when one side is destroyed.

To reconcile drift manually, start by inspecting live changes without modifying state or cloud resources, then let your triage decision determine the command loop:

The Misuse of -refresh-only

A common temptation is to run terraform apply -refresh-only to accept drift. While terraform apply -refresh-only updates terraform.tfstate to match observed cloud state without modifying live infrastructure, it does not update your HCL code.

If you run apply -refresh-only to acknowledge a console change but delay editing the .tf file, your HCL remains stale. The next time a colleague executes a standard terraform apply, Terraform will still attempt to revert the live resource to match the stale HCL configuration. Furthermore, apply -refresh-only will accept out-of-band resource deletions by removing those objects from state—meaning the subsequent terraform apply will attempt to recreate them from scratch. Use -refresh-only strictly as a state maintenance tool prior to refactoring, not as a shortcut for code codification.

Triage First, Sync Second

Emergency console fixes will always happen during major production incidents. However, turning every console edit into a pull request treats all drift as equally valid.

Before implementing automated drift synchronization, establish clear triage boundaries:

• Default to Revert: Treat emergency console edits as temporary deviations that must justify their existence before being codified.
• Isolate Dynamic Attributes: Exclude autoscaling ranges, runtime tags, and operator-managed attributes using ignore_changes or provider-level ignore blocks.
• Protect the Review Gate: Treat machine-generated PRs as unverified drafts. Establishing objective validation protocols ensures that human sign-off verifies whether a 2 AM quick fix belongs in the permanent baseline before it merges.

Drift detection tells you what your infrastructure is doing right now. Engineering governance determines what it should be doing tomorrow.