"Unexpected end of JSON input" error locking s3 backend

[petur]

Terraform Version

Terraform v1.3.8
on linux_amd64

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "<redacted>"
    key = "bug/unexpected-end-of-json-input/terraform.tfstate"
    encrypt = true
    kms_key_id = "<redacted>"
    dynamodb_table = "<redacted>"
  }
}

Debug Output

https://gist.github.com/petur/b66b6c7fce877ddb1bf3cc7629101de0

Expected Behavior

When terraform is run with -lock-timeout, it should either acquire the lock, or retry until the timeout expires.

Actual Behavior

Terraform fails with the error:

╷
│ Error: Error acquiring the state lock
│ 
│ Error message: 2 errors occurred:
│       * ConditionalCheckFailedException: The conditional request failed
│       * unexpected end of JSON input
│ 
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Steps to Reproduce

1. terraform init
2. Duplicate the folder containing the configuration
3. Run while TF_LOG=trace terraform plan -lock-timeout=60s; do : ; done in both folders (needs two shells).

(This is an artificial way to reproduce the issue. It was found in CI where multiple people push changes and plans get run in parallel.)

Additional Context

No response

References

No response

[petur]

This looks like a simple race condition in the locking code:

In s3/client.go:

	_, err := c.dynClient.PutItem(putParams)

	if err != nil {
		lockInfo, infoErr := c.getLockInfo()
		if infoErr != nil {
			err = multierror.Append(err, infoErr)
		}

		lockErr := &statemgr.LockError{
			Err:  err,
			Info: lockInfo,
		}
		return "", lockErr
	}

PutItem fails with a ConditionalCheckFailedException when another user has the state locked. The other user then unlocks the state before c.getLockInfo manages to fetch the lock.

In locker.go:

		if le == nil || le.Info == nil || le.Info.ID == "" {
			// If we don't have a complete LockError then there's something
			// wrong with the lock.
			return "", err
		}

Here le.Info.ID is presumably empty because the lock ID was gone before getLockInfo could read it from DynamoDB, and the locker exits the wait loop.

[skp33]

I am also facing the same. Is there any update on this issue?

[zhiweio]

same problem

[Amarsali01]

╷
│ Error: Error acquiring the state lock
│
│ Error message: 2 errors occurred:
│ * ConditionalCheckFailedException: The conditional request failed
│ * unexpected end of JSON input
│
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

[samaiyayashraj]

Getting exactly the same issue. Any update ?

[jbara-buk]

anyone have how to remove the same issue, seems to not have any lock but I am not able to interact with that path

[suchakra-texada]

How to reset the lock? Open to any concurrency related risks - as we can manually coordinate it in a small team.

[igoratencompass]

The issue is usually a lock left in DynamoDB from terraform plan or terraform apply that failed or have been uncleanly interrupted. In that case you will find 2 records in the DyamoDB table, one ending with .tfstate-md5 that usually looks like this:

{
  "LockID": {
    "S": "<some-path>/terraform.tfstate-md5"
  },
  "Digest": {
    "S": "d959b3a7bc1f0c2fc1dcbeb48bd3c17b"
  }
}

and one with .tfstate that usually looks like this:

{
  "LockID": {
    "S": "<some-path>/terraform.tfstate"
  },
  "Info": {
    "S": "{\"ID\":\"0f397c03-0058-fe56-a839-66afd1b912ed\",\"Operation\":\"OperationTypeApply\",\"Info\":\"<bla-bla>\"}"
  }
}

due to failed or interrupted plan (OperationTypePlan) or apply (OperationTypeApply) stage. Select the .tfstate one and delete it. Or run locally:

terraform force-unlock -force 0f397c03-0058-fe56-a839-66afd1b912ed

to remove the lock.

Otherwise, the error unexpected end of JSON input posted at the top:

│ Error message: 2 errors occurred:
│       * ConditionalCheckFailedException: The conditional request failed
│       * unexpected end of JSON input

is probably a result of someone manually tampering with the lock in the DynamoDB table and putting the state file's MD5 sum in the Digest field of a Info type of lock, something like this:

{
  "LockID": {
    "S": "<some-path>/terraform.tfstate"
  },
  "Digect": {
    "S": "d959b3a7bc1f0c2fc1dcbeb48bd3c17b"
  }
}

or vice-versa resulting in wrong (or in this case an empty) Info field content being returned. Obviously, if this was done by terraform itself it is a bug.