This series of commands will deploy every non-prod service for you. Run them from the top level directory (where this file is located). If you want to run them all quickly, then run each block of bash in a new terminal. If you want to be more careful, run them all one at a time, from top to bottom, inspecting the output on every step.

terraform -chdir="infra/api/service" init -backend-config="dev.s3.tfbackend" -reconfigure
terraform -chdir="infra/api/service" apply -var "environment_name=dev"

terraform -chdir="infra/api/service" init -backend-config="staging.s3.tfbackend" -reconfigure
terraform -chdir="infra/api/service" apply -var "environment_name=staging"

terraform -chdir="infra/frontend/service" init -backend-config="dev.s3.tfbackend" -reconfigure
terraform -chdir="infra/frontend/service" apply -var "environment_name=dev"

terraform -chdir="infra/frontend/service" init -backend-config="staging.s3.tfbackend" -reconfigure
terraform -chdir="infra/frontend/service" apply -var "environment_name=staging"

terraform -chdir="infra/analytics/service" init -backend-config="dev.s3.tfbackend" -reconfigure
terraform -chdir="infra/analytics/service" apply -var "environment_name=dev"

terraform -chdir="infra/analytics/service" init -backend-config="staging.s3.tfbackend" -reconfigure
terraform -chdir="infra/analytics/service" apply -var "environment_name=staging"

1. Install tfenv
2. Get the terraform version to install from terraform_version this file: https://github.com/HHS/simpler-grants-gov/blob/main/.github/workflows/deploy.yml
3. Follow tfenv instructions to install and utilize the given terraform version

Terraform state locks happen when multiple terraform deployments try to roll out simultaneously.

You can fix them on CLI by:

1. Finding the job (via Github Action or otherwise) where the deployment failed. If you aren't sure, then it was probably in a Github Action. You can find a list of failing actions here: https://github.com/HHS/simpler-grants-gov/actions
2. Wait for the deployment that caused the state lock to finish. If you can't find it, just wait 30 minutes.
3. Identify the folder in which the state lock is happening. The Path attribute on the Lock Info block will identify this.
4. Open up your terminal, set up AWS (eg. aws sso login --sso-session grants-sso && export AWS_PROFILE=staging, using the profile for the account that holds the environment), and cd into the folder identified above
5. Run terraform init -backend-config=<ENVIRONMENT>.s3.tfbackend, where <ENVIRONMENT> can be identified by the Path above.
6. Run terraform force-unlock -force <LOCK_ID> where <LOCK_ID> is the value of ID in your state lock message.
7. Re-run your deploy job

Sometimes CLI unlock won't work, that will look like (for example) the following error message:

terraform force-unlock -force <LOCK_ID> Failed to unlock state: failed to retrieve lock info for lock ID <LOCK_ID>: unexpected end of JSON input

When that happens, you need to unlock it via DynamoDB in the AWS console.

1. Login to AWS
2. Open the DynamoDB console
3. Open the tables tab
4. Click on the state locks table. There should only be one.
5. Click the Explore Table Items button
6. Find the item that corresponds to the currently locked state, you can get that by again looking at the Path attribute in your locked job.
7. Remove the Digest key, Save and close
8. Re-run your deploy job

bin/run-command runs a one-off application command as an ECS task, reusing the service's cluster, task definition, and network configuration.

export AWS_PROFILE=staging   # the account holding the environment — see below
./bin/run-command api infra-staging '["flask", "some-command"]'

The script passes no --profile and does not verify the account, so export the right profile first and confirm with aws sts get-caller-identity.

By default the task runs with the service's task role (<service_name>-app), which has read-only OpenSearch access. Some scheduled jobs declare a role_override in infra/api/app-config/env-config/scheduled_jobs.tf, and the scheduler applies it as a Step Functions TaskRoleArn override. bin/run-command does not pick that up — you have to pass it yourself with --task-role-arn.

Without it, an OpenSearch sync job fails with a 403 from the security plugin rather than a permissions error you can act on:

AuthorizationException(403, 'security_exception', 'no permissions for [indices:admin/aliases/get]
and User [name=arn:aws:iam::<account_id>:role/api-<env>-app, ...]')

The two jobs affected today are load-search-opportunity-data and load-search-agency-data, both of which need the <service_name>-opensearch-write role:

export AWS_PROFILE=staging
account_id=$(aws sts get-caller-identity --query Account --output text)

./bin/run-command \
  --task-role-arn "arn:aws:iam::${account_id}:role/api-infra-staging-opensearch-write" \
  api infra-staging \
  '["flask", "load-search-data", "load-opportunity-data", "--no-full-refresh"]'

Deriving account_id from the active profile rather than hardcoding it means a wrong AWS_PROFILE produces an ARN that fails loudly instead of quietly pointing at another account.

Two gotchas:

• The flag is --task-role-arn (hyphens) and must come before the positional arguments.
• --no-full-refresh only syncs opportunities changed since the last run, so records_loaded: 0 right after the hourly scheduled job is expected, not a failure. Omit the flag to rebuild the whole index.

All scaling options can be found in the following files:

API:

• infra/api/app-config/dev.tf
• infra/api/app-config/staging.tf
• infra/api/app-config/prod.tf

Frontend:

• infra/frontend/app-config/dev.tf
• infra/frontend/app-config/staging.tf
• infra/frontend/app-config/prod.tf

Scaling is handled by configuring the following values:

• instance desired instance count
• instance scaling minimum capacity
• instance scaling maximum capacity
• instance CPU
• instance memory

Our ECS instances auto scale based on both memory and CPU. You can view the autoscaling configuration here: infra/modules/service/autoscaling.tf

Scaling is handled by configuring the following values:

• Database minimum capacity
• Database maximum capacity
• Database instance count

In prod, the database maximum capacity is as high as it goes. Further scaling past the point will require scaling out the instance count. Effectively using the instance count scaling might require changes to our application layer.

• Search master instance type
• Search data instance type
• Search data volume size
• Search data instance count
• Search availability zone count

When scaling openSearch, consider which attribute changes will trigger blue/green deploys, versus which attributes can be edited in place. You can find that information here. Requiring blue/green changes for the average configuration change is a notable constraint of OpenSearch, relative to ECS and the Database.

We manage several secret values that need to be rotated yearly.

🚨🚨🚨 - Unclear how a cert rotation works with terraform pointing to a certificate in ACM by domain name. Leave a little extra time until we figure that out and document better below.

bin/create-csr <domain prefix> # anything you pass as the prefix will be prepended to .simpler.grants.gov
bin/create-csr soap.training # would yield a CSR for soap.training.simpler.grants.gov

bin/create-csr # called without any domain will generate a request for the root, simpler.grants.gov domain

The process starts by generating a cert like so. You will need to generate the certs one at a time.

openssl genrsa -out api_simpler_grants_gov.key 2048
openssl req -new \
    -key api_simpler_grants_gov.key \
    -out api_simpler_grants_gov.csr \
    -subj "/C=US/O=Simpler Grants Gov/CN=api.simpler.grants.gov" \
    -addext "subjectAltName=DNS:api.simpler.grants.gov"

You give the {url}.csr file to HHS so that they can perform the next step. When they get back to you, proceed to part 2a

You will get a zip file back from HHS containing the certificate. Inside the zip there will be one file. If it isn't called {url}.cer then rename it so that that's its name. That's the cert. Here you have two options. If the cert isn't plain text, you need to turn it into plain text. How to determine if something is plain text or not can be done by cat'ing the file and seeing if it has a bunch of �'s in it, that means it's binary. Binary certs can be turned into plain text, then uploaded, like so:

openssl x509 -inform DER \
    -in api_simpler_grants_gov.cer \
    -out api_simpler_grants_gov.pem # the file extensions here are functionally arbitrary

Alternatively, if you get an error (or it's a p7b file), you might need to run:

openssl pkcs7 -inform DER -print-certs \
    -in nofos_simpler_grants_gov_chain.p7b \
    -out nofos_simpler_grants_gov_chain.pem

Once you have plain text certs, you need to split it into the chain and the main cert. The main cert is the first certificate in the file, and the chain is the remaining (likely 2 certificates). So split it into a certificate.pem and certificate_chain.pem.

Later on (eg. during part 3), you might get the following error:

NET:ERR_CERT_AUTHORITY_INVALID

If that happens, then that means that something went wrong with this step, eg. step 2a. In that case, you need to redo your work in form of step 2b.

If there are multiple certificates (eg. a chain certificate is included) then there will be given another file called something like {url}_chain.cer and the commands look more like this

openssl x509 -inform DER \
    -in api_simpler_grants_gov.cer \
    -out api_simpler_grants_gov.pem

openssl x509 -inform DER \
    -in api_simpler_grants_gov_chain.cer \
    -out api_simpler_grants_gov_chain.pem

aws acm import-certificate --certificate fileb://api_simpler_grants_gov.pem \
    --certificate-chain fileb://api_simpler_grants_gov_chain.pem \
    --private-key fileb://api_simpler_grants_gov.key

Or if one of the certs is in plain text, then the command looks like this

aws acm import-certificate --certificate fileb://api_simpler_grants_gov.pem \
    --certificate-chain fileb://api_simpler_grants_gov_chain.cer \
    --private-key fileb://api_simpler_grants_gov.key

In summary, Inspect which certs are binary, which are plain text, and decrypt them as needed. Then import the file with multiple certs into --certificate-chain.

See #5261 for an example of committing this change

Set up your console for AWS Credentials

cd infra/<app name>/service
terraform init -backend-config <env name>.s3.tfbackend
terraform apply

Hit the service at the domain name with https://

Otherwise the next deploy from main will tear back down HTTPS

This is the last step where the cert actually starts being used. Login to the AWS console and open up this page:

https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LoadBalancers:v=3

From there:

• Click on the load balancer whose cert you are updating, like api-prod
• Scroll down to "Listeners and rules"
• Click HTTPS:443
• Click Certificates
• Click Add certificate
• Choose the certificate that you just uploaded, you can determine this by looking at the CertificateArn CLI output, or by otherwise inspecting the AWS console here.
• Add the certificate. This UI is confusing and likely subject to change on AWS's end, so the exact text of the buttons of the buttons isn't included in this documentation.
• Remove the old certificate.

At this point the new certificate should start being served... after 5 ~ 10 minutes. Check your website periodically during that turn to confirm the rotation of your cert.

Certificates were last rotated:

• Dev - December 2025
• Staging - December 2025
• Training - August 2025
• Prod - February 2026

We need to manage a public certificate with login.gov for private_jwt_auth in each of our environments.

To generate a certificate run:

openssl req -nodes -x509 -days 365 -newkey rsa:2048 -keyout private.pem -out public.crt -subj "/C=US/ST=Washington DC/L=Washington DC/O=Nava PBC/OU=Engineering/CN=Simpler Grants.gov/emailAddress=grantsteam@navapbc.com"

Navigate to the login.gov service provider page and for each application edit it, and upload the public.crt file. Leave any prior cert files alone until we have switched the API to using the new one.

Go to SSM parameter store and change the value that maps to the LOGIN_GOV_CLIENT_ASSERTION_PRIVATE_KEY value for the given environment to be the value from the private.pem key you generated.

After the next deployment in an environment, we should be using the new keys, and can cleanup the old certificate.

Remember to update these docs to mention when the certificates were last rotated and update the keys in our 1password vault. The vault should have an item for each environment named something like Login.gov Certificates - {environment}.

Prod login.gov does not update immediately, and you must request a deployment to get a certificate rotated.

For Prod, assume it will take at least two weeks from creating the certificate, before it is available for the API, and until it is, do not change the API's configured key.

Additionally, we don't currently own the account that the prod login.gov is configured in, and we need to ask MicroHealth to make the updates. Plan to rotate certificates at least a month in advance to be safe.

To sign the JWTs the API uses to track authentication with requests from the FE we need a public/private key. You can see the steps run via the Makefile for the setup-env-overide-file.sh that does this locally.

1. Generate a new public/private key pair
2. Set them in the /api//api-jwt-private-key and /api//api-jwt-public-key parameters in the Parameter Store.
3. Force redeploy the API Service to pick up the new values.

User API keys are stored as an HMAC-SHA256 hash rather than plaintext. The key for that HMAC (the pepper) is held in Parameter Store, outside the database, so that read access to the database alone isn't enough to brute force the stored hashes.

This is the one secret we deliberately never rotate. Changing a pepper invalidates every hash already stored in that environment, and those hashes cannot be re-derived because the plaintext key is no longer kept anywhere. Treat /api/<env>/api-key-pepper as write-once per environment.

To provision it in a new environment:

1. Generate a value: openssl rand -hex 32.
2. Store it as a SecureString: aws ssm put-parameter --name "/api/<env>/api-key-pepper" --type SecureString --value "<value>".
3. Force redeploy the API Service to pick up the value.

Because the parameter is referenced with manage_method = "manual", Terraform reads it through a data source — terraform apply fails outright in any environment where the parameter does not yet exist, so create it before the next API deploy. The ten environments that are stood up today each need one:

infra-sgg2, infra-sgg3, and infra-sgg1 are configured in infra/api/app-config/ but not serving traffic, so they are skipped here. Whoever stands one up provisions the whole secrets block for it, this parameter included.

grantor1 must be set to staging's pepper, not its own. That environment restores its database from the staging snapshot, so a distinct pepper would leave every restored key hash unresolvable. .github/scripts/copy-ssm-params.sh keeps it in sync on each run (api-key-pepper is intentionally not in its SKIP_PARAMS list), but that sync happens after the API deploy in the team-environment flow, so the parameter still has to pre-exist.

Locally and in tests the pepper comes from API_KEY_PEPPER in api/local.env, which is a fixed placeholder, not a real secret.

There are three ways to interact with New Relic: UI, CLI, or API. Most interactions will be done via the UI.

We use the New Relic via means of Terraform. You can via the New Relic Terraform API documentation here. To setup Terraform for New Relic, perform the following steps:

1. Login to New Relic
2. Navigate to the API keys page
3. Create a key of key type "user"
4. Copy the key value
5. Set the key value, region, in your .zshrc .bashrc or similar:

export NEW_RELIC_ACCOUNT_ID=1234 # Found in the URL among other places. eg. https://one.newrelic.com/nr1-core?account=< ACCOUNT ID HERE>
export NEW_RELIC_API_KEY="EXAMPLE"
export NEW_RELIC_REGION="US" # Always "US".

You will then be able to interact with New Relic via Terraform. There's some New Relic Terraform configuration inside of the infra/accounts/ folder for example. From this point you can use normal Terraform CLI commands to interact with New Relic, terraform init terraform apply etc.

cd infra/analytics/metabase

terraform init --reconfigure --backend-config="dev.s3.tfbackend"

terraform apply -var="image_tag=<target_version_number_here>"

Version numbers are prefixed with the letter "v". For example, if we wanted to upgrade to version 1.55.22 the terraform apply command would be:

terraform apply -var="image_tag=v1.55.22"