Sending email with Amazon SES

Send BugSnag On-premise email through Amazon SES using an IAM role, with no stored email credentials.

This guide sets up BugSnag to send its emails (alerts, invites, password resets) through Amazon SES using an IAM role, so no email username or password is stored anywhere. It takes about 30 to 45 minutes of work, plus waiting time while AWS verifies your domain and, for new SES accounts, reviews your request for production access.

How it works

1. BugSnag sends email from a component called sidekiq, which runs in your Kubernetes cluster.
2. EKS Pod Identity gives sidekiq an AWS identity: Role A, the pod identity role.
3. Role A is only allowed to switch to Role B, the SES sending role.
4. Role B is allowed to send email through Amazon SES.

You enter Role B’s ARN in the BugSnag settings. BugSnag does the rest.

sidekiq (BugSnag's email program, in your cluster)
   │  EKS Pod Identity gives it Role A (steps 3, 4, 7)
   ▼
Role A: BugSnagSidekiq — pod identity role, can only switch to Role B
   │  switches to Role B (steps 5, 6)
   ▼
Role B: BugSnagSESSender — SES sending role; its ARN is entered in BugSnag (step 8)
   │  sends the email from your verified domain (step 1)
   ▼
Amazon SES — delivers the email to your team's inboxes

Terms used in this guide

Before you start

Collect these six values first. Throughout this guide, replace each placeholder with your value.

AWS permissions you need: a user that can manage IAM roles, EKS clusters and SES. An AWS administrator has all of these.

This guide assumes your EKS cluster and your SES sending domain are in the same AWS account. If they are not, see Using an existing SES account.

New installation or existing one?

• New installation: do steps 1 to 7 in AWS before installing BugSnag. The association you create in step 7 works even though BugSnag is not installed yet, as long as you use the namespace you will choose during install. Then install BugSnag and enter the settings from step 8 on the config screen. No restart is needed.
• Existing installation: do steps 1 to 8, then restart the email program as shown at the end of step 8.

Wait until your domain shows Verified (step 1) before you install or switch over. Otherwise the first emails, such as the admin invite, fail and are retried until SES accepts them.

Step 1: Verify your sending domain in Amazon SES

This proves to AWS that you own the domain, so SES will send email from it. You need a verified sending domain even if your SES account already has production access.

1. Open the AWS Console and switch to <REGION> (top right).
2. Go to Amazon Simple Email Service (SES).
3. In the left menu, choose Configuration → Identities. If your domain is already listed as Verified with DKIM Successful, skip to step 2.
4. Otherwise click Create identity, select Domain and enter <DOMAIN>.
5. Leave Easy DKIM selected and click Create identity.
6. SES shows three CNAME records. Add them to your domain’s DNS. Your DNS or IT team can do this for you.
7. Wait until the identity’s status shows Verified. This usually takes minutes, but can take up to 72 hours.

Warning: use your own domain with DKIM. Many organizations publish a DMARC policy that tells mail servers to reject email that is not properly signed for their domain. SES still accepts these messages, so the only sign of a problem is a bounce from MAILER-DAEMON@amazonses.com. Sending from a domain verified with Easy DKIM in this step avoids this. Do not use an address on a domain you have not verified here.

Step 2: Allow SES to email anyone

New SES accounts start in a sandbox. In the sandbox, SES only delivers to addresses you have verified one by one, and sends at most 200 emails in 24 hours. Production access is granted separately for each region.

Check your account in <REGION>: in SES, open Account dashboard from the left menu.

• No sandbox banner: your account already has production access in this region. Skip to step 3.
• Sandbox banner shown: request production access as below before going live.

1. Click Request production access.
2. Fill in the form. For example: “Transactional notifications (error alerts, account invites, password resets) from our self-hosted BugSnag installation to our own staff.”
3. Submit the form. AWS usually replies within 24 hours.

You can carry on with the next steps while you wait.

Note: testing before production access. While the account is in the sandbox, you can still test by verifying the addresses you will send to. In Configuration → Identities, click Create identity, choose Email address and enter the recipient’s address. The recipient must click the link in the verification email from AWS, and the address must also be added to Role B’s permissions policy (see step 5). Emails to anyone else fail with Email address is not verified until production access is granted.

Step 3: Turn on EKS Pod Identity for your cluster

EKS Pod Identity needs a small helper, the Pod Identity Agent, installed in the cluster.

1. Go to Elastic Kubernetes Service → Clusters and click <CLUSTER_NAME>.
2. Open the Add-ons tab.
3. If Amazon EKS Pod Identity Agent is already listed, skip to step 4.
4. Otherwise click Get more add-ons, tick Amazon EKS Pod Identity Agent and click Next.
5. Keep the default settings, click Next, then click Create.
6. Wait until its status shows Active.

Step 4: Create Role A, the pod identity role

This is the role BugSnag’s sidekiq program receives. Create it first, because Role B has to name it.

1. Go to IAM → Roles and click Create role.
2. Set Trusted entity type to AWS service.
3. Under Service or use case, choose EKS, then choose the use case EKS - Pod Identity. Click Next.
4. Do not add any permissions on this page. Click Next.
5. Set Role name to BugSnagSidekiq.
6. Check that the Trust policy preview matches the policy below. It lets EKS Pod Identity use the role.
7. Click Create role.

Expected trust policy for Role A:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}

Role A gets its one permission in step 6, once Role B exists.

Step 5: Create Role B, the SES sending role

This role is allowed to send email. BugSnag switches to it from Role A.

1. In IAM → Roles, click Create role.
2. Set Trusted entity type to Custom trust policy.
3. Delete the text in the editor and paste the trust policy for Role B (below), replacing <ACCOUNT_ID>.
4. Click Next. Do not add any permissions on this page. Click Next.
5. Set Role name to BugSnagSESSender and click Create role.
6. Open the new BugSnagSESSender role. Click Add permissions → Create inline policy, then switch to the JSON tab.
7. Delete the text in the editor and paste the permissions policy for Role B (below), replacing <REGION>, <ACCOUNT_ID> and <DOMAIN>.
8. Click Next, name the policy SendEmailViaSES and click Create policy.
9. At the top of the role page, copy the ARN (arn:aws:iam::<ACCOUNT_ID>:role/BugSnagSESSender). You will enter it in BugSnag in step 8.

Trust policy for Role B. It allows only Role A to switch to this role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/BugSnagSidekiq" },
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}

Permissions policy for Role B. It allows sending only from your verified domain. BugSnag needs both actions:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["ses:SendEmail", "ses:SendRawEmail"],
    "Resource": ["arn:aws:ses:<REGION>:<ACCOUNT_ID>:identity/<DOMAIN>"]
  }]
}

If you plan to use an SES configuration set (optional, step 8), add its ARN to the Resource list: "arn:aws:ses:<REGION>:<ACCOUNT_ID>:configuration-set/<CONFIGURATION_SET>".

Warning: while your account is in the sandbox. In the sandbox, SES also checks Role B’s permission against each recipient, because recipients are verified identities too. Add the ARN of every address you verified for testing to the Resource list: "arn:aws:ses:<REGION>:<ACCOUNT_ID>:identity/<RECIPIENT_ADDRESS>". Otherwise every email fails with AccessDenied naming the recipient. You can remove these entries once production access is granted.

Step 6: Allow Role A to switch to Role B

1. Go to IAM → Roles and open BugSnagSidekiq (Role A).
2. Click Add permissions → Create inline policy and switch to the JSON tab.
3. Delete the text in the editor and paste the policy below, replacing <ACCOUNT_ID>.
4. Click Next, name the policy AssumeBugSnagSESSender and click Create policy.

Permissions policy for Role A:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["sts:AssumeRole", "sts:TagSession"],
    "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/BugSnagSESSender"
  }]
}

Step 7: Attach Role A to BugSnag’s email program

This step tells EKS: whenever the sidekiq program runs in the BugSnag namespace, give it Role A.

1. Go to Elastic Kubernetes Service → Clusters and click <CLUSTER_NAME>.
2. Open the Access tab. Under Pod Identity associations, click Create.
3. Set IAM role to BugSnagSidekiq.
4. Set Kubernetes namespace to <NAMESPACE>.
5. Set Kubernetes service account to sidekiq, typed exactly like this, in lowercase.
6. Click Create.

This works even if BugSnag is not installed yet. EKS applies the role to any program that later starts as sidekiq in that namespace. The namespace must match the one BugSnag is, or will be, installed into.

Step 8: Configure BugSnag

Open the BugSnag admin console and go to Config → Email. For a new installation, this is on the config screen during install. Fill in the settings below, click Save config, then Deploy.

Also check that Config → Service access → Dashboard starts with https://. BugSnag uses this address for the links in its emails.

Existing installations only: a program only picks up its AWS identity when it starts. If BugSnag was already running before step 7, restart the email program:

kubectl rollout restart deployment/sidekiq -n <NAMESPACE>

Step 9: Check that email works

1. In BugSnag, invite a user or use Forgot password with your own address. Check that the email arrives. If your account is still in the SES sandbox, use an address you verified in step 2.
2. If it does not arrive, check that the email program has an AWS identity. The first command below should print a web address.
3. Then look for errors in the email program’s logs with the second command below, and match the error against the troubleshooting table.

kubectl exec -n <NAMESPACE> deploy/sidekiq -c sidekiq -- printenv AWS_CONTAINER_CREDENTIALS_FULL_URI

kubectl logs -n <NAMESPACE> deploy/sidekiq -c sidekiq | grep -i -E "ses|sts|mail|denied"

BugSnag retries emails that fail. Once you fix the cause, waiting emails are sent automatically, so there is no need to invite users again.

Troubleshooting

Using an existing SES account

If your organization already sends email with SES, you can reuse that account. If it has production access in your SES region and your domain is verified with DKIM there, you only need steps 3 to 9. Keep in mind:

• Production access is per region. The Amazon SES region in BugSnag must be a region where the account has production access. Any other region is still in the sandbox.
• Sending limits are shared. BugSnag email counts toward the account’s daily sending quota and maximum send rate, alongside your other applications.
• Reputation is shared. Bounces and complaints from BugSnag email affect the account’s sending reputation. Use an SES configuration set to track BugSnag email separately and to receive bounce and complaint notifications.
• Domain in a different AWS account. If your verified domain is in a different account from your EKS cluster, create Role B in the account that owns the domain and Role A in the cluster’s account. In Role B’s trust policy and Role A’s permissions policy, use each role’s own account ID.

Good to know

• No passwords are stored. AWS gives BugSnag short-lived credentials that renew automatically.
• Grafana alert emails stop. BugSnag’s built-in Grafana dashboards can only send email over SMTP, so their email alerts are turned off in SES mode. BugSnag’s own emails are not affected.
• Failed emails are retried. If an email fails because of a setup problem, BugSnag retries it automatically once the problem is fixed.
• Switching back: set Email delivery method to SMTP, enter your SMTP details and deploy. The IAM roles and pod identity association are then unused and can be removed.

Appendix: the same setup using the AWS CLI

For administrators who prefer the command line, this covers steps 3 to 7. Replace the placeholders in the first five lines with your values, then run the rest. Steps 1 and 2 involve DNS changes and an AWS review, so do them in the console.

ACCOUNT_ID="<ACCOUNT_ID>"
REGION="<REGION>"
CLUSTER="<CLUSTER_NAME>"
NAMESPACE="<NAMESPACE>"
DOMAIN="<DOMAIN>"

# Step 3: Pod Identity Agent
aws eks create-addon --cluster-name $CLUSTER --addon-name eks-pod-identity-agent --region $REGION

# Step 4: Role A (pod identity role)
aws iam create-role --role-name BugSnagSidekiq --assume-role-policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Principal":{"Service":"pods.eks.amazonaws.com"},
  "Action":["sts:AssumeRole","sts:TagSession"]}]}'

# Step 5: Role B (SES sending role)
aws iam create-role --role-name BugSnagSESSender --assume-role-policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Principal":{"AWS":"arn:aws:iam::'$ACCOUNT_ID':role/BugSnagSidekiq"},
  "Action":["sts:AssumeRole","sts:TagSession"]}]}'
aws iam put-role-policy --role-name BugSnagSESSender --policy-name SendEmailViaSES --policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Action":["ses:SendEmail","ses:SendRawEmail"],
  "Resource":["arn:aws:ses:'$REGION':'$ACCOUNT_ID':identity/'$DOMAIN'"]}]}'

# Step 6: allow Role A to switch to Role B
aws iam put-role-policy --role-name BugSnagSidekiq --policy-name AssumeBugSnagSESSender --policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Action":["sts:AssumeRole","sts:TagSession"],
  "Resource":"arn:aws:iam::'$ACCOUNT_ID':role/BugSnagSESSender"}]}'

# Step 7: attach Role A to the sidekiq service account
aws eks create-pod-identity-association --region $REGION \
  --cluster-name $CLUSTER --namespace $NAMESPACE --service-account sidekiq \
  --role-arn arn:aws:iam::$ACCOUNT_ID:role/BugSnagSidekiq

# The ARN to enter in BugSnag (step 8)
echo arn:aws:iam::$ACCOUNT_ID:role/BugSnagSESSender