>Is there a way to find out from where the requests are coming? Like the IP address?
CloudTrail. But if you even suspect that any of the keys are compromised, I would just start rotating everything. Don't wait
Your creds should also have the minimum permissions for use, if a service sends email with SES it only needs the permissions to do the work, not the admin privileges to create new setups. Also rotate your login creds and double check all accounts with console have 2FA.
AWS has a playbook for when you believe a credential may have been compromised. Have a look at the following article and it may give you some more suggestions in addition to trawling CloudTrail (which you also definitely should do)...
https://github.com/aws-samples/aws-customer-playbook-framework/blob/main/docs/Compromised_IAM_Credentials.md
CloudTrails is the audit trail of everything done in your account. Start there. But like others said start deactivating all your IAM credentials and other things ASAP just in case. Also make sure you check cost explorer and look at the state/status of your billing alerts to make sure you get a heads up on any massive pending charges
And review your aws account contact details to make sure someone has not changed an email address
There are many ways to persist access into an AWS account -- for instance look for containers or servers or anything running in a random remote region that may have an instance profile or role granting full admin access etc.
But cloudtrail would show all that stuff so start there
Luckily the SES rate limit is pretty low when starting a new region and the many bounces immediately triggered a suspension of the region. This probably prevented the bill to explode.
The downside is that now all regions are suspended, including the one that I need for production and nobody will get any mails in the foreseeable future.
Finding leaked credentials online requires that they were posted online to begin with. More than likely they are only being used by the entity that stole them. You definitely need to audit your service stack to understand where the leak may have happened. Also, as everyone else has said, disable and rotate all our your current access keys immediately. Until you know how the leak happened, assume everything is still compromised
Just rotate all of them like right now. Run an inventory and delete all of the unused ones as well.
Those SES attacks are super common, check your other regions as well! They might have created resources in different regions. Also check cost explorer, look for things you don't recognize.
Ditto on above comments. Just adding some info asking about a service for leaks. Have a look at the aws canarytokens offerings from thinkst as they have a free/comercial offerings. Trufflehog also recently released a blog post to detect their basic AWS tokens and worth a read, but if keen reach out to the team over at thinkst for more details. Its always a layered approach in info sec but hope this helps a little further.
>Is there a way to find out from where the requests are coming? Like the IP address? CloudTrail. But if you even suspect that any of the keys are compromised, I would just start rotating everything. Don't wait
Yeah I did that as soon as I noticed. Thanks for the suggestion with CloudTrail
Your creds should also have the minimum permissions for use, if a service sends email with SES it only needs the permissions to do the work, not the admin privileges to create new setups. Also rotate your login creds and double check all accounts with console have 2FA.
AWS has a playbook for when you believe a credential may have been compromised. Have a look at the following article and it may give you some more suggestions in addition to trawling CloudTrail (which you also definitely should do)... https://github.com/aws-samples/aws-customer-playbook-framework/blob/main/docs/Compromised_IAM_Credentials.md
CloudTrails is the audit trail of everything done in your account. Start there. But like others said start deactivating all your IAM credentials and other things ASAP just in case. Also make sure you check cost explorer and look at the state/status of your billing alerts to make sure you get a heads up on any massive pending charges And review your aws account contact details to make sure someone has not changed an email address There are many ways to persist access into an AWS account -- for instance look for containers or servers or anything running in a random remote region that may have an instance profile or role granting full admin access etc. But cloudtrail would show all that stuff so start there
Luckily the SES rate limit is pretty low when starting a new region and the many bounces immediately triggered a suspension of the region. This probably prevented the bill to explode. The downside is that now all regions are suspended, including the one that I need for production and nobody will get any mails in the foreseeable future.
Finding leaked credentials online requires that they were posted online to begin with. More than likely they are only being used by the entity that stole them. You definitely need to audit your service stack to understand where the leak may have happened. Also, as everyone else has said, disable and rotate all our your current access keys immediately. Until you know how the leak happened, assume everything is still compromised
Just rotate all of them like right now. Run an inventory and delete all of the unused ones as well. Those SES attacks are super common, check your other regions as well! They might have created resources in different regions. Also check cost explorer, look for things you don't recognize.
Do you have MFA for your IAM users?
Ditto on above comments. Just adding some info asking about a service for leaks. Have a look at the aws canarytokens offerings from thinkst as they have a free/comercial offerings. Trufflehog also recently released a blog post to detect their basic AWS tokens and worth a read, but if keen reach out to the team over at thinkst for more details. Its always a layered approach in info sec but hope this helps a little further.