It Happened to Me: How I Suddenly Owed AWS $13,000…
When my Amazon Web Services account was suspended I was very confused. It said my password was invalid but my account was suspended for nonpayment. Having never used AWS for anything other than exploration, I didn’t have anything to pay for. No email in my inbox.
Then I checked my trash folder and there it was. A past due bill for $12,989.60.
The panic set in. What did I do? How did this happen? What did I accidentally click to go from a $0.00 bill to a $12,989.60 overnight? Fraud didn’t even enter my mind. Then calm. There’s no way I can pay this and it’s my first offense so there has to be some way we can work this out. And to the live chat, I went. A support ticket that included the invoice and my journalist credentials, just in case that helped.
If the account was active, I would see what the support agents saw: 14 closed tickets from Nov. 29 for Increased EC2 Spot Instance Service Limits in multiple availability zones worldwide which never made it to my inbox because I was hacked and there was a new email, jlwachtel@thailandoc.com, on my AWS account.
I also didn’t see the fraud alert sent by AWS alerting me of a potential hack on Nov. 30 because it was blocked by Gmail. So I waited for a reply from AWS support completely in the dark.
The first reply from AWS support was the only misstep in the process. I received the following email:
At this moment, I still believed this error was caused by me so I hit verify on the card I had on file. As soon as I did, it was followed by an uneasy feeling which was confirmed when I got a fraud alert text from my bank asking if I approve the charges from Amazon Web Services.
I replied “No” and back to the support console I went.
Luckily I was finally in contact with security and they confirmed my account had been grossly mishandled up until that point. I wasn’t sure what that meant until later when my bank confirmed that the $12,989.60 was attempted on my card four times in a row in about 45 minutes.
After that, it was smooth sailing. The account was restored; I was able to log in and see the fraud for myself. I was in good hands with the security team. They walked me through the account sanitization and security best practices implementation. I combed through the account and deleted the access keys, security groups, key pairs, and launch templates in the many regions they were set up in.
At first, I was mad at AWS. How did they approve the limit increases on a new email address? Why did they let the bill get so high? Why didn’t they turn the account off once they flagged potential fraud? But they did their part and I take responsibility for my part.
It wasn’t hard to see how common AWS fraud is and how it can be avoided to a point. I have since added MFA, Budgets, and CloudTrail. All of those things definitely should have been added on Day One but since I’m being reflective, I will say it didn’t cross my mind. I’ve never been hacked before. I usually follow best practices but since I set that account up so quickly and never went back to it, it didn’t cross my mind until it was too late.
Set up MFA. Set up CloudTrail. Check regularly.
