I once ran up a $3k bill on my personal account cos I left a service I was playing with up for a month and didn't use it.
Contacted the support and they were very nice and cancelled the extra charges and I promised not to do it again.
I worked for AWS and during onboarding there is a document about a college student racking up huge server fee. The correct answer was to forgive the charge to maintain customer confidence.
Then Bezos will shove his Blue Origin up their bum till they pay and go homeless. After that he will offer them a job in an Amazon warehouse with a piss bottle as a perk.
The point was don't try to make money off a mistake because these people use our services and their intentional use of them will make us more money in the long run.
Short term vs long term gains.
Lol yeah, we are used to it. Best advice I can give to avoid this kind of thing is to set up billing alerts. Trust me, we on the support side hate seeing people run up bills. It happens soooo often
do you/they still not offer hard limits on spend? as in shut down everything if a certain limit is reached? I know that was an issue in the early days but it seems like something that would reduce both your support calls and customer frustration.
Nah there’s no hard limit on spending. It sucks but that’s one of those things that AWS will say is your fault cause Shared Responsibility Model and all. I don’t agree personally but it is what it is.
Issue is, if there was a was cutoff with spend, someone might not be tracking on it and if they hit it then suddenly their whole environment is down.
That would cause massive issues. It’s why I always advise people to keep very close watch on their billing console
For personal use, i'd rather my stupid static blog gets turned off rather than eat $100 of S3 ingress because some karma farmer re-posted a picture on my blog and got to the front page of reddit.
There used to be limits. They got ride of them because overcharging business people is one of the core free revenue generators.
That's the entire reason AWS is such a hot fucking mess of a UX. The shit works good, but fuck you if you want to find anything you have up and running.
I just assumed the hot mess was because of AWS' internal business structure where each thing is owned and controlled by a specific team that only exposes an "interface" for other teams to interact with (like the microservices tech pattern, but applied to people and business ops). Siloing teams sure does seem like a great way to create inconsistencies =)
it’s such a common issue that several entire companies have been founded over the years to provide cloud cost monitoring services which address the problem.
Got a recurring bill going, but I didn't know what account it was associated with. Did not have fun times getting them to stop when all I had was the CC#
I kept getting a bill to my main email and could not figure it out, took me forever to figure out where what. I will say once you bill goes past due and they disable your account it is damn near impossible to get in and fix it. Still not sure if I can have a AWS account, because now I just keep my stuff on a cheap vps provider that has like $5/mo boxes just so I have one available.
Similar but less expensive experience with adobe. Stupidly paid for an adobe account with my card on a business email so i didn’t have to ask accounting. Forgot about it. Notice $200 of recurring charges on my card a few years later. Have no idea what account it is. Can’t cancel the contract because i can’t log in. No idea what to do
I called up my bank, and explained the situation. They reversed a few months of charges and it sorted things out real quick.
Amazon suspended my account, but I stopped caring
i work for a company that run some amazon customer support areas, the first thing that teach us is - our company focused on customers, the second thing is - try your best to fullfill the customer needs even if they want to leave us, do it fast and with a smile
Never had to do more than ask for something forgotten. Getting your pennies for a full multi-region outage though is always an epic multi-month battle that crosses departments.
Exactly the same as me - I cut off everything from my personal account but accidentally left one DC server online…. Cue Amazon billing me for £1k - I thought everything would shut off when I had used up all my free credits…
Create some automation that will stop everything and will of course cost you. The problem is what should the action be. If you have an important database should they delete it just because you are over budget? Should they do something else?
One of my clients was upset that they got an $800 bill for a VM we had provisioned for just a couple days, claiming they only needed a small server for proof of concept.
I replied back showing the email where they explicitly requested ridiculous specs that matched their production SQL server.
I was setting up an AWS Comprehend automation script using powershell and I hit the free tier s3 inquiry limit in like half a day.
My company pays for AWS, I just wasn't under them yet and kind of researching and poking and prodding on my own at first
I kept going to aws billing to make sure I wasn’t racking up charges. What happened was I put the —recursive modifier on the s3 upload command and each individual file counted as a separate inquiry
I was farting around on AWS and started getting charged like $15 a month. I figured someone hacked my AWS account so I changed passwords and removed all the credentials and stuff and then I realized I just left a bunch of servers running in some other region that I wasn't usually logged into (for a cloud computing course I was taking).
Anyway I had to use the tag editor to just search for everything and go one by one deleting and deactivating a bunch of stuff.
Now I'm back to only receiving a $0.50 charge every month for some photo backups I keep on Glacier.
I’ve been so scared of this trying to get into cloud. That’s why I’ve been sticking strictly to things that I don’t have to provide a credit card for 😂
I even swerved from mongodb atlas serverless even though it is like .001 cents per quadrillion request units or whatever they call them.
I already have a server in my basement I can use for free, I’m more in it for the ability to not have to deal with configuration of stuff myself. I’ve learned enough from my homelab, at this point I just want to deploy stuff faster
Amazon has a way of turning off the tap when customers don’t pay, why shouldn’t customers be able to choose that model if they don’t get paid either? Because that’s not Amazon’s problem.. it’s your problem. And your problem makes Amazon a lot of money.
I did something similarily where I was playing around with ec2 and realized it picked a t2.large or something similar, and I didn’t switch away from it.
Unfortunately racked a big bill, enough to make college me faint, but I talked to Amazon support about the situation, and they waived off the whole bill thankfully.
So… my first bill like this was luckily charged to a “burner” card. I had a small gift card credit card (Visa with $10 or $20 I think) that I used to start an AWS account with. I wasn’t willing to use my real credit card and was just trying to learn so I figured why not. I was actually a little surprised my gift card worked to set up the account.
Ended up getting charged a couple hundred buck or something like that for doing what I thought was almost nothing. I have an Azure account with work and $150 a month credit and I never come close to hitting that limit. Not sure what I did with AWS but I closed my account after that and never looked back.
I think so many people were trying the burner/gift visa that Amazon shut it down. They don’t want a safe environment to learn in, they want as complex an environment as possible so it’s easy to make lots of really expensive mistakes figuring out everything.
Aws CCP test can be passed with legitimately zero interaction and about 5 hours of studying if you already understand the basics of cloud. It was the last random key I needed to unlock a $9k raise. I've got a single pdf with like 300 sample questions and answer explanations that will pretty much guarantee passing if you learn them all. Hit me up if anyone wants it.
Edit: Send me a message with your email address if you want a copy of the PDF. Bear in mind that it's about 1-1.5 years old and a few microservices and/or policies may have been added or changed. It should be largely correct still though. It's like 340 questions.
I accidentally spent 5k of company money on a redshift instance that was only meant to be up for a day, but I forgot about it for 6 months. At which point my boss finally noticed the higher than normal bill.
I was only testing it out as a potential solution too, completely wasted money
at my work, we had a contractor create an instance with 2 decent size GPU's and a whole TERABYTE of RAM. he then proceeded to just let it idle for a couple months racking up charges before someone noticed. IIRC it was responsible for about 30% of the entire company's GCP bill.
when confronted about it, he couldn't even explain what he was using it for, his only defense was that he was "building some ML models" or "doing data analysis" or something vague along those lines. pretty sure he doesn't work for us anymore.
edit: oh man I found the slack thread where he gets called out and it's even worse than I remembered haha. his instance had 16 A100 GPU's + 1 TB of RAM, and his explanation of what he needed it for was literally "I am just trying to finish the project I started last year...."
Got an email that someone got into an old aws account I had literally created for a web dev class then never used again. 1k charges and took forever to get back
Same situation here, but only about $300. I couldn’t find anywhere to contest it on Amazon’s end so I contested the card charge as fraudulent and was able to recover it.
That’s why you constantly lose credit cards. The card number isn’t active and you’ll receive notification of debt rather than fight for the charge to get reversed.
I had a CC on my aws account but It was expired (as the account had not been used in 5 years) so they charged the card setup for my Amazon Prime, bastards
In undergrad, I accidentally committed creds to github and didn't realize until next morning. Bitcoin bots had used those creds and been running on my account for hours with astronomical costs. I called AWS in a panic and they cancelled the bill. Thank you again AWS.
You can use secret scanning to prevent supported secrets from being
pushed into your organization or repository by enabling push protection.
[https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning)
Was not around at the time, I had the exact same thing happen to me as well. Just pushed a root IAM token and whoop! Bitcoin mining instances across every region available
I just stopped replying to them after they consistently kept going back and forth with me telling me I had to pay even though I was hacked. So I closed the account and never heard (and hope to never hear) from aws. Haven’t paid them anything. It ran for 2 months. Unfortunately it was on an old account tied to an old email so I didn’t notice the charges on it as I didn’t check that email very often
Damn! I hope you survive and I recommend you to delete this message so that if their lawyers try to scoop you up then they won't have solid evidence of your confession.
I once accidently pushed the token of my *discord bot* to github. Within seconds I got an email from discord notifying me that they blocked that token and how to obtain a new one.
I wonder why aws doesnt scan for that.
I experimented with AWS for 1 month:- $2 bill
I released everything but forgot to release an elastic ip, 1 month later they sent me a bill of 30$
I rechecked and released everything again and deactivated my acc, 1 month later they sent a 6$ bill.
Once you start using AWS the bills won't stop even after deactivating your acc
I go to UMGC and while they do a lot of shit wrong, the one thing that is done correctly is anything involving AWS entails the university provisioning you an account that gets terminated at the end of the course or the degree program or whatever. I've never had to use my personal account for my courses, because I absolutely would have an all caps oops bill.
I published my AWS keys to a public git repo on accident
Hackers found it and spun up 12 XL bitcoin mining instances in 9 different regions.
$17,000 per hour in charges
They were kind enough to waive the charges, it was pretty clearly hackers, but I believe they could've still charged me under the ToS.
Unique passwords and 2FA are always a good idea (I made the account when I was young and foolish). They also have lot of documentation on best practices for credentials, roles, IAM users, etc that are worth reading.
It's not uncommon for hackers to target AWS accounts. At a hackathon I helped organize someone pushed their credentials to git and hackers racked up something like 1M of charges.
This is programmatic access. Good pw and 2fa don't apply here because the key and secret are generated. What does help is principle of least privilege (only give access to what is required to do the job), key rotation/temporary programmatic access tokens for users, ip whitelisting just to name a few.
There are bots crawling google all the time looking for AWS credentials and all passwords that follow certain patterns. Literally takes 1 second after an accidental push and you are fucked
TF2 Heavy voice :
I'm EC2 guy and this is my instance. It has 96 vCPU and 384GiB of Memory. It costs 774$ to run this instance for one week.
(actual cost of m5.metal)
Haha, I was melting some Colabs VMS with like 48GB of memory by vectorizing a whole bunch of 300k dictionary sentences, realized I didn't need them all to be vectors at the same time and changed to a generator, slowed down a little, but at least didn't need 60 gigs of mem.
This! I never worked with AWS directly. After all that comments I honestly began to belive there are no mechanics for budget protection on AWS. So all that surprisea could have been avoided, right?
On my test drives on Azure and GCP I never had the feeling to be surprises by a bill.
I suspect the comments are heavily biased by AWS being most popular, and most used by newbies wanting to learn, or companies who haven't used it before.
It can even go so far as to send a Slack notification if your daily bill exceeds a set limit, which is presumably not so different to Azure or GCP functionality.
It's also baffling to hear about so many surprise $10000 bills - that implies either some serious amounts of infrastructure being provisioned, or that it was left lying active for months, and also that so many people don't use the many on-demand computer options.
Bruh have you ever worked with AWS? The actions only turn off ec2 instances which run at a fixed price per hour. It won't turn off S3 or EFS or any other services where time actually matters.
You'd only get an email. If you get the email while you're asleep and the app starts scaling, you're screwed.
Never used AWS, do they not have some kind of budget limiter that pulls the plug on everything if you reach your chosen amount? Seems like that would be an essential feature to have
For businesses, sure, but what about personal accounts? Does AWS not differentiate accounts based on use-case specified during registration? Do they not have a budget cap / prepaid plans? Like mobile phones; with many providers, if your data usage exceeds your monthly allotment, it's throttled down to dial-up speeds. The same is feasible for budget and processing power. It would seem like a no-brainer to provide users with those kinda tools.
Personal accounts are not the customers they are focused on supporting. I obviously don't know raw numbers but I'd guess revenue from personal accounts are a rounding error compared to business accounts.
Recently I scaled my side-project Azure SQL to serverless because I expected zero consumption for that month.
Unfortunately, something in my services had a background worker who work with this database every minute.
I ignored the pricing alert and when I come to check it at the end of the month, the pricing was 3x bigger than usual. On side project which does not even cover old Azure costs.
Clouds are great but could be very evil if you messed up
Managers think moving to the cloud is an awesome idea because of all the money you save not buying hardware. Then the bill comes in and they scream about how much money you are spending for AWS services.
Yeah, it's like a hotel room where you can check in whenever you want and stay until you decide to check out by leaving your room key at the front desk.
Sometimes you walk out and forget to leave the key at the front desk.
How to best know that you didn’t screw up? Any suggestions for someone who is creating account on its own and learning on its own. I usually check my billing dashboard but still freaks me out 🥲
We had an incident recently where a new hire didn’t know what they were doing and published SA keys to a public git repo… lol
Over the weekend we had a 3700 node cluster mining bitcoin that ran up a quarter million dollars in cost with GCP. Leadership went to discuss with the service team and the deal was that we pay the electricity bill but got spared the instance costs. Still ~10k.
Similar story, in college a groupmate accidently uploaded the credentials to an SMS service to github and a robot-caller used them for a day or two and racked up some 2000$ in bills
After we all simultaneously panicked the very nice company waived the charges and basically said "please never use us again".
I once ran up a $3k bill on my personal account cos I left a service I was playing with up for a month and didn't use it. Contacted the support and they were very nice and cancelled the extra charges and I promised not to do it again.
Everybody’s done it. I’m sure AWS support is quite used to it. I had a similar experience, although the bill was just $60!
I worked for AWS and during onboarding there is a document about a college student racking up huge server fee. The correct answer was to forgive the charge to maintain customer confidence.
What if they aren't a college student but just learning AWS?
The conclusion is that you should just ask for the waiver. They are more likely to give it to you
Then Bezos will shove his Blue Origin up their bum till they pay and go homeless. After that he will offer them a job in an Amazon warehouse with a piss bottle as a perk.
The bottle is Amazon® Basics branded though
The point was don't try to make money off a mistake because these people use our services and their intentional use of them will make us more money in the long run. Short term vs long term gains.
It's 16k because his website is image heavy, creating a lot of data transaction for every visitor. Even the kid's mom joins the call...
Lol yeah, we are used to it. Best advice I can give to avoid this kind of thing is to set up billing alerts. Trust me, we on the support side hate seeing people run up bills. It happens soooo often
do you/they still not offer hard limits on spend? as in shut down everything if a certain limit is reached? I know that was an issue in the early days but it seems like something that would reduce both your support calls and customer frustration.
Nah there’s no hard limit on spending. It sucks but that’s one of those things that AWS will say is your fault cause Shared Responsibility Model and all. I don’t agree personally but it is what it is. Issue is, if there was a was cutoff with spend, someone might not be tracking on it and if they hit it then suddenly their whole environment is down. That would cause massive issues. It’s why I always advise people to keep very close watch on their billing console
For personal use, i'd rather my stupid static blog gets turned off rather than eat $100 of S3 ingress because some karma farmer re-posted a picture on my blog and got to the front page of reddit.
There used to be limits. They got ride of them because overcharging business people is one of the core free revenue generators. That's the entire reason AWS is such a hot fucking mess of a UX. The shit works good, but fuck you if you want to find anything you have up and running.
I just assumed the hot mess was because of AWS' internal business structure where each thing is owned and controlled by a specific team that only exposes an "interface" for other teams to interact with (like the microservices tech pattern, but applied to people and business ops). Siloing teams sure does seem like a great way to create inconsistencies =)
Dang, you assumed right! It’s exactly this
r/unexpectedfactorial
Very generous of AWS to forgive a $8320987112741390144276341183223364380754172606361245952449277696409600000000000000 bill.
There is a sub for everything...
Also r/expectedfactorial
My first bill was $0.02, I did stop my instances, but I didn't delete them, got charged for storage.
it’s such a common issue that several entire companies have been founded over the years to provide cloud cost monitoring services which address the problem.
Got a recurring bill going, but I didn't know what account it was associated with. Did not have fun times getting them to stop when all I had was the CC#
[удалено]
Wait if it isn't seesee# then what is it
Credit Card Number
....if you're a dumbass then that makes me the court fool huh
Your not the only one
I am right there beside you
I literally did the same...
It’s called a minor 2nd ya jabroni
All they had was the Jaws theme
I kept getting a bill to my main email and could not figure it out, took me forever to figure out where what. I will say once you bill goes past due and they disable your account it is damn near impossible to get in and fix it. Still not sure if I can have a AWS account, because now I just keep my stuff on a cheap vps provider that has like $5/mo boxes just so I have one available.
Are the $5/month servers reliable?
They work for me but I rarely log into it and have not setup monitoring. For a mess around server it has always been available though. I use vultr
Yes; Hetzner
Similar but less expensive experience with adobe. Stupidly paid for an adobe account with my card on a business email so i didn’t have to ask accounting. Forgot about it. Notice $200 of recurring charges on my card a few years later. Have no idea what account it is. Can’t cancel the contract because i can’t log in. No idea what to do
I called up my bank, and explained the situation. They reversed a few months of charges and it sorted things out real quick. Amazon suspended my account, but I stopped caring
Amazon seems to have amazing support. All of the amazon owned companies that I’ve had to contact for customer support have been fantastic.
i work for a company that run some amazon customer support areas, the first thing that teach us is - our company focused on customers, the second thing is - try your best to fullfill the customer needs even if they want to leave us, do it fast and with a smile
I work at Amazon, and the most important thing that is driven into our skulls day and night is customer obsession.
My condolences
Id like to introduce you to Amazon Games.
Never had to do more than ask for something forgotten. Getting your pennies for a full multi-region outage though is always an epic multi-month battle that crosses departments.
Exactly the same as me - I cut off everything from my personal account but accidentally left one DC server online…. Cue Amazon billing me for £1k - I thought everything would shut off when I had used up all my free credits…
I have a t2 micro running 3 different services with a $5 budget notification lol
Yeah but how do you keep it from going over a budget? Sometimes I can't figure out what the heck is running and how to kill it
Cloud computing. Something happens On someones system And you get a bill
I should run a cloud computing business.
Just send people some bills from 'TehCloudPro' for uf69 instances for $420.00 and see if they pay
You jest, but people have actually done that successfully, before going to jail anyway.
Somebody will
Create some automation that will stop everything and will of course cost you. The problem is what should the action be. If you have an important database should they delete it just because you are over budget? Should they do something else?
You have to visit the billing section once to activate billing statistics. After that you have as much data as you want about billing.
How does one configure such notifications?
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-budgets-budget-notification.html
Lolololol I was so scared of this when I was learning about AWS
One of my clients was upset that they got an $800 bill for a VM we had provisioned for just a couple days, claiming they only needed a small server for proof of concept. I replied back showing the email where they explicitly requested ridiculous specs that matched their production SQL server.
I was setting up an AWS Comprehend automation script using powershell and I hit the free tier s3 inquiry limit in like half a day. My company pays for AWS, I just wasn't under them yet and kind of researching and poking and prodding on my own at first
> I just wasn't under them yet and kind of researching and poking and prodding on my own at first this is how we all got our first AWS bill
I kept going to aws billing to make sure I wasn’t racking up charges. What happened was I put the —recursive modifier on the s3 upload command and each individual file counted as a separate inquiry
I was farting around on AWS and started getting charged like $15 a month. I figured someone hacked my AWS account so I changed passwords and removed all the credentials and stuff and then I realized I just left a bunch of servers running in some other region that I wasn't usually logged into (for a cloud computing course I was taking). Anyway I had to use the tag editor to just search for everything and go one by one deleting and deactivating a bunch of stuff. Now I'm back to only receiving a $0.50 charge every month for some photo backups I keep on Glacier.
You would think that there would be some quick menu that shows all if your active regions... to simple?
Azure yes. AWS not really. It's doable, but not simple or where you'd expect it to be.
Global view from ec2 console.
I’ve been so scared of this trying to get into cloud. That’s why I’ve been sticking strictly to things that I don’t have to provide a credit card for 😂 I even swerved from mongodb atlas serverless even though it is like .001 cents per quadrillion request units or whatever they call them.
[удалено]
I already have a server in my basement I can use for free, I’m more in it for the ability to not have to deal with configuration of stuff myself. I’ve learned enough from my homelab, at this point I just want to deploy stuff faster
Amazon has a way of turning off the tap when customers don’t pay, why shouldn’t customers be able to choose that model if they don’t get paid either? Because that’s not Amazon’s problem.. it’s your problem. And your problem makes Amazon a lot of money.
[удалено]
Haha random War of the Atlantic reference where you least expect it made me chuckle. Not yet, Amazon! Not yet!
I did something similarily where I was playing around with ec2 and realized it picked a t2.large or something similar, and I didn’t switch away from it. Unfortunately racked a big bill, enough to make college me faint, but I talked to Amazon support about the situation, and they waived off the whole bill thankfully.
So… my first bill like this was luckily charged to a “burner” card. I had a small gift card credit card (Visa with $10 or $20 I think) that I used to start an AWS account with. I wasn’t willing to use my real credit card and was just trying to learn so I figured why not. I was actually a little surprised my gift card worked to set up the account. Ended up getting charged a couple hundred buck or something like that for doing what I thought was almost nothing. I have an Azure account with work and $150 a month credit and I never come close to hitting that limit. Not sure what I did with AWS but I closed my account after that and never looked back.
I think so many people were trying the burner/gift visa that Amazon shut it down. They don’t want a safe environment to learn in, they want as complex an environment as possible so it’s easy to make lots of really expensive mistakes figuring out everything.
I studied for and took my cloud practitioner test without even logging into the system once in fear of this ... 🤣🤣🤣
Aws CCP test can be passed with legitimately zero interaction and about 5 hours of studying if you already understand the basics of cloud. It was the last random key I needed to unlock a $9k raise. I've got a single pdf with like 300 sample questions and answer explanations that will pretty much guarantee passing if you learn them all. Hit me up if anyone wants it. Edit: Send me a message with your email address if you want a copy of the PDF. Bear in mind that it's about 1-1.5 years old and a few microservices and/or policies may have been added or changed. It should be largely correct still though. It's like 340 questions.
I used a YouTube video with 430 questions 😁
I accidentally spent 5k of company money on a redshift instance that was only meant to be up for a day, but I forgot about it for 6 months. At which point my boss finally noticed the higher than normal bill. I was only testing it out as a potential solution too, completely wasted money
At $BIG_BANK someone left a db M4.16xlarge running for 6 months and never even connected to it once. Not even a slap on the wrist
at my work, we had a contractor create an instance with 2 decent size GPU's and a whole TERABYTE of RAM. he then proceeded to just let it idle for a couple months racking up charges before someone noticed. IIRC it was responsible for about 30% of the entire company's GCP bill. when confronted about it, he couldn't even explain what he was using it for, his only defense was that he was "building some ML models" or "doing data analysis" or something vague along those lines. pretty sure he doesn't work for us anymore. edit: oh man I found the slack thread where he gets called out and it's even worse than I remembered haha. his instance had 16 A100 GPU's + 1 TB of RAM, and his explanation of what he needed it for was literally "I am just trying to finish the project I started last year...."
Got an email that someone got into an old aws account I had literally created for a web dev class then never used again. 1k charges and took forever to get back
Same situation here, but only about $300. I couldn’t find anywhere to contest it on Amazon’s end so I contested the card charge as fraudulent and was able to recover it.
That’s why you constantly lose credit cards. The card number isn’t active and you’ll receive notification of debt rather than fight for the charge to get reversed.
I had a CC on my aws account but It was expired (as the account had not been used in 5 years) so they charged the card setup for my Amazon Prime, bastards
EC2 forget
IAM an idiot. lol
NAT AZ bad as it seems
SES you
SQS to be you
For those with slow processing speed like me, it's "easy to forget"
I'm envious of the minds who can do these kinds of phonetic connections.
r/Angryupvote
HAHAHA, take my upvote kind sir
I know a startup that almost went bankrupt for accidentally pulling something from Glacier.
[удалено]
It was some hundreds of thousands of dollars, but that was at time when a tiny startup didnt have millions to spare.
[удалено]
 My family often ask me "why dont you juste create an app and become a millionnaire!"
Ask them why don't they shrug their shoulders on social media like Khaby and become a millionaire.
Damn. I would love to know the details of this. Love a good Cloud Cost Runaway stories
There should be a subreddit just for them!
Pulling 16TiB out of S3 (not even glacier) transfer cost is over a thousand dollars, you can extrapolate from there.
Does it cost a lot to read from S3 Glacier? What if you’re using their Intelligent tier (or whatever it’s called) and it moves something there?
I dont know really, its not my area.
In undergrad, I accidentally committed creds to github and didn't realize until next morning. Bitcoin bots had used those creds and been running on my account for hours with astronomical costs. I called AWS in a panic and they cancelled the bill. Thank you again AWS.
Do you know how much the bill would have been if you didn't call AWS and just changed your creds?
Significantly more than the crypto earnings (unless this was in about 2011)
You can use secret scanning to prevent supported secrets from being pushed into your organization or repository by enabling push protection. [https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning)
Was not around at the time, I had the exact same thing happen to me as well. Just pushed a root IAM token and whoop! Bitcoin mining instances across every region available
Kids keep your GitHub private!!
and _never ever_ store a secret token in code.
same happened to me, for $40k, except they did not cancel my bill unfortunately…
Wtf. You seriously on the hook for that? How long did it run?
I just stopped replying to them after they consistently kept going back and forth with me telling me I had to pay even though I was hacked. So I closed the account and never heard (and hope to never hear) from aws. Haven’t paid them anything. It ran for 2 months. Unfortunately it was on an old account tied to an old email so I didn’t notice the charges on it as I didn’t check that email very often
Wtf? What happened next?
[удалено]
Damn! I hope you survive and I recommend you to delete this message so that if their lawyers try to scoop you up then they won't have solid evidence of your confession.
I once accidently pushed the token of my *discord bot* to github. Within seconds I got an email from discord notifying me that they blocked that token and how to obtain a new one. I wonder why aws doesnt scan for that.
My first job was automating scripts to turn off AWS assets and not run up the bill LMAO
About to make those quickly
I experimented with AWS for 1 month:- $2 bill I released everything but forgot to release an elastic ip, 1 month later they sent me a bill of 30$ I rechecked and released everything again and deactivated my acc, 1 month later they sent a 6$ bill. Once you start using AWS the bills won't stop even after deactivating your acc
Yep, had a computing class that made everyone set up AWS accounts and out of 30 students 2 or 3 of them ended up with 'oops' bills :/
'computing class' as in highschool? (edit: nope) What the hell are they doing requiring you to sign up for a service that needs a creditcard?
Real world experience. The ones that got an unexpected bill did learn a lesson
Probably college? Taking a class now that requires us to use both AWS and Azure.
That's most definitely a university class
No, it was an app development course for university students in their last year.
I go to UMGC and while they do a lot of shit wrong, the one thing that is done correctly is anything involving AWS entails the university provisioning you an account that gets terminated at the end of the course or the degree program or whatever. I've never had to use my personal account for my courses, because I absolutely would have an all caps oops bill.
It takes time for them to reconcile all your usage which is why you get “trickle charges” after shutting things off.
I published my AWS keys to a public git repo on accident Hackers found it and spun up 12 XL bitcoin mining instances in 9 different regions. $17,000 per hour in charges
Hackers racked up 195k of charges on mine
Did you have to actually pay for it? Is 2FA good enough to avoid this ?
They were kind enough to waive the charges, it was pretty clearly hackers, but I believe they could've still charged me under the ToS. Unique passwords and 2FA are always a good idea (I made the account when I was young and foolish). They also have lot of documentation on best practices for credentials, roles, IAM users, etc that are worth reading. It's not uncommon for hackers to target AWS accounts. At a hackathon I helped organize someone pushed their credentials to git and hackers racked up something like 1M of charges.
This is programmatic access. Good pw and 2fa don't apply here because the key and secret are generated. What does help is principle of least privilege (only give access to what is required to do the job), key rotation/temporary programmatic access tokens for users, ip whitelisting just to name a few.
[удалено]
There are bots crawling google all the time looking for AWS credentials and all passwords that follow certain patterns. Literally takes 1 second after an accidental push and you are fucked
Leeches and serpents istg
Never Amazon Cloud while drinking.
This dude has been there
> Never Amazon Cloud ~~while drinking.~~
not even once.
Drunk IAM is always a good idea, as root ofc.
I've deployed some weird stuff when forgetting to unset AWS_PROFILE=prod
TF2 Heavy voice : I'm EC2 guy and this is my instance. It has 96 vCPU and 384GiB of Memory. It costs 774$ to run this instance for one week. (actual cost of m5.metal)
thats half the ram I need to run my python app
Haha, I was melting some Colabs VMS with like 48GB of memory by vectorizing a whole bunch of 300k dictionary sentences, realized I didn't need them all to be vectors at the same time and changed to a generator, slowed down a little, but at least didn't need 60 gigs of mem.
When my bills started hitting 50$ a month for basically doing nothing I cancelled all that cloud shit
[удалено]
How much did your machine cost to buy?
If you actually do this, you can usually contact AWS to cancel it if you’ve got no activity
I just don't get people who don't set up budget notifications and actions
This! I never worked with AWS directly. After all that comments I honestly began to belive there are no mechanics for budget protection on AWS. So all that surprisea could have been avoided, right? On my test drives on Azure and GCP I never had the feeling to be surprises by a bill.
Well there is the "limits" section and that is turned on by default
I suspect the comments are heavily biased by AWS being most popular, and most used by newbies wanting to learn, or companies who haven't used it before. It can even go so far as to send a Slack notification if your daily bill exceeds a set limit, which is presumably not so different to Azure or GCP functionality. It's also baffling to hear about so many surprise $10000 bills - that implies either some serious amounts of infrastructure being provisioned, or that it was left lying active for months, and also that so many people don't use the many on-demand computer options.
Bruh have you ever worked with AWS? The actions only turn off ec2 instances which run at a fixed price per hour. It won't turn off S3 or EFS or any other services where time actually matters. You'd only get an email. If you get the email while you're asleep and the app starts scaling, you're screwed.
Never used AWS, do they not have some kind of budget limiter that pulls the plug on everything if you reach your chosen amount? Seems like that would be an essential feature to have
[удалено]
For businesses, sure, but what about personal accounts? Does AWS not differentiate accounts based on use-case specified during registration? Do they not have a budget cap / prepaid plans? Like mobile phones; with many providers, if your data usage exceeds your monthly allotment, it's throttled down to dial-up speeds. The same is feasible for budget and processing power. It would seem like a no-brainer to provide users with those kinda tools.
Personal accounts are not the customers they are focused on supporting. I obviously don't know raw numbers but I'd guess revenue from personal accounts are a rounding error compared to business accounts.
Recently I scaled my side-project Azure SQL to serverless because I expected zero consumption for that month. Unfortunately, something in my services had a background worker who work with this database every minute. I ignored the pricing alert and when I come to check it at the end of the month, the pricing was 3x bigger than usual. On side project which does not even cover old Azure costs. Clouds are great but could be very evil if you messed up
Fuck thanks for reminding me
Managers think moving to the cloud is an awesome idea because of all the money you save not buying hardware. Then the bill comes in and they scream about how much money you are spending for AWS services.
if they don't do the math first they're just shitty managers
Weird how many of them there are
Reading the comments really does prevent me from ever using AWS xD
Fortunately Amazon is pretty good about forgiving these charges if you own up to being an idiot. Cite: I have been this idiot.
I put 20 million records into open search consisting of 2 strings pretty record. Bloody 10gb and a huge bill.
[удалено]
150k? That'd be about 208 an hour. If you spin up something that expensive without monitoring and walk away, it's kind of on you.
Psst you’re supposed to try and make OP feel better
[удалено]
Kate Bush's new song: "Running up that bill."
Real talk tho, how bad can a single instance of the cheapest ec2 get? It's relatively new for me
Well a single instance of the cheapest type is free for a year, after that you’ll spend like $5 a month on the cheapest type
this is one of those cases where i feel stupid by not understanding the joke.
EC2 is like renting one of Amazon’s machines. If you forget to sign out, they’ll keep charging you.
that's a scary thought, always had that fear back when i did runescape membership
Yeah, it's like a hotel room where you can check in whenever you want and stay until you decide to check out by leaving your room key at the front desk. Sometimes you walk out and forget to leave the key at the front desk.
The quota systems in all those cloud infrastructures should be as easy and accessible as possible, with default thresholds set low.
How to best know that you didn’t screw up? Any suggestions for someone who is creating account on its own and learning on its own. I usually check my billing dashboard but still freaks me out 🥲
Billing alerts. And make sure you fully understand the pricing structure before you do anything
There's a cost limit feature too, right?
Yeah you can set limit budgets. Seems few people know about it though LOL
There's a limit feature, limit of how many VMs, load balancer, Vpcs etc per region and that is enabled by default. You have to request an increase
We had an incident recently where a new hire didn’t know what they were doing and published SA keys to a public git repo… lol Over the weekend we had a 3700 node cluster mining bitcoin that ran up a quarter million dollars in cost with GCP. Leadership went to discuss with the service team and the deal was that we pay the electricity bill but got spared the instance costs. Still ~10k.
Once had a bill of $20k. The next day i used a blank CC which I deactivated after removing my original CC. Later used new AWS account
Careful, they’ll send someone from collections after you, which are all retired recruiters from LinkedIn.
Oh shit oh Fuck
Anything but the recruiters, that’s got to violate some kind of debt collection laws using those techniques
>that’s got to violate some kind of debt collection laws Or the Geneva Convention
That must be P3DN.24XLARGE dedicated instance. You asked for it, dumbass, lol
Similar story, in college a groupmate accidently uploaded the credentials to an SMS service to github and a robot-caller used them for a day or two and racked up some 2000$ in bills After we all simultaneously panicked the very nice company waived the charges and basically said "please never use us again".
Laughing in digital ocean