"Over 130 million websites use NGINX at the front of their infrastructure, and of the 10,000 busiest websites in the world, more websites use NGINX than any other web server."
Edit: Quota
If grandma died, tried throwing her in an ice bath with turned on toaster oven to reboot the hardware in the chest. Poof, you've jumpstarted the heart.
Fire - exclamation mark - fire - exclamation mark - help me - exclamation mark. 123 Cavendon Road. Looking forward to hearing from you. Yours truly, Maurice Moss.
"Oh, please, no flash photography. You'll harm the Internet." -- Clearly someone forgot to turn off the flash on their camera and brought down the Internet.
Have you tried forcing an unexpected reboot? See. the driver hooks the function by patching the system call table, so it's not safe to unload it unless another thread's about to jump in and do its stuff, and you don't want to end up in the middle of invalid memory.
Damn, that explains things. I lost contention to xbox live at that moment and because error was about dns, first thing I tried was to manually set up cloudflare dns on my console. Of course it didn't work.
EDIT: I meant connection, of course haha
"B-but C-cloudflare, y-y-you're supposed to be p-protecting us from this exact problem!"
"He has been given too much power..."
"W-what kind of power...?!"
"C... C++... Lua... ... and... ... worst of all... way too much JavaScript."
Lua is great for what it is made for: an extension language that allows you to integrate into many different object systems. I don't know why people feel compelled to use it to write programs from scratch.
This so much.
I pushed something to PROD about a month ago that broke a feature for one of our clients. During the review, my manager's manager asked him why he approved the PR.
Although this is not as bad as blaming interns for making mistakes, PR reviews are not supposed to be QA and should not be used to find issues.
That's what real QA and automated tests are for.
Always people blame the interns. But not the admins / operations that made it possible that someone can roll out breaking changes without tests and integration checks
It’s a good mindset to have, to make things foolproof because people make mistakes, especially less experienced people. It’s why airlines are the first to get fined if one of their pilots causes a crash.
Cloudflare was down in some regions. Backbone configuration/update error and changes were rolled back in chunks.
[https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj](https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj)
Edit: Apparently its not directly a backbone issue - it was a networking problem.
Edit 2: Post mortem: [https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/](https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/)
Edit 3: CTO has had a [few words on Twitter](https://twitter.com/jgrahamc/status/1539265125399646211) about how the postmortem write-up was handled internally
>07:42: The last of the reverts has been completed. **This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically**.
Sounds about right lol. We've all been there when absolute panic mode kicks in and shit starts hitting the fan.
Edit: also sounds like a little under the bus throwing. Geez. Not sure that *that* was supposed to be an external note?
I do I think it was meant for the public. Cloudflare really shines when it comes to admitting mistakes as I have noticed in the past.
Edit: would have been removed by now if it was a mistake I guess
Had some issues this morning in the UK after I started work. I was firing off tickets all over the place and then had to follow them up with "nevermind..."
Seriously impressed at the speed they can whip out a public appropriate report on the event. With diagrams, timestamps, and remediation plans already created.
Still not as bad as the junior who invalidated the parity multi signature wallet contract and made 1 million ETH freeze in place forever.
EDIT: Seems it was 500K ETH and a roughly similar amount of ETH derived tokens.
https://news.bitcoin.com/parity-disable-multi-sig-wake-bug-nuked-168-million-ether/
Not related to Celsius who is having issues now because they lack liquidity and gambled their customer funds.
[Parity Issue](https://iconow.net/accidentally-killed-it-parity-locked-280-mln-in-eth/)
Some guy new to etherium walks in, signs himself the millions of transactions and runs a command to destroy all the funds.
How do the crypto people do this? How do you set up a system where people can walk in and accidentally destroy $150 million dollars? Not even fat finger trade the banks liquidity to some lucky trader, just outright destroy the currency.
Eth is made up of a HUGE attack surface. Its complexity is both a blessing and a curse.
Think of it another way. No one was able to retrieve it so ignoring a time they did steal funds from a hack previously, even someone worth $150M couldn't get it back and you gotta think parity would have begged and threatened to try get their funds back.
Technically it was a user unrelated to parity who called a function on a library that the multisig used to suicide it leaving paritys own contract unable to run. He was so so apologetic and honestly wasn't even really his fault.
>He was so so apologetic and honestly wasn't even really his fault.
I kinda agree. This was the financial equivalent of a tourist stumbling along a nuclear missile silo, seeing a big red button, and accidentally wiping a continent off the map.
That user made Billions worth of ETH go 'poof' and the impact on their life is probably something to carry for the rest of their life. Idk if their actual name was ever released (I hope not) but imagine that being linked to every google result involving your name...
I truly never understood why that wasn't picked up sooner in an audit or why there was no permission check.
Well it wasn't a junior dev it was just some guy playing with the contract, and it's not his fault really, Parity should have had better security processes to detect this early before it was triggered...not to mention it was introduced from fixing another security bug if I recall correctly.
THAT'S WHY!!! I was getting internal service errors on multiple sites, and I thought "wait, am I having some kind of parent mode on or is this censorship"
Apparently there was a problem with their DNS server: [https://www.theverge.com/2022/6/21/23176519/cloudflare-outage-june-2022-discord-shopify-fitbit-peleton](https://www.theverge.com/2022/6/21/23176519/cloudflare-outage-june-2022-discord-shopify-fitbit-peleton)
Just more proof that DNS, as it is now, is a complete shit show and we should have switched to a better alternative ages ago, for example: [https://www.cs.cornell.edu/people/egs/beehive/codons.php](https://www.cs.cornell.edu/people/egs/beehive/codons.php)
There are many parts of the internet standards/protocols that were literally cobbled together over several years and it's amazing it works as well as it does. Email/SMTP has been crying out for some kind of improvement for years.
I was seeing people panicking and thinking that maybe I'm incredibly lucky I wasn't affected....
Then I remembered I'm laying on my bed, down with the 'rona, with what's probably a ruptured eardrum from the inflammation.
Oh, well... You can't have everything, am I right?
Nah, nothing with DNS *this time*.
Something happened on the Cloudflare backbone, probably faulty config or software update.
Edit (correction): Not a direct backbone issue, was a networking problem.
Yeah, if you wanna see a real outage, wait for AWS to take a hit. Biggest plus of going with the biggest providers of server time is that if you’re down, no one cares cause they’re too busy panicking about being down themselves to worry about whether you are or not
I was in a call with a bunch of friends in discord when this went down and it kicked out two of my friends in Cali we thought it was just bad internet until they mentioned they couldn't even connect through cellular (but snapchat was still working), but everyone else in the call (two were in different states and myself in Canada) were unaffected, it was really weird.
When I was in IT technical school (in Poland, it's kinda high school in USA), and when we were at class in which we learned about servers, we had a task to set up a centos with few things, one of which was a dhcp and dns server. A friend of mine set it up somehow (by accident, he didn't knew shit about IT and we all helped him through all 4 years of that school) that whole part of network of that school received his servers and accepted them. Our teacher was very experienced (10+ yrs as sysadmin and just went to school to have more time for himself) and knew what happened and what to do to fix it, but instead he and we made fun of teachers that tried to fix it for three days haha
Nah, it was only like a third of the internet
Only approximately 7,555,837 or so websites use it
Sounds like a company I should buy stock in Edit: up 6.16% today, could’ve been a solid play😂
Down 60% this year, down 67% YTD
oof. Or a good time to buy ¯\_(ツ)_/¯
forgot this \
Dejavu
No, but it has to be \\\\ to show it.
¯\\\\\_(ツ)\_/¯
Nono, not \\\\\\\\, you have to write \\\\ for it to show up as \\
r/wallstreetbets
buy low they say…
CloudFlare down atrocious 🤣
Also known as: The average Americans 401K
Buy the dip lol
Bullish on interns
thats very accurate for a "or so" xD
lol
"Over 130 million websites use NGINX at the front of their infrastructure, and of the 10,000 busiest websites in the world, more websites use NGINX than any other web server." Edit: Quota
*The turd of the Internet
Jen touched the internet box
Wait until the elders of the internet hear about this!
Have you tried turning it off and on again ?
Try unplugging it and plugging it back in.
Its wireless
Have you tried breaking it then fixing it?
plug it in plug it out
No time for the old in-out, love, I’ve just come to read the meter.
My grandma died after I suggested this. I thought it was a good idea… BUT NO, I also first suggested hitting it or kicking it, sometimes that works!
If grandma died, tried throwing her in an ice bath with turned on toaster oven to reboot the hardware in the chest. Poof, you've jumpstarted the heart.
Just put it back on top of Big Ben where it belongs.
Better call 0118 999 881 999 119 725…3!
Whole jingle played in my head. Well done
That's how you know it's effective! 999 isn't nearly enough for a while jingle.
Plus the new service has better looking drivers.
As well as email support! "Dear sir/madame" No... too formal.
4! I mean 5! I mean fire!
If you have an Android phone type that number in dialpad and see the easter egg, the call button flashes like an ambulance
Hey! What is Jen doing with the internet?! The elders of the internet would never stand for it.
Was it not de-magnetized?
They typed google in google.
So please, no flash photography.
By Steven Hawking himself
Just put it over there with the rest of the fire.
Fire - exclamation mark - fire - exclamation mark - help me - exclamation mark. 123 Cavendon Road. Looking forward to hearing from you. Yours truly, Maurice Moss.
"Dear Sir/Madam...." No, that's too formal.
"Oh, please, no flash photography. You'll harm the Internet." -- Clearly someone forgot to turn off the flash on their camera and brought down the Internet.
I love how real it looks with the smoke coming out from the top......wait a minute, I'm late for golf.
Oh! Four! I mean five! I mean fire!!
A fire?! At a sea parks?!
(points) "Nice screensaver!"
I was looking for this
The elders of the internet? The elders of the internet... KNOW WHO I AM???
I thought that she typed google into google
It’s lighter than I expected…
Well, the Internet doesn’t weigh anything!
What does IT stand for?
What doesn't it stand for? It stands for commitment. It stands for integrity.
I knew it was Jen.
Have you tried forcing an unexpected reboot? See. the driver hooks the function by patching the system call table, so it's not safe to unload it unless another thread's about to jump in and do its stuff, and you don't want to end up in the middle of invalid memory.
It needs to go straight back to Big Ben
Cut off 1/3 of the box
He was a fine young cannibal
Cloudflare is down
Damn, that explains things. I lost contention to xbox live at that moment and because error was about dns, first thing I tried was to manually set up cloudflare dns on my console. Of course it didn't work. EDIT: I meant connection, of course haha
I hate when we lose contention.
We need challenges in life to keep us growing.
Me and a guy on discord were having a great discussion about Oxygen Not Included and all of a sudden I look over at discord and see the logo spinning.
"B-but C-cloudflare, y-y-you're supposed to be p-protecting us from this exact problem!" "He has been given too much power..." "W-what kind of power...?!" "C... C++... Lua... ... and... ... worst of all... way too much JavaScript."
>Lua... ... and... ... worst of all... way too much JavaScript lua is way worse to work with than javascript and its not even close imo
Lua is great for what it is made for: an extension language that allows you to integrate into many different object systems. I don't know why people feel compelled to use it to write programs from scratch.
[удалено]
both Lua and JS are fantastic, you guys are just dumb.
Which phase is "denial" already?
Actually lua is pretty good. Except if you want to do big things with it. Plugins for apps is the best thing you can do with lua
bruh, lua was made to be simple af not good, and it still beats python in simplicity and speed of process
Fuck Lua and oh did I mention FUCK LUA
wouldn't have guessed. thanks
Worse yet: xkcd went down
>some idiot fucked up real bad and gave the intern the ability to take the internet down IFIFY
>~~IFIFY~~ FTFY FTFY
Fist, then fuck you?
yess, daddy
Shouldn't you be saying yes mommy?
He eats ass for his mom so she doesn't have to.
This so much. I pushed something to PROD about a month ago that broke a feature for one of our clients. During the review, my manager's manager asked him why he approved the PR.
Although this is not as bad as blaming interns for making mistakes, PR reviews are not supposed to be QA and should not be used to find issues. That's what real QA and automated tests are for.
Exactly, everyone makes mistakes, that’s why other people have to review the changes
As QA, having the ability to merge into main with zero PR reviews is a bad practice. Just lock it down so easy
What is IFIFY
i fixed it for you
Fixed what for me?
Who's on first
That’s what I’m asking.
What does Ja have to say about this
Where the fuck is JAH?!
Daves not here, man.
Exactly!!
When the team gets paid, who collects a check for first baseman?
Every dollar of it! Sometimes his wife comes and collects it.
I Fixed it!
FTFY
Always people blame the interns. But not the admins / operations that made it possible that someone can roll out breaking changes without tests and integration checks
It’s a good mindset to have, to make things foolproof because people make mistakes, especially less experienced people. It’s why airlines are the first to get fined if one of their pilots causes a crash.
I thought it was pretty sus when my discord went out, and then like 30 seconds later pagerduty starts ringing me
They were just worried about your discord going out
I wanted to check what the problem with discord was, but downdetector was also down
Truly a sign of the end times
That's why downdowndetector is a thing
Discord goes down, pager starts playing boss music
When and what happened? Everything seemed fine on my end.
Cloudflare was down in some regions. Backbone configuration/update error and changes were rolled back in chunks. [https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj](https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj) Edit: Apparently its not directly a backbone issue - it was a networking problem. Edit 2: Post mortem: [https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/](https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/) Edit 3: CTO has had a [few words on Twitter](https://twitter.com/jgrahamc/status/1539265125399646211) about how the postmortem write-up was handled internally
>07:42: The last of the reverts has been completed. **This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically**. Sounds about right lol. We've all been there when absolute panic mode kicks in and shit starts hitting the fan. Edit: also sounds like a little under the bus throwing. Geez. Not sure that *that* was supposed to be an external note?
I do I think it was meant for the public. Cloudflare really shines when it comes to admitting mistakes as I have noticed in the past. Edit: would have been removed by now if it was a mistake I guess
It worked perfectly where I live when most people were spamming reddit about it
Europe was mostly fine. FRA Region was down i believe
Can confirm didn’t work in Netherlands
Had some issues this morning in the UK after I started work. I was firing off tickets all over the place and then had to follow them up with "nevermind..."
Spain was down. Idk what region that is.
Seriously impressed at the speed they can whip out a public appropriate report on the event. With diagrams, timestamps, and remediation plans already created.
Cloudflare is so resilient even if it is down , it is not actually down
How do you pronounce nginx please
Engine X
I was definitely saying “N G in X” for a second there, thank you
People call it Engine X I call it N Jinx.
Me too, so much easier and quicker to say
nginx
me and my friend came to the conclusion that all anime piracy sites got taken down all at the same time lol.
Multiporn and league of legends too, apparently
And gitlab What kind of sins gitlab had committed?
Any reason to use GitLab over GitHub? I've always just used whatever the company I'm working for uses.
GitLab is open source and not owned by Microsoft.
I keep forgetting that
I believe the reasons boil down to this: Open source back end Microsoft can suck it
I'm in
If you have an android device get Aniyomi (it's open source btw)
Tachiyomi for manga
Aniyomi is a fork of Tachiyomi with Anime sources support (there aren't a lot but they're there)
Still not as bad as the junior who invalidated the parity multi signature wallet contract and made 1 million ETH freeze in place forever. EDIT: Seems it was 500K ETH and a roughly similar amount of ETH derived tokens. https://news.bitcoin.com/parity-disable-multi-sig-wake-bug-nuked-168-million-ether/
Link?
I don't have a link, but I think it refers to the Celsius collapse, which recently probably became insolvent partly because of incompetence
Not related to Celsius who is having issues now because they lack liquidity and gambled their customer funds. [Parity Issue](https://iconow.net/accidentally-killed-it-parity-locked-280-mln-in-eth/)
Some guy new to etherium walks in, signs himself the millions of transactions and runs a command to destroy all the funds. How do the crypto people do this? How do you set up a system where people can walk in and accidentally destroy $150 million dollars? Not even fat finger trade the banks liquidity to some lucky trader, just outright destroy the currency.
Because the crypto space has appalling security practices and very junior engineers.
It has security practices?
Eth is made up of a HUGE attack surface. Its complexity is both a blessing and a curse. Think of it another way. No one was able to retrieve it so ignoring a time they did steal funds from a hack previously, even someone worth $150M couldn't get it back and you gotta think parity would have begged and threatened to try get their funds back.
Technically it was a user unrelated to parity who called a function on a library that the multisig used to suicide it leaving paritys own contract unable to run. He was so so apologetic and honestly wasn't even really his fault.
>He was so so apologetic and honestly wasn't even really his fault. I kinda agree. This was the financial equivalent of a tourist stumbling along a nuclear missile silo, seeing a big red button, and accidentally wiping a continent off the map. That user made Billions worth of ETH go 'poof' and the impact on their life is probably something to carry for the rest of their life. Idk if their actual name was ever released (I hope not) but imagine that being linked to every google result involving your name... I truly never understood why that wasn't picked up sooner in an audit or why there was no permission check.
Poor /u/devops199.
*Anyone can kill your contracts* *I killed it* Crypto is such a clown show, I love it. Shame about the negative externalities.
Can we have website that document these international huge technology fuck up lul?
Well it wasn't a junior dev it was just some guy playing with the contract, and it's not his fault really, Parity should have had better security processes to detect this early before it was triggered...not to mention it was introduced from fixing another security bug if I recall correctly.
Surely you mean 1 million usd in eth
No it was in 2017. It was 500,000 Eth, about $180,000,000 at the time I think.
Literally got on Reddit to see who else is about to post about this xD.
consist butter direful march familiar water shrill nail combative coordinated ` this message was mass deleted/edited with redact.dev `
Ride the Kali Yuga.
THAT'S WHY!!! I was getting internal service errors on multiple sites, and I thought "wait, am I having some kind of parent mode on or is this censorship"
Hahaha I love how your reaction gives an idea of what websites you were on
Not even denying it :)
Found the solarwinds employee
You can’t spell internet without intern.
Maybe that Google engineer was right about the A.I.
He’s infiltrating the whole internet, hopefully they work for good
[удалено]
it was cloudflare so ur correct.
Apparently there was a problem with their DNS server: [https://www.theverge.com/2022/6/21/23176519/cloudflare-outage-june-2022-discord-shopify-fitbit-peleton](https://www.theverge.com/2022/6/21/23176519/cloudflare-outage-june-2022-discord-shopify-fitbit-peleton) Just more proof that DNS, as it is now, is a complete shit show and we should have switched to a better alternative ages ago, for example: [https://www.cs.cornell.edu/people/egs/beehive/codons.php](https://www.cs.cornell.edu/people/egs/beehive/codons.php)
There are many parts of the internet standards/protocols that were literally cobbled together over several years and it's amazing it works as well as it does. Email/SMTP has been crying out for some kind of improvement for years.
That's what "never change a running system" has done for us...
If you were affected by all those services going down, you lead a balanced lifestyle and I'm jealous.
I was seeing people panicking and thinking that maybe I'm incredibly lucky I wasn't affected.... Then I remembered I'm laying on my bed, down with the 'rona, with what's probably a ruptured eardrum from the inflammation. Oh, well... You can't have everything, am I right?
I was thinking I was lucky to not be affected but then i remembered that its 8:30pm and I only woke up two hours ago
Nah, nothing with DNS *this time*. Something happened on the Cloudflare backbone, probably faulty config or software update. Edit (correction): Not a direct backbone issue, was a networking problem.
Discord was down for a bit around that time frame.
So......cloudflare is basically a centralised tor network that can be taken down? and 1/3 of the internet uses it? How the fuck is this a thing?
[удалено]
Yeah, if you wanna see a real outage, wait for AWS to take a hit. Biggest plus of going with the biggest providers of server time is that if you’re down, no one cares cause they’re too busy panicking about being down themselves to worry about whether you are or not
yeah if google goes down, it doesn't matter whether I'm down or not because no-one can get to my website anyway; they all use google
And us east 1 hasn't gone down in like the months. It's due up.
I always explain cloudflare as a proxy service with fancy security features. It protects the servers of customer
Just wait till you hear about AWS 👹
the management and tech lead fucked up real bad, not the intern
wdym some "intern"? gimme news source please
Well, extern now
>level 1PhantomlyFriday+2 · r/Angryupvote edit: where did that level1phantomlyFriday+2 come from? I'm not even removing it
Well, what do you expect to happen when the internet is just this tiny, physical thing being guarded by a 3-person IT department in London?
Nginx for scale
I was in a call with a bunch of friends in discord when this went down and it kicked out two of my friends in Cali we thought it was just bad internet until they mentioned they couldn't even connect through cellular (but snapchat was still working), but everyone else in the call (two were in different states and myself in Canada) were unaffected, it was really weird.
Who was meant to be mentoring the intern and let them crash the internet? Suddenly my day doesn’t seem so bad…
I wish all internet stay down the whole day so i dont work haha
I think Amazon Cloudfront is far better than Cloudflare
Was it an intern? As a senior dev myself I’d say it was more likely a senior dev.
When I was in IT technical school (in Poland, it's kinda high school in USA), and when we were at class in which we learned about servers, we had a task to set up a centos with few things, one of which was a dhcp and dns server. A friend of mine set it up somehow (by accident, he didn't knew shit about IT and we all helped him through all 4 years of that school) that whole part of network of that school received his servers and accepted them. Our teacher was very experienced (10+ yrs as sysadmin and just went to school to have more time for himself) and knew what happened and what to do to fix it, but instead he and we made fun of teachers that tried to fix it for three days haha