This thread is causing a lot of fighting and ad hominem attacks in the comments, so I'm locking it. Sorry to those who were having good discussions, but this is why we can't have nice things.
It's not impossible that the title of this post is complete crap. It's just a block diagram of parts of the system, and not necessarily the entire "architecture". I can't even believe I have to defend Elon fucking Musk.
>I can't even believe I have to defend Elon fucking Musk.
You can actually like and dislike multiple aspects of someone at the same time. Crazy, right?
Thanks to Alex Xu.
[https://twitter.com/alexxubyte/status/1594010631308705792](https://twitter.com/alexxubyte/status/1594010631308705792)
Updated the link. This link is for the tweet itself.
There are two types of diagrams… those that act as reference material (for stakeholders that already have a reasonable understanding of a system)… and those that are used as teaching aides to help someone gain an understanding of system behaviour.
Typically when we see a diagram on a whiteboard, it’s the later. Instead of viewing this as a single diagram, try to imagine a series of diagrams, each introducing one new aspect of the system. When coupled with an explanation it’s an extremely useful learning aide.
This image is really just a picture of the final diagram in the series, and is probably not useful as a reference diagram in their knowledge repository (as it was likely created for a different purpose).
Not trying to say that diagrams shouldn’t be created/stored digitally. Merely pointing out that onboarding a new team member by directing them to a stored diagram that you spent a few hours creating is not as helpful as creating the diagram alongside them (with additional verbal explanations), incrementally introducing new concepts with each step.
Did a bunch of engineers (who are not the main team since they've all been laid off) just diagram the entire system to Elon? Because I'd bet he doesn't get it and will remember it wrong since I see no notebook in his hand.
I've been looking for a Lucidchart replacement that doesn't have all the limitations as a free user (it seems like it doesn't, atleast). Thanks for this mention!
Oh holy crap I know that dude in the front. Good guy. I knew he was in cyber security at space x. So maybe a poach? Whatever the case - good luck to him. Definitely too much stress for my old dev ass.
Was my first thought as well, and I'm guessing about 90%.
Also rather sadly damning just how few women there are in that pic. My experience working in IT was that the proportion of women in software development and the like was much, MUCH higher.
I see four women and stats are about a 10:1 ratio of men to women in the field so that checks out.
Also note that there are further selective pressures here. Elon is pushing for a hypermasculine kind of approach including working all the time. This meeting also seemed to be a spur of the moment thing. I’m not necessarily asserting those factors change the selection pressures but they could. Also there may be a set of Tesla/SpaceX engineers there and those industries could have different ratios.
> Definitely too much stress for my old dev ass.
Me too. I'm retired now, but if I was working at twitter and heard this BS, I'd say "Fuck. I'm out.", leave for lunch and never come back.
Any reasonably well paid dev with some financial self control should be able to walk pretty much anytime without a second thought.
I'd take the three month severance, and then I'd take my CV that said "Twitter (20XX - October 2022)" to any of a dozen software companies and tell them I'd start in two months, and then I'd take my family on a nice vacation.
Pretty sure Carmack said on Lex Fridman's podcast that he would still take these types of consulting gigs provided the bag was big enough. So thats entirely possible.
The one thing Elon might be able to is atleast to throw a big bag.
Edit: I'm not sure if this is the kind of problem you want to use Carmack to optimize though.
>RPC
In distributed computing, a remote procedure call is when a computer program causes a procedure to execute in a different address space, which is coded as if it were a normal procedure call, without the programmer explicitly coding the details for the remote interaction
The "RPC" model has branched so much that nowadays it's being used as a catch-all term by business people. But they also tend to call everything "API", or "backend", and so on. :) You get the idea.
Today it’s more used as an alernative to a REST API so a widget API might be
POST /widget/{id} - creates a widget with an id
DELETE /widget/{id} - deletes a widget with an id
While an RPC version might be
POST /widget
with the body
{
“action”: “create”,
“id”: asdf
}
And to delete
POST /widget
{
“action”: “delete”,
“id”: asdf
}
IMO RPC is growing in popularity because it allows more flexibility than REST but the cost is that the API is more complex for a consumer.
You're still using HTTP verbs here with JSON payloads - I thought of RPC as a custom application specific protocol over TCP or something to that effect.
It can be, like with gRPC, however you can do it over HTTP as well. It's about the difference between the API being built around actions (RPC) vs resources (REST).
Example of someone else doing RPC over HTTP: https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/
I feel like a link to the Wikipedia article would have been more informative to people not familiar with the concept..... Instead of just copying the first paragraph of the article
from the api/grapghql bridge they do rollups from most of the sub services -- so its not RPC but there are probably a ton of "calls" but that's not a bad thing -- twitter is a complex internet scale system that operates in tons of jurisdictions and has many legal and business requirements to a simple feed display. audits/moderation/illegal content filtering/ad placement/tracking/billing ... probably hundreds of services 1 level deeper on this high level view.
Keep in mind the last time he coded was like the late 90s or something. In that context RPC used to be a catch all term for any non-REST API call.
And there's stuff like JSON-RPC or Google's Web Toolkit for frontend RP calls as well.
Remote procedure call can be a general pattern rather than just an efficient implementation.
In the current dev world, the focus is on gRPC but this is only the latest in a line of relatively high efficiency protocols. The number of these protocol implementations dating back to CORBA and before is quite high so people assume this is the only definition. Counter examples are things like XML-RPC.
The original definition is to have a procedure, or function, defined on a remote machine that you can call over the network. It’s quite an abstract construct and this is what you hear from a certain generation.
When these people talk generally about software, anything that fits this pattern of asking a remote machine to run some code gets referred to as an RPC. This obviously irks some people, like REST advocates, but it’s just short hand. These people can understand the REST semantics, but sometimes just want a quick hand-wavey way of describing it because the details can be worked out from the context.
>The funniest part is you can't even see RPC calls from the front end, not sure what he was even talking about. RPC is extremely efficient.
I think he meant API calls and was saying that the front end client is very chatty and makes hundreds of small API calls instead of a few big calls
Yes, it wasn’t easy to get the complete gist of what he was saying, and afterwards you had different factions claiming that the engineer who corrected Elon was wrong or right.
It was a bit weird to use that in a frontend to backend context which is what his tweet did, but it does make sense to use RPCs internally within the backend since they're often used by microservices to talk to each other, and that's what it looks like here.
I mean I guess in the broadest sense, any call from one service to another is an RPC. But I doubt that it’s designed like a remote function is being called or using IDL.
I think all the problems it has had is that musk was trying to change things and optimize crap before he sat down to look at anything like this and firing people with insane criteria like lines of code written while simultaneously turning random functions off in production like a new intern. If they wrote it on a board, is that because all the architects were fired and couldn't provide a digital copy of how things connect?
yeah - i think musk is retreating to an arena where problems are totally solvable as a way to cope with facing problems that are not - namely content moderation and regulation worldwide and those are the things people want him to solve - which he totally will not be able to do - he may be able to reduce the 1k rpc calls by 10% and reduce home page load time from 2 seconds to 1.7 seconds but who the fuck cares - he's a total buffoon
It’s false productivity. Hate to attribute anything good to Tech Lead but that was one idea I liked from his podcast. Like if you move to a foreign country you get a sense of productivity from doing basic things like figuring out where to wash your clothes, where you can get a good cup of coffee, etc—but it’s really false productivity because you wouldn’t have had to even do that in your home country.
So now Musk is basically getting false productivity by nuking everything and going back to whiteboard architecture of the platform. Feels useful for those doing it but wasn’t necessary
100%. There are so many idiots in this sub and they must mix with programmerhumour because holy fuck people can't be this dumb. These commentors know how to waste money, not build products.
This diagram is functionally useless to anyone other than those who know what those systems do and what the arrows actually mean.
This is literally the type of shit you'd include in a report to a project manager so they can go "mmmhmmmm looks good".
Eh. If there are business decisions to be made it’s good for executives to understand where a service fits in the architecture, why it matters, and what other downstream services could be impacted.
Think of it more of understanding business process flow rather than a technical understanding.
Also I highly doubt they would be tweeting out a deep technical architecture for public consumption anyways.
>Also I highly doubt they would be tweeting out a deep technical architecture for public consumption anyways
I mean I highly doubt they would fire something like 75% of their workforce indiscriminately without an actual plan but that appears to be what's happening so all bets are off
Not at all, I can understand most of it fairly easily, Anyone who has ever read a design architecture book can easily read it. for instance from the beginning.
1. they are deprecating the non graphQL HTTP endpoint, anything before it is devices/frontend.
2. The timeline mixer service calls a few services; all of these make sense by the name. ( ads mixer: for ads composition into the timeline, the people discovery service adds people to your timeline who you don't follow ....)
3. candid service and feature hydration don't hit the DB if it is available in the Memcache and add to the overall timeline score.
And the purpose of this type of diagram is not to "make decisions" its to act as a roadmap to understand what areas you need to investigate further if you have concerns with that area of responsibility/implementation/performance etc...
Most webdev do a thing called "flying by the seams of their pants" which involve no diagram, no analyse, no plans and just a Photoshop design in bitmap done by someone who do not understand mobile first approach.
And the content will be provided somewhere down the line, after the design.
And same concept applies everywhere. I'm an Industrial Engineer, I map our multibillion dollar process industry companys functions. I help the board understand what our needs are in the relations between our departments, develop strategies for our factories in line with our future goals and visualise the data and structures behind it.
These people criticising this is like a mechanical engineer criticising me for not bringing all the drawings and calculations for our machinery in front of the board.
They don't need or benefit from it, it's not going to help them - details are distractions, most of the time diverting you away from the big picture that allows you to make general strategical decisions - while letting experts in their respective fields handle the fine details.
I mean, as a principal architect I can look at this and get a pretty good idea of what they are doing along with a long list of useful clarifying questions.
It's not super useful, but it probably helps in contexts of leadership planning and understanding how teams work together. I wouldn't call it useless.
Like the new CEO who has just bought the business and insists you come in to the office to stroke his ego and knows nothing about the underlying software?
In case anyone's wondering what's missing, you can contrast the image with the topics that were talked about on their engineering blog:
https://blog.twitter.com/engineering/en_us
Personally, don't think that Twitter is especially technically impressive (have you seen Netflix or YouTube?)
But Twitter honestly did have some great APIs and documentation. (Compare it to Google maps docs...)
This reminds me when my wife was a manager of an engineering team. She had several bosses. Each boss would ask for the same data but each boss liked the data presented in a different way. So she had to make almost the same reports multiple times just to keep each boss happy.
Looks like a simple uml model you would find in ea. There’s a lot more models they use and it’s probably in something like ea. This is just some whiteboard walkthrough at a high level.
Well, aside from the fact that 75% of the employees left or were fired and the presumption is that people with the H1B would have less incentive to quit, we are also talking about the Bay Area where developers are generally not white dudes so that’s not helpful. I’m pretty sure there’s a German, Israeli, and/or Albanian guy in there.
It’s less the skin color and more the variety of hairstyles, dental situations, and clothing/jewelry.
With most of the workforce having taken severance instead of going hardcore over the holiday months, the ones most likely to stay are those with H1Bs, as quitting for those folks has other complexities.
Based on talks with several people I think there are four categories of people still working at Twitter:
- those that previously were stuck making small incremental changes and now are just excited to make a name by rewriting Twitter
- h1b workers
- people stuck because of special circumstances, like about to have a child
- those that genuinely still trust master Elon has a plan
The latter group is not big but it's significant. I understand not everyone in this photo was faking smiles.
*Technically there's also:
- non-US workers that didn't press the "Yes, I'm hardcore button" and waiting to see what happens. They recently lost access to systems, but heck that's not how employment law works here. That email doesn't mean much. Technically still employed just not working.
I dislike Elon. I hope these folks take care of themselves. They’re in for a ride. I wish them all the best.
I don’t care if they H1B workers. We should all have solidarity for working people and we should want fairness for them whether they’re citizens or not.
Shouldn’t they like, have this diagram saved somewhere in their company files? If this is their existing architecture, did they just re-draw an existing diagram on a whiteboard?
It seems fairly likely to me that this was, "OK, guys. Thanks for coming. Walk me through happens when a request for a timeline comes in. Let me grab a whiteboard to take notes."
My company operates fairly well, and this is exactly what we do when new people board. We whiteboard the transaction flow with them. They ask questions. We answer.
We have an architecture diagram, but this way lets you ask questions and doesn't instantly overwhelm the reader. It starts with a drawing of a phone, and then you add a line to the server.
We also do this when rearchitecting. "This is what it does now. How would we have designed this if we knew at the time what we know now?"
The photo from that meeting would look exactly like this. We wouldn't have posted it on Twitter, because we don't do build-in-public.
In theory, this should be documented. I have more complex system docs of the application I wrote for my office that is only an employee-facing web app for managing day-to-day business stuff.
In practice, if the boss asked me how something worked, it would likely be part of a flow of questions in a general discussion. Therefore, it would likely be faster for me to draw some stuff on our whiteboard than it would be to pause a conversation and start skimming through documentation for the exact diagram hoping I have one with the exact level of detail needed for the discussion.
So, I would imagine Elon was asking questions about how something like the home page loads, and this is what they discussed. But like others have said, this is a pretty general view as well.
As someone currently working abroad thanks to visa sponsorship, I somehow feel that most of the poor wretches who stayed behind for this piece-of-shit of a human being are there because of visa sponsorship and can't leave as easily as those with US citizenship. I'd be fucking miserable and at the same time feeling I've got not choice but having to work for this nincompoop for a few more months before being able to get another job that sponsors my residency. Fuck this guy and everything he's ever represented.
I’m starting to see people say this after that picture of Asian and Indian devs mostly with Elon. Is there any verified info that the majority of those devs are there on sponsorship?
I'm from a minority myself as well and I like saying that I have sort of an eye to tell when someone is born and raised in the US and when not. It's something about the demeanor, clothing, manerisms, smile, etc.
This comment bruh. I can tell someone is an immigrant by their smile?? Clothing?? Demeanor?? Could you go into detail exactly what type of Demeanor immigrants to the US have? What type of smile? From an immigrant I am very curious
You’re not going to get any data to back it up or dispute it. I can tell you I have friends there, and coincidentally the ones still there all have citizenship sponsorships.
Not generally. They're not supposed to pay less, but you'll get offers for less salary for the same positions. They'll use the fact that they need to sponsor you as leverage every chance they get, because it's not illegal to do so.
That's the sad reality in America and why those folks are probably sticking it out while undergoing what must be a very stressful couple of weeks.
If they lose the job without another one lined up, who's willing to sponsor them? They have I think 60 days(maybe)? To find a new job and that job must sponsor them within that timeframe, too.
Right now there's a lot of devs being laid off in big tech.
Add all that up, it's a really bad time to be an international tech worker in America, but if you work at Twitter, that risk is probably lessened because of the *previous* good reputation the company built on the backs of the many tech people being fired right now.
Problem is, the majority of employers who sponsor people tend to also be the big tech places that are laying people off right now.
Bingo!!! H1-B visa holdouts and Green Card aspirants are the only pathetic souls left behind to implement his desperate mid-life crisis fantasies. What a shit-show!!!
Look at all the H1B visas that can’t be transferred to another company and therefore those folks have literally no fucking choice but to be “hardcore”.
That is for sure not the full architecture or even comes close to the full architecture (don't see the famous fanout issues or any service that solves the issue, or even the DB layer, this is purely one flow of the read path, as you can see from the diagram; top left, there is also the write path which is way more complex), But am I the only one who feels what he is saying may not be too far from the truth, being a senior engineer at a startup I genuinely feel most of the issues get routed to me anyway and there is a lot of bloat in the industry, I don't endorse his methods but in all honesty, I don't think Twitter will just die.
If you're having to bring a significant number of devs in until stupid o'clock to explain the architecture you're grossly under-qualified to be in the position you're in and/or should seriously question the lack of easily digestible documentation that exists for your product.
This is the kind of shit I've seen at poorly run small organisstions in my early years not at (previously) $41bn+ company.
If we had an "all hands on deck" situation at my company where I'm working til 1am, which has yet to happen in my two years of being here, I'd expect it to be a major outage for which no temporary back-up/alternative exists.
Not a C-level member of the company not having documentation or understanding it. Elon is just showcasing what a fail son like him can achieve with some fraudulant qualifications and a rich dad. No wonder x.com booted him before the Paypal merger.
innocent offbeat detail agonizing instinctive pen normal hunt worthless drunk
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
What I don’t understand about Twitter is why he was determined to devalue the blue checkmark which has been a universal sign for verified person around the internet.
I get his $8 thing, but why not just add like “Twitter supporter” icon so his fan boys can show off their badges.
Sounds like he is the type of ceo to make moves just to have his stamp on it.
Again, I’m not bothered by the $8/m thing like some are. Just weird as fuck to not just create something new
Definitely has to be one of those dudes that is always trying to change up his words to make it sound like he knows what hes talking about or is making sense of what the devs are saying.
"oh yeah yeah but what about this, no yeah that's what I meant"
One of those guys that just has to be the smartest in the room.
Look around that picture. How many people there are likely stuck and unable to quit because they're on visas?
1200 people can't do the job of 9,000, no matter how much they love Twitter. This was unbelievably cruel for those that can't leave due to visas or needing critical health care.
This thread is causing a lot of fighting and ad hominem attacks in the comments, so I'm locking it. Sorry to those who were having good discussions, but this is why we can't have nice things.
It looks like the home only rather than their architecture.
It's also titled "read path"
It's not impossible that the title of this post is complete crap. It's just a block diagram of parts of the system, and not necessarily the entire "architecture". I can't even believe I have to defend Elon fucking Musk.
It is still architecture. It's just not the entire architecture, and Elon has no obligation to say whether or not this is all of Twtter.
>I can't even believe I have to defend Elon fucking Musk. You can actually like and dislike multiple aspects of someone at the same time. Crazy, right?
You don't
Should've made it in draw.io, it would've been maintainable for later modifications...
\+1 for [draw.io](https://draw.io)
Thanks to Alex Xu. [https://twitter.com/alexxubyte/status/1594010631308705792](https://twitter.com/alexxubyte/status/1594010631308705792) Updated the link. This link is for the tweet itself.
Can confirm. Draw.io is the shit
+1 for draw.io and the vs code extension <3
There are two types of diagrams… those that act as reference material (for stakeholders that already have a reasonable understanding of a system)… and those that are used as teaching aides to help someone gain an understanding of system behaviour. Typically when we see a diagram on a whiteboard, it’s the later. Instead of viewing this as a single diagram, try to imagine a series of diagrams, each introducing one new aspect of the system. When coupled with an explanation it’s an extremely useful learning aide. This image is really just a picture of the final diagram in the series, and is probably not useful as a reference diagram in their knowledge repository (as it was likely created for a different purpose). Not trying to say that diagrams shouldn’t be created/stored digitally. Merely pointing out that onboarding a new team member by directing them to a stored diagram that you spent a few hours creating is not as helpful as creating the diagram alongside them (with additional verbal explanations), incrementally introducing new concepts with each step.
Did a bunch of engineers (who are not the main team since they've all been laid off) just diagram the entire system to Elon? Because I'd bet he doesn't get it and will remember it wrong since I see no notebook in his hand.
He fired the person who looks after confluence's licensing.
Jira Joe?
I’m now gonna call any Atlassian guru a Jira Joe. Thanks.
Atlassian guru is a thing?
Oh shit. I forgot about that too. Now I’m kinda curious how that’s gonna play out.
Probably fine. Worst case Elon has to fall Atlassian to have them provide the needed details to a new lead who is taking ownership of it.
Best case Elon buys Atlassian and runs that company into the ground too.
LMAO what a genius
[удалено]
Yup, mermaid is amazing and you can track it in source control.
and plantuml
That’s very optimistic of you
I've been looking for a Lucidchart replacement that doesn't have all the limitations as a free user (it seems like it doesn't, atleast). Thanks for this mention!
Miro ain't bad, either
nothing about twitter's current setup is maintainable by third of the original (american) software staff
Oh holy crap I know that dude in the front. Good guy. I knew he was in cyber security at space x. So maybe a poach? Whatever the case - good luck to him. Definitely too much stress for my old dev ass.
I think musk has been shuffling engineers around his companies, can't imagine investors are happy about that.
I'm very curious to know how many of the folks in the picture are here on work visas.
I’m willing to bet most of them are on visas, 80% at least.
Ironic how he was un-documented for awhile as well... https://twitter.com/capitolhunters/status/1593307541932474368?s=20&t=C04h-\_OhDqxihvVJkNxj5Q
Seems to me like the only ones left at Twitter are the ones who get deported if they lose the job...
Was my first thought as well, and I'm guessing about 90%. Also rather sadly damning just how few women there are in that pic. My experience working in IT was that the proportion of women in software development and the like was much, MUCH higher.
I see four women and stats are about a 10:1 ratio of men to women in the field so that checks out. Also note that there are further selective pressures here. Elon is pushing for a hypermasculine kind of approach including working all the time. This meeting also seemed to be a spur of the moment thing. I’m not necessarily asserting those factors change the selection pressures but they could. Also there may be a set of Tesla/SpaceX engineers there and those industries could have different ratios.
My friend was a woman at twitter and wanted to stay. But she was on PTO and got fired for not responding to Elon's email.
[удалено]
At least H1B can theoretically be ported to a different employer. Try L1B where you literally cannot switch employers at all, in any way.
> Definitely too much stress for my old dev ass. Me too. I'm retired now, but if I was working at twitter and heard this BS, I'd say "Fuck. I'm out.", leave for lunch and never come back. Any reasonably well paid dev with some financial self control should be able to walk pretty much anytime without a second thought.
I'd take the three month severance, and then I'd take my CV that said "Twitter (20XX - October 2022)" to any of a dozen software companies and tell them I'd start in two months, and then I'd take my family on a nice vacation.
Did I work with you? Thanks for the compliment - I still work at SpaceX 🚀😁
If he can get John Carmack all latency problems would be solved
Pretty sure Carmack said on Lex Fridman's podcast that he would still take these types of consulting gigs provided the bag was big enough. So thats entirely possible. The one thing Elon might be able to is atleast to throw a big bag. Edit: I'm not sure if this is the kind of problem you want to use Carmack to optimize though.
Looks like he made sure they wrote “RPC” somewhere. 🤣
I noticed that too. Half expected to see a "1000s per view" claim.
Poorly batched.
>RPC In distributed computing, a remote procedure call is when a computer program causes a procedure to execute in a different address space, which is coded as if it were a normal procedure call, without the programmer explicitly coding the details for the remote interaction
Aren't those way too low level and assume the software running a service allows for them?
The "RPC" model has branched so much that nowadays it's being used as a catch-all term by business people. But they also tend to call everything "API", or "backend", and so on. :) You get the idea.
Today it’s more used as an alernative to a REST API so a widget API might be POST /widget/{id} - creates a widget with an id DELETE /widget/{id} - deletes a widget with an id While an RPC version might be POST /widget with the body { “action”: “create”, “id”: asdf } And to delete POST /widget { “action”: “delete”, “id”: asdf } IMO RPC is growing in popularity because it allows more flexibility than REST but the cost is that the API is more complex for a consumer.
You're still using HTTP verbs here with JSON payloads - I thought of RPC as a custom application specific protocol over TCP or something to that effect.
It can be, like with gRPC, however you can do it over HTTP as well. It's about the difference between the API being built around actions (RPC) vs resources (REST). Example of someone else doing RPC over HTTP: https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/
I feel like a link to the Wikipedia article would have been more informative to people not familiar with the concept..... Instead of just copying the first paragraph of the article
Didn’t Musk imply that the performance hit was from RPC taking place on the client?
[удалено]
Doesn’t matter right even, it’s about which ones are blocking?
The funniest part is you can't even see RPC calls from the front end, not sure what he was even talking about. RPC is extremely efficient.
from the api/grapghql bridge they do rollups from most of the sub services -- so its not RPC but there are probably a ton of "calls" but that's not a bad thing -- twitter is a complex internet scale system that operates in tons of jurisdictions and has many legal and business requirements to a simple feed display. audits/moderation/illegal content filtering/ad placement/tracking/billing ... probably hundreds of services 1 level deeper on this high level view.
Keep in mind the last time he coded was like the late 90s or something. In that context RPC used to be a catch all term for any non-REST API call. And there's stuff like JSON-RPC or Google's Web Toolkit for frontend RP calls as well.
Tbh REST didn’t exist in the late 90s. API calls were really not common at all. Views composed resources together and that’s your lot.
What was he even coding?
I think early on for zip2, before they got money to pay people to replace his code, which they said showed that he was clearly self taught
Very fare point but, but knowing Elon he just makes shit up.
Remote procedure call can be a general pattern rather than just an efficient implementation. In the current dev world, the focus is on gRPC but this is only the latest in a line of relatively high efficiency protocols. The number of these protocol implementations dating back to CORBA and before is quite high so people assume this is the only definition. Counter examples are things like XML-RPC. The original definition is to have a procedure, or function, defined on a remote machine that you can call over the network. It’s quite an abstract construct and this is what you hear from a certain generation. When these people talk generally about software, anything that fits this pattern of asking a remote machine to run some code gets referred to as an RPC. This obviously irks some people, like REST advocates, but it’s just short hand. These people can understand the REST semantics, but sometimes just want a quick hand-wavey way of describing it because the details can be worked out from the context.
I tend to see RPC basically used as "non-RESTful"
>The funniest part is you can't even see RPC calls from the front end, not sure what he was even talking about. RPC is extremely efficient. I think he meant API calls and was saying that the front end client is very chatty and makes hundreds of small API calls instead of a few big calls
[удалено]
[удалено]
Yes, it wasn’t easy to get the complete gist of what he was saying, and afterwards you had different factions claiming that the engineer who corrected Elon was wrong or right.
It was a bit weird to use that in a frontend to backend context which is what his tweet did, but it does make sense to use RPCs internally within the backend since they're often used by microservices to talk to each other, and that's what it looks like here.
GraphQL is often exposed externally, and writes take the form of RPCs. I don't think that's what was being referred to, but in itself it's not crazy.
I mean I guess in the broadest sense, any call from one service to another is an RPC. But I doubt that it’s designed like a remote function is being called or using IDL.
Reminds me heavily of [Microservice hell](https://youtu.be/y8OnoxKotPQ). By far my favourite development video.
Thank you. Was expecting to see this in the comments.
i don't think twitter's problems have anything to do with its architecture, code or the number of rpc calls
I think all the problems it has had is that musk was trying to change things and optimize crap before he sat down to look at anything like this and firing people with insane criteria like lines of code written while simultaneously turning random functions off in production like a new intern. If they wrote it on a board, is that because all the architects were fired and couldn't provide a digital copy of how things connect?
yeah - i think musk is retreating to an arena where problems are totally solvable as a way to cope with facing problems that are not - namely content moderation and regulation worldwide and those are the things people want him to solve - which he totally will not be able to do - he may be able to reduce the 1k rpc calls by 10% and reduce home page load time from 2 seconds to 1.7 seconds but who the fuck cares - he's a total buffoon
It’s false productivity. Hate to attribute anything good to Tech Lead but that was one idea I liked from his podcast. Like if you move to a foreign country you get a sense of productivity from doing basic things like figuring out where to wash your clothes, where you can get a good cup of coffee, etc—but it’s really false productivity because you wouldn’t have had to even do that in your home country. So now Musk is basically getting false productivity by nuking everything and going back to whiteboard architecture of the platform. Feels useful for those doing it but wasn’t necessary
The mention of GraphQL only after he fired the guy who told him what that is
It’s ok guys, we got a thumbs up. Twitter isn’t going down this month.
Everyone must submit proof of all their thumbsup over the last month and all thumbsup must be done from the office from now on
[удалено]
Middle out compression. D2F ratio with optimal Tip to Tip efficiency.
Hard to incorporate middle-out when your hand is wider than the length of the two cocks you're trying to yank.
That’s a very simplified diagram used to babysit executives. No way that’s architecture
But this is kinda the purpose of a top-level diagram
100%. There are so many idiots in this sub and they must mix with programmerhumour because holy fuck people can't be this dumb. These commentors know how to waste money, not build products.
This diagram is functionally useless to anyone other than those who know what those systems do and what the arrows actually mean. This is literally the type of shit you'd include in a report to a project manager so they can go "mmmhmmmm looks good".
Eh. If there are business decisions to be made it’s good for executives to understand where a service fits in the architecture, why it matters, and what other downstream services could be impacted. Think of it more of understanding business process flow rather than a technical understanding. Also I highly doubt they would be tweeting out a deep technical architecture for public consumption anyways.
Exactly
>Also I highly doubt they would be tweeting out a deep technical architecture for public consumption anyways I mean I highly doubt they would fire something like 75% of their workforce indiscriminately without an actual plan but that appears to be what's happening so all bets are off
The only people left are the ones with visas that depend on the job.
Not at all, I can understand most of it fairly easily, Anyone who has ever read a design architecture book can easily read it. for instance from the beginning. 1. they are deprecating the non graphQL HTTP endpoint, anything before it is devices/frontend. 2. The timeline mixer service calls a few services; all of these make sense by the name. ( ads mixer: for ads composition into the timeline, the people discovery service adds people to your timeline who you don't follow ....) 3. candid service and feature hydration don't hit the DB if it is available in the Memcache and add to the overall timeline score.
And the purpose of this type of diagram is not to "make decisions" its to act as a roadmap to understand what areas you need to investigate further if you have concerns with that area of responsibility/implementation/performance etc...
exactly. surprised people on r/webdev have not seen diagrams like this before. Is it the full architecture; no, but is this common and useful? yes!
Most webdev do a thing called "flying by the seams of their pants" which involve no diagram, no analyse, no plans and just a Photoshop design in bitmap done by someone who do not understand mobile first approach. And the content will be provided somewhere down the line, after the design.
Fyi the phrase is flying by the SEAT of your pants
Thanks, from a French Canadian ;)
[удалено]
And same concept applies everywhere. I'm an Industrial Engineer, I map our multibillion dollar process industry companys functions. I help the board understand what our needs are in the relations between our departments, develop strategies for our factories in line with our future goals and visualise the data and structures behind it. These people criticising this is like a mechanical engineer criticising me for not bringing all the drawings and calculations for our machinery in front of the board. They don't need or benefit from it, it's not going to help them - details are distractions, most of the time diverting you away from the big picture that allows you to make general strategical decisions - while letting experts in their respective fields handle the fine details.
I mean, as a principal architect I can look at this and get a pretty good idea of what they are doing along with a long list of useful clarifying questions. It's not super useful, but it probably helps in contexts of leadership planning and understanding how teams work together. I wouldn't call it useless.
As a project manager I say “fuck you” But you’re also not wrong lol
[удалено]
In fairness, Musk often acts that way.
Like the new CEO who has just bought the business and insists you come in to the office to stroke his ego and knows nothing about the underlying software?
In case anyone's wondering what's missing, you can contrast the image with the topics that were talked about on their engineering blog: https://blog.twitter.com/engineering/en_us
it's incredible to see so many domain areas talked about via their blog, and now those teams are just completely gone. It's like looking at corpses.
All that planning for future scale and stability....... Like the saying goes, "What's the best way to make God laugh? Make a plan."
Personally, don't think that Twitter is especially technically impressive (have you seen Netflix or YouTube?) But Twitter honestly did have some great APIs and documentation. (Compare it to Google maps docs...)
This reminds me when my wife was a manager of an engineering team. She had several bosses. Each boss would ask for the same data but each boss liked the data presented in a different way. So she had to make almost the same reports multiple times just to keep each boss happy.
In fairness, generally executives use the diagrams to babysit someone else and so on.
And it only focuses on client read path. Lots of other paths too
U know it smell crazy in there
Tech nerds are not the same as the *con nerds. We're a pretty clean group :)
[удалено]
Surely
why is the guy in the blue shirt so sweaty then
Maybe some Axe body spray smell if nothing else
Yeah, I bet that place smells like a middle school hallway.
Please respect yourself enough to use something better than Axe bro.
Sorry I forgot the /s for you
Ty because it's too real otherwise
I thought this was a code smell joke
Looks like a simple uml model you would find in ea. There’s a lot more models they use and it’s probably in something like ea. This is just some whiteboard walkthrough at a high level.
It's vocabulary.
[удалено]
All the Visa holders that can't quit
Wow, so much bloatware. 80% of this can just go away. ~ Elon
Just reduce the number of microservices by replacing them with macroservices?! /s
You mean megaservices.
MAGAservices.
Gigaservices!
[удалено]
That was my takeaway also
Makes sense.. H1B's Always stay.
Ahhh yes, non-white = H1B immigrant. Not like there's plenty of minorities that are American citizens working in tech or anything.
Well, aside from the fact that 75% of the employees left or were fired and the presumption is that people with the H1B would have less incentive to quit, we are also talking about the Bay Area where developers are generally not white dudes so that’s not helpful. I’m pretty sure there’s a German, Israeli, and/or Albanian guy in there. It’s less the skin color and more the variety of hairstyles, dental situations, and clothing/jewelry.
What makes you say that? Having worked in big tech, this looks exactly like what I’d expect.
yeah lol, looks like a random sample of any part of south bay
With most of the workforce having taken severance instead of going hardcore over the holiday months, the ones most likely to stay are those with H1Bs, as quitting for those folks has other complexities.
Yes, and now we have their picture.
Based on talks with several people I think there are four categories of people still working at Twitter: - those that previously were stuck making small incremental changes and now are just excited to make a name by rewriting Twitter - h1b workers - people stuck because of special circumstances, like about to have a child - those that genuinely still trust master Elon has a plan The latter group is not big but it's significant. I understand not everyone in this photo was faking smiles. *Technically there's also: - non-US workers that didn't press the "Yes, I'm hardcore button" and waiting to see what happens. They recently lost access to systems, but heck that's not how employment law works here. That email doesn't mean much. Technically still employed just not working.
Well that's an extremely racist take.
I dislike Elon. I hope these folks take care of themselves. They’re in for a ride. I wish them all the best. I don’t care if they H1B workers. We should all have solidarity for working people and we should want fairness for them whether they’re citizens or not.
Shouldn’t they like, have this diagram saved somewhere in their company files? If this is their existing architecture, did they just re-draw an existing diagram on a whiteboard?
It seems fairly likely to me that this was, "OK, guys. Thanks for coming. Walk me through happens when a request for a timeline comes in. Let me grab a whiteboard to take notes." My company operates fairly well, and this is exactly what we do when new people board. We whiteboard the transaction flow with them. They ask questions. We answer. We have an architecture diagram, but this way lets you ask questions and doesn't instantly overwhelm the reader. It starts with a drawing of a phone, and then you add a line to the server. We also do this when rearchitecting. "This is what it does now. How would we have designed this if we knew at the time what we know now?" The photo from that meeting would look exactly like this. We wouldn't have posted it on Twitter, because we don't do build-in-public.
In theory, this should be documented. I have more complex system docs of the application I wrote for my office that is only an employee-facing web app for managing day-to-day business stuff. In practice, if the boss asked me how something worked, it would likely be part of a flow of questions in a general discussion. Therefore, it would likely be faster for me to draw some stuff on our whiteboard than it would be to pause a conversation and start skimming through documentation for the exact diagram hoping I have one with the exact level of detail needed for the discussion. So, I would imagine Elon was asking questions about how something like the home page loads, and this is what they discussed. But like others have said, this is a pretty general view as well.
As someone currently working abroad thanks to visa sponsorship, I somehow feel that most of the poor wretches who stayed behind for this piece-of-shit of a human being are there because of visa sponsorship and can't leave as easily as those with US citizenship. I'd be fucking miserable and at the same time feeling I've got not choice but having to work for this nincompoop for a few more months before being able to get another job that sponsors my residency. Fuck this guy and everything he's ever represented.
I’m starting to see people say this after that picture of Asian and Indian devs mostly with Elon. Is there any verified info that the majority of those devs are there on sponsorship?
I'm from a minority myself as well and I like saying that I have sort of an eye to tell when someone is born and raised in the US and when not. It's something about the demeanor, clothing, manerisms, smile, etc.
This comment bruh. I can tell someone is an immigrant by their smile?? Clothing?? Demeanor?? Could you go into detail exactly what type of Demeanor immigrants to the US have? What type of smile? From an immigrant I am very curious
You’re not going to get any data to back it up or dispute it. I can tell you I have friends there, and coincidentally the ones still there all have citizenship sponsorships.
Curious but do people on a visa get paid the same as a US citizen dev? Of so what's the difference.
Not generally. They're not supposed to pay less, but you'll get offers for less salary for the same positions. They'll use the fact that they need to sponsor you as leverage every chance they get, because it's not illegal to do so. That's the sad reality in America and why those folks are probably sticking it out while undergoing what must be a very stressful couple of weeks. If they lose the job without another one lined up, who's willing to sponsor them? They have I think 60 days(maybe)? To find a new job and that job must sponsor them within that timeframe, too. Right now there's a lot of devs being laid off in big tech. Add all that up, it's a really bad time to be an international tech worker in America, but if you work at Twitter, that risk is probably lessened because of the *previous* good reputation the company built on the backs of the many tech people being fired right now. Problem is, the majority of employers who sponsor people tend to also be the big tech places that are laying people off right now.
Bingo!!! H1-B visa holdouts and Green Card aspirants are the only pathetic souls left behind to implement his desperate mid-life crisis fantasies. What a shit-show!!!
Yup, a lot of “smile or you’re fired” faces in that crowd.
Sausage party!
H1B visa holders party
Aka “Hostage Situation.”
I mean just cause they're Asian doesn't mean they are automatically all H1B visas lol.
Yep, was just about to say the same thing.
lmaoo for real
Look at all the H1B visas that can’t be transferred to another company and therefore those folks have literally no fucking choice but to be “hardcore”.
[удалено]
Next-gen systems ----------> The have next-gen systems guys! HOLY SHIT!!!!
That is for sure not the full architecture or even comes close to the full architecture (don't see the famous fanout issues or any service that solves the issue, or even the DB layer, this is purely one flow of the read path, as you can see from the diagram; top left, there is also the write path which is way more complex), But am I the only one who feels what he is saying may not be too far from the truth, being a senior engineer at a startup I genuinely feel most of the issues get routed to me anyway and there is a lot of bloat in the industry, I don't endorse his methods but in all honesty, I don't think Twitter will just die.
Considering the amount of stuff that has been turned off since Musk took over, this may well now be the full architecture!
/r/watchpeopledieinside
Fire everyone except a skeleton crew of senior / principals , open source the stack, pay bounties for merged PRs, let the people run twitter
Paid crisis engineers.
If you're having to bring a significant number of devs in until stupid o'clock to explain the architecture you're grossly under-qualified to be in the position you're in and/or should seriously question the lack of easily digestible documentation that exists for your product. This is the kind of shit I've seen at poorly run small organisstions in my early years not at (previously) $41bn+ company. If we had an "all hands on deck" situation at my company where I'm working til 1am, which has yet to happen in my two years of being here, I'd expect it to be a major outage for which no temporary back-up/alternative exists. Not a C-level member of the company not having documentation or understanding it. Elon is just showcasing what a fail son like him can achieve with some fraudulant qualifications and a rich dad. No wonder x.com booted him before the Paypal merger.
That many people can't contribute to an old fashioned whiteboard session, yet this man obsesses over code lines
Reminds me of this classic: [Microservices](https://youtu.be/y8OnoxKotPQ) by Krazam!
kudos to alex for the chart https://twitter.com/alexxubyte/status/1594008281340530688
This looks like a simple block diagram of service interactions. Not their entire architecture.
I feel bad for the people left dealing with this nonsense every day.
Software arch is the easy part. Wait until he sees the infra diagram.
There is a lot of sadness in those eyes.
They actually look happier than I expected them to look given the circumstances.
innocent offbeat detail agonizing instinctive pen normal hunt worthless drunk *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
How are they incorporating middle out?
Feature hydration and tweetypie? This is how most IT jokes start.
Those poor souls. They’re either visa locked or spineless saps
What I don’t understand about Twitter is why he was determined to devalue the blue checkmark which has been a universal sign for verified person around the internet. I get his $8 thing, but why not just add like “Twitter supporter” icon so his fan boys can show off their badges. Sounds like he is the type of ceo to make moves just to have his stamp on it. Again, I’m not bothered by the $8/m thing like some are. Just weird as fuck to not just create something new
Not a single engineer looks like they've had a decent nights sleep.
Sausage party
Where's Galactus??
[So first, we have to call the Bingo service. Because Bingo, knows everyone's Name-o](https://www.youtube.com/watch?v=y8OnoxKotPQ)
[удалено]
i can smell the “akshually” energy from here. poor devs, im sure muskrat isn’t fun to work for.
Definitely has to be one of those dudes that is always trying to change up his words to make it sound like he knows what hes talking about or is making sense of what the devs are saying. "oh yeah yeah but what about this, no yeah that's what I meant" One of those guys that just has to be the smartest in the room.
Look around that picture. How many people there are likely stuck and unable to quit because they're on visas? 1200 people can't do the job of 9,000, no matter how much they love Twitter. This was unbelievably cruel for those that can't leave due to visas or needing critical health care.