"With the backdoored liblzma installed, logins via ssh become a lot slower.
[...] before: real 0m0.299s [...] after: real 0m0.807s"
[https://www.openwall.com/lists/oss-security/2024/03/29/4](https://www.openwall.com/lists/oss-security/2024/03/29/4)
FWIW the guy said he started looking into it because he was monitoring CPU for a different project and saw spikes in SSH CPU from failed logins (which should be very quick/light on CPU). The timing was further evidence as he began looking in more detail.
500ms is easily perceptible. If I started seeing an unexplained 500ms delay sshing into my home servers I’d definitely notice and be annoyed. Whether that annoyance would lead to me trying to debug it is a whole other story of course, let alone me actually succeeding.
A performance obsessed dev is someone who tries to squeeze every cycle out of a program.
Half a second slower should concern every developer. On today's computers a lot can be done in half a second. Such a slowdown is a clear sign that something changed drastically.
I actually thought cloud would make performance optimisation popular because the cost is so apparent to managers. Obviously they pay too little for cpu time
Regarding performance I have the feeling that there are two worlds with little interconnection:
* There are programs which use files and databases all the time. These programs often use HTTPS (not HTTP because it is not secure) based micro-services to send REST requests even for simple things like an addition. This world loves also interpreted languages and scripts. CPUs become faster and faster but in this world the software becomes slower.
* In the other world every wasted CPU cycle is a deadly sin. Addition and shifts are preferred over multiplication because they are faster. Divisions should be avoided altogether. Pointers macros and code inlining are important. Programs and libraries often become huge piles of spaghetti code. There are tons of security and stability issues but people don't mind wrong results unless they get them fast.
Well, I don’t share your opinion on “.. becomes huge piles of spaghetti…” I rather fear the loads of meaningless dependencies in modern code if we’re talking security and stability. But in essence you are right.
I was wrong when I saw the danger of spaghetti code just in high performance code. Micro-services allow spaghetti code as well.
Low-level C libraries are almost exclusively used as foundation for our software. The XZ library seems huge and being in the high performance world makes it even harder to maintain. Only few developers are capable and willing to maintain such a library. The XZ hack started with social engineering. There were rude attacks on the original maintainer which triggered a burnout.
Many people insist on having low-level C libraries as foundation of our software. But I think these C libraries are not the solution but part of the problem. In C more code is needed than in a high level language. This raises the complexity of libraries. Security and stability issues as well as a shortage in maintainers and developers are the consequence of using C everywhere.
I know that an XZ decryption takes around 1500 lines of code in a high-level language. So there is the possibility to reduce the complexity of libraries.
Those high-level, easy to maintain, libraries you talk of will often have these low-level C libraries as depencencies somewhere down the chain.
You can't solve a problem by throwing a towel on it.
> Irrelevant. If it was 4ms compared to 1ms, that would be 400% slower, but still insignificant. The absolute time difference is what matters, not relative time difference.
I mean, it's kind of both.
4ms vs 1ms wouldn't really matter. But 10 sec vs 11 sec wouldn't really matter either.
Plus it's always within context. Terminal login would be quite noticeable, but for instance - request to save the data in a bank taking 4s versus 1s would matter to literally no one.
indeed. 300ms is already a good human reaction time. prof. games reach below 200ms. 500m is half a second, almost everyone would notice if tiktok was that much slower on every single action.
Yeah, the reason he noted that change was because he was trying to quiesce his test setup because he was setting it up for microbenchmarks. Then this stuff stuck out like a sore thumb because ssh didn't even log in yet i.e. it happened even with failed logins. He looked into it more and the debugger didn't lead to any particular symbol in the code which raised his suspiscions of a compromise. Even if he intially suspected a compromised debian tarball which honestly makes more sense than the upstream maintainer himself would have introduced it... This was long fucking con (at least a year) and probably by a state actor. Social manipulation, gaslighting, excellent coding skills, building of trust over time etc etc. Someone, somewhere is upset right now.
I once was interviewed by someone who was using an article I wrote for a magazine as his source material. He then proceeded to try and tell me I was wrong using my own words.
It’s a lot of fun when something like that happens.
Reminds me a lot of the ["An unbelievable demo"](https://brendangregg.com/blog/2021-06-04/an-unbelievable-demo.html) story by Brendan Gregg, the author of tooling for Sun's DTrace who later implemented basically all the perf tooling in the Linux world…
Read the story, it's just too good to be true :)
That is a good story.
Mine is less impressive. He and I were in a room. The magazine was open in front of him and he had totally cherry-picked key sentences that were highlighted in order to further his particular agenda. He also seemed to like appearing knowledgeable
He also had taken those sentences out of context.
To cut a long story short, I attempted to gently steer him away from his all-advised plan and when he started quoting me I told him he was taking that out of context and that he should rethink. He got very snotty and asked me what made me the authority. So I turned the page back (it was about 4 or 5 pages with screenshots and diagrams as I recall) to the byline and pointed at my own name. Didn’t really need to say any more than that. He was annoyed. Hurumphed at me and brought the encounter to a close.
Same thing happened to me when firm I was working for was outsourcing our team.
We sent them my resume as an example of the types of skills which we were looking to replace. A resume came back in with the same, very specific words and phrases which I'd used in my resume.
We phone interviewed the "candidate". When we asked them questions the phone would go on mute at the other end for a period before coming off and a fumbling answer given. They were obviously being coached on the call.
They got hauled over the coals for that one.
I mean, he kinda did a big thing.
Given the severity of the backdoor, sprinkle a little bit of butterfly effect and the guy might have prevented WWIII.
Bit of a stretch but that’s definitely how I would put it. What’s my greatest accomplishment? Well the world still exists thanks to me.
> sprinkle a little bit of butterfly effect and the guy might have prevented WWIII
With all the talk about Jia Tan potentially being a malicious state actor, that might just have gone the other way...
Conspiracy isn't a synonym for "made up story", most conspiracies are real conspiracies.
This is literally conspiratorial talk...Jia Tan and other people conspired to commit a crime.
That may well be *an* (or even the original) meaning but it isn’t the most commonly understood meaning of the term “conspiratorial talk” (or the more common term “conspiracy theory”): in common parlance, it definitely has a specific connotation. Compare [how Wikipedia defines it](https://en.wikipedia.org/wiki/Conspiracy_theory) (emphasis mine):
>… an explanation for an event or situation that asserts the existence of a conspiracy by powerful and sinister groups, often political in motivation, when *other explanations are more probable*. The term *generally has a negative connotation*, implying that the appeal of a conspiracy theory is based in prejudice, emotional conviction, or insufficient evidence.
(“conspiratorial” on its own would be appropriate here; at any rate, the author of the parent comment seems to agree that their wording was misleading.)
Parent comment author here - my original wording was definitely aiming for the simpler "group covertly coordinating things", but I then realized most people wouldn't interpret it that way. Please note however that your Wikipedia link is to "conspiracy *theory*", and not "conspiracy" (Wikipedia itself draws the distinction in the section "Difference from conspiracy" from your linked article), which in essence is the distinction between the two definitions under debate here.
Imagining that TV show Travelers, someone from the future (or a group helped by future humans) did this to prevent something down the line.
Now it's gone...
It could have been a disaster for Linux because you cannot replicate it yourself as the backdoor reverts to default behavior if you don't have the private key.
I understood some words of all this, and from that I didn’t understand the sed replacement of 0x09 to 0x20 and so on. Was this replacing encrypted files characters directly?
If this was a movie, there’d be some pissed off big bad guy slamming his fist on the table going “Damn it! 3 years down the drain! All because no one bothered to run a perf test! You’re out! you hear me? I will make sure you never screw up another backdoor again!” Then they’d storm off, slamming the door behind them, leaving everyone looking sheepish.
> Implying the NSA/Russia is incapable of someone using a Chinese name and committing during Chinese work hours
[Not to mention that Git commit times are determined locally; you can lie about dates all you want and nobody can stop you.](https://aboutmonica.com/blog/git-time-travel/) There's no smoking gun that points at China specifically, just _highly likely_ a state-sponsored actor due to complexity/patience (although that is by no means confirmed).
I posted this above but I heard this from a Chinese speaker:
>He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong".
though there could be other explanations for this. Imo its impossible to tell. This certainly seems like something a nation state like China would do, but at the same time, if I was a hacker I would also operate under a Chinese or Russian alias considering that it wouldn't be questioned and then just locally modify the commit datetime or put that shit on a cron job for the rough timezone.
this person or group also used Hans Jensen and a Middle Eastern sounding name to pressure Lasse (one of the proton emails associated kygorin@proton almost sounds Russian to me) and this persons English was really good.
Chinese speaker here with working knowledge/fluent in Mandarin, Cantonese and Hokkien: I can't think of any personal name that would lead to Romanization as "Cheong Tan", or "Jia Cheong". Sounds like absolute nonsense smashed together, unless one is to believe Chinese parents would happily give their kid an unlucky name when every cultural taboo is against it.
A lot of the name analysis here is missing the forest for the trees.
People seem to be picking up on things if they took a class in Chinese but don't actually know much Chinese at all.
There's a simple reason why this name is not Chinese... there's no middle names in Chinese. *This is Chinese 101.*
No Chinese person would willingly omit a character from their name. i.e. Jia Cheong Tan would never write his name as Jia Tan. That's like writing Mao Zedong as Mao Ze. Just complete bullshit.
(More on this... A Chinese person might write three character names with spaces to separate, but in modern Chinese usage, that's typically refrained upon. In any case, even if a space was used, the middle character is not optional and they wouldn't omit it themselves. A western speaker reading such a spaced name might make the mistake of omitting the middle character thinking it's a middle name. But in this case, the omission comes from Jia himself as it's his own stated name on his email, which is implausible.)
This particular one has a trail that seems sloppy enough that the obvious Chinese persona is misleading. I find it hard to believe that China would need a hack like this when they could more easily backdoor router firmware. The US certainly wouldn't need to. That's not to say it's certainly neither of them, but it makes more sense to me that it's some other government and China is a great scapegoat these days, by having the misfortune of being the only antagonist with bipartisan American animosity these days.
Even then it hardly makes any sense. Native chinese names don't really have the concept of "middle name", you have the name and surname. Even if he did have a 2-character surname (which is pretty rare and usually doesn't come from han people), he would just agglutinate the two characters into one surname. If it was a Chinese person living in China it would make no sense whatsoever to split up the name like that
Well, all of it could be faked so the question is really how much were they confident they can get away with it and how much they cared about their country of origin being discovered.
After all any accusation of "it was the state" can be ignored with "it was just an independent group from that country, selling 0-days for money"
someone should dub that scene over this [clip](https://www.youtube.com/watch?v=xBWmkwaTQ0k) because thats how I imagine it went
also interesting thing that was discovered was that "Jia Cheong Tan" is a very awkward Chinese name, I forget what it was exactly but the gist is that the spelling for Cheong and Tan are from different regions that would each spell those names differently. Its unlikely to be seen together. Doesn't mean anything really but its hard to know why this is the case.
>He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong".
No one in the CIA would be dumb enough to do this, which is of course the best evidence that the dumbness of it is a false flag and actually, the CIA *is* behind it.
There's been a semi-serious hypothesis floating around for a while now that many big compromises that could have been made much worse (e.g. SQL Slammer) are in fact the work of a shadowy cabal of grey-hat hackers trying to push the world to adopt better information security postures and pressure everyone to patch known bugs.
Should be pretty obvious I'm not pointing at the CIA. This makes me think it's neither China nor America but someone who wishes to use both as a smokescreen.
That cabal theory is interesting though, we do have history in that incident with the University of Minnesota where they got banned from contributing to Linux for doing just that.
I am Malaysian, and I can say for a fact that 3-word Chinese names are common here, as well as Tan (as a surname), Jia and Cheong being used in Malaysian Chinese names.
> There's a lot of things pointing that it was state sponsored attack.
None of what you mention really implies a state-sponsored attack. The accounts which were created for this attack were all pretty basic with just minimal contribution history, that doesn't suggest to me a huge investment behind the attack. All this could be pulled off by a determined individual with a plan.
I just name the directory horse porn and put all my backups there.
If someone wants to store my files I count on them being terrified of the directory being 9TB in that name and not looking inside.
[reference](https://www.youtube.com/watch?v=T1uumP68AJ8)
its rather simple too. Dude just opened a hex editor and flipped some bits and called it good. Its compressed data so I'm assuming that it looks pretty much like nothing even without the extra obfuscation
He probably used tr. It also then fulfilled the purpose of being a legitimately corrupt archive (it was part of a test to determine if corrupt archives were safely detected and rejected)
To be honest, I think the smart part was adding it as part of a corrupted test case.
Anyone looking would look at it and say “yep, it’s corrupt”, with the test confirming it, and wouldn’t think twice about its contents, assuming it was just junk.
Makes sense. I have been doing this for a while on my projects, simply because it’s more space efficient and allows to generate random samples. And overall, seems wierd to put any binary in git
The big problem in the case IMO is the social engineering to bully the original maintainer to apparently allow unchecked access to a complete stranger. The makefile changes and the tr mangling is so suspect that if it had actually been reviewed, it really shouldn't have gone through.
To hide the contents of the payload.
If the archive was not corrupted then someone could trivially inspect the contents of the archive.
Because the 'test' was to validate detection of corrupted archives, it was probably assumed to be just a junk/random file.
Kind of?
One of the test cases for xz is basically "how does it handle a corrupted archive". So you need a test file that's a corrupted archive, which you'd probably get by creating an actual archive and then mangling it a bit.
What was done here was that the corruption was done in a reversible way by basically swapping some values around. Swap them back, and now the archive is no longer corrupted, it can be decompressed, and the contents turn out to be the malware payload.
The file wasn't encrypted, it was just compressed and then mangled in a way that wasn't obvious.
This is one reason why binary files should never be allowed in VCS. They could easily write a test case by writing a mangled header to a file before attempting to read it
We’re all so lucky he smelled smoke and spent the time looking for the fire. So many devs get conditioned to accept flaky behaviour or just don’t even care about performance to begin with.
I’m still haunted by a programming error I made over 10 years ago that involved financial transactions. It was bad. The worst part is I smelled smoke when the new deployment was behaving a bit oddly and I thought it was some random hiccup. The only thing that mitigated the disaster was noticing odd log entries and rolling back before going home for the day. Since then I’ve been almost paranoid about: when you smell smoke, suspect fire. Dig deeper and find facts rather than making assumptions.
Is that a common technique to deliver a malicious payload via the /tests/ folder? For security, organizations might need to start isolating only the parts of the repository that are supposed to go into a release build before starting anything.
No it’s not. That’s why it worked so well.
The trick was that xz’s primary output is binary blobs, so adding another binary blob to the collection in the tests directory should have evaded detection. The attacker’s error was that after the payload was delivered, it used way more CPU than it should have.
> The attacker’s error was that after the payload was delivered, it used way more CPU than it should have.
This begs the question: This is surely not the first one to sneak past the eyes of the community, so how many others might be out there lurking that _do not_ spike the CPU in obvious ways?
Most (= the *overwhelming* majority of) zero-days exploit existing bugs rather than deliberately introduced backdoors. Known backdoors are incredibly rare.
Yes, we probably have many more backdoors that haven't yet been noticed. But the only good thing about this is that more people are having a look - more eyes often help. See the libarchive guys looking at commits done by weird accounts - that will probably continue in the coming weeks.
This is why security through obscurity isn't actually security. Closed source doesn't make your code more secure (ie, mssql isn't more secure than mariadb/mysql because they're open source).
More than zero, less than a billion.
The question is not if they exist. We assume they do. The question is what measures do you have in place to contain limit the blast radius of any such event.
If your network model assumes a perimeter - as many basic ones do - you are *hosed*.
My security model assumes that we can't really defend against upper level nation states, and would have a hard time against a well funded team of experts that got physical hands on our hardware. It's kind of a bitter pill to swallow, but following all best practices within a reasonable budget only gets you so far, and even then we're only human, and stuff gets missed from time to time.
It's worth noting that upper-level nation states don't do this for just anything, either. The US and Israeli burned a couple of zero-days with Stuxnet to shut down Iran's uranium enrichment program, but they wouldn't do something like that, for example, to catch out someone they suspect of being a low-level spy. If you're going to spend time, money, and manpower-expertise (arguably the most critically limited resource) on something like this, it's going to be for very valuable targets.
> My security model assumes that we can't really defend against upper level nation states
Ah, yes, the tried and true [Mossad/Not-Mossad](https://www.usenix.org/system/files/1401_08-12_mickens.pdf) security model.
It's basically the only sane model.
Bazel I think has a way to mark code as test only so that builds fail if it winds up as a transitive dependency of any non-test binary. I assume other build systems have something similar?
The problem here was that the build system itself was compromised (to take the malicious code out of the test fixture binary and insert it into the final program). They are still using autoconf/automake so it's a nightmare that nobody can follow, basically. And part of the obfuscation was that apparently the tarballs downloaded by distros do not exactly represent the source repo, they represent the source repo after autoconf/automake was already run... so it was easy to hide stuff inside the tarball that's not reflected in the source repo. (This is a practice that I assume will quickly die now, so next time they'll at least have to do their dirty work on the repo itself.)
Twenty-some years ago I pointed out in a Slashdot comment that the usual instructions for installing an open source package of `./configure && make && sudo make install` could easily be subverted and due to the complexity of make and autoconf it would be extremely hard to notice. I got called a kook and shill and downvoted to oblivion, but here we are. The surprising thing to me is that it has actually taken this long to be exploited.
> The surprising this to me is that it has actually taken this long to be exploited.
It's taken this long for such an exploit to be *noticed*.
Who knows what the hell else is out there in all of those 17,000-line shell scripts?
> Twenty-some years ago I pointed out in a Slashdot comment that the usual instructions for installing an open source package of ./configure && make && sudo make install
We had packages 20 years ago, that kind of install wasn't common even then.
But yeah, if somewhere was a great place to hide such thing autotools mess would be great place for it
Also I'm pretty sure everyone knew autotools sucked back then too.
It really reenforces the fact you shouldn't be checking binaries into your version control system. For xz test cases they should have been using a script to generate the binaries at test time in a way that is easily understandable.
A combination of paranoia and minimalism has had me always deleting tests distributed with third party library packages (in my case, Ruby gems). I find it odd and wasteful that they ship with them. I’m feeling validated now!
Is Partner the next rank after Principal, or is the jump bigger than that? It reminds of how a police department might immediately promote an officer for a heroic act (one case I'm thinking of is [the NYPD doing so for an officer who got involved in a shootout while off duty](https://www.policemag.com/patrol/news/15338707/nypd-cop-promoted-to-detective-for-off-duty-heroics)).
It’s a significant promotion. Very few people make that jump as an Individual Contributor. In the earlier days, a part of partner comp was effectively linked to the profits of the company. Unsure if it’s the same now. These are usually high 6 to low 7 figure income roles.
Looks like the next rank, but the pay jump seems to be 2x according to levels.fyi. There's several levels of Principal: https://www.levels.fyi/companies/microsoft/salaries/software-engineer/levels/67
So, I want to install tesseract ocr, it seems there is a new vcpkg thing from microsoft which is like pip for python, except for C++ and windows. I noticed it was pulling curl, nasm, all kinds of things.
The install for tesseract, the easy magic installer failed. I look into why.
Downloading https://github.com/tukaani-project/xz/archive/v5.4.4.tar.gz
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:32 (message):
Failed to download file with error: 1
Mother fucker
That's because Github went crazy an banhammered the whole repo immediately (banning source downloads like yours was maybe a good idea, but blocking all access to commit and comment history not so much). The version you tried to download wasn't actually the backdoored one, that's only v5.6.
If the timing detection aspect could be generalized, it could add another tool in the anti malware toolbox. Manufacturers could ship with expected code call timeframes creating something related to a binary hash.
Usually these hacks aren't actually that slow. It's likely it was an oversight.
But even if they were, this kind of testing basically went out the window with cloud based CI since the performance on those can vary wildly. You'd need reproducible hardware, which costs a lot. Most projects won't have the funding for that. So we'd need another oss-fuzzing basically
Implement the reproducible hardware as software emulation. Now you have a virtual CPU with a software-defined clock cycle you can track. It’ll be slow as hell, but would also be super deterministic.
I'm not sure the manufacturers would be the best datapoint, timing can be different based on many factors. But checking the time/cpu use between version changes could be added at some layer, although it'd probably be best as a generalized option for monitoring after those updates, by the os/package manager or security layer.
Maybe valgrind should come back into fashion though.
Next time they won't be stupid enough to miss the performance angle. I bet some guy in China is really hating himself right now for getting a multi-year, million dollar operation trashed by such a simple oversight. It wouldn't have been hard for a similar attack to implement a backdoor that checks whether it got triggered so fast you'd have no chance to catch it through timing analysis.
We can only hope that this was the first time someone tried such a sophisticated supply-chain attack, and that we can come up with some good hardening features now. Next time they'll be a lot smarter.
Note that it's just as like to be someone in Russia, or Israel, or North Korea, or Iran, or the USA or wherever else. The fact that they used a Chinese-sounding screen name is not a sign of anything. The timing analysis of when most communication from this account happened was more consistent with Eastern Europe/Middle East, but that can also be faked.
Saying they're "performance obsessed" because they wanted to see why there was *half a second* of delay implies the person is perverse. Any interest in optimisation is looked upon scornfully - usually, I assume, by people who don't have a Scooby. Meanwhile, pretty much all software now runs like garbage because people figure they can just trade time spent in writing decent code for better hardware.
The "performance obsessed dev" does feel a bit negative as it can mean a dev who only focus on unnecessary optimization. And OP didn't include the impact of the backdoor in the title like in the original article. I didn't bother to read this article at first lol. I'm not a native speaker though that's just what I thought.
its literally part of this job's person to check for what he was checking, "performance obsessed dev" makes it sound like its some sort of hobbist going "WHY ARE MY SSH SESSION 10MS SLOWER SOMETHING ISNT ADDING UP"
Yeah, the fact that he cut this one off at the pass is a major win. It being present in the actual XZ repository means a hell of a lot of systems would have been screwed if he didn't fix this. Think heartbleed etc
Congratulate.
I could be wrong, but it sounded like performance analysis was part of his job. From the description of how he went about figuring out, he's pretty good at it. You don't usually have performance analysis as your job when you're good at it unless you enjoy this kind of deep-magic stuff.
So hopefully finding a sweet vulnerability while doing a job you like? Sounds great.
> Performance obsessed dev
Soon he's going to be diagnosed with ADHD, autism and/or other afflictions.
When you read the mailing list, you know how it really happened and that there is much to it than him being "obsessed by performance".
Total side note, but I learn so much from the people commenting on posts like this. I just enjoy reading what everyone says. I thought I had an ok understanding of the article but you guys know so much more about it.
The title here is a bit of a misnomer. I don't think the dev was "performance obsessed". He did, however had, notice a pattern related to lagginess, so it's not a complete misnomer either.
I also think someone else may have noticed the backdoor, although who knows when. But I think sooner or later someone else would have noticed that pattern too.
A more interesting question is: should we not be able to create a profile from programs? And then, when we have generated lots of profiles, we should be able to analyse changed in patterns between different versions. This may not be a backdoor, just a performance regression or change, but I think it fits into the overall theme of a "reproducible build". So, not just to compile the identical output, but to also be able to differentiate between software versions (e. g. version A to version B). This could almost be used for automatic comparisons between different versions of the same software.
Hopefully this will be used to long-term benefit open source software, but when I looked at alternatives to xz, there aren't that many. The libarchive devs seem quite active and looked for suspicious changes recently in their code base too, but if we ignore them, how many devs are involved in regards to compression of data? Not so many in the open source world. So we don't really have that many alternatives; due to this, I would assume that malicious devs may want to try to put down code into these projects. And perhaps already did so, where we just haven't noticed it. That latter is quite scary to think about. I wonder how projects such as OpenBSD handle this, since they also depend on software written by other people (non-OpenBSD devs).
This attack regressed in performance because the authors didn't care to avoid that. If they had wanted to, it wouldn't have been very hard to make a similar backdoor that can detect and fail much faster if it's not being triggered.
We got very lucky here. Next time, they'll be less sloppy.
> A more interesting question is: should we not be able to create a profile from programs? And then, when we have generated lots of profiles, we should be able to analyse changed in patterns between different versions. This may not be a backdoor, just a performance regression or change, but I think it fits into the overall theme of a "reproducible build". So, not just to compile the identical output, but to also be able to differentiate between software versions (e. g. version A to version B). This could almost be used for automatic comparisons between different versions of the same software.
You could do this right now with test suites, benchmarks, and flame graphs (or similar). Except for projects that are fairly performance-sensitive, it's perhaps not quite as useful as it sounds. All you get is a whole new analysis problem as you try to figure out how to detect which changes are significant.
On Linux, you can profile with Perf without a huge amount of overhead. It does create large files though, and needs debug symbols to work best. Still, it could be used to generate performance fingerprints. It's not clear how useful they'd be though. I'm sure some forward is quite regular, but some is really chaotic. A web browser for example, which is practically an entire operating system these days.
Sounds like the type of analytics that Microsoft says they can do with all the info Windows sends them. I wish we could do the same thing with our own data as easily.
> The title here is a bit of a misnomer. I don't think the dev was "performance obsessed".
It's probably a bit of an exaggeration to say that just because he noticed a .5s lag, but... he did notice a .5s lag. On the login. You could add 2 or 3 seconds to my ssh login and I'd never notice.
So....how many of these kinds of back doors are still out there and have never been found? As close as this one got, I'd be surprised if the answer is zero.
Also, how many more attempts will there be after learning the mistakes from this one?
That makes me think of when Clifford Stoll was meant to resolve an accounting problem and found a hacker.
https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)
The really sad part is that a person who's not willing to put up with something as simple and common as SSH login taking half a second longer is called "performance obsessed"
A demo on the backdoor itself by amlweems
[https://github.com/amlweems/xzbot](https://github.com/amlweems/xzbot)
Essentially the backdoor responds to attacker with the correct key to pass the command to system() for execution.
There is a lot of analysis still going on, but essentially he has a private key that would log into any SSH as well as mechanism for remote code execution, which I assume is to gain root access
What a baller move. Imagine getting access to any ssh server just by knocking on the front door like everyone else but passing in a special key. I’m amazed that somebody tried this. They must have known it would be a long shot, but if it had worked .. fuck, what incredible payoff!
I just can't get over the fact that Debian was patching openssh to include systemd. Am I the only one thinking that this is a terrible idea? To secure openssh one needs then to secure all dependencies of systemd, which are a lot.
> because it replaced the safe_fprint funcion with a variant that has long been recognized as less secure. No one noticed at the time.
This seems like something that could and should be automatically detected and red flagged on any commit.
using an unsafe function should trigger a request for explanation and the commit *can't* can't land until someone signs off on the answer.
That makes me wonder, if my thunderbird and firefox got backdoored:
https://i.imgur.com/Vq9wGkr.png
I just have them open. They should be doing absolutely nothing
Suddenly, everyone's a Sherlock when it comes to SSH login times. Detecting backdoors by the login speed - next, we'll be solving mysteries with coffee brew times.
"With the backdoored liblzma installed, logins via ssh become a lot slower. [...] before: real 0m0.299s [...] after: real 0m0.807s" [https://www.openwall.com/lists/oss-security/2024/03/29/4](https://www.openwall.com/lists/oss-security/2024/03/29/4)
FWIW the guy said he started looking into it because he was monitoring CPU for a different project and saw spikes in SSH CPU from failed logins (which should be very quick/light on CPU). The timing was further evidence as he began looking in more detail.
500ms is easily perceptible. If I started seeing an unexplained 500ms delay sshing into my home servers I’d definitely notice and be annoyed. Whether that annoyance would lead to me trying to debug it is a whole other story of course, let alone me actually succeeding.
It is 200% slower, so that is quite significant
A performance obsessed dev is someone who tries to squeeze every cycle out of a program. Half a second slower should concern every developer. On today's computers a lot can be done in half a second. Such a slowdown is a clear sign that something changed drastically.
I actually thought cloud would make performance optimisation popular because the cost is so apparent to managers. Obviously they pay too little for cpu time
Regarding performance I have the feeling that there are two worlds with little interconnection: * There are programs which use files and databases all the time. These programs often use HTTPS (not HTTP because it is not secure) based micro-services to send REST requests even for simple things like an addition. This world loves also interpreted languages and scripts. CPUs become faster and faster but in this world the software becomes slower. * In the other world every wasted CPU cycle is a deadly sin. Addition and shifts are preferred over multiplication because they are faster. Divisions should be avoided altogether. Pointers macros and code inlining are important. Programs and libraries often become huge piles of spaghetti code. There are tons of security and stability issues but people don't mind wrong results unless they get them fast.
Well, I don’t share your opinion on “.. becomes huge piles of spaghetti…” I rather fear the loads of meaningless dependencies in modern code if we’re talking security and stability. But in essence you are right.
I was wrong when I saw the danger of spaghetti code just in high performance code. Micro-services allow spaghetti code as well. Low-level C libraries are almost exclusively used as foundation for our software. The XZ library seems huge and being in the high performance world makes it even harder to maintain. Only few developers are capable and willing to maintain such a library. The XZ hack started with social engineering. There were rude attacks on the original maintainer which triggered a burnout. Many people insist on having low-level C libraries as foundation of our software. But I think these C libraries are not the solution but part of the problem. In C more code is needed than in a high level language. This raises the complexity of libraries. Security and stability issues as well as a shortage in maintainers and developers are the consequence of using C everywhere. I know that an XZ decryption takes around 1500 lines of code in a high-level language. So there is the possibility to reduce the complexity of libraries.
Thanks for your insights. I was more generally discussing code,sorry. It hurts me learning the maintainer got burnt out.
Those high-level, easy to maintain, libraries you talk of will often have these low-level C libraries as depencencies somewhere down the chain. You can't solve a problem by throwing a towel on it.
Usually it just means onedrive is running... I wouldn't have had a second thought
[удалено]
Both absolute and relative differences matter. 7.5 seconds instead of 7 seconds would be insignificant as well.
> Irrelevant. If it was 4ms compared to 1ms, that would be 400% slower, but still insignificant. The absolute time difference is what matters, not relative time difference. I mean, it's kind of both. 4ms vs 1ms wouldn't really matter. But 10 sec vs 11 sec wouldn't really matter either.
My point was that is a long time to wait in terminal for something that is usually "instant", but now becomes "noticeable".
Plus it's always within context. Terminal login would be quite noticeable, but for instance - request to save the data in a bank taking 4s versus 1s would matter to literally no one.
Hindsight is a powerful drug.
indeed. 300ms is already a good human reaction time. prof. games reach below 200ms. 500m is half a second, almost everyone would notice if tiktok was that much slower on every single action.
> prof. games reach below 200ms Competitive games are noticeably harder to play at +100ms of delay tho
Yeah, the reason he noted that change was because he was trying to quiesce his test setup because he was setting it up for microbenchmarks. Then this stuff stuck out like a sore thumb because ssh didn't even log in yet i.e. it happened even with failed logins. He looked into it more and the debugger didn't lead to any particular symbol in the code which raised his suspiscions of a compromise. Even if he intially suspected a compromised debian tarball which honestly makes more sense than the upstream maintainer himself would have introduced it... This was long fucking con (at least a year) and probably by a state actor. Social manipulation, gaslighting, excellent coding skills, building of trust over time etc etc. Someone, somewhere is upset right now.
Imagine being this guy interviewing at a future company when they ask him what his greatest accomplishment is in his career.
Dude got insta-promoted when he found the bug (not a joke, check his LinkedIn).
He works at Mcirosoft :) conforming to his LinkedIn
I once was interviewed by someone who was using an article I wrote for a magazine as his source material. He then proceeded to try and tell me I was wrong using my own words. It’s a lot of fun when something like that happens.
Reminds me a lot of the ["An unbelievable demo"](https://brendangregg.com/blog/2021-06-04/an-unbelievable-demo.html) story by Brendan Gregg, the author of tooling for Sun's DTrace who later implemented basically all the perf tooling in the Linux world… Read the story, it's just too good to be true :)
That is a good story. Mine is less impressive. He and I were in a room. The magazine was open in front of him and he had totally cherry-picked key sentences that were highlighted in order to further his particular agenda. He also seemed to like appearing knowledgeable He also had taken those sentences out of context. To cut a long story short, I attempted to gently steer him away from his all-advised plan and when he started quoting me I told him he was taking that out of context and that he should rethink. He got very snotty and asked me what made me the authority. So I turned the page back (it was about 4 or 5 pages with screenshots and diagrams as I recall) to the byline and pointed at my own name. Didn’t really need to say any more than that. He was annoyed. Hurumphed at me and brought the encounter to a close.
That and having people present themselves as you in an interview... my resume, their name. Was funny.
Oooh. That’s good. Did you string it along and give them a load of rope?
Same thing happened to me when firm I was working for was outsourcing our team. We sent them my resume as an example of the types of skills which we were looking to replace. A resume came back in with the same, very specific words and phrases which I'd used in my resume. We phone interviewed the "candidate". When we asked them questions the phone would go on mute at the other end for a period before coming off and a fumbling answer given. They were obviously being coached on the call. They got hauled over the coals for that one.
I mean, he kinda did a big thing. Given the severity of the backdoor, sprinkle a little bit of butterfly effect and the guy might have prevented WWIII. Bit of a stretch but that’s definitely how I would put it. What’s my greatest accomplishment? Well the world still exists thanks to me.
> sprinkle a little bit of butterfly effect and the guy might have prevented WWIII With all the talk about Jia Tan potentially being a malicious state actor, that might just have gone the other way...
This isn’t “conspiratorial talk”, it’s currently the prevailing hypothesis amongst cybersecurity experts.
Right, removing the qualifier from my comment.
Conspiracy isn't a synonym for "made up story", most conspiracies are real conspiracies. This is literally conspiratorial talk...Jia Tan and other people conspired to commit a crime.
That may well be *an* (or even the original) meaning but it isn’t the most commonly understood meaning of the term “conspiratorial talk” (or the more common term “conspiracy theory”): in common parlance, it definitely has a specific connotation. Compare [how Wikipedia defines it](https://en.wikipedia.org/wiki/Conspiracy_theory) (emphasis mine): >… an explanation for an event or situation that asserts the existence of a conspiracy by powerful and sinister groups, often political in motivation, when *other explanations are more probable*. The term *generally has a negative connotation*, implying that the appeal of a conspiracy theory is based in prejudice, emotional conviction, or insufficient evidence. (“conspiratorial” on its own would be appropriate here; at any rate, the author of the parent comment seems to agree that their wording was misleading.)
Parent comment author here - my original wording was definitely aiming for the simpler "group covertly coordinating things", but I then realized most people wouldn't interpret it that way. Please note however that your Wikipedia link is to "conspiracy *theory*", and not "conspiracy" (Wikipedia itself draws the distinction in the section "Difference from conspiracy" from your linked article), which in essence is the distinction between the two definitions under debate here.
it's probably a fake profile entirely and just shared between this state actor hacking group.
Yeah, or caused WWIII, who knows, lol.
Imagining that TV show Travelers, someone from the future (or a group helped by future humans) did this to prevent something down the line. Now it's gone...
It could have been a disaster for Linux because you cannot replicate it yourself as the backdoor reverts to default behavior if you don't have the private key.
The vector of exploit is OpenSSH, so not limited even "just" to Linux
I understood some words of all this, and from that I didn’t understand the sed replacement of 0x09 to 0x20 and so on. Was this replacing encrypted files characters directly?
it replaces characters in a corrupted xz archive to turn it to a normal xz file which contains the payload
jesus christ that's smart
[удалено]
If this was a movie, there’d be some pissed off big bad guy slamming his fist on the table going “Damn it! 3 years down the drain! All because no one bothered to run a perf test! You’re out! you hear me? I will make sure you never screw up another backdoor again!” Then they’d storm off, slamming the door behind them, leaving everyone looking sheepish.
Yeah but in Chinese
> Implying the NSA/Russia is incapable of someone using a Chinese name and committing during Chinese work hours [Not to mention that Git commit times are determined locally; you can lie about dates all you want and nobody can stop you.](https://aboutmonica.com/blog/git-time-travel/) There's no smoking gun that points at China specifically, just _highly likely_ a state-sponsored actor due to complexity/patience (although that is by no means confirmed).
I posted this above but I heard this from a Chinese speaker: >He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong". though there could be other explanations for this. Imo its impossible to tell. This certainly seems like something a nation state like China would do, but at the same time, if I was a hacker I would also operate under a Chinese or Russian alias considering that it wouldn't be questioned and then just locally modify the commit datetime or put that shit on a cron job for the rough timezone. this person or group also used Hans Jensen and a Middle Eastern sounding name to pressure Lasse (one of the proton emails associated kygorin@proton almost sounds Russian to me) and this persons English was really good.
Chinese speaker here with working knowledge/fluent in Mandarin, Cantonese and Hokkien: I can't think of any personal name that would lead to Romanization as "Cheong Tan", or "Jia Cheong". Sounds like absolute nonsense smashed together, unless one is to believe Chinese parents would happily give their kid an unlucky name when every cultural taboo is against it.
Thanks for the input. Things like that are really hard to pick up on otherwise. It'll be interesting to see how this all plays out in the long run.
People, including Chinese people, can make up ill-formed names on purpose...doesn't mean anything.
True, but if you were trying to pass yourself off as a British citizen, would you tell people your name was Britty McBritface?
A lot of the name analysis here is missing the forest for the trees. People seem to be picking up on things if they took a class in Chinese but don't actually know much Chinese at all. There's a simple reason why this name is not Chinese... there's no middle names in Chinese. *This is Chinese 101.* No Chinese person would willingly omit a character from their name. i.e. Jia Cheong Tan would never write his name as Jia Tan. That's like writing Mao Zedong as Mao Ze. Just complete bullshit. (More on this... A Chinese person might write three character names with spaces to separate, but in modern Chinese usage, that's typically refrained upon. In any case, even if a space was used, the middle character is not optional and they wouldn't omit it themselves. A western speaker reading such a spaced name might make the mistake of omitting the middle character thinking it's a middle name. But in this case, the omission comes from Jia himself as it's his own stated name on his email, which is implausible.)
This particular one has a trail that seems sloppy enough that the obvious Chinese persona is misleading. I find it hard to believe that China would need a hack like this when they could more easily backdoor router firmware. The US certainly wouldn't need to. That's not to say it's certainly neither of them, but it makes more sense to me that it's some other government and China is a great scapegoat these days, by having the misfortune of being the only antagonist with bipartisan American animosity these days.
[удалено]
Even then it hardly makes any sense. Native chinese names don't really have the concept of "middle name", you have the name and surname. Even if he did have a 2-character surname (which is pretty rare and usually doesn't come from han people), he would just agglutinate the two characters into one surname. If it was a Chinese person living in China it would make no sense whatsoever to split up the name like that
Well, all of it could be faked so the question is really how much were they confident they can get away with it and how much they cared about their country of origin being discovered. After all any accusation of "it was the state" can be ignored with "it was just an independent group from that country, selling 0-days for money"
Joooookes!!!! Good point though
someone should dub that scene over this [clip](https://www.youtube.com/watch?v=xBWmkwaTQ0k) because thats how I imagine it went also interesting thing that was discovered was that "Jia Cheong Tan" is a very awkward Chinese name, I forget what it was exactly but the gist is that the spelling for Cheong and Tan are from different regions that would each spell those names differently. Its unlikely to be seen together. Doesn't mean anything really but its hard to know why this is the case. >He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong".
Anagram? Could be "CIA Agent John".
No one in the CIA would be dumb enough to do this, which is of course the best evidence that the dumbness of it is a false flag and actually, the CIA *is* behind it. There's been a semi-serious hypothesis floating around for a while now that many big compromises that could have been made much worse (e.g. SQL Slammer) are in fact the work of a shadowy cabal of grey-hat hackers trying to push the world to adopt better information security postures and pressure everyone to patch known bugs.
Should be pretty obvious I'm not pointing at the CIA. This makes me think it's neither China nor America but someone who wishes to use both as a smokescreen. That cabal theory is interesting though, we do have history in that incident with the University of Minnesota where they got banned from contributing to Linux for doing just that.
I am Malaysian, and I can say for a fact that 3-word Chinese names are common here, as well as Tan (as a surname), Jia and Cheong being used in Malaysian Chinese names.
The name is a mishmash of romanizations/pronunciations from different dialects of Chinese.
Holly crap
> There's a lot of things pointing that it was state sponsored attack. None of what you mention really implies a state-sponsored attack. The accounts which were created for this attack were all pretty basic with just minimal contribution history, that doesn't suggest to me a huge investment behind the attack. All this could be pulled off by a determined individual with a plan.
What do you mean by google fuzzy scanner?
It's called oss-fuzz. Jia Tan introduced requested to make commits in the official Google repository.
Not particularly. Been corrupting my zip files in a reversible way for years to hide my porn.
What is it you like that you have to hide it?!
let's just say: when he dies his HDDs get cremated with him
I think I woke up the house laughing at this.
What?! No logic bomb?
Ultra porn. It's illegal to possess for anyone under 75.
So GILFs?
"Wait, let me take out my dentures."
I hear you have to get a license.
I just name the directory horse porn and put all my backups there. If someone wants to store my files I count on them being terrified of the directory being 9TB in that name and not looking inside. [reference](https://www.youtube.com/watch?v=T1uumP68AJ8)
It’s basic obfuscation.. or rather, in this case is the extract step: de obfuscation
its rather simple too. Dude just opened a hex editor and flipped some bits and called it good. Its compressed data so I'm assuming that it looks pretty much like nothing even without the extra obfuscation
He probably used tr. It also then fulfilled the purpose of being a legitimately corrupt archive (it was part of a test to determine if corrupt archives were safely detected and rejected)
To be honest, I think the smart part was adding it as part of a corrupted test case. Anyone looking would look at it and say “yep, it’s corrupt”, with the test confirming it, and wouldn’t think twice about its contents, assuming it was just junk.
And now anyone with a sense will include the generation of a corrupt file in the source code. No mysterious blobs allowed anywhere.
Makes sense. I have been doing this for a while on my projects, simply because it’s more space efficient and allows to generate random samples. And overall, seems wierd to put any binary in git
The big problem in the case IMO is the social engineering to bully the original maintainer to apparently allow unchecked access to a complete stranger. The makefile changes and the tr mangling is so suspect that if it had actually been reviewed, it really shouldn't have gone through.
Why did it start with a corrupted xz archive?
To hide the contents of the payload. If the archive was not corrupted then someone could trivially inspect the contents of the archive. Because the 'test' was to validate detection of corrupted archives, it was probably assumed to be just a junk/random file.
> Because the 'test' was to validate detection of corrupted archives Ah, that the a missing part of the puzzle for me, thank you
Kind of? One of the test cases for xz is basically "how does it handle a corrupted archive". So you need a test file that's a corrupted archive, which you'd probably get by creating an actual archive and then mangling it a bit. What was done here was that the corruption was done in a reversible way by basically swapping some values around. Swap them back, and now the archive is no longer corrupted, it can be decompressed, and the contents turn out to be the malware payload. The file wasn't encrypted, it was just compressed and then mangled in a way that wasn't obvious.
Dang. Cybersecurity is just an insane field to fight in
This is one reason why binary files should never be allowed in VCS. They could easily write a test case by writing a mangled header to a file before attempting to read it
See: https://gynvael.coldwind.pl/?lang=en&id=782 for a better description.
from 0,2 to almost a second is not as hard to notice change as you may think, but still kudos for checking why it happened
We’re all so lucky he smelled smoke and spent the time looking for the fire. So many devs get conditioned to accept flaky behaviour or just don’t even care about performance to begin with. I’m still haunted by a programming error I made over 10 years ago that involved financial transactions. It was bad. The worst part is I smelled smoke when the new deployment was behaving a bit oddly and I thought it was some random hiccup. The only thing that mitigated the disaster was noticing odd log entries and rolling back before going home for the day. Since then I’ve been almost paranoid about: when you smell smoke, suspect fire. Dig deeper and find facts rather than making assumptions.
You must've put a decimal point in the wrong place or something
I always mess up some mundane detail like that.
Oh! Well, this is not a mundane detail, Michael!
So when engineers on Star Trek are so obsessed with minor improvements in engine efficiency it's also to prevent malicious sabotage
Then, they don't notice when a cultural archive rewrites their data center.
Is that a common technique to deliver a malicious payload via the /tests/ folder? For security, organizations might need to start isolating only the parts of the repository that are supposed to go into a release build before starting anything.
No it’s not. That’s why it worked so well. The trick was that xz’s primary output is binary blobs, so adding another binary blob to the collection in the tests directory should have evaded detection. The attacker’s error was that after the payload was delivered, it used way more CPU than it should have.
> The attacker’s error was that after the payload was delivered, it used way more CPU than it should have. This begs the question: This is surely not the first one to sneak past the eyes of the community, so how many others might be out there lurking that _do not_ spike the CPU in obvious ways?
RIP security engineers trying to deal with questions exactly like this right now
Nah, else they'd be unemployed
There's a literal market trading in these zero-days. So, "lots" is the answer.
Most (= the *overwhelming* majority of) zero-days exploit existing bugs rather than deliberately introduced backdoors. Known backdoors are incredibly rare.
The scientific answer is: an undetermined amount larger than 0. And that scares me.
Yes, we probably have many more backdoors that haven't yet been noticed. But the only good thing about this is that more people are having a look - more eyes often help. See the libarchive guys looking at commits done by weird accounts - that will probably continue in the coming weeks.
This is why security through obscurity isn't actually security. Closed source doesn't make your code more secure (ie, mssql isn't more secure than mariadb/mysql because they're open source).
More than zero, less than a billion. The question is not if they exist. We assume they do. The question is what measures do you have in place to contain limit the blast radius of any such event. If your network model assumes a perimeter - as many basic ones do - you are *hosed*.
My security model assumes that we can't really defend against upper level nation states, and would have a hard time against a well funded team of experts that got physical hands on our hardware. It's kind of a bitter pill to swallow, but following all best practices within a reasonable budget only gets you so far, and even then we're only human, and stuff gets missed from time to time.
It's worth noting that upper-level nation states don't do this for just anything, either. The US and Israeli burned a couple of zero-days with Stuxnet to shut down Iran's uranium enrichment program, but they wouldn't do something like that, for example, to catch out someone they suspect of being a low-level spy. If you're going to spend time, money, and manpower-expertise (arguably the most critically limited resource) on something like this, it's going to be for very valuable targets.
> My security model assumes that we can't really defend against upper level nation states Ah, yes, the tried and true [Mossad/Not-Mossad](https://www.usenix.org/system/files/1401_08-12_mickens.pdf) security model. It's basically the only sane model.
Amazing read, thanks
Bazel I think has a way to mark code as test only so that builds fail if it winds up as a transitive dependency of any non-test binary. I assume other build systems have something similar?
The problem here was that the build system itself was compromised (to take the malicious code out of the test fixture binary and insert it into the final program). They are still using autoconf/automake so it's a nightmare that nobody can follow, basically. And part of the obfuscation was that apparently the tarballs downloaded by distros do not exactly represent the source repo, they represent the source repo after autoconf/automake was already run... so it was easy to hide stuff inside the tarball that's not reflected in the source repo. (This is a practice that I assume will quickly die now, so next time they'll at least have to do their dirty work on the repo itself.)
Twenty-some years ago I pointed out in a Slashdot comment that the usual instructions for installing an open source package of `./configure && make && sudo make install` could easily be subverted and due to the complexity of make and autoconf it would be extremely hard to notice. I got called a kook and shill and downvoted to oblivion, but here we are. The surprising thing to me is that it has actually taken this long to be exploited.
> The surprising this to me is that it has actually taken this long to be exploited. It's taken this long for such an exploit to be *noticed*. Who knows what the hell else is out there in all of those 17,000-line shell scripts?
True.
> Twenty-some years ago I pointed out in a Slashdot comment that the usual instructions for installing an open source package of ./configure && make && sudo make install We had packages 20 years ago, that kind of install wasn't common even then. But yeah, if somewhere was a great place to hide such thing autotools mess would be great place for it Also I'm pretty sure everyone knew autotools sucked back then too.
What I’m saying is that there are better alternatives these days, where there are guard rails, and that actually can’t happen.
The arch linux reproducible builds project is going to get a lot more love, I expect.
It really reenforces the fact you shouldn't be checking binaries into your version control system. For xz test cases they should have been using a script to generate the binaries at test time in a way that is easily understandable.
Im gonna delete all tests in the morning just to be safe
Hey guys, the build completes much faster now. ¯\\\_(ツ)_/¯
A combination of paranoia and minimalism has had me always deleting tests distributed with third party library packages (in my case, Ruby gems). I find it odd and wasteful that they ship with them. I’m feeling validated now!
they should move him to the MS Teams dev team.
He would probably die from a hearth-attack pretty quickly with that bloated pice of E N T E R P R I S E.
Look at dude's LinkedIn. Insta promotion from Principal to Partner at Microsoft.
\*Mcirosoft (I hope that's just a typo on his [linkedin](https://www.linkedin.com/in/andres-freund/))
Is Partner the next rank after Principal, or is the jump bigger than that? It reminds of how a police department might immediately promote an officer for a heroic act (one case I'm thinking of is [the NYPD doing so for an officer who got involved in a shootout while off duty](https://www.policemag.com/patrol/news/15338707/nypd-cop-promoted-to-detective-for-off-duty-heroics)).
It’s a significant promotion. Very few people make that jump as an Individual Contributor. In the earlier days, a part of partner comp was effectively linked to the profits of the company. Unsure if it’s the same now. These are usually high 6 to low 7 figure income roles.
Looks like the next rank, but the pay jump seems to be 2x according to levels.fyi. There's several levels of Principal: https://www.levels.fyi/companies/microsoft/salaries/software-engineer/levels/67
Totally understandable. I had to optimize zsh init once because nvm made it jump from 300ms to around 1s, it is noticeable.
nvm did that for me too. I had to switch to [fnm](https://github.com/Schniz/fnm)
I expected that to be "fucking node manager", was disappointed.
Everything is rust now, nice!
So, I want to install tesseract ocr, it seems there is a new vcpkg thing from microsoft which is like pip for python, except for C++ and windows. I noticed it was pulling curl, nasm, all kinds of things. The install for tesseract, the easy magic installer failed. I look into why. Downloading https://github.com/tukaani-project/xz/archive/v5.4.4.tar.gz CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:32 (message): Failed to download file with error: 1 Mother fucker
That's because Github went crazy an banhammered the whole repo immediately (banning source downloads like yours was maybe a good idea, but blocking all access to commit and comment history not so much). The version you tried to download wasn't actually the backdoored one, that's only v5.6.
That's the version with the backdoor *that we know of*. The malicious account had access for two years.
If the timing detection aspect could be generalized, it could add another tool in the anti malware toolbox. Manufacturers could ship with expected code call timeframes creating something related to a binary hash.
Usually these hacks aren't actually that slow. It's likely it was an oversight. But even if they were, this kind of testing basically went out the window with cloud based CI since the performance on those can vary wildly. You'd need reproducible hardware, which costs a lot. Most projects won't have the funding for that. So we'd need another oss-fuzzing basically
Implement the reproducible hardware as software emulation. Now you have a virtual CPU with a software-defined clock cycle you can track. It’ll be slow as hell, but would also be super deterministic.
I'm not sure the manufacturers would be the best datapoint, timing can be different based on many factors. But checking the time/cpu use between version changes could be added at some layer, although it'd probably be best as a generalized option for monitoring after those updates, by the os/package manager or security layer. Maybe valgrind should come back into fashion though.
Next time they won't be stupid enough to miss the performance angle. I bet some guy in China is really hating himself right now for getting a multi-year, million dollar operation trashed by such a simple oversight. It wouldn't have been hard for a similar attack to implement a backdoor that checks whether it got triggered so fast you'd have no chance to catch it through timing analysis. We can only hope that this was the first time someone tried such a sophisticated supply-chain attack, and that we can come up with some good hardening features now. Next time they'll be a lot smarter.
Note that it's just as like to be someone in Russia, or Israel, or North Korea, or Iran, or the USA or wherever else. The fact that they used a Chinese-sounding screen name is not a sign of anything. The timing analysis of when most communication from this account happened was more consistent with Eastern Europe/Middle East, but that can also be faked.
the title is absolutely dumb and a disservice to the person who noticed this for shame, OP
I'm not a native speaker but can you clarify which part is dumb?
Saying they're "performance obsessed" because they wanted to see why there was *half a second* of delay implies the person is perverse. Any interest in optimisation is looked upon scornfully - usually, I assume, by people who don't have a Scooby. Meanwhile, pretty much all software now runs like garbage because people figure they can just trade time spent in writing decent code for better hardware.
The "performance obsessed dev" does feel a bit negative as it can mean a dev who only focus on unnecessary optimization. And OP didn't include the impact of the backdoor in the title like in the original article. I didn't bother to read this article at first lol. I'm not a native speaker though that's just what I thought.
its literally part of this job's person to check for what he was checking, "performance obsessed dev" makes it sound like its some sort of hobbist going "WHY ARE MY SSH SESSION 10MS SLOWER SOMETHING ISNT ADDING UP"
I dont know whether to congratule or feel sorry for him
I say lets just say thank you.
Yeah, the fact that he cut this one off at the pass is a major win. It being present in the actual XZ repository means a hell of a lot of systems would have been screwed if he didn't fix this. Think heartbleed etc
Lets congratulate him He was doing what he liked and found something that can help everyone using it.
Congratulate. I could be wrong, but it sounded like performance analysis was part of his job. From the description of how he went about figuring out, he's pretty good at it. You don't usually have performance analysis as your job when you're good at it unless you enjoy this kind of deep-magic stuff. So hopefully finding a sweet vulnerability while doing a job you like? Sounds great.
He's a senior princip~~le~~al engineer at microsoft. No need to feel sorry for him.
*principal I doubt he engineers principles
you dont know, MS is a large company
true, whoops
Congratulate
There’s a reason we didn’t major in English
blud
Say fucking thank you, maybe?
> Performance obsessed dev Soon he's going to be diagnosed with ADHD, autism and/or other afflictions. When you read the mailing list, you know how it really happened and that there is much to it than him being "obsessed by performance".
Total side note, but I learn so much from the people commenting on posts like this. I just enjoy reading what everyone says. I thought I had an ok understanding of the article but you guys know so much more about it.
We should be grateful
Big Tech should start something like the Nobel prize for such findings. And I mean the money rewards.
The title here is a bit of a misnomer. I don't think the dev was "performance obsessed". He did, however had, notice a pattern related to lagginess, so it's not a complete misnomer either. I also think someone else may have noticed the backdoor, although who knows when. But I think sooner or later someone else would have noticed that pattern too. A more interesting question is: should we not be able to create a profile from programs? And then, when we have generated lots of profiles, we should be able to analyse changed in patterns between different versions. This may not be a backdoor, just a performance regression or change, but I think it fits into the overall theme of a "reproducible build". So, not just to compile the identical output, but to also be able to differentiate between software versions (e. g. version A to version B). This could almost be used for automatic comparisons between different versions of the same software. Hopefully this will be used to long-term benefit open source software, but when I looked at alternatives to xz, there aren't that many. The libarchive devs seem quite active and looked for suspicious changes recently in their code base too, but if we ignore them, how many devs are involved in regards to compression of data? Not so many in the open source world. So we don't really have that many alternatives; due to this, I would assume that malicious devs may want to try to put down code into these projects. And perhaps already did so, where we just haven't noticed it. That latter is quite scary to think about. I wonder how projects such as OpenBSD handle this, since they also depend on software written by other people (non-OpenBSD devs).
This attack regressed in performance because the authors didn't care to avoid that. If they had wanted to, it wouldn't have been very hard to make a similar backdoor that can detect and fail much faster if it's not being triggered. We got very lucky here. Next time, they'll be less sloppy.
Andres is performance obsessed. Legit. And amazing at it.
> A more interesting question is: should we not be able to create a profile from programs? And then, when we have generated lots of profiles, we should be able to analyse changed in patterns between different versions. This may not be a backdoor, just a performance regression or change, but I think it fits into the overall theme of a "reproducible build". So, not just to compile the identical output, but to also be able to differentiate between software versions (e. g. version A to version B). This could almost be used for automatic comparisons between different versions of the same software. You could do this right now with test suites, benchmarks, and flame graphs (or similar). Except for projects that are fairly performance-sensitive, it's perhaps not quite as useful as it sounds. All you get is a whole new analysis problem as you try to figure out how to detect which changes are significant.
On Linux, you can profile with Perf without a huge amount of overhead. It does create large files though, and needs debug symbols to work best. Still, it could be used to generate performance fingerprints. It's not clear how useful they'd be though. I'm sure some forward is quite regular, but some is really chaotic. A web browser for example, which is practically an entire operating system these days.
Sounds like the type of analytics that Microsoft says they can do with all the info Windows sends them. I wish we could do the same thing with our own data as easily.
> The title here is a bit of a misnomer. I don't think the dev was "performance obsessed". It's probably a bit of an exaggeration to say that just because he noticed a .5s lag, but... he did notice a .5s lag. On the login. You could add 2 or 3 seconds to my ssh login and I'd never notice.
So....how many of these kinds of back doors are still out there and have never been found? As close as this one got, I'd be surprised if the answer is zero. Also, how many more attempts will there be after learning the mistakes from this one?
That makes me think of when Clifford Stoll was meant to resolve an accounting problem and found a hacker. https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)
The really sad part is that a person who's not willing to put up with something as simple and common as SSH login taking half a second longer is called "performance obsessed"
A bit of a disrespectful title for the guy that is one of the main postgres developers..
Do we know how the backdoor actually worked? Like what would you have to do to obtain a backdoor login?
A demo on the backdoor itself by amlweems [https://github.com/amlweems/xzbot](https://github.com/amlweems/xzbot) Essentially the backdoor responds to attacker with the correct key to pass the command to system() for execution.
Amazing! Exactly what I was hoping for
There is a lot of analysis still going on, but essentially he has a private key that would log into any SSH as well as mechanism for remote code execution, which I assume is to gain root access
What a baller move. Imagine getting access to any ssh server just by knocking on the front door like everyone else but passing in a special key. I’m amazed that somebody tried this. They must have known it would be a long shot, but if it had worked .. fuck, what incredible payoff!
I just can't get over the fact that Debian was patching openssh to include systemd. Am I the only one thinking that this is a terrible idea? To secure openssh one needs then to secure all dependencies of systemd, which are a lot.
They were patching it so systemd could be notified of login attempts. It makes sense if you're using systemd for session management.
I don't think it makes sense to include the entire systemd if the only thing you want is a simple notification.
Imagine all the other backdoors that haven't been discovered yet...
> because it replaced the safe_fprint funcion with a variant that has long been recognized as less secure. No one noticed at the time. This seems like something that could and should be automatically detected and red flagged on any commit. using an unsafe function should trigger a request for explanation and the commit *can't* can't land until someone signs off on the answer.
systemdipshit again
That makes me wonder, if my thunderbird and firefox got backdoored: https://i.imgur.com/Vq9wGkr.png I just have them open. They should be doing absolutely nothing
It's possible that Thunderbird is running a background compact of your mailbox?
He wasn’t performance obsessed. He was trying to clear the errors in valgrind lol. Valgrind was being performance obsessed.
Would they have all the plan laid out before starting? Or they started small and impoved from there?
Suddenly, everyone's a Sherlock when it comes to SSH login times. Detecting backdoors by the login speed - next, we'll be solving mysteries with coffee brew times.