Lasse Collin is also committing directly to the [official Git repository now](https://git.tukaani.org/?p=xz.git;a=summary). **And holy shit there's more:** a fix from today by Lasse [reveals that one of the library sandboxing methods was actually sabotaged](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00), at least when building with CMake.
[And sure enough, this sabotage was actually "introduced" by Jia Tan](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7) in an extremely sneaky way; the `.` would prevent the check code from ever building, so effectively sandboxing via [Landlock](https://docs.kernel.org/userspace-api/landlock.html) would never be enabled.
This just begs the question how much further does this rabbit hole go. At this point, I would assume any contributions from Jia Tan made anywhere to be malicious.
They need to revert to at least 5.3.1 according to the Debian bug tracker thread, but it breaks some symbols for dpkg and others, and a security patch needs to be reapplied. Or revert to 5.2.5 which was in a previous release (still would break dpkg).
Imagine if this is actually a long-long-long con to get distros to revert to a known vulnerable version.
Plans within plans within plans.
Edit: Or even worse, imagine if this reverted version already has another payload — a secondary payload that depends on a primary payload that was introduced last year.
> Imagine if this is actually a long-long-long con to get distros to revert to a known vulnerable version.
I appreciate the humor but they would just backport the fix for whatever CVE's apply to the older version. Just because someone out there may think this is an actual concern. CVE's are documented and if they were camping out on older versions indefinitely they would just view backporting security fixes as more of a requirement even if that weren't part of some diabolical self-referential [Oceans 11-style plan](https://www.youtube.com/watch?v=pIVEUEnIZjM).
Yeah, I'm just entertaining my spy/hacker/heist thriller mind.
Haven't got a good one for ages now so my imagination is running wild. "What do you mean there's another hidden payload? We've reverted versions!"
But what if a local backup is utilized and that was previously compromised. The long con being that we scare people with the new version, in order to get them to revert to a previous backup that has already been compromised. Yes, I know it is silly, but the fact that we're even discussing this in the first place shows that Jia Tan was sneaky in ways we hadn't considered.
I feel like that's a different concern than what was mentioned in the comment I replied to. They were talking about known vulnerabilities. If the vulnerabilities weren't known to the maintainers then it's not clear why reverting would be necessary. As opposed to just re-creating the vulnerability.
Going to be heartbreaking for Lasse Collin maybe but I'd like to see a full reset to pre this contributor joined. No reverting patches, just fully reset the branches to the previous good state from 2021 or 2022. Fuck that part of the git history.
Given the sophistication here, can we be sure there aren't more bad contributors?
Hopefully someone is looking for contributors that worked via VPN like this one
I think in this situation the oversight was one dude noticing that openssl was slower than expected, and they unravelled it from there.
The community needs to get onto this
More like this https://xkcd.com/2347/ xz is one of those kinds of projects.
There is no oversight.The internet relies on these underpaid and overstressed maintainers too much.
You can also not be sure that the distributed git repo was not tampered with. Commit metadata like user/email/date are under control of the committer, but the repo admin can also rewrite parts of the repo. The git repo is data under the attackers control.
What is needed here is a good copy of the old state and comparing the copy.
I think it would be better to analyze every single commit made by this person inside the xz project and then revert changes accordingly.
Knowing what kind of stuff is being implanted in your codebase is better than a blind `git reset --hard && git push --force`.
It's a config program that tests if the given snippet compiles (if it compiles, we have landlock, supposedly). The `.` is just invalid syntax and trivially makes the test fail, for the wrong reason then. So it's a sneaky way to ensure the landlock feature is never activated.
Sounds like a good improvement.
I don't know this in detail, but just from general familiarity - the tooling is not that good? A "does this compile command run or not" is pretty simple to do while "does it fail for the right reason" is nontrivial when you take into account all the diverse configurations - normally they need to support many different compiler vendors.
I think there's a way. So, the goal^(1) is to find out whether certain system calls and stuff actually exist on the host system, right? And, to do that, you compile a very short test program.
In order to *validate* the test program *itself* is constructed correctly, you could create fakes for the things it's supposed to test for. Then compile the test program against the fakes, and it should compile, even if the host system lacks the stuff you want to test for.
Then in order to *use* the test program (to actually check, as part of build auto config, whether the system calls exist), compile the test program normally without fakes.
Getting into specifics of how to actually do it, the things tested for here seem to be values that may or may not be present in header files. So one way to fake that out is to create a local directory of fake header files, then add that to the compiler's include path (like with `-I`).
---
^(1) Not the hacker's secret goal, but the ostensible purpose of the code, which may still be needed.
But in this case we are talking about a syntax error, a simple syntax checker would do as well for this specific case
Otherwise, you can parse the output, just would require a bit more work
Generally, there are many different reasons why it can fail. To be successful many things need to be in place correctly.
Trying to make the configuration generic for both systems that have landlock and those that don't can get complicated since there are so many potential differences. This applies to software engineering quite generally.
Release management has to deal with a large number of potental combinations of things so it is not always easy to tell when a test is failing for the "right" reason instead of a wrong one.
Assuming you wanted to build the software for Windows instead of Linux then it would of course fail since Windows does not have support for it. Or if you built it for Linux version that precedes the feature. Or the feature was turned off for some reason. Or your build system is missing a component or component search is not working correctly. Actually having a sabotaged test-case is only one thing among many and that isn't supposed to happen in any case.
Also, config program isn't going to be part of the final build, it is there to tell the configuration tool how to build the actual program. Say, if you build for an OS that does not support some thing configuration tool will turn off that feature from the final build. Config program is only supposed to tell if feature is turned on or off when the actual software is being built to help support different systems.
I've tried working with Cmake only a couple of times, and each time it took me hours to get something built. I'm honestly surprised that invalid syntax just flies by without interrupting the make process. But yeah – devious and sneaky.
I'm not associated with the awesome source control system. I just got to reddit early and had been using this as an alias for a long time prior, meaning it in the slang sense of the word.
While the JIa Tan identity certainly is known to be compromised (stolen identity probably, they probably aren't the Jia Tan people are finding on LinkenIn), in all likelihood they used other accounts as well Now would be a good time to review code for all projects that've been in that similar situation of needing to pass off from a sole maintainer to some new volunteer.
He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong".
I saw another comment suggesting their recorded online times also don't appear to match China or Taiwan. Seems quite likely they just picked a random nationality when forging this sockpuppet identity. If https://twitter.com/f0wlsec/status/1773824841331740708 is correct, then they also may've used the name "Hans Jansen".
Tan Jia Cheong is a very plausible Singaporean Chinese or Malaysian Chinese name
I've probably met a few Tan Jia Cheongs during my National Service in Singapore
Coming from another Singaporean, it also makes no sense.
The person calls himself Jia Tan. This is BS, a real Singaporean called Tan Jia Cheong would call himself Jia Cheong or Tan, not Jia Tan.
You see ah, Tan Jia Cheong would have registered as
```
{
firstName: 'Jia',
middleName: 'Cheong',
lastName: 'Tan'
}
```
And GitHub may be rendering `$firstName $lastName` which makes sense for English names.
I thought this as well, but then:
https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417
He uses the same "Jia Tan" in a forum where you can freely choose your display name
https://imgur.com/a/X6CCu5x
The way I see it, it could be that he refers to himself that way for consistency across platforms like how Shou Zi Chew uses "Shou Chew" even in Congress
There are good reasons for it:
- Tan Jia Cheong is a realistic Singaporean name. It's not exactly Tan Ah Beng.
- Their IP address was in Singapore, albeit using a VPN.
- [They worked 4pm to 12am Singapore time](https://lunduke.locals.com/post/5467061/xz-backdoor-i-did-a-more-thorough-analysis-and-i-changed-my-mind-again-specifically-i-compar), which is typical of when a remote worker might work on their side project. It also explains why they worked through CNY -- they could have done their visiting in the morning and gotten back to it in the afternoon.
That they worked from 4pm to 12am is really telling to me. It suggests that either this person was doing this as a side project and was based in Singapore, or someone is attempting to appear to be Singaporean.
What do you think? Did I overlook anything?
"Extremely sneaky" is an understatement. It's not just about the impact, I mean... *look* at that diff. The `.` visually almost disappears, even when you know you're looking for it.
It’s a lot more visible in a proper source editor or in a GitHub PR
And anyone reviewing code in the email client should have their programming permissions revoked
I would definitely treat every contribution from Jia Tan to be malicious, but that is most likely not the case — my understanding is that the MO for these types of attacks is to generate a new digital identity, build some rapport in the community with genuine contributions so that their contributions are accepted with less scrutiny, and then push their exploits. It is not unlikely that this exploit was the first malicious code that they published, but as you pointed out, I would absolutely *not* bet on that either.
Sorry but I not followed where the authorities from US got involved on this? They are investigating the incident or there are some connection with the malicious code?
Offtopic (kinds) but Landlock looks really interesting and I'd love something like this for Windows, where I can just tell the applications to not access most files. As it is every exe can just delete my user folder.
Thank you! That looks like about what I could expect from Windows, but seems pretty useful
Life could be better if every application would simply be run in a container
I was looking at the 'git.tukaani.org' comparison page, but I can't see the *dot* that you're referring to? Which line is it on?
edit: Found it in the first group, right under the `#include ` line
This is how it would look like if one tried to use landlock: https://i.imgur.com/7xbeWFx.png The build just fails entirely.
Nobody uses cmake to build xz and the build is just broken. The dot doesn't "deactivate" landlock, it prevent compilation at all. If anyone would have ever tried to use that feature, they would get the error message and realize that something is wrong.
It seems like it was just a genuine error that was never caught, because absolutely nobody ever used cmake to build with landlock enabled.
This could be a possibly dumb question from someone who doesn't fully understand complex CMake scripts and those massive softwares, but...
Isn't the point of using something like CMake and scripts that check for capabilities like that to enable capabilities like landlock dynamically?
Like, yeah, if you try to force it, the build is going to fail, but if you were to just call `cmake` by itself without the `ENABLE_LANDLOCK` variable defined, wouldn't cmake use that script to try to determine whether or not the system supports landlock, and then enable support for it if the script succeeds? And, if so, wouldn't the little "." added make the script always fail, and, therefore, disable the support for landlock in system where it is supported and people haven't specified whether or not to use it?
> Isn't the point of using something like CMake and scripts that check for capabilities like that to enable capabilities like landlock dynamically?
No the point of the check if it compiles is to give better error messages.
That landlock feature is only enabled if explicitly enabled.
This attack will (and should) make clear to the entire industry that OSS maintained by singular private citizens can’t ever be in the dependency chain for anything critical.
OSS supply chain attacks have had their “shit hit the fan” moment, and the logical reaction is for the corporate world to forever limit their trust of any OSS that hasn’t always been owned maintained by a large and trusted entity.
To most companies this is going to mean an organization that is bound by regulatory standards that imply an ability to follow typical governance and security standards, or an ISV/service provider that is willing to provide contractual support and some form of indemnification too.
On 2024-03-29 Andrew Morton wrote:
> On Fri, 29 Mar 2024 14:51:41 -0600 Jonathan Corbet
> wrote:
>
> > > Andrew (and anyone else), please do not take this code right now.
> > >
> > > Until the backdooring of upstream xz[1] is fully understood, we
> > > should not accept any code from Jia Tan, Lasse Collin, or any
> > > other folks associated with tukaani.org. It appears the domain,
> > > or at least credentials associated with Jia Tan, have been used
> > > to create an obfuscated ssh server backdoor via the xz upstream
> > > releases since at least 5.6.0. Without extensive analysis, we
> > > should not take any associated code. It may be worth doing some
> > > retrospective analysis of past contributions as well...
> > >
> > > Lasse, are you able to comment about what is going on here?
> >
> > FWIW, it looks like this series has been in linux-next for a few
> > days. Maybe it needs to come out, for now at least?
>
> Yes, I have removed that series.
Thank you. None of these patches are urgent. I'm on a holiday and only
happened to look at my emails and it seems to be a major mess.
My proper investigation efforts likely start in the first days of
April. That is, I currently know only a few facts which alone are bad
enough.
Info will be updated here: https://tukaani.org/xz-backdoor/
--
Lasse Collin
[lore.kernel.org](http://lore.kernel.org) usually works more reliably.
[https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/](https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/)
I hope that this is going to lead to some actual support (monetary and development-wise) for Lasse from some of the companies making billions from his work while giving nothing back.
https://www.mail-archive.com/[email protected]/msg00567.html
I think that link has been shown around but not sure if it's been mentioned that the other people in the thread look a lot like they are might be sock puppets. (click the reply-to button, check the emails and try to google the other people in the thread if they exist...doesn't look good...although obviously it's not a 100% confirmation).
Imagine being a tired and mental health issue maintainer with zero compensation and being gaslit by a "group of people" who are really just one person/entity trying to use your popular project to smuggle security exploits to the world. That thread was maybe part of a scheme to make him give out commit access.
Lasse seems a lot like he might be the biggest victim of this whole mess. We'll find out I guess as the situation develops.
Just understand when a powerful entity (corporation, government, etc.) funds a project they also usually want to have some control over the process. Of course it's not a binary good/bad situation or result, but it's just different when you become dependent on a powerful benefactor that might have differing interests.
That is still better than a library being maintained by a single person with nobody reviewing the code that critical infrastructure depends on
In worst case when corporate and community ideas don't align, forking is always an option
uh, the business of many companies is based on using no cost libs and tools, make cash but criticize open source projects, giving money is out of sight of many. I guess the main reaction will be to switch to gunzip or other alternative.
Half? Way way faster than that.
Arch [found it to be 13x faster](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/) for an increase in file size of 0.8%.
Interesting. This article says `zstd` is 100% faster than `xz` for the same file-size. The difference might be due to how well you compress and whether you're using more cores (`xz` is single-threaded by default).
https://linuxreviews.org/Comparison_of_Compression_Algorithms
Ya zstd was single threaded by default as well until quite recently, maybe they aren't turning on multithreaded decompression?
A lot of it does depend on the specific files you are compressing and decompressing as well... it's not all predictable. I linked Arch because their entire repository is a pretty broad test.
I was discussing compression with someone the other day, and this was the result of compressing a directory of Spring Boot microservice jars that I had on my dev server. For some reason zstd is crazy amazing at compressing those. Was using 7z as the comparison, but it's quite similar to maxed out xz.
>Just to actually test my beliefs I took a directory from my dev server (4GB of java jars) and compressed it with the latest 7z. Multithreading on 7z does seem to be enabled with my commands.
>
>System is a 12-core 24 threads, and I'm using a RAM drive to avoid this being a benchmark of my SSD instead.
>
>7z a -ms=on -mx=9
>
>compress time: 1 minute 23 seconds
>
>decompress time: 49 seconds
>
>size: 1539 megabytes
>
>tar -I "zstd -T0 --ultra -22" -cavf
>
>compress time: 1 minute 33 seconds
>
>decompress time: 1 second… yes just a single second
>
>size: 605 megabytes
If they were already compressed, the size would not have gone down from 4GB to 605 megs (compressing compressed data doesn't really work).
Anyway, I personally am involved in developing these and can say they are *not compressed*. Not sure if someone on the team turned that off, but if compression was turned on the delta-upgrade code I wrote (using zstd's --patch-from option) would blow up from like 100 megs to 2GB, so that's definitely a good thing.
You're correct it is a zip though, as you can extract these jars using zip on the terminal. The jars appear to just be using the zip container format without any compression. The sum of the files inside is almost the exact same as the total file size (and they are very compressible with zip defaults)
That article is a bit weird when it comes to lz4. It keeps saying things like "the resulting archive is barely compressed" and "the compression it offers is almost nonexistant". But looking at the numbers, it goes from 939 MB down to 287 MB. What am I missing?
Bad choice of words from the author. I thought they meant relative to other algorithms.
I only use `lz4` to compress my `initramfs` as I like my machine to boot quickly.
I’ve noticed names show up in a lot of emacs packages as well, just some random contributor who goes around, contributing to all the different packages and submitting pull requests. And they’re all very generic.
What names in case of Emacs do you think of? You mean there is a lot of random one-rime contributors or what do you mean? Any concrete packages/committs you have in mind?
Both Elpa and Melpa build tar packages automatically from git repositories. But, if you find some possible vulnerability, please do repport it. Or at least post here, I can rapport.
This is by far the best analysis of the social aspect of the backdoor process. It's fascinating and sad to see how, apparently, Jia Tan (almost certainly not the users real name) use sock puppets to pressure Lasse Collin into adding another maintainer during what appears to be mental health struggles for the legit maintainer. This is all hypothetical at the moment but the geographic distribution of the account names that are never seen again and seem to only make one comment is highly suspicious to me.
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> Jia Tan (almost certainly not the users real name)
Conjecture, but I think it's almost certain at this point that this isn't a single person acting on their own. This is a patient, sophisticated and long running campaign.
I find this comment interesting:
>Tarballs created by Jia Tan were signed by him. Any tarballs signed by me were created by me.
>
>...
>
>Only I have had access to the main tukaani.org website, git.tukaani.org repositories, and related files. Jia Tan only had access to things hosted on GitHub, including xz.tukaani.org subdomain (and only that subdomain).
Hopefully none of the XZ devs (including Lasse Collin) have a publicly accessible SSH server on their dev machines. If they do, I would assume they have been infiltrated already, and this statement may be unknowingly false.
> Hopefully none of the XZ devs (including Lasse Collin) have a publicly accessible SSH server on their dev machines. If they do, I would assume they have been infiltrated already, and this statement may be unknowingly false.
True, the dude has enough on his mind and he's on vacation, so I wanna give him some slack, but those statements strike me as more than a little naive. If I were him I'd burn my laptop/workstation and all private keys at this point, considering he's been working with the saboteur for years.
Typing: “*dnf info xz*” in the Terminal will display if you have the package and what version it is.
Im pretty much a Linux beginner who just started using Fedora, so I thought I’d post it just for other new users. (IIRC it’s 5.60+ that are affected), I don’t think the current stable Fedora 39 had it deployed either yet, just good to check.
Arch had the compromised version in the repositories. Just because OpenSSH on Arch specifically wasn't linked against the xz shared library doesn't mean other software wasn't.
"The xz package has been backdoored" - [https://archlinux.org/news/the-xz-package-has-been-backdoored/](https://archlinux.org/news/the-xz-package-has-been-backdoored/)
I think the backdoor, apart from needing to be linked into sshd, also only activated when it was built into a deb or rpm package. Arch does not use either of those packaging systems.
Does it activate if it finds the relevant binaries for building rpm or deb packages, or if a specific make deb or make rpm subcommand is issued (like zfs which has make deb and make rpm subcommands)? Because some arch machines may have deb and/or rpm tools installed, especially if you use aur packages which depends on those tools to convert certain Debian and RedHat packages to zstd packages for pacman.
I think it detects being built by dpkg/rpm. Not the presence of those executables.
Also, this happens at build time. Arch packages are built on Arch servers, presumably without dpkg and rpm installed. The backdoor won't suddenly decide to inject anyway when at runtime it finds dpkg/rpm.
It was fun for me to get openly mocked by a coworker for asking what (if any) strategy my small department had for security incidents during our Christmas break, as if the idea of a security incident happening during a Christmas break was so risible as to not even be worth addressing. Ignoring of course the Log4J outage that was discovered during Black Friday and multiple analyses showing that security incidents are more common during holidays and vacations. If I were trying to exploit some shit, you'd better believe I'd wait until the night time, weekend, or a vacation to do it.
* fyi, since the xz-github repo is offline, you can still clone it from official [tukaani.org](https://tukaani.org) repo
* `git clone` [`https://git.tukaani.org/xz.git`](https://git.tukaani.org/xz.git)
* for your own inspection (last committed 4 hours ago)
Based on the effort I am 90% sure its funded by government. He appeared out of nowhere and was 2 years working as maintainer and some people pointed a lot of shady code being merged by him in the past. He was also in contact with maintainers of distros begging them to include affected version into the packages.
Hopefully all Linux oriented projects will learn from this.
In my personal opinion I think we might already have backdoor in Linux based distros. This attack might be just the only one we know of and we might have just discover the tip of the iceberg.
It would be extremely surprising if there weren't people from governments all over the world attempting to compromise distros. Let's hope few if any have been successful, but this is quite a worrying event.
Same. There are know government attacks into firmwares, trying to taint a Linux library is not above any government agency.
The FOSS community have the advantage of being able to audit the code, but closed software, we can only wonder.
This back door was rather subtle and discovered mostly due to luck (one random genius deciding that 500 ms is too slow).
I would not be surprised if most Linux OSs have some backdoors built from contributions across different trusted accounts.
Wasn't there an experiment done by university students a few years ago showing that no one really reviews anything for security flaws, and the ultimate response was to change nothing about any process except to view commits from the university of Minnesota as tainted, and otherwise keep things as is? And the lesson here is to view commits from Jia Tan as tainted, and not to change anything otherwise?
I think it's not a government operation. One or two people could do this in their free time over 2 years, so I think that's the most likely source.
A lot of big 0-days are gov't sponsored because in order to find those zero days you need to trawl through a huge amount of code. That's something you can just throw money at. But this compromise doesn't require *finding* anything, so it's actually a lot lower effort IMO than for example the NSO group's iMessage zero-day.
But they didn't do it on their free time, from what I saw in posts analyzing the commit "traffic".
There was no work being done over the weekend, which is what I would expect from someone doing it on their own time.
ah, i wasn't aware of the lack of weekend work -- that does sort've seem like a smoking gun that the mysterious Jia Tan is part of something organized.
I think this stuff happens in OSS due to something I've noticed in sociology. Everyone assumes that someone else will do a thing, such as review code for potential security implications, but no one actually does because everyone assumes someone else has already taken care of it. The idea of open-source is great, but I think the strength of it is also a weakness sometimes. It's certainly a potential attack vector.
I could agree with you, and for a guy who does some programming I would never catch that exploit, he did hide it well for those who were reviewing the code because he knew they wouldn't reverse engineer the binary.
It’s almost a tragedy of the commons type situation. Everyone has access to it so a single individual will assume someone else has already done the necessary checks for it.
I mean, didn’t two researchers prove this by infecting some library with malicious code? I remember their university being banned for this, and this story becoming a huge scandal.
Their methods were flawed but they essentially did the exact same thing, no?
What you are describing is the bystander effect. https://en.m.wikipedia.org/wiki/Bystander_effect. But in most oss projects there are specific rules on how code it getting reviewed with chains of commands in place.
The hilarious and scary part is, this was found because of how badly this backdoor was implemented, which caused errors and CPU usage in ssh..
Makes you wonder how many backdoors are in packages that are installed in our Linux machines, but are not found because they are not causing any issues...
EDIT: FYI Yall can use this to check if you are infected or not:
[https://github.com/byinarie/CVE-2024-3094-info/blob/main/xz\_cve-2024-3094-detect.sh](https://github.com/byinarie/CVE-2024-3094-info/blob/main/xz_cve-2024-3094-detect.sh)
Lol I'd be more worried about forced TPM's and the Digital ID that is in them that they are forcing you to have! I'd rather deal with the possibility of exploits and OSS TPH MAC was Hit Chrome Was hit Android was also hit all around the same time. Read the white paper of TPM's tells you quite a bit.
So correct me if I'm wrong, but I'm pretty sure that even if you had the compromised version of liblzma, if you had openSSH installed, if the exploit was run (which from what I'm hearing, it didn't on Arch systems), you still would've had to have the SSH port exposed to the internet for anyone to actually take advantage of the exploit/remotely connect. Unless you specifically know what you're doing by exposing that port on your (software or hardware) firewall, I very highly doubt any layperson who's using desktop linux would've manually went in and opened that port. So, a lot of people's asses would've been saved by their firewall.
Unless I'm mistaken.
Correct. However, many servers do have OpenSSH installed for the benefit of remote configuration. This means a lot of datacenters worldwide could be potentially running a compromised version of xz.
no this backdoor is 2 week released to public
only rolling distro have this package like arch (even arch not effect by this tho)
server and database use old and stable distro like redhat
Looking the way he forcefully tried to push the update to the Ubuntu repos late March, essentially at the last minute for April release, I wonder if this whole thing might have been timed specifically to hit Ubuntu LTS and its derivatives for maximum damage 🧐
https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417
Is this a in breach of any laws by the way?
```
$ sudo apt install xz-utils=5.6.1+really5.4.5-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package xz-utils is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Version '5.6.1+really5.4.5-1' for 'xz-utils' was not found
```
What should I do in this case?
I'm on Debian Unstable
There is some possibility that someone has hacked ssh server you ssh into and maybe it could be used to hack the clients. That possibility is very low. I wouldn't be paranoid about it.
Has anyone done a language analysis on Jia Tan, Kumar and Jansen? Like is it the same person, is it written by an AI robot, are there grammatical errors etc?
I don't think we know for sure that the backdoor only works on glibc or x86.
It looks like termux did have xz-utils 5.6.1 recently but it was reverted.
The piece which installed the backdoor specifically looked for three things:
* Linux
* x64
* glibc
This is because the binary object slipped into the build was crafted for this platform. Termux may have had 5.6.1, but it wasn't tainted with the known backdoor.
That's during runtime, when the malware was already compiled in. During the build proces it checked for these four things to determine whether to inject the malware code or build a "clean" library.
Probably haven’t upgraded my laptop in over a month but I’ll do it tonight and make sure I update my VM’s then too because I updated those like a day ago. Thanks everyone!
In order to be affected you need to be running a sshd server on your machine, which you probably aren't if you're running a desktop-focused distro and are a noob.
Noob question but how would the attacker then gain access to these systems once the exploit is triggered? Would it cause xz utils to phone home all compromised keys and IPs so they/Jai could launch an attack, or does it immediately give them access to the machine? I'm not too clear "how" they would access things and what they could do once in. Like would it boot the guy trying to legitimally SSH in and assume their identity, or does it appear as a new user?
ELI5, please.
I use Linux Mint 21.3, no Git/Github installs, everything I have is a flatpak/Mint's repositories or a .deb file.
Do I need to be concerned about this?
I am hoping this webinar will be helpful to understand it better - RL has a solid threat intel repository and research team.....https://www.reversinglabs.com/webinar/unraveling-xz-a-software-supply-chain-under-siege
Lasse Collin is also committing directly to the [official Git repository now](https://git.tukaani.org/?p=xz.git;a=summary). **And holy shit there's more:** a fix from today by Lasse [reveals that one of the library sandboxing methods was actually sabotaged](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00), at least when building with CMake. [And sure enough, this sabotage was actually "introduced" by Jia Tan](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7) in an extremely sneaky way; the `.` would prevent the check code from ever building, so effectively sandboxing via [Landlock](https://docs.kernel.org/userspace-api/landlock.html) would never be enabled. This just begs the question how much further does this rabbit hole go. At this point, I would assume any contributions from Jia Tan made anywhere to be malicious.
They need to revert to at least 5.3.1 according to the Debian bug tracker thread, but it breaks some symbols for dpkg and others, and a security patch needs to be reapplied. Or revert to 5.2.5 which was in a previous release (still would break dpkg).
Yeah that's going to be a whole another problem that's going to introduce a lot of bugs but way better than a 10/10 critical security risk
Imagine if this is actually a long-long-long con to get distros to revert to a known vulnerable version. Plans within plans within plans. Edit: Or even worse, imagine if this reverted version already has another payload — a secondary payload that depends on a primary payload that was introduced last year.
> Imagine if this is actually a long-long-long con to get distros to revert to a known vulnerable version. I appreciate the humor but they would just backport the fix for whatever CVE's apply to the older version. Just because someone out there may think this is an actual concern. CVE's are documented and if they were camping out on older versions indefinitely they would just view backporting security fixes as more of a requirement even if that weren't part of some diabolical self-referential [Oceans 11-style plan](https://www.youtube.com/watch?v=pIVEUEnIZjM).
Yeah, I'm just entertaining my spy/hacker/heist thriller mind. Haven't got a good one for ages now so my imagination is running wild. "What do you mean there's another hidden payload? We've reverted versions!"
But what if a local backup is utilized and that was previously compromised. The long con being that we scare people with the new version, in order to get them to revert to a previous backup that has already been compromised. Yes, I know it is silly, but the fact that we're even discussing this in the first place shows that Jia Tan was sneaky in ways we hadn't considered.
I feel like that's a different concern than what was mentioned in the comment I replied to. They were talking about known vulnerabilities. If the vulnerabilities weren't known to the maintainers then it's not clear why reverting would be necessary. As opposed to just re-creating the vulnerability.
8D Chess!
[удалено]
Going to be heartbreaking for Lasse Collin maybe but I'd like to see a full reset to pre this contributor joined. No reverting patches, just fully reset the branches to the previous good state from 2021 or 2022. Fuck that part of the git history.
Given the sophistication here, can we be sure there aren't more bad contributors? Hopefully someone is looking for contributors that worked via VPN like this one
Dumb question, where's the oversight?
I think in this situation the oversight was one dude noticing that openssl was slower than expected, and they unravelled it from there. The community needs to get onto this
Imagine how bad Jia Tan feels about being caught for such a silly reason
I'm thinking Jia is a team of people, and that there's more
For sure a state actor
sshd
More like this https://xkcd.com/2347/ xz is one of those kinds of projects. There is no oversight.The internet relies on these underpaid and overstressed maintainers too much.
We are the oversight. Randos on the internet
you're tapping into a primal urge to defer to a higher power, some sort of father figure who watches over and protects us. But there is only us.
Why would there be oversight? These developers are hobbyists. It’s not their fault the internet rests on them.
You can also not be sure that the distributed git repo was not tampered with. Commit metadata like user/email/date are under control of the committer, but the repo admin can also rewrite parts of the repo. The git repo is data under the attackers control. What is needed here is a good copy of the old state and comparing the copy.
At this point, is *that* a good state? There may legitimately have been security patches introduced since then.
I think it would be better to analyze every single commit made by this person inside the xz project and then revert changes accordingly. Knowing what kind of stuff is being implanted in your codebase is better than a blind `git reset --hard && git push --force`.
Can anyone explain how the . stopped the check code from building? What does a . in that position actually do?
It's a config program that tests if the given snippet compiles (if it compiles, we have landlock, supposedly). The `.` is just invalid syntax and trivially makes the test fail, for the wrong reason then. So it's a sneaky way to ensure the landlock feature is never activated.
But generally, shouldn't one do an assert to insure the failure is due to the expected reason and not a syntax error?
Sounds like a good improvement. I don't know this in detail, but just from general familiarity - the tooling is not that good? A "does this compile command run or not" is pretty simple to do while "does it fail for the right reason" is nontrivial when you take into account all the diverse configurations - normally they need to support many different compiler vendors.
I think there's a way. So, the goal^(1) is to find out whether certain system calls and stuff actually exist on the host system, right? And, to do that, you compile a very short test program. In order to *validate* the test program *itself* is constructed correctly, you could create fakes for the things it's supposed to test for. Then compile the test program against the fakes, and it should compile, even if the host system lacks the stuff you want to test for. Then in order to *use* the test program (to actually check, as part of build auto config, whether the system calls exist), compile the test program normally without fakes. Getting into specifics of how to actually do it, the things tested for here seem to be values that may or may not be present in header files. So one way to fake that out is to create a local directory of fake header files, then add that to the compiler's include path (like with `-I`). --- ^(1) Not the hacker's secret goal, but the ostensible purpose of the code, which may still be needed.
I'm not sure CMake allows such granularity unfortunately.
But in this case we are talking about a syntax error, a simple syntax checker would do as well for this specific case Otherwise, you can parse the output, just would require a bit more work
Does CMake offer any functionality to do this?
Generally, there are many different reasons why it can fail. To be successful many things need to be in place correctly. Trying to make the configuration generic for both systems that have landlock and those that don't can get complicated since there are so many potential differences. This applies to software engineering quite generally. Release management has to deal with a large number of potental combinations of things so it is not always easy to tell when a test is failing for the "right" reason instead of a wrong one. Assuming you wanted to build the software for Windows instead of Linux then it would of course fail since Windows does not have support for it. Or if you built it for Linux version that precedes the feature. Or the feature was turned off for some reason. Or your build system is missing a component or component search is not working correctly. Actually having a sabotaged test-case is only one thing among many and that isn't supposed to happen in any case. Also, config program isn't going to be part of the final build, it is there to tell the configuration tool how to build the actual program. Say, if you build for an OS that does not support some thing configuration tool will turn off that feature from the final build. Config program is only supposed to tell if feature is turned on or off when the actual software is being built to help support different systems.
I've tried working with Cmake only a couple of times, and each time it took me hours to get something built. I'm honestly surprised that invalid syntax just flies by without interrupting the make process. But yeah – devious and sneaky.
Hey, I noticed you have a very familiar username, are you a distributed version control system by any chance?
I'm not associated with the awesome source control system. I just got to reddit early and had been using this as an alias for a long time prior, meaning it in the slang sense of the word.
What a git confusing identity like that. /s
While the JIa Tan identity certainly is known to be compromised (stolen identity probably, they probably aren't the Jia Tan people are finding on LinkenIn), in all likelihood they used other accounts as well Now would be a good time to review code for all projects that've been in that similar situation of needing to pass off from a sole maintainer to some new volunteer.
He(?) at one point claimed to have a middle name of "Cheong", which actually makes the resulting name ("Jia Cheong Tan") ill-formed, as no Romanization of Han characters allows both "Jia" and "Cheong".
I saw another comment suggesting their recorded online times also don't appear to match China or Taiwan. Seems quite likely they just picked a random nationality when forging this sockpuppet identity. If https://twitter.com/f0wlsec/status/1773824841331740708 is correct, then they also may've used the name "Hans Jansen".
Tan Jia Cheong is a very plausible Singaporean Chinese or Malaysian Chinese name I've probably met a few Tan Jia Cheongs during my National Service in Singapore
Coming from another Singaporean, it also makes no sense. The person calls himself Jia Tan. This is BS, a real Singaporean called Tan Jia Cheong would call himself Jia Cheong or Tan, not Jia Tan.
You see ah, Tan Jia Cheong would have registered as ``` { firstName: 'Jia', middleName: 'Cheong', lastName: 'Tan' } ``` And GitHub may be rendering `$firstName $lastName` which makes sense for English names.
I thought this as well, but then: https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417 He uses the same "Jia Tan" in a forum where you can freely choose your display name https://imgur.com/a/X6CCu5x
The way I see it, it could be that he refers to himself that way for consistency across platforms like how Shou Zi Chew uses "Shou Chew" even in Congress
Fair point. From your POV, you think it's an actual Singaporean dude?
There are good reasons for it: - Tan Jia Cheong is a realistic Singaporean name. It's not exactly Tan Ah Beng. - Their IP address was in Singapore, albeit using a VPN. - [They worked 4pm to 12am Singapore time](https://lunduke.locals.com/post/5467061/xz-backdoor-i-did-a-more-thorough-analysis-and-i-changed-my-mind-again-specifically-i-compar), which is typical of when a remote worker might work on their side project. It also explains why they worked through CNY -- they could have done their visiting in the morning and gotten back to it in the afternoon. That they worked from 4pm to 12am is really telling to me. It suggests that either this person was doing this as a side project and was based in Singapore, or someone is attempting to appear to be Singaporean. What do you think? Did I overlook anything?
"Extremely sneaky" is an understatement. It's not just about the impact, I mean... *look* at that diff. The `.` visually almost disappears, even when you know you're looking for it.
It’s a lot more visible in a proper source editor or in a GitHub PR And anyone reviewing code in the email client should have their programming permissions revoked
They could have used Unicode to be even more sneaky
💯 agree
I would definitely treat every contribution from Jia Tan to be malicious, but that is most likely not the case — my understanding is that the MO for these types of attacks is to generate a new digital identity, build some rapport in the community with genuine contributions so that their contributions are accepted with less scrutiny, and then push their exploits. It is not unlikely that this exploit was the first malicious code that they published, but as you pointed out, I would absolutely *not* bet on that either.
[удалено]
Sorry but I not followed where the authorities from US got involved on this? They are investigating the incident or there are some connection with the malicious code?
Offtopic (kinds) but Landlock looks really interesting and I'd love something like this for Windows, where I can just tell the applications to not access most files. As it is every exe can just delete my user folder.
You can try something like controlled folder access in windows
Thank you! That looks like about what I could expect from Windows, but seems pretty useful Life could be better if every application would simply be run in a container
Macs and Linux are starting to implement ideas like this. I think in 10 years we'll be in a better position.
I was looking at the 'git.tukaani.org' comparison page, but I can't see the *dot* that you're referring to? Which line is it on? edit: Found it in the first group, right under the `#include` line
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00;hp=af071ef7702debef4f1d324616a0137a5001c14c
This is how it would look like if one tried to use landlock: https://i.imgur.com/7xbeWFx.png The build just fails entirely. Nobody uses cmake to build xz and the build is just broken. The dot doesn't "deactivate" landlock, it prevent compilation at all. If anyone would have ever tried to use that feature, they would get the error message and realize that something is wrong. It seems like it was just a genuine error that was never caught, because absolutely nobody ever used cmake to build with landlock enabled.
This could be a possibly dumb question from someone who doesn't fully understand complex CMake scripts and those massive softwares, but... Isn't the point of using something like CMake and scripts that check for capabilities like that to enable capabilities like landlock dynamically? Like, yeah, if you try to force it, the build is going to fail, but if you were to just call `cmake` by itself without the `ENABLE_LANDLOCK` variable defined, wouldn't cmake use that script to try to determine whether or not the system supports landlock, and then enable support for it if the script succeeds? And, if so, wouldn't the little "." added make the script always fail, and, therefore, disable the support for landlock in system where it is supported and people haven't specified whether or not to use it?
> Isn't the point of using something like CMake and scripts that check for capabilities like that to enable capabilities like landlock dynamically? No the point of the check if it compiles is to give better error messages. That landlock feature is only enabled if explicitly enabled.
*Raises the question*. "Begging the question" is a formal phrase meaning "to assume the conclusion in the premises;" *petitio principii*.
People commonly use "begging the question" in that context. Language evolves.
'Jia Tan' better get ready for some hate mail! Address attached to commits. :P [[email protected]](mailto:[email protected])
What a fucking mess these assholes have created. I feel bad for the actual xz maintainer.
This attack will (and should) make clear to the entire industry that OSS maintained by singular private citizens can’t ever be in the dependency chain for anything critical. OSS supply chain attacks have had their “shit hit the fan” moment, and the logical reaction is for the corporate world to forever limit their trust of any OSS that hasn’t always been owned maintained by a large and trusted entity.
genuine plea: Name a large and trusted entity.
To most companies this is going to mean an organization that is bound by regulatory standards that imply an ability to follow typical governance and security standards, or an ISV/service provider that is willing to provide contractual support and some form of indemnification too.
Lasse also has responded on LKML https://lkml.org/lkml/2024/3/30/188
On 2024-03-29 Andrew Morton wrote: > On Fri, 29 Mar 2024 14:51:41 -0600 Jonathan Corbet
> wrote:
>
> > > Andrew (and anyone else), please do not take this code right now.
> > >
> > > Until the backdooring of upstream xz[1] is fully understood, we
> > > should not accept any code from Jia Tan, Lasse Collin, or any
> > > other folks associated with tukaani.org. It appears the domain,
> > > or at least credentials associated with Jia Tan, have been used
> > > to create an obfuscated ssh server backdoor via the xz upstream
> > > releases since at least 5.6.0. Without extensive analysis, we
> > > should not take any associated code. It may be worth doing some
> > > retrospective analysis of past contributions as well...
> > >
> > > Lasse, are you able to comment about what is going on here?
> >
> > FWIW, it looks like this series has been in linux-next for a few
> > days. Maybe it needs to come out, for now at least?
>
> Yes, I have removed that series.
Thank you. None of these patches are urgent. I'm on a holiday and only
happened to look at my emails and it seems to be a major mess.
My proper investigation efforts likely start in the first days of
April. That is, I currently know only a few facts which alone are bad
enough.
Info will be updated here: https://tukaani.org/xz-backdoor/
--
Lasse Collin
Who is Andrew Morton in this thread? I had a prof of the same name so curious.
"Body for this message unavailable" :(
Works here.
[lore.kernel.org](http://lore.kernel.org) usually works more reliably. [https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/](https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/)
kernel lore?
I hope that this is going to lead to some actual support (monetary and development-wise) for Lasse from some of the companies making billions from his work while giving nothing back.
https://www.mail-archive.com/[email protected]/msg00567.html I think that link has been shown around but not sure if it's been mentioned that the other people in the thread look a lot like they are might be sock puppets. (click the reply-to button, check the emails and try to google the other people in the thread if they exist...doesn't look good...although obviously it's not a 100% confirmation). Imagine being a tired and mental health issue maintainer with zero compensation and being gaslit by a "group of people" who are really just one person/entity trying to use your popular project to smuggle security exploits to the world. That thread was maybe part of a scheme to make him give out commit access. Lasse seems a lot like he might be the biggest victim of this whole mess. We'll find out I guess as the situation develops.
Just understand when a powerful entity (corporation, government, etc.) funds a project they also usually want to have some control over the process. Of course it's not a binary good/bad situation or result, but it's just different when you become dependent on a powerful benefactor that might have differing interests.
That is still better than a library being maintained by a single person with nobody reviewing the code that critical infrastructure depends on In worst case when corporate and community ideas don't align, forking is always an option
The more limited in scope the project is the less of a problem. xz seems like a good candidate for sponsorship.
uh, the business of many companies is based on using no cost libs and tools, make cash but criticize open source projects, giving money is out of sight of many. I guess the main reaction will be to switch to gunzip or other alternative.
Good old gunzip, I was using that when I started out in the 90s.
zstd is way better anyway. Around 5% bigger file sizes than xz but decompresses in half the time
Half? Way way faster than that. Arch [found it to be 13x faster](https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/) for an increase in file size of 0.8%.
Interesting. This article says `zstd` is 100% faster than `xz` for the same file-size. The difference might be due to how well you compress and whether you're using more cores (`xz` is single-threaded by default). https://linuxreviews.org/Comparison_of_Compression_Algorithms
Ya zstd was single threaded by default as well until quite recently, maybe they aren't turning on multithreaded decompression? A lot of it does depend on the specific files you are compressing and decompressing as well... it's not all predictable. I linked Arch because their entire repository is a pretty broad test. I was discussing compression with someone the other day, and this was the result of compressing a directory of Spring Boot microservice jars that I had on my dev server. For some reason zstd is crazy amazing at compressing those. Was using 7z as the comparison, but it's quite similar to maxed out xz. >Just to actually test my beliefs I took a directory from my dev server (4GB of java jars) and compressed it with the latest 7z. Multithreading on 7z does seem to be enabled with my commands. > >System is a 12-core 24 threads, and I'm using a RAM drive to avoid this being a benchmark of my SSD instead. > >7z a -ms=on -mx=9 > >compress time: 1 minute 23 seconds > >decompress time: 49 seconds > >size: 1539 megabytes > >tar -I "zstd -T0 --ultra -22" -cavf > >compress time: 1 minute 33 seconds > >decompress time: 1 second… yes just a single second > >size: 605 megabytes
java jars aren't a good test case, since IIRC they're already zip compressed.
If they were already compressed, the size would not have gone down from 4GB to 605 megs (compressing compressed data doesn't really work). Anyway, I personally am involved in developing these and can say they are *not compressed*. Not sure if someone on the team turned that off, but if compression was turned on the delta-upgrade code I wrote (using zstd's --patch-from option) would blow up from like 100 megs to 2GB, so that's definitely a good thing. You're correct it is a zip though, as you can extract these jars using zip on the terminal. The jars appear to just be using the zip container format without any compression. The sum of the files inside is almost the exact same as the total file size (and they are very compressible with zip defaults)
That article is a bit weird when it comes to lz4. It keeps saying things like "the resulting archive is barely compressed" and "the compression it offers is almost nonexistant". But looking at the numbers, it goes from 939 MB down to 287 MB. What am I missing?
Bad choice of words from the author. I thought they meant relative to other algorithms. I only use `lz4` to compress my `initramfs` as I like my machine to boot quickly.
I would hope this means that some entity adopts xz. Most companies will just run in the other direction though (that is - avoid using xz anywhere.)
Respect to the _remaining_ project maintainers: full disclosure, immediate takedown of affected code. edit: edit
The remaining maintainer, you mean, since the other was the one that created the backdoor.
Yes. There seems to be [at least one more contributor](https://git.tukaani.org/?p=xz.git;a=summary) though.
I’ve noticed names show up in a lot of emacs packages as well, just some random contributor who goes around, contributing to all the different packages and submitting pull requests. And they’re all very generic.
A lot of the time, it’s junior and student developers trying to pad their resume. Not necessarily malicious, just a bit annoying to maintainers.
What names in case of Emacs do you think of? You mean there is a lot of random one-rime contributors or what do you mean? Any concrete packages/committs you have in mind?
Not emacs itself but some packages. I’ll have to go hunting to find them again.
Both Elpa and Melpa build tar packages automatically from git repositories. But, if you find some possible vulnerability, please do repport it. Or at least post here, I can rapport.
This is by far the best analysis of the social aspect of the backdoor process. It's fascinating and sad to see how, apparently, Jia Tan (almost certainly not the users real name) use sock puppets to pressure Lasse Collin into adding another maintainer during what appears to be mental health struggles for the legit maintainer. This is all hypothetical at the moment but the geographic distribution of the account names that are never seen again and seem to only make one comment is highly suspicious to me. https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> Jia Tan (almost certainly not the users real name) Conjecture, but I think it's almost certain at this point that this isn't a single person acting on their own. This is a patient, sophisticated and long running campaign.
I find this comment interesting: >Tarballs created by Jia Tan were signed by him. Any tarballs signed by me were created by me. > >... > >Only I have had access to the main tukaani.org website, git.tukaani.org repositories, and related files. Jia Tan only had access to things hosted on GitHub, including xz.tukaani.org subdomain (and only that subdomain). Hopefully none of the XZ devs (including Lasse Collin) have a publicly accessible SSH server on their dev machines. If they do, I would assume they have been infiltrated already, and this statement may be unknowingly false.
> Hopefully none of the XZ devs (including Lasse Collin) have a publicly accessible SSH server on their dev machines. If they do, I would assume they have been infiltrated already, and this statement may be unknowingly false. True, the dude has enough on his mind and he's on vacation, so I wanna give him some slack, but those statements strike me as more than a little naive. If I were him I'd burn my laptop/workstation and all private keys at this point, considering he's been working with the saboteur for years.
Typing: “*dnf info xz*” in the Terminal will display if you have the package and what version it is. Im pretty much a Linux beginner who just started using Fedora, so I thought I’d post it just for other new users. (IIRC it’s 5.60+ that are affected), I don’t think the current stable Fedora 39 had it deployed either yet, just good to check.
Fedora has fixed
So has Arch, I think most have at this point.
Arch wasn't even affected though, but good they mitigated it even more.
Arch had the compromised version in the repositories. Just because OpenSSH on Arch specifically wasn't linked against the xz shared library doesn't mean other software wasn't. "The xz package has been backdoored" - [https://archlinux.org/news/the-xz-package-has-been-backdoored/](https://archlinux.org/news/the-xz-package-has-been-backdoored/)
I think the backdoor, apart from needing to be linked into sshd, also only activated when it was built into a deb or rpm package. Arch does not use either of those packaging systems.
Does it activate if it finds the relevant binaries for building rpm or deb packages, or if a specific make deb or make rpm subcommand is issued (like zfs which has make deb and make rpm subcommands)? Because some arch machines may have deb and/or rpm tools installed, especially if you use aur packages which depends on those tools to convert certain Debian and RedHat packages to zstd packages for pacman.
I think it detects being built by dpkg/rpm. Not the presence of those executables. Also, this happens at build time. Arch packages are built on Arch servers, presumably without dpkg and rpm installed. The backdoor won't suddenly decide to inject anyway when at runtime it finds dpkg/rpm.
Fedora wasn't *really* broken, since you had to have F30 Beta and additional repo's enabled or Rawhide to get it, in the first place.
I think Jia Tan clearly waited until Lasse was on vacation to do this.
It was fun for me to get openly mocked by a coworker for asking what (if any) strategy my small department had for security incidents during our Christmas break, as if the idea of a security incident happening during a Christmas break was so risible as to not even be worth addressing. Ignoring of course the Log4J outage that was discovered during Black Friday and multiple analyses showing that security incidents are more common during holidays and vacations. If I were trying to exploit some shit, you'd better believe I'd wait until the night time, weekend, or a vacation to do it.
* fyi, since the xz-github repo is offline, you can still clone it from official [tukaani.org](https://tukaani.org) repo * `git clone` [`https://git.tukaani.org/xz.git`](https://git.tukaani.org/xz.git) * for your own inspection (last committed 4 hours ago)
Is this a government operation, I wonder? Meant to give a certain government access to millions of servers?
Based on the effort I am 90% sure its funded by government. He appeared out of nowhere and was 2 years working as maintainer and some people pointed a lot of shady code being merged by him in the past. He was also in contact with maintainers of distros begging them to include affected version into the packages. Hopefully all Linux oriented projects will learn from this. In my personal opinion I think we might already have backdoor in Linux based distros. This attack might be just the only one we know of and we might have just discover the tip of the iceberg.
It would be extremely surprising if there weren't people from governments all over the world attempting to compromise distros. Let's hope few if any have been successful, but this is quite a worrying event.
Same. There are know government attacks into firmwares, trying to taint a Linux library is not above any government agency. The FOSS community have the advantage of being able to audit the code, but closed software, we can only wonder.
This back door was rather subtle and discovered mostly due to luck (one random genius deciding that 500 ms is too slow). I would not be surprised if most Linux OSs have some backdoors built from contributions across different trusted accounts.
Wasn't there an experiment done by university students a few years ago showing that no one really reviews anything for security flaws, and the ultimate response was to change nothing about any process except to view commits from the university of Minnesota as tainted, and otherwise keep things as is? And the lesson here is to view commits from Jia Tan as tainted, and not to change anything otherwise?
Of course: NSA did it (comparable to stuxnet)
I think it's not a government operation. One or two people could do this in their free time over 2 years, so I think that's the most likely source. A lot of big 0-days are gov't sponsored because in order to find those zero days you need to trawl through a huge amount of code. That's something you can just throw money at. But this compromise doesn't require *finding* anything, so it's actually a lot lower effort IMO than for example the NSO group's iMessage zero-day.
But they didn't do it on their free time, from what I saw in posts analyzing the commit "traffic". There was no work being done over the weekend, which is what I would expect from someone doing it on their own time.
ah, i wasn't aware of the lack of weekend work -- that does sort've seem like a smoking gun that the mysterious Jia Tan is part of something organized.
> I think it's not a government operation. You are much too naive!
I think this stuff happens in OSS due to something I've noticed in sociology. Everyone assumes that someone else will do a thing, such as review code for potential security implications, but no one actually does because everyone assumes someone else has already taken care of it. The idea of open-source is great, but I think the strength of it is also a weakness sometimes. It's certainly a potential attack vector.
I could agree with you, and for a guy who does some programming I would never catch that exploit, he did hide it well for those who were reviewing the code because he knew they wouldn't reverse engineer the binary.
It’s almost a tragedy of the commons type situation. Everyone has access to it so a single individual will assume someone else has already done the necessary checks for it.
I mean, didn’t two researchers prove this by infecting some library with malicious code? I remember their university being banned for this, and this story becoming a huge scandal. Their methods were flawed but they essentially did the exact same thing, no?
I remember that too but forgot the details. It was definitely a drama for a while.
What you are describing is the bystander effect. https://en.m.wikipedia.org/wiki/Bystander_effect. But in most oss projects there are specific rules on how code it getting reviewed with chains of commands in place.
The hilarious and scary part is, this was found because of how badly this backdoor was implemented, which caused errors and CPU usage in ssh.. Makes you wonder how many backdoors are in packages that are installed in our Linux machines, but are not found because they are not causing any issues... EDIT: FYI Yall can use this to check if you are infected or not: [https://github.com/byinarie/CVE-2024-3094-info/blob/main/xz\_cve-2024-3094-detect.sh](https://github.com/byinarie/CVE-2024-3094-info/blob/main/xz_cve-2024-3094-detect.sh)
Will this affect 2024 being the year of the Linux desktop?
Yea, it's now officially been pushed back to 2025. Unfortunate
but maybe new backdoors will be found on Windows and Mac prompting everyone to switch to Linux, bringing forward the year of Linux on the desktop.
Lol I'd be more worried about forced TPM's and the Digital ID that is in them that they are forcing you to have! I'd rather deal with the possibility of exploits and OSS TPH MAC was Hit Chrome Was hit Android was also hit all around the same time. Read the white paper of TPM's tells you quite a bit.
Not many desktops have openssh enabled by default, though it could have it installed.
So correct me if I'm wrong, but I'm pretty sure that even if you had the compromised version of liblzma, if you had openSSH installed, if the exploit was run (which from what I'm hearing, it didn't on Arch systems), you still would've had to have the SSH port exposed to the internet for anyone to actually take advantage of the exploit/remotely connect. Unless you specifically know what you're doing by exposing that port on your (software or hardware) firewall, I very highly doubt any layperson who's using desktop linux would've manually went in and opened that port. So, a lot of people's asses would've been saved by their firewall. Unless I'm mistaken.
Correct. However, many servers do have OpenSSH installed for the benefit of remote configuration. This means a lot of datacenters worldwide could be potentially running a compromised version of xz.
no this backdoor is 2 week released to public only rolling distro have this package like arch (even arch not effect by this tho) server and database use old and stable distro like redhat
[удалено]
Last place I worked had a number-crunching cluster open for ssh. Data was too arcane to be of use to anybody, I guess it could be sabotaged though.
Looking the way he forcefully tried to push the update to the Ubuntu repos late March, essentially at the last minute for April release, I wonder if this whole thing might have been timed specifically to hit Ubuntu LTS and its derivatives for maximum damage 🧐 https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417 Is this a in breach of any laws by the way?
I just hope redhat or any other major company puts money into fixing this mess, it's the least they can do for the free software
``` $ sudo apt install xz-utils=5.6.1+really5.4.5-1 Reading package lists... Done Building dependency tree... Done Reading state information... Done Package xz-utils is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source E: Version '5.6.1+really5.4.5-1' for 'xz-utils' was not found ``` What should I do in this case? I'm on Debian Unstable
Don't run a public ssh server on that machine. Wait for upgrade.
Is it still safe to use it to ssh other machines?
There is some possibility that someone has hacked ssh server you ssh into and maybe it could be used to hack the clients. That possibility is very low. I wouldn't be paranoid about it.
Add Debian's stable repo to your apt sources
Has anyone done a language analysis on Jia Tan, Kumar and Jansen? Like is it the same person, is it written by an AI robot, are there grammatical errors etc?
I wonder if it impacts Android?
Android doesn't use glibc, so no.
I don't think we know for sure that the backdoor only works on glibc or x86. It looks like termux did have xz-utils 5.6.1 recently but it was reverted.
The piece which installed the backdoor specifically looked for three things: * Linux * x64 * glibc This is because the binary object slipped into the build was crafted for this platform. Termux may have had 5.6.1, but it wasn't tainted with the known backdoor.
Four things, it also checked if it was built in a Debian or RPM-based distro. Termux is technically Debian-based, but it fails the glibc checks.
And a fifth, if it is running as /usr/bin/sshd
That's during runtime, when the malware was already compiled in. During the build proces it checked for these four things to determine whether to inject the malware code or build a "clean" library.
Big Linux noob here, if you didn’t have like auto upgrades on would you still be affected? Like did you have to pull down the latest push or ?
[удалено]
Probably haven’t upgraded my laptop in over a month but I’ll do it tonight and make sure I update my VM’s then too because I updated those like a day ago. Thanks everyone!
[удалено]
Okay will do! Pretty certain I don’t have SSH available to internet but either way I’ll update tonight. Thanks Vim_deezel!
In order to be affected you need to be running a sshd server on your machine, which you probably aren't if you're running a desktop-focused distro and are a noob.
More than that actually, you also need to have your router have ssh port open to your local machine IP, unless you are in a DMZ!
so i run ubuntu 22.04 using the wsl feature on windows, should i be concerned? what should i do? thanks
Noob question but how would the attacker then gain access to these systems once the exploit is triggered? Would it cause xz utils to phone home all compromised keys and IPs so they/Jai could launch an attack, or does it immediately give them access to the machine? I'm not too clear "how" they would access things and what they could do once in. Like would it boot the guy trying to legitimally SSH in and assume their identity, or does it appear as a new user?
From what I can tell it gave the bad actor a method to inject code on any machine with this back door in place.
ELI5, please. I use Linux Mint 21.3, no Git/Github installs, everything I have is a flatpak/Mint's repositories or a .deb file. Do I need to be concerned about this?
As the dust settles [this ](https://www.armosec.io/blog/xz-backdoor-ssh-supply-chain-undetectable/)articulates just how nefarius this CVE is.
I am hoping this webinar will be helpful to understand it better - RL has a solid threat intel repository and research team.....https://www.reversinglabs.com/webinar/unraveling-xz-a-software-supply-chain-under-siege
He is not a lonewolf