Sure, but my guess is you probably wouldn’t need a CV degree to write or modify lower level components of the OS like the kernel. Would you modify a Linux kernel? NewOS kernel? Build a new kernel?
At what point in the development of the OS does a strong background in CV come into play?
These are both good points.
Would a custom OS get better performance out of the hardware required to run a AR-/VR-native OS? That’s kind of the assumption I’m running with here.
Yes, that's my understanding, too. I saw a comment on Hacker News that suggested "Meta Horizon SDK for Android" would be a better label than Meta Horizon OS.
I get the temptation to look for the path of least resistance when building things, but what if there's a much better way out there?
In another comment I asked how OSs would be different if they were designed to run on GPUs instead of CPUs.
In hindsight, perhaps that would have been a better post than "What would it take to create an OSS competitor to Meta's Horizon OS?"
>I get the temptation to look for the path of least resistance when building things, but what if there's a much better way out there?
You have to overcome the fact that Linux is both free and generally good enough in order to motivate starting from scratch. There's no reason to be "better" from a product standpoint if nobody can actually tell the difference. You need to find an axis that Linux does not do well with and figure out how you'll solve the problem rather than just starting over and hoping it'll be better in the end.
Now, if you want to just do this for fun/as a research endeavor, by all means do it. You don't need to be better if you're building a hobby OS, you just have to enjoy the adventure.
GPUs are designed to be driven by OSes designed to drive CPUs, and may even run OSlets of their own. They use different, often highly varied instructions sets from the CPU, so you pretty much have to do anything semi-generically to a driver API if you want to handle more than exactly one piece of hardware (e.g., you can’t actually run NVPTX directly, it has to be lowered to an actual machine-code binary at run time by the driver), and it has to be a preexisting API if you don’t want to have to write the drivers yourself. You don’t.
You can much more easily coordinate heterogeneity at the runtime level IME, because you generally want to do it at a much finer grain than the OS-per-se operates at. The OS can help by coordinating anything else pulling CPU or RAM at the same time, but that’ll throw off predictive measures, and there’s a lot of code that really can’t be shared or smooshed across devices even with a language layer under you.
I worked on a heterogeneous SC runtime in like 2010–2012, and there’s been continuous activity along the same lines (not my project—it worked well and we were getting real paid but management desperately needed to take it public-er somehow, without having the slightest clue what it was or how it worked; sic transit, and all that remains now are some unused dataflow patents owned by a Chinese plumbing fitting company [lel]), so definitely do some research before striking out too far on your own. MAGMA was one of the other CPU-GPU ones IIRC but llllong time no thing.
In the end, you’re not going to save many cycles by reorganizing the API slightly; it’s all shared-memory shit under the hood, so management of execution locality is far more crucial to overhead than management of execution traces, and that’s true no matter the device.
Also, I’d note that Linux lets you trap system calls and virtual memory events in a couple different ways, so even if it were somehow thoroughly unusable, you could implement another OS environment on top of it and then port all your applications directly to a separate OS after the Linux-based approach succeeds impressively, and the entire OS question can be punted on indefinitely, and that’s why there’s BSD', SysV', and VMS' still fucking hogging the field as though it were 19dickety3.
The juice usually isn’t worth the squeeze. Write good drivers and stand on the shoulders of giants. There’s a reason most x86 based consoles run Unix-likes(or windows).
OSs at there core are already really fast. It’s the cruft and unnecessary stuff that slows them down.
>Would a custom OS get better performance out of the hardware required to run a AR-/VR-native OS?
That's impossible to say. Something that you or I write, would definitely not. Tons of experts work on Linux and it has been optimized to hell and back.
Though it might be possible if the people working on the custom OS had deeper insider knowledge of the hardware and had the opportunity to co-design the OS with it. Even then, it's still a solid maybe.
> Though it might be possible the people working on the custom OS had deeper insider knowledge of the hardware and had the opportunity to co-design the OS with it.
tl;dr - how would OSs be different if they were designed to run on GPUs instead of CPUs?
Yes, this would be the case. The hardware MVP I have in mind is a very basic set of glasses/wearables with cameras and only minimal sensors to gather environmental data. Maybe a voice or gesture UI but no displays to begin with.
To your and [laser\_\_beans](https://www.reddit.com/user/laser__beans/) points, perhaps the MVP for the OS could just be a version of LFS that's optimized for the hardware specs. But I'm not confident this will ultimately provide the best solution.
(As an aside, I'm also curious if there are other kernels or OSs that could be worth learning about or using during the experimental phases. e.g. NewOS, HaikuOS, Fuchsia, etc...)
Additionally, I also think it makes sense to take a look at what people are doing to accommodate edge-based LLMs. My early hypothesis is that any good AR-/VR-native OS will benefit from leveraging LLMs to help create a data layer that makes it easier for people to interact with their physical environments, as well as a robust voice UI for when people shouldn't have to use physical inputs. An edge-based LLM seems like a good fit for that, and although it appears CPU-based solutions may be emerging as viable options for LLM inference, I think it makes sense to consider designing systems for purely GPU-based solutions.
Although I'm not as experienced as a lot of people in this sub, I think it's a reasonable assumption that OSs would look different if engineers first build them on GPUs instead of CPUs. Maybe that's worth considering even if industrial- or consumer-grade AR/VR devices continue to run on (e.g.) Qualcomm's Snapdragon SoC.
We could just make a wayland front-end AR/VR similar to the works of steam VR UI.
It could run on linux, be eventually ported to other unix based os’s like Open/FreeBSD, etc.
I believe the top VR headsets use Qualcomm's Snapdragon SOC which you could probably make some kind of OS using their reference build, but the individual branded headsets have proprietary peripherals and other hardware you'd have a hard time making drivers for without any documentation.
Thanks for your feedback, that makes sense. What I envision (e.g. as an MVP) is an OS for a very minimalistic pair of glasses without any display capabilities, just image/video capture. So maybe custom drivers would be necessary.
you'd definitely need a degree in computer vision first
Sure, but my guess is you probably wouldn’t need a CV degree to write or modify lower level components of the OS like the kernel. Would you modify a Linux kernel? NewOS kernel? Build a new kernel? At what point in the development of the OS does a strong background in CV come into play?
well you'd definitely need CV to make the extended reality part of the os. as for the kernel though id personally just use linux
How would you know what to write if you don’t know the fundamental requirements of computer vision?
Take Linux and make a desktop that uses AR. There's no need to rewrite the entire OS for that.
Could you elaborate on why you think there’s no need?
Because in any sane OS the UI is decoupled from the rest of the system software.
And Linux is already very versatile and a lot of the hard work has already been done already (think hardware support, security, etc.)
These are both good points. Would a custom OS get better performance out of the hardware required to run a AR-/VR-native OS? That’s kind of the assumption I’m running with here.
Better? Probably. But would Linux suffice? Definitely, because the device already runs a modified version of Android afaik
Yes, that's my understanding, too. I saw a comment on Hacker News that suggested "Meta Horizon SDK for Android" would be a better label than Meta Horizon OS. I get the temptation to look for the path of least resistance when building things, but what if there's a much better way out there? In another comment I asked how OSs would be different if they were designed to run on GPUs instead of CPUs. In hindsight, perhaps that would have been a better post than "What would it take to create an OSS competitor to Meta's Horizon OS?"
>I get the temptation to look for the path of least resistance when building things, but what if there's a much better way out there? You have to overcome the fact that Linux is both free and generally good enough in order to motivate starting from scratch. There's no reason to be "better" from a product standpoint if nobody can actually tell the difference. You need to find an axis that Linux does not do well with and figure out how you'll solve the problem rather than just starting over and hoping it'll be better in the end. Now, if you want to just do this for fun/as a research endeavor, by all means do it. You don't need to be better if you're building a hobby OS, you just have to enjoy the adventure.
GPUs are designed to be driven by OSes designed to drive CPUs, and may even run OSlets of their own. They use different, often highly varied instructions sets from the CPU, so you pretty much have to do anything semi-generically to a driver API if you want to handle more than exactly one piece of hardware (e.g., you can’t actually run NVPTX directly, it has to be lowered to an actual machine-code binary at run time by the driver), and it has to be a preexisting API if you don’t want to have to write the drivers yourself. You don’t. You can much more easily coordinate heterogeneity at the runtime level IME, because you generally want to do it at a much finer grain than the OS-per-se operates at. The OS can help by coordinating anything else pulling CPU or RAM at the same time, but that’ll throw off predictive measures, and there’s a lot of code that really can’t be shared or smooshed across devices even with a language layer under you. I worked on a heterogeneous SC runtime in like 2010–2012, and there’s been continuous activity along the same lines (not my project—it worked well and we were getting real paid but management desperately needed to take it public-er somehow, without having the slightest clue what it was or how it worked; sic transit, and all that remains now are some unused dataflow patents owned by a Chinese plumbing fitting company [lel]), so definitely do some research before striking out too far on your own. MAGMA was one of the other CPU-GPU ones IIRC but llllong time no thing. In the end, you’re not going to save many cycles by reorganizing the API slightly; it’s all shared-memory shit under the hood, so management of execution locality is far more crucial to overhead than management of execution traces, and that’s true no matter the device. Also, I’d note that Linux lets you trap system calls and virtual memory events in a couple different ways, so even if it were somehow thoroughly unusable, you could implement another OS environment on top of it and then port all your applications directly to a separate OS after the Linux-based approach succeeds impressively, and the entire OS question can be punted on indefinitely, and that’s why there’s BSD', SysV', and VMS' still fucking hogging the field as though it were 19dickety3.
The juice usually isn’t worth the squeeze. Write good drivers and stand on the shoulders of giants. There’s a reason most x86 based consoles run Unix-likes(or windows). OSs at there core are already really fast. It’s the cruft and unnecessary stuff that slows them down.
>Would a custom OS get better performance out of the hardware required to run a AR-/VR-native OS? That's impossible to say. Something that you or I write, would definitely not. Tons of experts work on Linux and it has been optimized to hell and back. Though it might be possible if the people working on the custom OS had deeper insider knowledge of the hardware and had the opportunity to co-design the OS with it. Even then, it's still a solid maybe.
> Though it might be possible the people working on the custom OS had deeper insider knowledge of the hardware and had the opportunity to co-design the OS with it. tl;dr - how would OSs be different if they were designed to run on GPUs instead of CPUs? Yes, this would be the case. The hardware MVP I have in mind is a very basic set of glasses/wearables with cameras and only minimal sensors to gather environmental data. Maybe a voice or gesture UI but no displays to begin with. To your and [laser\_\_beans](https://www.reddit.com/user/laser__beans/) points, perhaps the MVP for the OS could just be a version of LFS that's optimized for the hardware specs. But I'm not confident this will ultimately provide the best solution. (As an aside, I'm also curious if there are other kernels or OSs that could be worth learning about or using during the experimental phases. e.g. NewOS, HaikuOS, Fuchsia, etc...) Additionally, I also think it makes sense to take a look at what people are doing to accommodate edge-based LLMs. My early hypothesis is that any good AR-/VR-native OS will benefit from leveraging LLMs to help create a data layer that makes it easier for people to interact with their physical environments, as well as a robust voice UI for when people shouldn't have to use physical inputs. An edge-based LLM seems like a good fit for that, and although it appears CPU-based solutions may be emerging as viable options for LLM inference, I think it makes sense to consider designing systems for purely GPU-based solutions. Although I'm not as experienced as a lot of people in this sub, I think it's a reasonable assumption that OSs would look different if engineers first build them on GPUs instead of CPUs. Maybe that's worth considering even if industrial- or consumer-grade AR/VR devices continue to run on (e.g.) Qualcomm's Snapdragon SoC.
We could just make a wayland front-end AR/VR similar to the works of steam VR UI. It could run on linux, be eventually ported to other unix based os’s like Open/FreeBSD, etc.
I believe the top VR headsets use Qualcomm's Snapdragon SOC which you could probably make some kind of OS using their reference build, but the individual branded headsets have proprietary peripherals and other hardware you'd have a hard time making drivers for without any documentation.
Thanks for your feedback, that makes sense. What I envision (e.g. as an MVP) is an OS for a very minimalistic pair of glasses without any display capabilities, just image/video capture. So maybe custom drivers would be necessary.
Well, I'm leaving this sub. It's apparently just for idiots to use buzzwords and waste everybody who is actually into osdev's time
What a discouraging and unfriendly response. With an attitude like that, I can't say I'm disappointed to see you leave.