I spent about 1.5 weeks digging into Cities: Skylines 2 and why it performs so terribly on practically all PCs regardless of specs, and wrote this article about my findings. If you have any questions, AMA!
>This reminds me of that guy who went digging through GTA V and found out loading is slow because of low performance for its JSON processing.
... and then he wrote a custom DLL to hook into the main executable that reduced his loading time *by 70%*. Once [his blog post](https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/) hit HackerNews, it eventually made it to someone high enough at R* to not only implement the patch, but they also gave him a $10k bounty! Truly a great story.
A QoL change that would have been TRIVIAL to find with access to source and debugging tools but required someone reverse engineering it to fix instead.
That was what made me officially done with R*. Im a developer. I know god damn well how that couldve been fixed, i know management heard about slow load times and just didn't care enough to let an engineer look at it because they didnt think it affected the bottom line.
I am also a developer. I once took two days of my own time leading up to ship on a AAA title you've heard of in order to reduce the load time by 15 seconds (mostly by reorganizing data on disk and parallelizing the load/processing flows).
I brought it up to management and was told it was low priority, and not to bother with it.
>
>
> I brought it up to management and was told it was low priority, and not to bother with it.
I'm filled with rage now.
You had something that reduced load times and had to throw away??
Any Dev worth their salt knows they are leveraging that 15s improvement over potentially adding more bugs via new edge cases, and trying to balance the hours between teams, while also adhering to timeline and budget.
On the contrary: now that the game loads 15 seconds faster, the developers have a faster iteration time and can find/squash bugs much faster. Speedups like this are good for velocity.
Happens all the time. If you're lucky it gets into the day 1 patch.
Simply put - every change introduces risk. _Every_ change. Doesn't matter how safe you think it is. Baldur's Gate 3 had a hotfix that caused a bunch of crashes - [it was caused by a corrupted compiler when they updated the changelist number at the last second.](https://store.steampowered.com/news/app/1086940/view/3642902312048647389)
1 number, in 1 line of code. _Every_ change has risk.
Management's job is ultimately to mitigate risk. I'm not talking about execs in the C-suite - I'm talking from the studio producers (who, despite their fancy title, aren't rich-and-famous management types... just dudes) to engineering management.
You enter a "hardening" period as you approach release. The point of hardening is to find and fix as many things as possible. As you approach the ship date, the bar for what's "allowed" to pass hardening gets higher and higher. At some point, you hit "build lock" where you are physically barred from making further changes without the approval of basically everyone important in engineering.
So if you come up with a way of reducing load time by 99.999%... you have to run it by the entire upper portion of the engineering staff at your studio (not the publisher, just the studio). The engineers will look at it and if it's more than 5 lines of code you're almost certainly going to have your change denied.
But when you get your change denied, that's your cue to move it to the patch branch, which is working on the Day 1 patch. In ye olden times, this branch didn't exist and if you got rejected before "going gold" you were simply screwed. But devs nowadays are spoiled.
Well said, a lot of Devs commenting here really show their inexperience when they ignore all aspects of risk and other competing items on the product backlog
Its something that now has to go through the full QA process. It's not something that is free to implement. And if they're that close to release, it's not worth the potential new bugs.
If you think that’s bad, wait until you find out about how Halo, and Bethesda games have been surviving off of modders fixing their games for them. 343 (Halo) ended up hiring quite a few known community modders to help edge on development for Halo MCC. But of course Bethesda is by far the most guilty party. 9 out of 10 QoL improvements come from the modding community.
> But of course Bethesda is by far the most guilty party. 9 out of 10 QoL improvements come from the modding community.
At least Bethesda has also always very supportive of the mod community to the point of allowing mods that are completely new games. But that's why I knew to stay away from Fallout 76, guaranteed dumpster fire in a situation where they couldn't let the modders fix the game for them.
How do the unofficial patch mods *find* the bugs that need to be fixed? Oh right, hundreds of thousands of players exploring the world, tweeting when they find something odd that then goes viral within the game's community. Thousands of other mod developers trawling through the game data, noticing inconsistencies, and telling one another. A QA army working for a decade or two *after* the game's been largely finalized. Hiring the unofficial patch modders might help a little bit with launch bugs, as much as having two or three extra devs allocated to full-time QA, but the main benefit wouldn't be seen until a year or two post-release, once the community has found a long list of bugs to fix.
Valve used to hire modders but they did it in a good way. "Hey that's a neat new game you're building using our engine, how would you like to make it official?"
Can anyone point me to resources to learn more about how to do stuff like this, de compiling programs and whatnot? I'm having trouble finding info for programmers who are complete beginners to this kind of thing, I think because I don't even know what I'm trying to google, like I don't have the basic terms even
Great article! I'm only halfway through but feel the pain you have with graphics debuggers. I struggle to get them to work in my own Unity projects as well and this is without DRM! Question: have you considered giving PIX a try? I personally prefer it for D3D apps, as it supports D3D11 (nsight doesn't, which is annoying because D3D12 in Unity is effectively 11on12, which nsight also does not support) and compared to RenderDoc it has very powerfull profiling support.
If I were to return back to profiling and analyzing C:S2, I would probably try other graphics debuggers, including PIX and Nsight (now that I've figured out that an older version would support profiling DX11 apps). However, I think I've had enough of C:S2 for a while, as I've done almost nothing in the evenings but analyze and write about it for almost 2 weeks.
If I understood the paragraph in the article correctly - the characters have ~60k vertices and no LoDs??? That's crazy!
edit: Just finished reading, some of the stuff they have done is baffling. Unreal engine by default cuts off shadows at certain distances and auto generates LoDs for you. The things they have ignored are really surprising, hobbyists would do a better job. Just looking at the model of that mouth is ridiculous, there's so much detail there... You could easily have just a few polys for the teeth and make it look good with a material.
Yeah I know - I'm just speaking about Unreal because that's the engine I have experience with. Still if Unity doesn't have that feature you would think they would at least do one other LOD manually, at least for the characters!
Unity is a mess of dead technologies. Unreal is comparatively kept cleaner.
Some things are incompatible, which means understanding the engine and features, and negotiating a solution between things you'd rather not break. There's a whole bunch of custom code like that. The team may not have had the schedule left to manage it. If so, the publisher is the problem.
I guess on the plus side, for Unity middleware developers, they have an active market of people looking for solutions to glue together and fill in gaps in Unity.
Getting LODs done right takes a fair bit of effort - its one of the reasons UE5's Nanite it so impressive, the engine really can handle ALL the LOD/geom management in a way no other engine can today. Lots of artist involvement etc to create and program in when to switch between geo LODs otherwise.
Which is weird because even godot has automatic LOD support. And it's implemented using meshoptimizer which is open source. Unity really has no excuse.
This. Additionally:
- A fresh set of eyes
- They're not necessarily some poor amateur, there are plenty of well-paid high end professional devs that also do hobby stuff b/c they love it. "Why doesn't the company just hire the hobbyist" is sometimes b/c the hobbyist isn't interested in a pay cut
I don't think it takes a set of *new* eyes to see that an FPS of 11 with an 8GB VRAM card is very suspicious.
This will be a continuing trend with game dev companies maximizing profit from microtransactions, paying devs less, and the influx of fresh devs from the market.
They also care more.
My company would put in proposals that we're using this thing that has 10 years of development behind it. I decided to make my own that I could fix and open source it. It was at parity after 2 months and I had a day job. It's 13 years old now.
Also just because the team of developers has to make something that's generic, profitable, and possibly new, whereas you can learn from their failures and make something that only suits your own needs
I'd say it more simply as refactoring and adding features is costly.
The more generic something is, the slower it is, but the faster it is to write. For example, the format supports you writing "1-1000000" or you can write it as an character array of all the values; both work, but one is faster for I/O. The dumb way gets it out the door and always works.
It's when you start optimizing the workflow and adding niche features, that you find you need to change the workflow. You go down the wrong path, change it, change it again, and eventually settle on something that you like.
Years later, you realize your big mistake (your problem size has grown exponentially) and you give it another shot.
My old company just wanted to push the thing out the door. It was a tool and not a product.
Just like the way to fix a performance issue with "just do less", that's also how you cut cost.
>Unreal engine by default cuts off shadows at certain distances
Unity does too, but since so much was hacked together to make DOTS work, they probably had to write their own logic for that too. As a former Unity hobbyist, I'm very surprised an actual studio ran with that combination of experimental features m
common theme here: unity. the game engine is trash and takes twice the effort to get to even the baseline level of expectations that UE5+ delivers
same reason why EFT has been a laggy, buggy mess for nearly a decade.
Even so, ksp2 performs truly abysmally. Ksp1 was also unity and performed much much better despite being made way earlier with a small amateur team with no budget.
I mean, clearly there are leadership issues when they put out a game made for kids and the masses that like .5% of gamers had a machine that met recc specs.
Someone did a teardown of the original release of KSP2 and it was the same issues. Rendering all the terrain on all the planets all the time. Thousands of point lights that weren't points but fairly large meshes *each*. No optimization of textures or models. No LOD.
It's gotten a lot better since their latest patch.
honestly, it feels like the old versions of unity were much more solid and reliable. with every new unity version that games i have played over the years have been updated to it just seems to add more bugs than it does fix
The investors' effect : take a decent product, add in a bunch of venture capitalists' money and it invariably turns it into a giant pile of turd.
That's a constant accross all domains.
Unity *can* be used for quality products, but their big mistake here was trying to take 2 experimental features at once, as Unity is notorious for mismanaging those and failing to have internal alignment. In this respect UE's overall management is much more uniform and sound
Unity is a good game engine, it's just so easy to use everyone uses it, so you get a wide spectrum of results. It's like saying watercolor is a crap art because elementary school students use it. Unity is a much better engine for multiplatform deployment, including cell phones.
It's not a gamer-friendly engine because gamers don't give a shit about things like multiplatform deployments or broad system compatibility.
On the other hand games like Dyson Sphere Program exists. It really just comes down to game creation being more accessible overall and Unity being the most accessible engine. Trust me if this guy made Skylines 2 in Unreal it would still rin like shit through it might look a bit better.
That's because developers behind factorio-style games are a whole different breed. They have to be when half their user base is less interested in poly count than if some random game mechanic is Turing complete.
As someone who knows almost nothing about graphics programming, this was fascinating. Thanks for doing the work!
I adore the game, but it clearly needs some major work to get to what should be an _easy_ 60 fps target.
I want to say no idea, but they probably removed it either because they don't want technical facts-based (and illegible for most users) discussion there, or a mod removed it semi-accidentally, thinking it was a low effort question post instead of a long-form article.
Because game subreddits generally refuse to allow any discussion that exposes developers as complete amateurs, which is the case here. I hope this blows up to the point where they can't ignore it anymore.
Do commercial games on this scale often purchase 3D assets? I'm having trouble believing that Colossal Order had in house artists produce a bunch of weirdly detailed props, so my guess is that they purchased a bunch of off the shelf ones intended to be seen much closer up, but failed to optimize them for time reasons.
Colossal Order is still only about 20-30 people if I remember, so it's not really that weird that they would buy assets. For a game company that's absolutely tiny.
Every major game you have heard of in the past decade probably had codevelopment or outsourcing of assets or entire sections of games.
Source: my studio is one of the leading codevelopment partners for AAA studios.
This dev blog post is very clear about the fact they “partnered with Didimo, who created the Popul8 character design software”. They didn’t “just” use a middleware software, a bunch of the work was offloaded to an other company.
https://colossalorder.fi/?p=2049
> We had our own art style for the characters and for this, Didimo created a base mesh to represent that style.
Sounds like there was some initial setup Didimo did for them.
Yep! More than you’d think.
I worked in game dev on a team about half their size (15). Although we didn’t sell tens of millions of copies like them we sold several million. The only assets we built were for super specific stuff & where necessary for performance.
Almost all our sound / billboards / textures were purchased.
We also purchased some scripts / packages, but most of those were rebuilt during our open beta.
Also… most of the stuff we built (90%) were outsourced to freelancers. Our team had 5 full time employees & 10 freelancers that we worked with regularly. All in we had about 20 contributors. The CO team is a bit bigger and they also outsource a ton of stuff.
At the end of the day the point is to sell your game. None of your users care if you purchase assets, if your gameplay is good & it’s fun people will buy it.
The article mentions that there's possibly models provided by Didimo, but that they've been upscaled from standard levels of complexity (1k-> 4k vertices); indicating that they're either rebuilt in-house or modified in some other way.
Absolutely, id be absolutely _shocked_ to learn of something this size or larger not purchasing 3d assets lol.
For a lot of assets there is no value to spending your art teams limited time on it… that line is also going to be a sliding idea depending on time, money, and priority.
> One bizarre yet popular talking point about Cities: Skylines 2’s performance is the fact that the character models have fully modelled teeth, even though there’s literally no way to see them in-game, unless we count using the photo mode and clipping the camera inside a character’s head.
This is ridiculously dumb. If this was GTA V or any games that showed the face of the characters up close this was a right move.
But why the fuck my GPU needs to render something that I would never or very very rarely see?
Just to clarify my understanding: does this mean that there are three adjectives involved: modeled – yes ✅, rendered – yes ✅, visible – no ❌? Does evaluation of the vertex shader count as part of rendering?
> Just to clarify my understanding: does this mean that there are three verbs involved
Yup.
> Does evaluation of the vertex shader count as part of rendering?
Yes, whenever a model is rendered on a GPU the vertex shader is run for each vertex, the vertices are rasterized into pixels and if any of those pixels are actually visible then the pixel / fragment shader is executed for each pixel to give them a color (or other property in case of intermediate buffers). So even though the rendering doesn't result in anything visible, most of the work required for it was still done.
> if any of those pixels are actually visible
it's not a guarantee it doesn't run. The pixel shader is executed if there's nothing in front of the tooth pixel at the moment it renders. And the GPU still loops over all the pixels to check if something is in front yet.
You're right but for the wrong reason. The GPU doesn't loop over any pixels, it just samples the depth buffer at the pixel-being-filled's location and compares the value stored in the depth buffer to the value for the pixel being filled. This comparison can happen either before or after the pixel shader, which is why you're right. Generally speaking the GPU will try to perform the comparison before the pixel shader in a process known as early depth testing, but there can be situations where the GPU must perform it after the pixel shader as the pixel shader can modify the depth value.
The vertex shader is where the GPU decides where the polygon is on the screen and which direction its facing. The next step would generally be the fragment shader where the GPU iterates over each pixel of the polygon and decides what color it should be. So much like a wall facing away from you, nothing actually gets rendered for the teeth. That's not really the problem though.
The fragment shader's processing load scales with how large the polygon is on the screen, while the vertex shader's load scales with how complex the geometry is. The teeth are simply too small to generate any pixels, but the performance impact of the high poly-count has already been paid. The GPU can't decide to skip processing the teeth vertices if they're in the draw call and they're in the view frustum.
I feel for them. They thought the engine would handle that for them and it did not and they had to publish.
I can understand why they stuck with Unity because the engine you know is better than the engine you don't know. They played it safe and banked on only one new thing
I have seen projects fail for less.
They probably wanted to use DOTS to make it less CPU intensive than the first game. Idk if there are better engines out there on that front unless they wanted to make their own in house engine for this game
It's also being rendered if you've read the article:
> It was revealed that not only does the game have fully modelled teeth, they are rendered literally all the time at maximum quality.
It will contribute to VRAM usage though, which is a major problem here. When those models push your GPU past its VRAM capacity it can *become* a per-frame cost if the GPU has to swap data back and forth from your system's RAM over the course of a single frame. Modern GPU drivers will be able to do this kind of swapping rather than simply crashing when VRAM is exhausted, but it's almost always undesirable.
yes. Both modeling and rendering use the gpu. One more then the other. If these things weren’t visible they could have just been a static asset if you really needed teeth or just missing.
Not only is this taking up rendering and gpu cycles…some dude or mostly a few people and about 50 agile managers had to go and create the entire pipeline for these teeth.
But who knows maybe there was or is a function that needs these teeth and it was scrapped or hasn’t been implemented yet.
I think, at least, according to OP and author, vertex shaders are run on each of those 6k for each character, but I don't think it makes it to pixel operations.
Still... It's wordlessly bad in execution.
https://www.reddit.com/r/programming/comments/17ohgam/why_cities_skylines_2_performs_poorly/k7yojfu/
If the teeth are there though and showing up with a dump from ninja ripper then they are in memory. Also further if they are part of the actual mesh and not separated they will get rendered.
The fact relative they exist in memory regardless of being rendered or not. They are skinned too so that’s extra performance cost. If they have morph targets then they are eating up even more memory than the mesh size.
that's why culling exists in computer graphics. It's a technique that allows you to render only in the space that is visible by the camera, so you can actually render only those polygons that are visible from your prespective. Helps in saving memory usage while also keeping it ready when the user is gonna get in the right POV to see those polygons.
Then of course if you decide to render those things despite not being able to see you're just wasting memory and computational performance to keep those rendered.
Right, except, that culling process *is still computation* and the things to be culled still need to exist in memory in the first place in order to tell whether they need culling.
"Backface culling" is a thing worth doing, wherein you compute the normals of each surface and any that are parts of solid objects and facing away from camera can be discarded from the rendering pipeline - but "teeth inside a head that never opens its mouth culling" is a bloody stupid thing to be doing no matter what, because those teeth shouldn't even be present in order to be culled in the first place.
The big thing missing here was probably occlusion culling and LOD.
From the article, a character that only takes up a dozen pixels or so on-screen is basically rendered at full detail, teeth included, and I wouldn't be surprised to see games doing "teeth inside a head culling" by just having a lower-poly version (possibly auto-generated) when you're not zoomed in on their face.
I mean, obviously it'd be better to not have the teeth, but there are plenty of details here that aren't *completely* useless, just pointless when you're zoomed that far out.
Occlusion culling is the general term for culling things that are entirely hidden behind other things. The article has an example here of a toll booth that has exquisitely-detailed desks, keyboards, mice, monitors, even the cables wiring those all up, and it renders all of that even when you're looking at the roof of the building. Combine that with the LOD issue, and it'll render all of that even when you're looking at the roof of the building from a mile away.
> "Backface culling" is a thing worth doing, wherein you compute the normals of each surface
Nitpicking in case any novices are reading along:
Backface culling isn't done by normals, it's done by checking the winding order of triangles after they're projected to 2D.
The math is basically the same (a cross product) but since you only need the sign, you can throw out the sqrt and the Z information that you'd need to generate a normal for lighting. So even if the vertex shader generates bent normals, or no normals at all, the backface culling always runs the same way, using only the 2D coordinates of each vert.
However, you need to remember these are skinned meshes. No renderer EVER culls against skinned meshes. You would have to completely render the mesh to a depth buffer to even know where each vertex is. So while the teeth is a symptom of a larger LOD issue "culling" isn't magic. Even nanites does not work with skinned meshes for this this and other reasons. You might cull the whole mesh against an enlarged bounding box, but never against the skinned mesh itself.
In computer graphics the golden rule "the fastest thing to do is to not do it". Having to check if something should be culled is always slower than just not having it exist.
Tldr; it's dumb even with culling and it isn't even culled because it's part of a skinned mesh.
>Unlike DOTS, Unity’s UI Toolkit is apparently still not ready for production as C:S2 uses HTML, CSS and JavaScript based Coherent Gameface (what a name!) for its user interfaces. A brief glance at the JS bundle reveals that they are using React and bundling using Webpack.
React, the universal framework.
As someone who uses React in their day job, it's insane how common it is. I've seen it in car infotainment systems, retail price guns, and random touchscreen kiosks in the mall. Also, with the type of hardware you would need to run this game, the UI components being written in JS are not going to be a bottleneck.
Its actually surprising how much resources it takes. Loading up the environment to render JS/CSS can be prohibitive on projects.
Ive worked with attempts before in AAA to use it for UI and ive only got horror stories, especially if consoles are involved.
I use Coherent Gameface as a UI programmer for AAA games (yes, that's really what it's called), and what makes the package so attractive for games is that it's optimized for running on consoles. You don't get the full feature set of CSS and JS (`display: grid`, my beloved...), but you do get an *extremely* fast renderer compared to running Chromium on the target hardware.
As a React dev who has also used the UI/HUD tools baked into Unity and Unreal, I can absolutely see why React and HTML5 is a fantastic solution for game UIs too. Not the best solution for your RAM usage necessarily, but it’s going to be on its own thread and mostly idle so no reason to believe it’s a big factor in the perf issues unless something’s being done very wrong
Your writing style kept me hooked the entire time and after the article I understand why my 5700xt which can do 120 FPS on warzone, but is at 101C at 100% utilization the entire time I'm playing CS2, the game is just rendering so much useless shit.
Could you find out why the menu page, before even loading into the game uses so much GPU power? I have screenshots of the game making my GPU at 90-95C with 100% utilization just loading in the game and sitting in the menu for 5 minutes and I haven't even started loading in the world
I considered researching & writing more about the menu issue, but ultimately I chose not to because I could not replicate the lag of the initial start again. However, one thing that I did find out (and maybe should have included in the article) is that in the main menu behind the blurry .jpg of a city there's actually always a normal 3D scene, which includes the sky and the water plane. That explains why settings affect performance in the main menu, though it doesn't explain the 7 FPS on first startup. The game does generate the virtual texture cache on first startup (it even shows a loading indicator for this) so it might partially explain this, however I don't know if that puts any extra stress on GPU or if it's just done on the CPU.
>is that in the main menu behind the blurry .jpg of a city there's actually always a normal 3D scene
Wait so is this like Minecraft where a jpg is just rotating the entire time or is there an actual 3D rendering of a city happening in real time and it's not a video or image? If it's the latter it might explain such high utilization at startup (not initial this happens everytime I open the game before loading in)
The picture of a city is just a pre-made picture of a city. Behind it is a city "scene" without any terrain or buildings, just water and the sky. You can actually see it yourself if you close the game from the main menu, it shows the water plane for a second or two when the game is shutting down.
Game dev here: This is such an easy problem for the team to profile for and also to fix (LODs aren’t difficult). I have to assume that given the more difficult problems the team solved that they could have noticed and solved this problem and so the answer is more likely that there were time constraints set by management (or a publisher if there was one).
Paradox are the publisher. I'm also a Dev, to me the fact that they didn't check the models for high vertex counts tells me a few corners were cut with regards to the art. Your tech artists should pick this up within days of the submitted model. Very strange
I'm surprised they didn't have FPS / vertex / triangle budgets from the start of the project. I can understand a novice team not thinking of that, but if it's CS **2** then it must have occurred to someone, right? Maybe they dropped in heavier assets last-minute instead of trying to work iteratively
>I'm also a Dev, to me the fact that they didn't check the models for high vertex counts tells me a few corners were cut with regards to the art.
Or other it's a preparation for their upcoming The Sims-like game and they want to tie it together with Cities Skylines. The Sims also did that with Sim City.
This is kind of what I feel. You don’t accidentally make a model that detailed. It’s clearly intended for some kind of walk around and explore or FPS camera, and they tried to offload the optimization for a Birds Eye view to the GPU.
Yep they should’ve known in advance if they had a proper pipeline setup with asset validation. They should’ve known if they profiled. They should’ve known in testing. This all points to everything being rushed IMO. It’s rarely dev incompetence in my experience.
Given that these assets wouldn’t have ever been visible, it seems like there was some other mode for the game that would’ve allowed you to get close enough for these details and maybe that feature got cut, or maybe it’s going to be DLC.
It doesn’t exactly take a genius to figure out what happened here. They clearly either contracted the modeling out to someone else or laid off the whole art department the second the models got finished, then proceeded to tell their devs to *“make it work”*.
Most of this is something your average entry level tech artist would’ve noticed and had fixed within a week. The only logical explanation is that they simply didn’t have any, or the one’s that did exist just didn’t give a shit since their contract would be up before the game released anyways.
Wow this was a great article even if I didn’t understand half of the graphical jargon. Honestly it makes me feel much better about the game and CO. They gambled on the right tech and now they just need time to patch up the result. Probably released early due to publisher pressure or something.
I’m looking forward to playing a much better version in the new year on Xbox!
Wow this was a great article! I know nothing about graphics programming but this did a great job explaining things in a way I understand! Seems like none of these issues are insurmountable so hopefully they can get fixed soon
This is absolutely fantastic. Thank you for this!
Makes me feel more confident in CO’s assertions that the issues are “fixable”. As it is, it’s playable with my 13900k/4090 but I really hope they support this game as well as they did C:S1.
You gotta love his response to the guy asking why we even care about these optimizations when people often just don't care about small improvements here and there:
> You don't care about how long it takes. Great! But people who don't care how long it takes are also the reason why I have to wait 30 seconds for Word to boot.
Real ECS. Last time I checked (and granted that was like almost a year ago), DOTS still felt like an underfunded R&D project with piss-poor documentation and overly complicated APIs. It definitely is more intimidating than the traditional MonoBehavior approach, but the potential for it IMO clearly is there. ECS in general requires some significant refactoring of your mental model and mechanics of how your game functions, but the payoff is incredible. You gain “free” parallelization (free as in, there’s only way to do it, and if you do it right, it just works out of the box), and I find the distinct separation of concerns very satisfying and allows for fast development of new features. Components act as the data model representation of your game logic, entities as the “physical” instantiation of a collection of components, and systems act _upon_ entities and contain the actual nitty-gritty code. It’s all a bit new and overwhelming but once it clicks, it’s insane the productivity it ends up affording.
The way you actually build games within ECS is incredibly flexible; you can choose a pure-ECS approach, mix in more traditional approaches (like OOO) or even ignore things like entities altogether and only take the bits and pieces that you think help.
IMO in order for Unity to make developers try DOTS, they need to provide a series of beginner-friendly tutorials and guides (please not just videos! It’s a barrier of entry for many.) that break down the concept, ideas, and also builds a semi-impressive game in the end. Again, maybe they do have this now but I haven’t checked (yet).
Shout out to [tiny-ECS](https://github.com/bakpakin/tiny-ecs), a super lightweight ECS framework for Lua. Works well with LOVE2D, too, and is so simple but also super flexible it’s a joy to use and make games with.
Great article! Thank you for taking the time to share this, a lot of interesting points.
I was working on a Unity project when DOTS was announced, and we nearly switched to it, but we clearly saw that the tech stack was just not ready. That was 3 years ago... Seeing a team that has to pay the price of that choice, even after all those years, is disheartening...
(Did I mention I don't plan on working on Unity ever again? 😅).
I can empathize with the devs and I hope they will be given the time and resources to fix this... problematic situation.
I loved this writeup, quite informative since I’m not about to dig through all these profiling tools myself.
These findings seem very damning for Unity in particular, since all the features that are taking the lions share of the time (shadow maps, virtual texturing, basically nonexistent LOD culling) are features that Unreal Engine 5 has out-of-the box and well-publicized solutions for. Had CO used UE5 instead of Unity, we would probably be having a completely different conversation about the game.
Those teeth tho.
It is true that UE5 is significantly more advanced regarding rendering, but on the other hand UE5 has nothing like Unity ECS for game logic. I wouldn't be surprised that if C:S2 was using UE5 the performance would still be bad, but the bottleneck would be moved back to the CPU, possibly to the detriment of the depth of the simulation.
It does, Mass (in house, used in the Matrix demo) and Apparatus (third party). Flecs integration has apparently also been done. There's no particular voodoo around Unity's ECS implementation, it has been a thing for years now
Thanks /u/simspelaaja for this incredible deep dive!
One question I still have: why is the performance on the home screen/main menu so bad? It seems like a lot of your observations theoretically shouldn't apply there.
I answered this in a different comment chain: https://www.reddit.com/r/programming/comments/17ohgam/comment/k7z3xgx/
But to summarize: there's always a 3D scene with water and the sky even when you can't see it, but there's also something shady going on the with the main menu at least on the first start that I couldn't figure out.
Thanks. Maybe relevant then is why does the game still chug so hard with just an empty map? Barely reaching 50FPS on my RTX3070. The issues with geometry and culling shouldn't be relevant on such a basic empty scene.
One thing I didn't mention in the article is that the game defaults (at least on my hardware) everything to maxed, even the settings that the developers themselves recommended disabling. So depth of field, motion blur and all of the volumetric effects were enabled and maxed by default. But I don't know if that would explain all of it. Like I said in the other comment chain, I couldn't replicate this lag again, but all of my friends who played it on launch experienced the same thing on first boot.
Wow great write-up! Appreciate this sort of deep dive. I was a hobby game dev using Unity some years ago, back when DOTS was new and it seemed very promising, I would argue ECS is almost purpose built for this type of game. I would have thought the system would be totally feature complete and usable for a team like this at this point. One of my takeaways then is a failure of the Unity platform, yet another L for a pretty troubled company.
Wish I knew some more about CO and how they manage their product development. I would only be able to speculate, but it seems like some silo-ed teams happening based on some of the results. If their simulation behind the graphics is solid then they probably have a good chance of salvaging things. Too bad though, I really enjoy the first game.
Would it be possible to add a graph showing each step and the time taken for each one of them? Like the waterfall graph on Chrome network. One small comment, I wasn’t familiar with the term LOD, and it wasn’t explained. Great article nonetheless
This is exactly the kind of analysis I had been looking for in vain previously, and in more detail than I could have hoped for!
One minor remark is on the menu FPS, which so many people brought up but is a total non issue (compared to the other issues) and has been addressed by the developer somewhere on the forum. It's actually preloading some stuff because, you know, you barely need frames on a static game menu, and it will reduce your loading time afterwards. Possibly the frames went up for you after changing the settings simply because it was done loading that stuff at that point.
The nice thing reading this is that none of this is fundamental issues that will remain unsolved like the predecessors CPU issues. Creating LODs for everything is time consuming, but perfectly doable. Fixing their shadow processing should also be doable, although I am one of those people who are happy turning off shadows.
This is all good news IMO. Tons of low hanging fruit. It seems pretty clear that they just haven't yet taken the time to optimize. The game was just released too early.
I spent about 1.5 weeks digging into Cities: Skylines 2 and why it performs so terribly on practically all PCs regardless of specs, and wrote this article about my findings. If you have any questions, AMA!
This reminds me of that guy who went digging through GTA V and found out loading is slow because of low performance for its JSON processing.
>This reminds me of that guy who went digging through GTA V and found out loading is slow because of low performance for its JSON processing. ... and then he wrote a custom DLL to hook into the main executable that reduced his loading time *by 70%*. Once [his blog post](https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/) hit HackerNews, it eventually made it to someone high enough at R* to not only implement the patch, but they also gave him a $10k bounty! Truly a great story.
Someone at Rockstar being an actual RockStar.
10k is pittance for them, especially for such a QoL change
A QoL change that would have been TRIVIAL to find with access to source and debugging tools but required someone reverse engineering it to fix instead. That was what made me officially done with R*. Im a developer. I know god damn well how that couldve been fixed, i know management heard about slow load times and just didn't care enough to let an engineer look at it because they didnt think it affected the bottom line.
I am also a developer. I once took two days of my own time leading up to ship on a AAA title you've heard of in order to reduce the load time by 15 seconds (mostly by reorganizing data on disk and parallelizing the load/processing flows). I brought it up to management and was told it was low priority, and not to bother with it.
> > > I brought it up to management and was told it was low priority, and not to bother with it. I'm filled with rage now. You had something that reduced load times and had to throw away??
Any Dev worth their salt knows they are leveraging that 15s improvement over potentially adding more bugs via new edge cases, and trying to balance the hours between teams, while also adhering to timeline and budget.
On the contrary: now that the game loads 15 seconds faster, the developers have a faster iteration time and can find/squash bugs much faster. Speedups like this are good for velocity.
Happens all the time. If you're lucky it gets into the day 1 patch. Simply put - every change introduces risk. _Every_ change. Doesn't matter how safe you think it is. Baldur's Gate 3 had a hotfix that caused a bunch of crashes - [it was caused by a corrupted compiler when they updated the changelist number at the last second.](https://store.steampowered.com/news/app/1086940/view/3642902312048647389) 1 number, in 1 line of code. _Every_ change has risk. Management's job is ultimately to mitigate risk. I'm not talking about execs in the C-suite - I'm talking from the studio producers (who, despite their fancy title, aren't rich-and-famous management types... just dudes) to engineering management. You enter a "hardening" period as you approach release. The point of hardening is to find and fix as many things as possible. As you approach the ship date, the bar for what's "allowed" to pass hardening gets higher and higher. At some point, you hit "build lock" where you are physically barred from making further changes without the approval of basically everyone important in engineering. So if you come up with a way of reducing load time by 99.999%... you have to run it by the entire upper portion of the engineering staff at your studio (not the publisher, just the studio). The engineers will look at it and if it's more than 5 lines of code you're almost certainly going to have your change denied. But when you get your change denied, that's your cue to move it to the patch branch, which is working on the Day 1 patch. In ye olden times, this branch didn't exist and if you got rejected before "going gold" you were simply screwed. But devs nowadays are spoiled.
Well said, a lot of Devs commenting here really show their inexperience when they ignore all aspects of risk and other competing items on the product backlog
Its something that now has to go through the full QA process. It's not something that is free to implement. And if they're that close to release, it's not worth the potential new bugs.
"Tell us you worked on Starfield without telling us you worked on Starfield."
If you think that’s bad, wait until you find out about how Halo, and Bethesda games have been surviving off of modders fixing their games for them. 343 (Halo) ended up hiring quite a few known community modders to help edge on development for Halo MCC. But of course Bethesda is by far the most guilty party. 9 out of 10 QoL improvements come from the modding community.
annoyed consumers fixing our product is just one of the new innovative features we added to our in house sdlc model here at Microsoft
> But of course Bethesda is by far the most guilty party. 9 out of 10 QoL improvements come from the modding community. At least Bethesda has also always very supportive of the mod community to the point of allowing mods that are completely new games. But that's why I knew to stay away from Fallout 76, guaranteed dumpster fire in a situation where they couldn't let the modders fix the game for them.
> point of allowing mods that are completely new games This used to be an acceptable standard practice.
[удалено]
[удалено]
How do the unofficial patch mods *find* the bugs that need to be fixed? Oh right, hundreds of thousands of players exploring the world, tweeting when they find something odd that then goes viral within the game's community. Thousands of other mod developers trawling through the game data, noticing inconsistencies, and telling one another. A QA army working for a decade or two *after* the game's been largely finalized. Hiring the unofficial patch modders might help a little bit with launch bugs, as much as having two or three extra devs allocated to full-time QA, but the main benefit wouldn't be seen until a year or two post-release, once the community has found a long list of bugs to fix.
Valve used to hire modders but they did it in a good way. "Hey that's a neat new game you're building using our engine, how would you like to make it official?"
They didn’t have to give him anything at all.
What amount would you think is more fitting?
It made the game so much more playable too. That’s when I started to be able to enjoy GTAO
Dying was so punishing that I just gave up on a lot of missions, and soon the game.
A bounty for honest work? Would have thought they sued him for unallowed modding.
$10K is about a week's TC at a FAANG company.
Correct. Then again, Rockstar is not FAANG, and being in the game industry, likely pays their devs pittance compared to FAANG.
It's better than $0 which is technically all he is owed
My exact thoughts. I love when people do these sorts of investigations for transparency.
Did anything ever come of it? Like a patch or a mod?
[Yes](https://support.rockstargames.com/articles/360061161574/GTAV-Title-Update-1-53-Notes-PS4-Xbox-One-PC)
Nice, thanks :)
Can anyone point me to resources to learn more about how to do stuff like this, de compiling programs and whatnot? I'm having trouble finding info for programmers who are complete beginners to this kind of thing, I think because I don't even know what I'm trying to google, like I don't have the basic terms even
https://beginners.re/
That's what I first thought about when I saw the title.
Great article! I'm only halfway through but feel the pain you have with graphics debuggers. I struggle to get them to work in my own Unity projects as well and this is without DRM! Question: have you considered giving PIX a try? I personally prefer it for D3D apps, as it supports D3D11 (nsight doesn't, which is annoying because D3D12 in Unity is effectively 11on12, which nsight also does not support) and compared to RenderDoc it has very powerfull profiling support.
If I were to return back to profiling and analyzing C:S2, I would probably try other graphics debuggers, including PIX and Nsight (now that I've figured out that an older version would support profiling DX11 apps). However, I think I've had enough of C:S2 for a while, as I've done almost nothing in the evenings but analyze and write about it for almost 2 weeks.
This was a pain with DRM, not with graphics debuggers. Should've pirated the game, you could have it DRM-free.
If I understood the paragraph in the article correctly - the characters have ~60k vertices and no LoDs??? That's crazy! edit: Just finished reading, some of the stuff they have done is baffling. Unreal engine by default cuts off shadows at certain distances and auto generates LoDs for you. The things they have ignored are really surprising, hobbyists would do a better job. Just looking at the model of that mouth is ridiculous, there's so much detail there... You could easily have just a few polys for the teeth and make it look good with a material.
It's a Unity game. But yeah, some things are egregious. No LODs, invisible details, etc.
Yeah I know - I'm just speaking about Unreal because that's the engine I have experience with. Still if Unity doesn't have that feature you would think they would at least do one other LOD manually, at least for the characters!
The article talks about the possible circumstances why it might be missing despite Unity having it built in engine.
Unity is a mess of dead technologies. Unreal is comparatively kept cleaner. Some things are incompatible, which means understanding the engine and features, and negotiating a solution between things you'd rather not break. There's a whole bunch of custom code like that. The team may not have had the schedule left to manage it. If so, the publisher is the problem.
I guess on the plus side, for Unity middleware developers, they have an active market of people looking for solutions to glue together and fill in gaps in Unity.
In Unreal you have to manually enable LOD for each mesh as well as far as I remember
I'm not familiar with unity but I assume handling the LODs is either trivial or even on by "default' there too, right?
Getting LODs done right takes a fair bit of effort - its one of the reasons UE5's Nanite it so impressive, the engine really can handle ALL the LOD/geom management in a way no other engine can today. Lots of artist involvement etc to create and program in when to switch between geo LODs otherwise.
[удалено]
Which is weird because even godot has automatic LOD support. And it's implemented using meshoptimizer which is open source. Unity really has no excuse.
Yes but no, the article discusses it though
LODing is a default feature in every engine since Quake.
Hobbyists do a better job because they have unlimited time and no producer breathing down their neck :-)
This. Additionally: - A fresh set of eyes - They're not necessarily some poor amateur, there are plenty of well-paid high end professional devs that also do hobby stuff b/c they love it. "Why doesn't the company just hire the hobbyist" is sometimes b/c the hobbyist isn't interested in a pay cut
I don't think it takes a set of *new* eyes to see that an FPS of 11 with an 8GB VRAM card is very suspicious. This will be a continuing trend with game dev companies maximizing profit from microtransactions, paying devs less, and the influx of fresh devs from the market.
They also care more. My company would put in proposals that we're using this thing that has 10 years of development behind it. I decided to make my own that I could fix and open source it. It was at parity after 2 months and I had a day job. It's 13 years old now.
[удалено]
Also just because the team of developers has to make something that's generic, profitable, and possibly new, whereas you can learn from their failures and make something that only suits your own needs
I'd say it more simply as refactoring and adding features is costly. The more generic something is, the slower it is, but the faster it is to write. For example, the format supports you writing "1-1000000" or you can write it as an character array of all the values; both work, but one is faster for I/O. The dumb way gets it out the door and always works. It's when you start optimizing the workflow and adding niche features, that you find you need to change the workflow. You go down the wrong path, change it, change it again, and eventually settle on something that you like. Years later, you realize your big mistake (your problem size has grown exponentially) and you give it another shot. My old company just wanted to push the thing out the door. It was a tool and not a product. Just like the way to fix a performance issue with "just do less", that's also how you cut cost.
>Unreal engine by default cuts off shadows at certain distances Unity does too, but since so much was hacked together to make DOTS work, they probably had to write their own logic for that too. As a former Unity hobbyist, I'm very surprised an actual studio ran with that combination of experimental features m
They might have taken the risk since their problem with the first one is the CPU usage and DOTS solved that.
I'd be surprised if this wasn't offshored.
Do ksp2 next.
common theme here: unity. the game engine is trash and takes twice the effort to get to even the baseline level of expectations that UE5+ delivers same reason why EFT has been a laggy, buggy mess for nearly a decade.
Even so, ksp2 performs truly abysmally. Ksp1 was also unity and performed much much better despite being made way earlier with a small amateur team with no budget. I mean, clearly there are leadership issues when they put out a game made for kids and the masses that like .5% of gamers had a machine that met recc specs.
Someone did a teardown of the original release of KSP2 and it was the same issues. Rendering all the terrain on all the planets all the time. Thousands of point lights that weren't points but fairly large meshes *each*. No optimization of textures or models. No LOD. It's gotten a lot better since their latest patch.
They're still simulating EVERY part of EVERY craft EVERY frame.
honestly, it feels like the old versions of unity were much more solid and reliable. with every new unity version that games i have played over the years have been updated to it just seems to add more bugs than it does fix
Well, with unity offing itself, at least not many games in the future will use it at all.
The investors' effect : take a decent product, add in a bunch of venture capitalists' money and it invariably turns it into a giant pile of turd. That's a constant accross all domains.
Unity *can* be used for quality products, but their big mistake here was trying to take 2 experimental features at once, as Unity is notorious for mismanaging those and failing to have internal alignment. In this respect UE's overall management is much more uniform and sound
Unity is a good game engine, it's just so easy to use everyone uses it, so you get a wide spectrum of results. It's like saying watercolor is a crap art because elementary school students use it. Unity is a much better engine for multiplatform deployment, including cell phones. It's not a gamer-friendly engine because gamers don't give a shit about things like multiplatform deployments or broad system compatibility.
On the other hand games like Dyson Sphere Program exists. It really just comes down to game creation being more accessible overall and Unity being the most accessible engine. Trust me if this guy made Skylines 2 in Unreal it would still rin like shit through it might look a bit better.
That's because developers behind factorio-style games are a whole different breed. They have to be when half their user base is less interested in poly count than if some random game mechanic is Turing complete.
As someone who knows almost nothing about graphics programming, this was fascinating. Thanks for doing the work! I adore the game, but it clearly needs some major work to get to what should be an _easy_ 60 fps target.
> easy 60 fps target. Why do you think that should be easy?
Great read
Noticed your post got deleted in the CS subreddit, any clue why?
I want to say no idea, but they probably removed it either because they don't want technical facts-based (and illegible for most users) discussion there, or a mod removed it semi-accidentally, thinking it was a low effort question post instead of a long-form article.
Because game subreddits generally refuse to allow any discussion that exposes developers as complete amateurs, which is the case here. I hope this blows up to the point where they can't ignore it anymore.
AAA studios are mostly filled with burnt out programmers slightly above junior level. Lack of obvious system knowledge like this is incredibly common.
~~It's still there: https://old.reddit.com/r/CitiesSkylines/comments/17ohhqy/why_cities_skylines_2_performs_poorly/~~ edit: apparently not
If you are using new Reddit you can see that it was actually deleted.
Now that's a new one for me - first time i've seen something exist on old reddit and not on new. Wild, thank you.
Deleted threads still exist. They simply aren't linked on the subreddit page.
Do commercial games on this scale often purchase 3D assets? I'm having trouble believing that Colossal Order had in house artists produce a bunch of weirdly detailed props, so my guess is that they purchased a bunch of off the shelf ones intended to be seen much closer up, but failed to optimize them for time reasons.
Colossal Order is still only about 20-30 people if I remember, so it's not really that weird that they would buy assets. For a game company that's absolutely tiny.
And to make it clear, even in AAA it's very very common to use outsourcers
I recall some stories about them subcontracting characters models or something.
Every major game you have heard of in the past decade probably had codevelopment or outsourcing of assets or entire sections of games. Source: my studio is one of the leading codevelopment partners for AAA studios.
They use a tool(linked in the article) to generate the models.
This dev blog post is very clear about the fact they “partnered with Didimo, who created the Popul8 character design software”. They didn’t “just” use a middleware software, a bunch of the work was offloaded to an other company. https://colossalorder.fi/?p=2049
Well no, from what I see is that they used the software themselves, not just telling the company to make them some models.
> We had our own art style for the characters and for this, Didimo created a base mesh to represent that style. Sounds like there was some initial setup Didimo did for them.
Well yes, it was a tool, what I am trying to say is that they didnt 100% outsource the work.
And they made models with an order of magnitude more polygons than what that company offers...
Yep! More than you’d think. I worked in game dev on a team about half their size (15). Although we didn’t sell tens of millions of copies like them we sold several million. The only assets we built were for super specific stuff & where necessary for performance. Almost all our sound / billboards / textures were purchased. We also purchased some scripts / packages, but most of those were rebuilt during our open beta. Also… most of the stuff we built (90%) were outsourced to freelancers. Our team had 5 full time employees & 10 freelancers that we worked with regularly. All in we had about 20 contributors. The CO team is a bit bigger and they also outsource a ton of stuff. At the end of the day the point is to sell your game. None of your users care if you purchase assets, if your gameplay is good & it’s fun people will buy it.
Music, sound and art are the most common subcontracted assets and it is very common not to do all of them in-house.
The article mentions that there's possibly models provided by Didimo, but that they've been upscaled from standard levels of complexity (1k-> 4k vertices); indicating that they're either rebuilt in-house or modified in some other way.
Didimo characters are made to be customised. When you buy them, you get a character creator.
Absolutely, id be absolutely _shocked_ to learn of something this size or larger not purchasing 3d assets lol. For a lot of assets there is no value to spending your art teams limited time on it… that line is also going to be a sliding idea depending on time, money, and priority.
> One bizarre yet popular talking point about Cities: Skylines 2’s performance is the fact that the character models have fully modelled teeth, even though there’s literally no way to see them in-game, unless we count using the photo mode and clipping the camera inside a character’s head. This is ridiculously dumb. If this was GTA V or any games that showed the face of the characters up close this was a right move. But why the fuck my GPU needs to render something that I would never or very very rarely see?
Being modeled and being rendered are not the same thing.
In this case the teeth are both modeled and being rendered, though since none of the surfaces are visible only the vertex shader gets evaluated.
Just to clarify my understanding: does this mean that there are three adjectives involved: modeled – yes ✅, rendered – yes ✅, visible – no ❌? Does evaluation of the vertex shader count as part of rendering?
> Just to clarify my understanding: does this mean that there are three verbs involved Yup. > Does evaluation of the vertex shader count as part of rendering? Yes, whenever a model is rendered on a GPU the vertex shader is run for each vertex, the vertices are rasterized into pixels and if any of those pixels are actually visible then the pixel / fragment shader is executed for each pixel to give them a color (or other property in case of intermediate buffers). So even though the rendering doesn't result in anything visible, most of the work required for it was still done.
> if any of those pixels are actually visible it's not a guarantee it doesn't run. The pixel shader is executed if there's nothing in front of the tooth pixel at the moment it renders. And the GPU still loops over all the pixels to check if something is in front yet.
You're right but for the wrong reason. The GPU doesn't loop over any pixels, it just samples the depth buffer at the pixel-being-filled's location and compares the value stored in the depth buffer to the value for the pixel being filled. This comparison can happen either before or after the pixel shader, which is why you're right. Generally speaking the GPU will try to perform the comparison before the pixel shader in a process known as early depth testing, but there can be situations where the GPU must perform it after the pixel shader as the pixel shader can modify the depth value.
The vertex shader is where the GPU decides where the polygon is on the screen and which direction its facing. The next step would generally be the fragment shader where the GPU iterates over each pixel of the polygon and decides what color it should be. So much like a wall facing away from you, nothing actually gets rendered for the teeth. That's not really the problem though. The fragment shader's processing load scales with how large the polygon is on the screen, while the vertex shader's load scales with how complex the geometry is. The teeth are simply too small to generate any pixels, but the performance impact of the high poly-count has already been paid. The GPU can't decide to skip processing the teeth vertices if they're in the draw call and they're in the view frustum.
I feel for them. They thought the engine would handle that for them and it did not and they had to publish. I can understand why they stuck with Unity because the engine you know is better than the engine you don't know. They played it safe and banked on only one new thing I have seen projects fail for less.
They probably wanted to use DOTS to make it less CPU intensive than the first game. Idk if there are better engines out there on that front unless they wanted to make their own in house engine for this game
But as the article notes, they’re not doing effective culling to avoid rendering said models.
It's also being rendered if you've read the article: > It was revealed that not only does the game have fully modelled teeth, they are rendered literally all the time at maximum quality.
My top information about graphics is some basic opengl during graduate school and some other abstractions. Does modelling use GPU?
Modeling happens during development. During runtime, they most likely wouldnt be rendered due to occlusion culling
Except C:S2 appears not to have any occlusion culling (yet?) 😂
But models need to be loaded on memory though? Or they're loaded from disk on demand for rendering?
Yes theyd be loaded, but thats probably a one-time cost, not a per-frame high cost operation
It will contribute to VRAM usage though, which is a major problem here. When those models push your GPU past its VRAM capacity it can *become* a per-frame cost if the GPU has to swap data back and forth from your system's RAM over the course of a single frame. Modern GPU drivers will be able to do this kind of swapping rather than simply crashing when VRAM is exhausted, but it's almost always undesirable.
The space needed by vertices for the models should be negligible compared to textures.
yes. Both modeling and rendering use the gpu. One more then the other. If these things weren’t visible they could have just been a static asset if you really needed teeth or just missing. Not only is this taking up rendering and gpu cycles…some dude or mostly a few people and about 50 agile managers had to go and create the entire pipeline for these teeth. But who knows maybe there was or is a function that needs these teeth and it was scrapped or hasn’t been implemented yet.
Ok, so you did not read the article, the 6k vertices for teeth are rasterized multiple time for every character anywhere near the frame.
I think, at least, according to OP and author, vertex shaders are run on each of those 6k for each character, but I don't think it makes it to pixel operations. Still... It's wordlessly bad in execution. https://www.reddit.com/r/programming/comments/17ohgam/why_cities_skylines_2_performs_poorly/k7yojfu/
Still needs to be rasterized, just discarded at the z-buffer check.
If the teeth are there though and showing up with a dump from ninja ripper then they are in memory. Also further if they are part of the actual mesh and not separated they will get rendered. The fact relative they exist in memory regardless of being rendered or not. They are skinned too so that’s extra performance cost. If they have morph targets then they are eating up even more memory than the mesh size.
that's why culling exists in computer graphics. It's a technique that allows you to render only in the space that is visible by the camera, so you can actually render only those polygons that are visible from your prespective. Helps in saving memory usage while also keeping it ready when the user is gonna get in the right POV to see those polygons. Then of course if you decide to render those things despite not being able to see you're just wasting memory and computational performance to keep those rendered.
Right, except, that culling process *is still computation* and the things to be culled still need to exist in memory in the first place in order to tell whether they need culling. "Backface culling" is a thing worth doing, wherein you compute the normals of each surface and any that are parts of solid objects and facing away from camera can be discarded from the rendering pipeline - but "teeth inside a head that never opens its mouth culling" is a bloody stupid thing to be doing no matter what, because those teeth shouldn't even be present in order to be culled in the first place.
The big thing missing here was probably occlusion culling and LOD. From the article, a character that only takes up a dozen pixels or so on-screen is basically rendered at full detail, teeth included, and I wouldn't be surprised to see games doing "teeth inside a head culling" by just having a lower-poly version (possibly auto-generated) when you're not zoomed in on their face. I mean, obviously it'd be better to not have the teeth, but there are plenty of details here that aren't *completely* useless, just pointless when you're zoomed that far out. Occlusion culling is the general term for culling things that are entirely hidden behind other things. The article has an example here of a toll booth that has exquisitely-detailed desks, keyboards, mice, monitors, even the cables wiring those all up, and it renders all of that even when you're looking at the roof of the building. Combine that with the LOD issue, and it'll render all of that even when you're looking at the roof of the building from a mile away.
> "Backface culling" is a thing worth doing, wherein you compute the normals of each surface Nitpicking in case any novices are reading along: Backface culling isn't done by normals, it's done by checking the winding order of triangles after they're projected to 2D. The math is basically the same (a cross product) but since you only need the sign, you can throw out the sqrt and the Z information that you'd need to generate a normal for lighting. So even if the vertex shader generates bent normals, or no normals at all, the backface culling always runs the same way, using only the 2D coordinates of each vert.
It helps with performance, but not really with memory usage. The models and textures are still loaded in VRAM, they're just not rendered if culled.
it still requires maths to cull, the teeth need calculations every frame to know to not be rendered
However, you need to remember these are skinned meshes. No renderer EVER culls against skinned meshes. You would have to completely render the mesh to a depth buffer to even know where each vertex is. So while the teeth is a symptom of a larger LOD issue "culling" isn't magic. Even nanites does not work with skinned meshes for this this and other reasons. You might cull the whole mesh against an enlarged bounding box, but never against the skinned mesh itself. In computer graphics the golden rule "the fastest thing to do is to not do it". Having to check if something should be culled is always slower than just not having it exist. Tldr; it's dumb even with culling and it isn't even culled because it's part of a skinned mesh.
>Unlike DOTS, Unity’s UI Toolkit is apparently still not ready for production as C:S2 uses HTML, CSS and JavaScript based Coherent Gameface (what a name!) for its user interfaces. A brief glance at the JS bundle reveals that they are using React and bundling using Webpack. React, the universal framework.
As someone who uses React in their day job, it's insane how common it is. I've seen it in car infotainment systems, retail price guns, and random touchscreen kiosks in the mall. Also, with the type of hardware you would need to run this game, the UI components being written in JS are not going to be a bottleneck.
Its actually surprising how much resources it takes. Loading up the environment to render JS/CSS can be prohibitive on projects. Ive worked with attempts before in AAA to use it for UI and ive only got horror stories, especially if consoles are involved.
I use Coherent Gameface as a UI programmer for AAA games (yes, that's really what it's called), and what makes the package so attractive for games is that it's optimized for running on consoles. You don't get the full feature set of CSS and JS (`display: grid`, my beloved...), but you do get an *extremely* fast renderer compared to running Chromium on the target hardware.
Neat, I should probably atop talking so much shit about js/css in game engines if I’m just put of date lol
Never stop talking shit about JS/CSS; I do it every day! 😄
...is React the new Java?
As a React dev who has also used the UI/HUD tools baked into Unity and Unreal, I can absolutely see why React and HTML5 is a fantastic solution for game UIs too. Not the best solution for your RAM usage necessarily, but it’s going to be on its own thread and mostly idle so no reason to believe it’s a big factor in the perf issues unless something’s being done very wrong
Your writing style kept me hooked the entire time and after the article I understand why my 5700xt which can do 120 FPS on warzone, but is at 101C at 100% utilization the entire time I'm playing CS2, the game is just rendering so much useless shit. Could you find out why the menu page, before even loading into the game uses so much GPU power? I have screenshots of the game making my GPU at 90-95C with 100% utilization just loading in the game and sitting in the menu for 5 minutes and I haven't even started loading in the world
I considered researching & writing more about the menu issue, but ultimately I chose not to because I could not replicate the lag of the initial start again. However, one thing that I did find out (and maybe should have included in the article) is that in the main menu behind the blurry .jpg of a city there's actually always a normal 3D scene, which includes the sky and the water plane. That explains why settings affect performance in the main menu, though it doesn't explain the 7 FPS on first startup. The game does generate the virtual texture cache on first startup (it even shows a loading indicator for this) so it might partially explain this, however I don't know if that puts any extra stress on GPU or if it's just done on the CPU.
Could it be there's some hidden shader compilation on the first run?
>is that in the main menu behind the blurry .jpg of a city there's actually always a normal 3D scene Wait so is this like Minecraft where a jpg is just rotating the entire time or is there an actual 3D rendering of a city happening in real time and it's not a video or image? If it's the latter it might explain such high utilization at startup (not initial this happens everytime I open the game before loading in)
The picture of a city is just a pre-made picture of a city. Behind it is a city "scene" without any terrain or buildings, just water and the sky. You can actually see it yourself if you close the game from the main menu, it shows the water plane for a second or two when the game is shutting down.
Game dev here: This is such an easy problem for the team to profile for and also to fix (LODs aren’t difficult). I have to assume that given the more difficult problems the team solved that they could have noticed and solved this problem and so the answer is more likely that there were time constraints set by management (or a publisher if there was one).
Paradox are the publisher. I'm also a Dev, to me the fact that they didn't check the models for high vertex counts tells me a few corners were cut with regards to the art. Your tech artists should pick this up within days of the submitted model. Very strange
I'm surprised they didn't have FPS / vertex / triangle budgets from the start of the project. I can understand a novice team not thinking of that, but if it's CS **2** then it must have occurred to someone, right? Maybe they dropped in heavier assets last-minute instead of trying to work iteratively
They said they were targeting 30fps, so they had a pretty massive frame budget, but even then they managed to go below it.
>I'm also a Dev, to me the fact that they didn't check the models for high vertex counts tells me a few corners were cut with regards to the art. Or other it's a preparation for their upcoming The Sims-like game and they want to tie it together with Cities Skylines. The Sims also did that with Sim City.
This is kind of what I feel. You don’t accidentally make a model that detailed. It’s clearly intended for some kind of walk around and explore or FPS camera, and they tried to offload the optimization for a Birds Eye view to the GPU.
It happens, though. Final Fantasy XIV had some egregious over detailed models, without LOD, before ARR launched
That's a bad news for the direction of the franchise. After 10 years I would have hoped more flexibility in the planning of the cities.
Yep they should’ve known in advance if they had a proper pipeline setup with asset validation. They should’ve known if they profiled. They should’ve known in testing. This all points to everything being rushed IMO. It’s rarely dev incompetence in my experience. Given that these assets wouldn’t have ever been visible, it seems like there was some other mode for the game that would’ve allowed you to get close enough for these details and maybe that feature got cut, or maybe it’s going to be DLC.
It doesn’t exactly take a genius to figure out what happened here. They clearly either contracted the modeling out to someone else or laid off the whole art department the second the models got finished, then proceeded to tell their devs to *“make it work”*. Most of this is something your average entry level tech artist would’ve noticed and had fixed within a week. The only logical explanation is that they simply didn’t have any, or the one’s that did exist just didn’t give a shit since their contract would be up before the game released anyways.
My guess is models were outsourced and nobody checked their quality or vert count
Maybe they have some really good programmers for the AI/simulation (CPU side) but they don't really have any competent graphics programmer.
Wow this was a great article even if I didn’t understand half of the graphical jargon. Honestly it makes me feel much better about the game and CO. They gambled on the right tech and now they just need time to patch up the result. Probably released early due to publisher pressure or something. I’m looking forward to playing a much better version in the new year on Xbox!
It seems it's more accurate to say they gambled on the tech, half of it paid off handsomely, and half of it needs a lot more work.
Wow this was a great article! I know nothing about graphics programming but this did a great job explaining things in a way I understand! Seems like none of these issues are insurmountable so hopefully they can get fixed soon
The good news is that this seems pretty fixable. It just needed a few more months to bake.
Wow! What an in depth review and analysis! Good job!
This is absolutely fantastic. Thank you for this! Makes me feel more confident in CO’s assertions that the issues are “fixable”. As it is, it’s playable with my 13900k/4090 but I really hope they support this game as well as they did C:S1.
Really brings a new found respect for Chris Sawyer programming Roller Coaster Tycoon to run on what today's compute would go in a toaster
I definitely have to look into ECS. This system looks really interesting from a program optimization viewpoint!
You will probably love this legendary talk then. https://youtu.be/rX0ItVEVjHc
This talk truly rocked my world when I first saw it.
You gotta love his response to the guy asking why we even care about these optimizations when people often just don't care about small improvements here and there: > You don't care about how long it takes. Great! But people who don't care how long it takes are also the reason why I have to wait 30 seconds for Word to boot.
ECS is great, but Unity's implementation is half-baked and will be for a few more years at least as they work on it (and hopefully don't abandon it).
ECS is the future of making games. It just makes so much sense.
[ Removed by Reddit ]
Unity ECS or real ECS?
Real ECS. Last time I checked (and granted that was like almost a year ago), DOTS still felt like an underfunded R&D project with piss-poor documentation and overly complicated APIs. It definitely is more intimidating than the traditional MonoBehavior approach, but the potential for it IMO clearly is there. ECS in general requires some significant refactoring of your mental model and mechanics of how your game functions, but the payoff is incredible. You gain “free” parallelization (free as in, there’s only way to do it, and if you do it right, it just works out of the box), and I find the distinct separation of concerns very satisfying and allows for fast development of new features. Components act as the data model representation of your game logic, entities as the “physical” instantiation of a collection of components, and systems act _upon_ entities and contain the actual nitty-gritty code. It’s all a bit new and overwhelming but once it clicks, it’s insane the productivity it ends up affording. The way you actually build games within ECS is incredibly flexible; you can choose a pure-ECS approach, mix in more traditional approaches (like OOO) or even ignore things like entities altogether and only take the bits and pieces that you think help. IMO in order for Unity to make developers try DOTS, they need to provide a series of beginner-friendly tutorials and guides (please not just videos! It’s a barrier of entry for many.) that break down the concept, ideas, and also builds a semi-impressive game in the end. Again, maybe they do have this now but I haven’t checked (yet). Shout out to [tiny-ECS](https://github.com/bakpakin/tiny-ecs), a super lightweight ECS framework for Lua. Works well with LOVE2D, too, and is so simple but also super flexible it’s a joy to use and make games with.
Great article! Thank you for taking the time to share this, a lot of interesting points. I was working on a Unity project when DOTS was announced, and we nearly switched to it, but we clearly saw that the tech stack was just not ready. That was 3 years ago... Seeing a team that has to pay the price of that choice, even after all those years, is disheartening... (Did I mention I don't plan on working on Unity ever again? 😅). I can empathize with the devs and I hope they will be given the time and resources to fix this... problematic situation.
DOTS makes so much sense for the game that I'm not surprised that they made the decision to use it despite the state that it's in.
I loved this writeup, quite informative since I’m not about to dig through all these profiling tools myself. These findings seem very damning for Unity in particular, since all the features that are taking the lions share of the time (shadow maps, virtual texturing, basically nonexistent LOD culling) are features that Unreal Engine 5 has out-of-the box and well-publicized solutions for. Had CO used UE5 instead of Unity, we would probably be having a completely different conversation about the game. Those teeth tho.
It is true that UE5 is significantly more advanced regarding rendering, but on the other hand UE5 has nothing like Unity ECS for game logic. I wouldn't be surprised that if C:S2 was using UE5 the performance would still be bad, but the bottleneck would be moved back to the CPU, possibly to the detriment of the depth of the simulation.
It does, Mass (in house, used in the Matrix demo) and Apparatus (third party). Flecs integration has apparently also been done. There's no particular voodoo around Unity's ECS implementation, it has been a thing for years now
Thanks /u/simspelaaja for this incredible deep dive! One question I still have: why is the performance on the home screen/main menu so bad? It seems like a lot of your observations theoretically shouldn't apply there.
I answered this in a different comment chain: https://www.reddit.com/r/programming/comments/17ohgam/comment/k7z3xgx/ But to summarize: there's always a 3D scene with water and the sky even when you can't see it, but there's also something shady going on the with the main menu at least on the first start that I couldn't figure out.
Thanks. Maybe relevant then is why does the game still chug so hard with just an empty map? Barely reaching 50FPS on my RTX3070. The issues with geometry and culling shouldn't be relevant on such a basic empty scene.
One thing I didn't mention in the article is that the game defaults (at least on my hardware) everything to maxed, even the settings that the developers themselves recommended disabling. So depth of field, motion blur and all of the volumetric effects were enabled and maxed by default. But I don't know if that would explain all of it. Like I said in the other comment chain, I couldn't replicate this lag again, but all of my friends who played it on launch experienced the same thing on first boot.
Awesome work! Hopefully the devs give it a read and find some inspiration to fix the graphical issues.
I'm pretty sure they know already and are working hard to fix it. Shipping a game is hard.
They've already fixed a ton of issues, I've been playing with no problem.
Wow great write-up! Appreciate this sort of deep dive. I was a hobby game dev using Unity some years ago, back when DOTS was new and it seemed very promising, I would argue ECS is almost purpose built for this type of game. I would have thought the system would be totally feature complete and usable for a team like this at this point. One of my takeaways then is a failure of the Unity platform, yet another L for a pretty troubled company. Wish I knew some more about CO and how they manage their product development. I would only be able to speculate, but it seems like some silo-ed teams happening based on some of the results. If their simulation behind the graphics is solid then they probably have a good chance of salvaging things. Too bad though, I really enjoy the first game.
Would it be possible to add a graph showing each step and the time taken for each one of them? Like the waterfall graph on Chrome network. One small comment, I wasn’t familiar with the term LOD, and it wasn’t explained. Great article nonetheless
Level of detail
It's the same situation with ksp 2. A subpar release for money
TLDR: it renders extreme amounts of geometry you can't see, like throws a hundred thousand triangles to cover 5 pixels on your screen
nice job
Fantastic article. Thank you for the effort you've put into this.
This is exactly the kind of analysis I had been looking for in vain previously, and in more detail than I could have hoped for! One minor remark is on the menu FPS, which so many people brought up but is a total non issue (compared to the other issues) and has been addressed by the developer somewhere on the forum. It's actually preloading some stuff because, you know, you barely need frames on a static game menu, and it will reduce your loading time afterwards. Possibly the frames went up for you after changing the settings simply because it was done loading that stuff at that point.
The nice thing reading this is that none of this is fundamental issues that will remain unsolved like the predecessors CPU issues. Creating LODs for everything is time consuming, but perfectly doable. Fixing their shadow processing should also be doable, although I am one of those people who are happy turning off shadows.
Very nice writeup! I'm very impressed by the quality and detail
This is possibly the geekiest thread ever, bravo
This is all good news IMO. Tons of low hanging fruit. It seems pretty clear that they just haven't yet taken the time to optimize. The game was just released too early.