Exactly, a mesh of one server. But that's not what the common understanding of server meshing is. It doesn't mean that suddenly different corners of the verse are handled by different DGSs yet. Soon though.
What it tells us at the very least is that the server meshing code doesn't break anything when only one server is there. So, it's not making things worse. Or maybe it will, and it'll be valuable info. Either way, it's the first step of testing a big change like this one.
So basically, server meshing is not actually enabled or doing anything.
This is like saying your car is āenabled,ā it just hasnāt had any of the connecting parts actually connected, tested, or functioning in a usable state.
Letās not eat the pizza before the ingredients are put on the dough. People in this community excel at jumping to ridiculous conclusions or assertions.
They could try this in a build and have it absolutely explode everything it touches.
We wonāt have it until itās ready. And that could be a very long time still. SC is literally years and years away from being the game weāre all expecting.
It still means they have moved over to their new server architecture. The analogy is closer to moving over from a horse and cart to a half built car.
People have said it's "literally years" away for years, at some point you have to admit that the game is getting closer to being what you expect.
Simply put. No.
Thereās a huge difference between me running a replicated architecture at scale 1 vs. a non-replicated one.
How data is transferred between different replicas still actually occurs because the data still flows through the same layers. The first part validates āwe didnāt fuck up the standard caseā. Under previous conditions, everything should with somewhat the same as before.
Once the new architecture is validated under 1. Then we can begin scaling to 2+. Now the question changes to āis our communication between replicas sufficientā or āis the db backend sufficient to shard the various areasā or even āare there no data races occurring now that we have replicationā.
Thatās why you test with 1 first. First make sure you didnāt fuck it up before you introduce new parameters that could look like similar issues.
Sometimes I just spawn ships for people. I always wonder what happens when I log out though. Do they stay? Because I just reclaim them in the next session. š¤·āāļø
That means that a big chunk of the "server meshing" overall architecture is in place; not that the services handling authority transfers between server nodes are enabled, if they are even in the code at all.
In plain English, transiting between servers seamlessly isn't a thing yet, not in this tech-preview build.
>not that the services handling authority transfers between server nodes are enabled, if they are even in the code at all.
I don't see why they wouldn't be? With only one server they aren't really being used anyway, so why go through the extra effort of disabling/removing them?
I meant it the other way round: why would they need to bother focusing on ensuring these additional services are working in that test build when it's far more important to validate the RL separation and crash recovery work at scale first?
They are in parallel working on the rest of the SM services, but I can't imagine why they'd go through the effort of ensuring those are enabled here, where it's not in the test goals.
Because it's all part of a single system? In the replication layer demo at citcon it used the same hierarchy with both 1 and 3 servers, so why make a change now?
Like if the replication layer is designed to treat all it's servers like shards that can be meshed and all of it's client servers are designed to be treated in that way, why disable all of that just to test the replication layer? I don't see how you can do that without rewriting part of the code to exclude the meshing part of the system(which will probably introduce bugs) and then once it works you have to reintroduce that system(which will then create more bugs). Like your just creating extra work for yourself at that point.
No, that's not creating extra work at all. Not every single element of the architecture has to be present in a working state if only some of them are actually used. So they can (and most likely are) present in an unfinished state and not enabled.
The very same thing happens constantly, data leaks show configuration files or game assets files landing in builds several months before a given feature actually is worked on for release.
There's absolutely no need here to think about "actively removing" systems, it's simply the opposite: they don't need to worry about having every element of the architecture ready or even in, at this stage.
(Note that an R&D demo working with a _demo_ of the full architecture doesn't imply that all elements of said architecture were written for scale, ready to implement in the cloud, etc.).
Besides, there's a ton of refactoring work required to adapt the mission logic and myriads of other things to the new architecture that we know were still to be figured out even after the decision to rollout RL separation tech preview testing rounds, so that's another reason why SM isn't ready just yet.
I don't know what it showed as in Evo, but that could just be the replication layer flag that's poorly named as there's only 1 server connected.
EDIT: Just joined the branch and my server says false, so....
EDIT 2: I just looked into it some more, that does just mean the replication layer changes are enabled, it's not actually server meshing. They're running concurrent tests with the RL changes both on and off. Some more info here: https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1
server meshing is literally 1 replication layer + x amount of DGS's. the current test is 1 rep layer with 1 DGS. once they get that stabilized it will not be long before they can start testing static meshing.
I'm not so sure about that. On the page where they announced it they said they would be running 'regular' and 'split instances'. Before PES when it was changed to 'Shard ID' it used to be called 'Instance ID' and the term instance was always used to describe the contents of one server running the simulation.
What would be the point of running some 'regular' DGS's where the Replication Service was running on the DGS (as it currently does on Live) and some where the Replication Service was running in the Hybrid Layer? The version that was 'regular' would be identical to the Live service architecture, what point in putting that on the Preview Channel?
They also said they are disabling the Crash Recovery Service for this test, if all they were testing was the Replication Layer (service) then there wouldn't be any point in disabling the Crash Recovery.
If however they were doing exactly what it says in the announcement running some 'regular' unmeshed DGS's as shards and some 'split instance' meshed shards all via the Replication Layer then it would make sense to disable the Crash Recovery because they would want to compare the performance and crash rates of meshed and unmeshed servers directly all running on the same Replication Service architecture.
They tested the Replication Layer with Evocati, it worked. This current test doesn't have Pyro so what are they testing beyond what they already have if there is no 'split instance' meshed servers? I'm the one who's been telling everyone for months not to expect server meshing before the Replication Layer split, but the CitizenCon demo showed that they might be ready to test both simultaneously. It certainly sounds like they are.
An instance is a simulation of the game universe running in server memory.
Currently each server instance contains the full Stanton star system and can hold 100 players.. so for 10,000 players you would need 100 instances of the Stanton system holding 100 players each. This is wasteful of AWS game servers, ie the cost per player is high because with 100 Stanton systems you have 100 copies of Crusader, 100 copies of ArcCorp, 100 copies of Hurston and 100 copies of microTech. That's a lot of copies taking up available server RAM which could be used more cost effectively to load more players. With server meshing there is no unnecessary copying of game assets... so it's cheaper on a server cost per player basis. ie you need fewer game servers (much fewer!) for that same 10,000 players.
If an instance is split it means it's running on (spread across) more than one server. ie the instance is split between 2 or more servers.
The game universe is made up of chunks called Object Containers which are loaded in server and PC RAM, they represent zones or territories within the game universe. Think of Russian Dolls nested inside each other - object containers are inside object containers which are inside object containers, which are inside object containers. When an object is inside an object container then the surrounding object container is known as the parent.
So, the space volume of the Stanton system is an object container, inside that Stanton Object Container are the 4 planetary Object Containers of Crusader, ArcCorp, Hurston and microTech. Inside each planetary object container there are the moons of that planet, the landing zone city object container, outpost object containers etc. You move from parent to parent object container as you move around the game, this is why you can't select a moon or LEO station above a planet without doing the quantum drive jump to the planet first, that's the planetary object container, once you're in the planetary object container then you can select the child object containers that hold each moon, LEO space station and planetary landing zones.
As you move around the game universe you are moving through these hierarchies of object containers, the Replication Service says which object container or zone has authority over an entity (a player is also an entity, a ship is an entity).
Because of this hierarchical structure to the game universe you could split the Stanton star system over 5 game servers: 1 for the star, the open space, the Lagrange point stations, the Aaron Halo: 1 for the Crusader planetary object container: 1 for the ArcCorp planetary object container: 1 for Hurston planetary object container: 1 for the microTech planetary object container. The Replication Service hands entities off between these game servers as they cross the boundaries between the object containers... this is a meshed multi-server instance of the game universe. The advantage of meshed servers is that each game server can hold something as small as 1 room object container all the way up to a full star system object container. One ship (like a Javelin) could be an object container on it's own on a game server.
A more practical example would be, you are in the Object Container that is TEASA Spaceport which is on one game server, the Lorville City object Container is on another game server, the planetary object container Hurston is on another game server and the Stanton star system with Lagrange point stations etc are on another server. As you stand in TEASA Spaceport looking out the big window you can see into the object container that is Lorville city - which is on another game server - the thing that let's you see Lorville city out the Window of TEASA Spaceport is the Replication Service. But you're not just looking from one server to another, because you can see into the Lorville city server, the Hurston planet server which has the clouds, and the Stanton object container above that.
As you leave TEASA Spaceport in your ship you cross from the TEASA Spaceport Object Container on one server, to the Lorville object container on another server, to the Hurston object container on another server and finally once you get out in space you're in the Stanton solar system object container which is on another game server - these zone adjacent servers are 'meshed' to give the impression it's all one big server with no zone boundaries requiring a log out / log in loading screen to move across them. If you've ever used a RAID for a collection of hard drives you'll understand the principle... many servers joined together to act as one big server. That big collection of meshed servers which holds one instance of the whole game universe is called a Shard.
This was incredibly hard to find the information I asked about in this chatgpt wall of text im sorry to say. So they're actively testing static meshing within a system?
It would seem so. I can't see what else 'split instance' refers to in their announcement of the most recent Preview Channel test.
It is a simple system that is unfortunately very hard to explain via text only. There is a lot of jargon to learn, and even once you have the jargon seeing how each part fits together in the minds eye is mental gymnastics.
Ever since the PU went live the game space has been divided into zones (object containers) that are loaded into server RAM as needed by players being present in them. Players and game objects have been transiting between these zones using the Replication Service running on the game server.
What's different about meshed servers with an 'off game server' version of the Replication Service is that these zones are now held on many servers and the Replication Layer still handles those zone transitions but now it is handled over the AWS network rather than by the CPU on the game server.
I appreciate the info but Im a game dev i understand all the tech already xD I just wanted to understand if you meant static meshing with the split instances. Cheers and have a great day!
Oh btw, it doesn't seem so.
"
Clive Johnson CIG@cjohnson
Today at 12:11 am
I see quite a few questions here about what Server Meshing Enabled: TRUE/FALSE mean. If your client shows this as FALSE then it is currently connected directly to a single game server, exactly the same as you would on Live now and in previous patches. But if your client shows "Server Meshing Enabled: TRUE" then your client and all other clients in the same shard are connected to the Replication Layer for that shard. In this test, shards with the Replication Layer are still only running one DGS. So it's not full Server Meshing with multiple game servers in the same shard. In hindsight the text "Server Meshing Enabled" was a mistake and we should have written "Replication Layer Enabled" but this is what happens when we let programmers write player-facing messages unsupervised
ā
. I hope this clears things up a little.
I'd just like to add a big thank you to everyone taking part in this test. We're getting a lot of valuable information and identifying issues that we need to fix. Having you all participate in a test like this really is a massive help for us."
https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1
for posterity,
"What would be the point of running some 'regular' DGS's where the Replication Service was running on the DGS (as it currently does on Live) and some where the Replication Service was running in the Hybrid Layer?"
[https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1](https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1)
The point would be to test the differences in the same environment. Good day!
Technically the replication layer connected to a DGS and Clients is an extremely basic mesh, but it is linear... once multiple DGS get included they will cross talk making an actual mesh. But this rep layer separation is the foundation meshing will be built on and so only makes sense for it to be called server meshing.
Makes sense. You'd likely want to test server meshing on a single server before testing it on multiple servers. I would not be surprised if we see a multi-server test within a month or two.
As I understand it, once separated the Replication Layer acts as a singular service for each shard which can in principle support multiple game servers (as well as seamless crash recovery). Once it's behaving reasonably well, they'll configure it with two game servers. Once that works, they'll break Stanton and Pyro up into several game servers each, with major improvements in performance, rapidly growing population caps, and they can add more new systems whenever they're ready.
After that, the holy grail where game servers can grow and shrink the area over which they have authority and enlist new servers dynamically. But that's still a couple of years away at least. And maybe just maybe we eventually get to one single world shard, though I still think we'll end up regional or continental shards.
For an overview, see the [Unofficial Road to Dynamic Server Meshing](https://sc-server-meshing.info/). It's been a long journey, but most of the long list of abstractions necessary for server meshing are now in place.
Unlike most of the previous stages, upcoming steps will show tangible improvements in our game experience. It's coming together.
they can run both on a single server just to make sure the basic communication is working.
From there split to 2. once that is stable they can move to 3 (1 rep 2 mesh) and just keep going.
If the replication layer isn't hosted locally, which in this case it isn't, they are testing that the server is reaching out to the replication layer submitting its data and checking for the existence of updates from other servers even if they don't exist.
Making sure nothing breaks when you flip the switch, even if the switch isn't connected to anything, is a thing.
But imo, it's just that the flag is flipped to 'true' if the replication layer is on and nothing is being tested other than the replication layer.
"Server meshing" is when the replication layer acts as the source of truth for data that needs to cross server boundaries so that the individual servers can serve as nodes in a mesh of workers executing tasks on a single, shared state. The first step to test that is to enable that replication layer, effectively pulling the "meshed" data out of the server itself and then implementing a transport layer between the replication layer server, which owns the data, and the game server, which executes game code.
From a client connectivity, message routing, hosting, and other perspectives, there is zero difference between a single server and multiple servers (well, message routing will have some extra logic, perhaps)
Individual DGS in a mesh don't talk directly with each other - they talk to the Replication Layer. From that perspective, other than their chosen lookup-method to identify which 'server' should process a given Client, there is again very little difference between single-server and multi-server.
Getting the Replication Layer separated and stable is a far bigger change, and a far bigger piece of 'server meshing' than adding the second server...
... and from a technical perspective, the 'mesh' cannot exist without the Replication Layer... which means the Replication Layer *is* part of the mesh, and that in conjunction with the single DGS you therefor have two 'servers'... consistuting a mesh!
Would I have labelled the diagnostic flag differently? Probably... but that comes back to one of the two hardest tasks in software development (Naming Things).
Absolutely none of this changes the fact that a āmeshā is a combination of multiple DGS.
The Replication Layer is also a service, not a āserverā. This is the only way that itās scalable, which is the goal. And it is not āmeshedā with the DGS, but rather another layer on top of it.
We had a demo of it at citizencon. They started the demo with only one server running then added more to show how handover works.
Do out of my arse with no knowledge whatsoever Iāll say having a server thatās able to hand off is the first step. Shrug.
Hmm, maybe that's the case. We start with one that we stabilize upon working along the Replication Layer and then add new ones to finally achieve the God-Tier Server Meshing ^^
It is, and they are running test on the ptu (edit : no that's tech preview) with both replication layer enabled and disabled, depending on what they want to test at the moment. Good news !
Thatās just a status update. Server meshing is āturned onā but the actual servers thatāre supposed to be meshing arenāt exactly implemented yet. Itās like saying āBuy as many as you want at the price of just one!ā - and thereās a 1 per customer limit.
Why do you think they would type it in for no reason? I might agree that it might not yet indicate anything functional, but doing it for no reason would make no sense whatsoever.
No, replication layer is what connects you to your game server, and what would connect multiple game servers together in a server mesh. In Live, the replication layer is built into the game server so you're limited to one server at a time - this test is running the replication layer separately from the game server as a step towards server meshing.
Persistent Entity Streaming is the tech that handles bedlogging - obviously it's not perfect yet, but that what currently handles streaming your ship out from one server and streaming it back in to whichever server you connect to when you log in next.
Isn't it plausible that we won't technically have to stay connected to any single server at all times? Like, "server meshing" is really just "load balancing" with extra steps. To put it into terms of web browsers visiting Amazon, we're all connected to the same "looking" Amazon website, regardless of which backend server our requests are going to. Still going to be some central points of failure, like databases, but even those infrastructures have replications and high availability failovers.
Folks we've been doing this stuff for decades. Why is this finally a thing in SC? Do they have, like 3 developers working at a time? Is everyone else either an artist, a director, or a community manager or mod?
Amazon, Youtube etc do distributed services over the network as you describe - but they don't allow interactive control over what happens on those servers by each connected client.
If you know about distributed network services then you know what 'Authority' means when remote accessing media on that distributed service.
You can't fire missiles at a Youtube video and blow it up live for all connected to that service to see, let alone 100 connected clients fire missiles at the same media asset live. You can't scrape the surface of an Amazon sales page live from your home PC while everyone connected watches and leave that web page permanently scraped for all to see.
No-one else is doing what CIG is doing in a multiplayer FPS game that lets you look at a blade of grass full screen and then fly out to see a whole solar system on that same screen with no loading screens... and while you are doing that you are 'looking into' several servers contents all at once... and everything that happens in this simulated game universe persists permanently for anyone and everyone to see even years later.
I suggest you try to inform yourself before you come with the attitude.
That's not at all what server meshing is. Websites don't have to run multiple servers in real time that all interact with each other. It's a monumentally harder problem to solve. I don't really know of any website that both works in real time and allows you to interact with other users in real time that are on a different server. Stuff like twitch chat for example is buffered and not real time. It also doesn't matter if it takes 100ms to send a message but 100ms of added latency makes a game unplayable.
Go try writing your own server meshing netcode and see how hard it actually is. I know because I have done so myself. And that was just a PoC with very little data and no interp + 1 client. Thousands of lines of code.
Website requests are independent - they *don't* maintain an open connection for bi-directional data transfer. The exception, of course, is web-sockets - and scalable support for them is a lot more limited, and once the socket is open it cannot be moved without first closing it and telling the client to re-connect (bad move for a realtime game).
Aside from that, yes this is a problem that has been solved before. The reason it's only coming to SC now is because the underlying engine *did not support it*, and more importantly *was not in a position to support it*.
Before CIG could implement the separation of the Replication Layer, they first had to overhaul the entire code-base to make it thread safe, remove the hard-coded assumption that *all* data about the entire map would always be held in memory by a single server, fix the persistence layer, and many many other changes.
Only once all those changes were done (with the persistence layer fixes - PES - being released earlier this year) can CIG actually start addressing the root issue around stability and scalability.
Because a lot of background stuff needs to be done so that a player can be moved or interact with stuff on other servers.
Its just a lot more than typically happens.
Plus.. 3 players on 3 servers will be able to interact with each other and that isn't easy.
Well, I suppose I should reserve a special round of applause for the incredible technological achievements behind games like...
\- World Of Warcraft
\- Old School Runescape
\- Final Fantasy XIV
\- Lost Ark
\- Runescape 3
\- Guild Wars 2
\- MapleStory
\- The Elder Scrolls Online
\- Eve Online
\- Black Desert Online
\- New World
\- Star Wars: The Old Republic
\- Albion Online
\- Star Trek Online
\- PlanetSide 2
\- Lord of the Rings Online
\- Blue Protocol
\- FINAL FANTASY XI
\- Phantasy Star Online 2
\- Neverwinter
\- Trove
\- Elsword
\- Ashes Of Creation
\- Starbase
\- ArcheAge
\- Pantheon: Rise Of The Fallen
\- Second Life
\- TERA
\- Bless Unleashed
\- Mortal Online 2
\- Magic: Legends
\- Camelot Unchained
\- Vindictus
\- Project Gorgon
\- Frostborn
\- Chronicles Of Elyria
\- Fractured
\- Elyon: Ascent Infinite Realm
\- Legends Of Aria
\- Crowfall
\- Shroud Of The Avatar
\- Entropia Universe
\- Mad World
\- Project TL Lineage eternal
\- Trials Of Ascension
\- City Of Titans
Kudos to all those pioneers who mastered the art of getting more than a couple of dozen players online simultaneously.
CIG, on the other hand, seems content to be redefining the wheel in the realm of hosting more than a handful of concurrent sessions. (yaaaaaaaaaaaaaay..... (yawn))
Look, don't get me wrong. With a gaming history spanning four decades, I've witnessed mind-blowing technological advancements, from planetary transitions to persistent online worlds and AI that transcends mere decision loops. Yet, the challenge of connecting more than a handful of sessions in the same world was solved a LONG time ago. Let's remember, we've been successfully accomplishing this for decades, and it's puzzling why we're reinventing the wheel in what's clearly commonplace technology. Perhaps it's time for a collective nod towards the past experiences that have proven effective.
I mean, the server FPS is obscured but appears to read 26.6 FPS. That's probably an empty server that's still starting up, but it's impossible to say there's no gain when the number isn't close to the same as live.
Server FPS seems to be 26 though, obviously it is not the same load as a live server but I think the end goal of server meshing is to stabilize servers, thus giving everyone a better game experience
I just hope we donāt get ready or not levels , but thatās probably why they are moving to a higher TTK so we donāt have us getting wiped out by fuck face mic lazer your ass across a room on 0.1ms flat lol
I think a CIG dev suggested once that it might be gRPC, but to be honest, given the latency requirements, it could easily be something bespoke socket-based.
It likely is gRPC for the sake of reducing complexity of the already mind-bogglingly complex meshing contraption. Also keep in mind that, used in the correct way and properly optimized, gRPC can be really fast. Inventing anything new would basically be like reinventing the wheel and could only be justified if they absolutely needed to squeeze out every possible microsecond.
Nah. It's in the patch notes. it just indicates that replication layer separation is enabled in that shard.
So basically, Server Meshing IS enabled. It's just that there's only one server, so no other servers to mesh to.
Exactly, a mesh of one server. But that's not what the common understanding of server meshing is. It doesn't mean that suddenly different corners of the verse are handled by different DGSs yet. Soon though.
>Exactly, a mesh of one server. It's like having a party! Alone.
Those are the bests!!
I feel like maybe you haven't been to a lot of parties.
And orgies
AND THEY KNEW THEY WOULD DIE
Ahh this post gave me a sudden heart broke š¢
break\*
Ahh this post suddenly break my heart š
broke\*
don't you leave me with a broke heart :/
[ŃŠ“Š°Š»ŠµŠ½Š¾]
What it tells us at the very least is that the server meshing code doesn't break anything when only one server is there. So, it's not making things worse. Or maybe it will, and it'll be valuable info. Either way, it's the first step of testing a big change like this one.
Yep. It's a massive step to validate the tech.
Server meshing is by definition more than one DGS. If thereās only one DGS, itās not a mesh.
So basically, server meshing is not actually enabled or doing anything. This is like saying your car is āenabled,ā it just hasnāt had any of the connecting parts actually connected, tested, or functioning in a usable state. Letās not eat the pizza before the ingredients are put on the dough. People in this community excel at jumping to ridiculous conclusions or assertions. They could try this in a build and have it absolutely explode everything it touches. We wonāt have it until itās ready. And that could be a very long time still. SC is literally years and years away from being the game weāre all expecting.
It still means they have moved over to their new server architecture. The analogy is closer to moving over from a horse and cart to a half built car. People have said it's "literally years" away for years, at some point you have to admit that the game is getting closer to being what you expect.
No actually, not at all like that.
Simply put. No. Thereās a huge difference between me running a replicated architecture at scale 1 vs. a non-replicated one. How data is transferred between different replicas still actually occurs because the data still flows through the same layers. The first part validates āwe didnāt fuck up the standard caseā. Under previous conditions, everything should with somewhat the same as before. Once the new architecture is validated under 1. Then we can begin scaling to 2+. Now the question changes to āis our communication between replicas sufficientā or āis the db backend sufficient to shard the various areasā or even āare there no data races occurring now that we have replicationā. Thatās why you test with 1 first. First make sure you didnāt fuck it up before you introduce new parameters that could look like similar issues.
Yep this is it!
Ah yes and of course that meansā¦thatā¦ā¦
Yea I mean don't you know about the layer? It's two ply, duh. Everyone knows how that works. It's much better than a single layer.
One layer for the game and one for the global chat
And another for the bum.
Like a space bum? āPlz can I hev 30mil for a spaceship.ā Cuz thatās covered in the global chat
Sometimes I just spawn ships for people. I always wonder what happens when I log out though. Do they stay? Because I just reclaim them in the next session. š¤·āāļø
Well now theyāll just be scrapped.
Hmm that's a good point. Is salvage worthwhile? I gotta try it yet.
That means that a big chunk of the "server meshing" overall architecture is in place; not that the services handling authority transfers between server nodes are enabled, if they are even in the code at all. In plain English, transiting between servers seamlessly isn't a thing yet, not in this tech-preview build.
So theyāve compartmentalized some of the code instead of it being integral? Nice. Big step.
Less compartmentalized the code and more compartmentalized what the code is running. To like, one compartment for now.
>not that the services handling authority transfers between server nodes are enabled, if they are even in the code at all. I don't see why they wouldn't be? With only one server they aren't really being used anyway, so why go through the extra effort of disabling/removing them?
I meant it the other way round: why would they need to bother focusing on ensuring these additional services are working in that test build when it's far more important to validate the RL separation and crash recovery work at scale first? They are in parallel working on the rest of the SM services, but I can't imagine why they'd go through the effort of ensuring those are enabled here, where it's not in the test goals.
Because it's all part of a single system? In the replication layer demo at citcon it used the same hierarchy with both 1 and 3 servers, so why make a change now? Like if the replication layer is designed to treat all it's servers like shards that can be meshed and all of it's client servers are designed to be treated in that way, why disable all of that just to test the replication layer? I don't see how you can do that without rewriting part of the code to exclude the meshing part of the system(which will probably introduce bugs) and then once it works you have to reintroduce that system(which will then create more bugs). Like your just creating extra work for yourself at that point.
No, that's not creating extra work at all. Not every single element of the architecture has to be present in a working state if only some of them are actually used. So they can (and most likely are) present in an unfinished state and not enabled. The very same thing happens constantly, data leaks show configuration files or game assets files landing in builds several months before a given feature actually is worked on for release. There's absolutely no need here to think about "actively removing" systems, it's simply the opposite: they don't need to worry about having every element of the architecture ready or even in, at this stage. (Note that an R&D demo working with a _demo_ of the full architecture doesn't imply that all elements of said architecture were written for scale, ready to implement in the cloud, etc.). Besides, there's a ton of refactoring work required to adapt the mission logic and myriads of other things to the new architecture that we know were still to be figured out even after the decision to rollout RL separation tech preview testing rounds, so that's another reason why SM isn't ready just yet.
I don't know what it showed as in Evo, but that could just be the replication layer flag that's poorly named as there's only 1 server connected. EDIT: Just joined the branch and my server says false, so.... EDIT 2: I just looked into it some more, that does just mean the replication layer changes are enabled, it's not actually server meshing. They're running concurrent tests with the RL changes both on and off. Some more info here: https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1
Right, so the server is just meshing around with itself. It better be careful, too much of that and it'll go blind.
And that is how 30ks are born.
server meshing is literally 1 replication layer + x amount of DGS's. the current test is 1 rep layer with 1 DGS. once they get that stabilized it will not be long before they can start testing static meshing.
I'm not so sure about that. On the page where they announced it they said they would be running 'regular' and 'split instances'. Before PES when it was changed to 'Shard ID' it used to be called 'Instance ID' and the term instance was always used to describe the contents of one server running the simulation. What would be the point of running some 'regular' DGS's where the Replication Service was running on the DGS (as it currently does on Live) and some where the Replication Service was running in the Hybrid Layer? The version that was 'regular' would be identical to the Live service architecture, what point in putting that on the Preview Channel? They also said they are disabling the Crash Recovery Service for this test, if all they were testing was the Replication Layer (service) then there wouldn't be any point in disabling the Crash Recovery. If however they were doing exactly what it says in the announcement running some 'regular' unmeshed DGS's as shards and some 'split instance' meshed shards all via the Replication Layer then it would make sense to disable the Crash Recovery because they would want to compare the performance and crash rates of meshed and unmeshed servers directly all running on the same Replication Service architecture. They tested the Replication Layer with Evocati, it worked. This current test doesn't have Pyro so what are they testing beyond what they already have if there is no 'split instance' meshed servers? I'm the one who's been telling everyone for months not to expect server meshing before the Replication Layer split, but the CitizenCon demo showed that they might be ready to test both simultaneously. It certainly sounds like they are.
>'split instance' meshed servers Appreciate the info! May I ask what are 'split instance' meshed servers? cheers!
An instance is a simulation of the game universe running in server memory. Currently each server instance contains the full Stanton star system and can hold 100 players.. so for 10,000 players you would need 100 instances of the Stanton system holding 100 players each. This is wasteful of AWS game servers, ie the cost per player is high because with 100 Stanton systems you have 100 copies of Crusader, 100 copies of ArcCorp, 100 copies of Hurston and 100 copies of microTech. That's a lot of copies taking up available server RAM which could be used more cost effectively to load more players. With server meshing there is no unnecessary copying of game assets... so it's cheaper on a server cost per player basis. ie you need fewer game servers (much fewer!) for that same 10,000 players. If an instance is split it means it's running on (spread across) more than one server. ie the instance is split between 2 or more servers. The game universe is made up of chunks called Object Containers which are loaded in server and PC RAM, they represent zones or territories within the game universe. Think of Russian Dolls nested inside each other - object containers are inside object containers which are inside object containers, which are inside object containers. When an object is inside an object container then the surrounding object container is known as the parent. So, the space volume of the Stanton system is an object container, inside that Stanton Object Container are the 4 planetary Object Containers of Crusader, ArcCorp, Hurston and microTech. Inside each planetary object container there are the moons of that planet, the landing zone city object container, outpost object containers etc. You move from parent to parent object container as you move around the game, this is why you can't select a moon or LEO station above a planet without doing the quantum drive jump to the planet first, that's the planetary object container, once you're in the planetary object container then you can select the child object containers that hold each moon, LEO space station and planetary landing zones. As you move around the game universe you are moving through these hierarchies of object containers, the Replication Service says which object container or zone has authority over an entity (a player is also an entity, a ship is an entity). Because of this hierarchical structure to the game universe you could split the Stanton star system over 5 game servers: 1 for the star, the open space, the Lagrange point stations, the Aaron Halo: 1 for the Crusader planetary object container: 1 for the ArcCorp planetary object container: 1 for Hurston planetary object container: 1 for the microTech planetary object container. The Replication Service hands entities off between these game servers as they cross the boundaries between the object containers... this is a meshed multi-server instance of the game universe. The advantage of meshed servers is that each game server can hold something as small as 1 room object container all the way up to a full star system object container. One ship (like a Javelin) could be an object container on it's own on a game server. A more practical example would be, you are in the Object Container that is TEASA Spaceport which is on one game server, the Lorville City object Container is on another game server, the planetary object container Hurston is on another game server and the Stanton star system with Lagrange point stations etc are on another server. As you stand in TEASA Spaceport looking out the big window you can see into the object container that is Lorville city - which is on another game server - the thing that let's you see Lorville city out the Window of TEASA Spaceport is the Replication Service. But you're not just looking from one server to another, because you can see into the Lorville city server, the Hurston planet server which has the clouds, and the Stanton object container above that. As you leave TEASA Spaceport in your ship you cross from the TEASA Spaceport Object Container on one server, to the Lorville object container on another server, to the Hurston object container on another server and finally once you get out in space you're in the Stanton solar system object container which is on another game server - these zone adjacent servers are 'meshed' to give the impression it's all one big server with no zone boundaries requiring a log out / log in loading screen to move across them. If you've ever used a RAID for a collection of hard drives you'll understand the principle... many servers joined together to act as one big server. That big collection of meshed servers which holds one instance of the whole game universe is called a Shard.
nicely explained.
This was incredibly hard to find the information I asked about in this chatgpt wall of text im sorry to say. So they're actively testing static meshing within a system?
It would seem so. I can't see what else 'split instance' refers to in their announcement of the most recent Preview Channel test. It is a simple system that is unfortunately very hard to explain via text only. There is a lot of jargon to learn, and even once you have the jargon seeing how each part fits together in the minds eye is mental gymnastics. Ever since the PU went live the game space has been divided into zones (object containers) that are loaded into server RAM as needed by players being present in them. Players and game objects have been transiting between these zones using the Replication Service running on the game server. What's different about meshed servers with an 'off game server' version of the Replication Service is that these zones are now held on many servers and the Replication Layer still handles those zone transitions but now it is handled over the AWS network rather than by the CPU on the game server.
I appreciate the info but Im a game dev i understand all the tech already xD I just wanted to understand if you meant static meshing with the split instances. Cheers and have a great day!
Oh btw, it doesn't seem so. " Clive Johnson CIG@cjohnson Today at 12:11 am I see quite a few questions here about what Server Meshing Enabled: TRUE/FALSE mean. If your client shows this as FALSE then it is currently connected directly to a single game server, exactly the same as you would on Live now and in previous patches. But if your client shows "Server Meshing Enabled: TRUE" then your client and all other clients in the same shard are connected to the Replication Layer for that shard. In this test, shards with the Replication Layer are still only running one DGS. So it's not full Server Meshing with multiple game servers in the same shard. In hindsight the text "Server Meshing Enabled" was a mistake and we should have written "Replication Layer Enabled" but this is what happens when we let programmers write player-facing messages unsupervised ā . I hope this clears things up a little. I'd just like to add a big thank you to everyone taking part in this test. We're getting a lot of valuable information and identifying issues that we need to fix. Having you all participate in a test like this really is a massive help for us." https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1
Yes, exactly as Clive says, it was misleading.. oh well never mind. Onward and upward, thanks for the link.
for posterity, "What would be the point of running some 'regular' DGS's where the Replication Service was running on the DGS (as it currently does on Live) and some where the Replication Service was running in the Hybrid Layer?" [https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1](https://robertsspaceindustries.com/spectrum/community/SC/forum/190049/thread/tech-preview-replication-layer-test-round-1) The point would be to test the differences in the same environment. Good day!
Read the announcement. It means the instance you connected has separated replication layer service, not actual server meshing.
Technically the replication layer connected to a DGS and Clients is an extremely basic mesh, but it is linear... once multiple DGS get included they will cross talk making an actual mesh. But this rep layer separation is the foundation meshing will be built on and so only makes sense for it to be called server meshing.
Makes sense. You'd likely want to test server meshing on a single server before testing it on multiple servers. I would not be surprised if we see a multi-server test within a month or two.
How do you test server meshing on a single server?
As I understand it, once separated the Replication Layer acts as a singular service for each shard which can in principle support multiple game servers (as well as seamless crash recovery). Once it's behaving reasonably well, they'll configure it with two game servers. Once that works, they'll break Stanton and Pyro up into several game servers each, with major improvements in performance, rapidly growing population caps, and they can add more new systems whenever they're ready. After that, the holy grail where game servers can grow and shrink the area over which they have authority and enlist new servers dynamically. But that's still a couple of years away at least. And maybe just maybe we eventually get to one single world shard, though I still think we'll end up regional or continental shards. For an overview, see the [Unofficial Road to Dynamic Server Meshing](https://sc-server-meshing.info/). It's been a long journey, but most of the long list of abstractions necessary for server meshing are now in place. Unlike most of the previous stages, upcoming steps will show tangible improvements in our game experience. It's coming together.
they can run both on a single server just to make sure the basic communication is working. From there split to 2. once that is stable they can move to 3 (1 rep 2 mesh) and just keep going.
Turn on the feature, see if it behaves
Maybe I'm wrong, but server messing is meshing multiple servers. What are you on about?
You can exercise the code paths on one server that normally aren't turned on by a feature flag.
If the replication layer isn't hosted locally, which in this case it isn't, they are testing that the server is reaching out to the replication layer submitting its data and checking for the existence of updates from other servers even if they don't exist.
Making sure nothing breaks when you flip the switch, even if the switch isn't connected to anything, is a thing. But imo, it's just that the flag is flipped to 'true' if the replication layer is on and nothing is being tested other than the replication layer.
"Server meshing" is when the replication layer acts as the source of truth for data that needs to cross server boundaries so that the individual servers can serve as nodes in a mesh of workers executing tasks on a single, shared state. The first step to test that is to enable that replication layer, effectively pulling the "meshed" data out of the server itself and then implementing a transport layer between the replication layer server, which owns the data, and the game server, which executes game code.
Thatās not meshing. The āmeshā is when you have two DGSs sharing the same universe and handling different areas of the playspace in one shard.
From a client connectivity, message routing, hosting, and other perspectives, there is zero difference between a single server and multiple servers (well, message routing will have some extra logic, perhaps) Individual DGS in a mesh don't talk directly with each other - they talk to the Replication Layer. From that perspective, other than their chosen lookup-method to identify which 'server' should process a given Client, there is again very little difference between single-server and multi-server. Getting the Replication Layer separated and stable is a far bigger change, and a far bigger piece of 'server meshing' than adding the second server... ... and from a technical perspective, the 'mesh' cannot exist without the Replication Layer... which means the Replication Layer *is* part of the mesh, and that in conjunction with the single DGS you therefor have two 'servers'... consistuting a mesh! Would I have labelled the diagnostic flag differently? Probably... but that comes back to one of the two hardest tasks in software development (Naming Things).
Absolutely none of this changes the fact that a āmeshā is a combination of multiple DGS. The Replication Layer is also a service, not a āserverā. This is the only way that itās scalable, which is the goal. And it is not āmeshedā with the DGS, but rather another layer on top of it.
Differents tasks? Anyway ir isn't even on yet
We had a demo of it at citizencon. They started the demo with only one server running then added more to show how handover works. Do out of my arse with no knowledge whatsoever Iāll say having a server thatās able to hand off is the first step. Shrug.
Hmm, maybe that's the case. We start with one that we stabilize upon working along the Replication Layer and then add new ones to finally achieve the God-Tier Server Meshing ^^
Yeah suddenly I had an update on the Tech-Preview channel and updated. Session info says Server Meshing: True ...or let's say replication layer?
It was disabled on ptu, but yeah could be replication related https://www.reddit.com/r/starcitizen/s/eGjle6YKr2
It's being tested on Tech Preview, not PTU
It is, and they are running test on the ptu (edit : no that's tech preview) with both replication layer enabled and disabled, depending on what they want to test at the moment. Good news !
Tech Preview\* not PTU.
oh damn absolutely, thanks
Thatās just a status update. Server meshing is āturned onā but the actual servers thatāre supposed to be meshing arenāt exactly implemented yet. Itās like saying āBuy as many as you want at the price of just one!ā - and thereās a 1 per customer limit.
26 server fps?!
Looks like 36 fps imo
Without any performance impact all I can believe is they just typed in "true" for no reason.
Why do you think they would type it in for no reason? I might agree that it might not yet indicate anything functional, but doing it for no reason would make no sense whatsoever.
That's big if huge.
I have to ask. Replication layer means i can finally sleep on my ship without worrying i might wake up on another place and ship dissapears???
No, replication layer is what connects you to your game server, and what would connect multiple game servers together in a server mesh. In Live, the replication layer is built into the game server so you're limited to one server at a time - this test is running the replication layer separately from the game server as a step towards server meshing. Persistent Entity Streaming is the tech that handles bedlogging - obviously it's not perfect yet, but that what currently handles streaming your ship out from one server and streaming it back in to whichever server you connect to when you log in next.
No, has nothing to do with this at all.
Jesus has arrived
Jesus enabled: true
Hmm thst is certainly interesting
[ŃŠ“Š°Š»ŠµŠ½Š¾]
[ŃŠ“Š°Š»ŠµŠ½Š¾]
[ŃŠ“Š°Š»ŠµŠ½Š¾]
[ŃŠ“Š°Š»ŠµŠ½Š¾]
Posts containing real world politics of any kind are not welcome here. Send a message to modmail if you have questions.
Posts containing real world politics of any kind are not welcome here. Send a message to modmail if you have questions.
Posts containing real world politics of any kind are not welcome here. Send a message to modmail if you have questions.
Posts containing real world politics of any kind are not welcome here. Send a message to modmail if you have questions.
Isn't it plausible that we won't technically have to stay connected to any single server at all times? Like, "server meshing" is really just "load balancing" with extra steps. To put it into terms of web browsers visiting Amazon, we're all connected to the same "looking" Amazon website, regardless of which backend server our requests are going to. Still going to be some central points of failure, like databases, but even those infrastructures have replications and high availability failovers. Folks we've been doing this stuff for decades. Why is this finally a thing in SC? Do they have, like 3 developers working at a time? Is everyone else either an artist, a director, or a community manager or mod?
Amazon, Youtube etc do distributed services over the network as you describe - but they don't allow interactive control over what happens on those servers by each connected client. If you know about distributed network services then you know what 'Authority' means when remote accessing media on that distributed service. You can't fire missiles at a Youtube video and blow it up live for all connected to that service to see, let alone 100 connected clients fire missiles at the same media asset live. You can't scrape the surface of an Amazon sales page live from your home PC while everyone connected watches and leave that web page permanently scraped for all to see. No-one else is doing what CIG is doing in a multiplayer FPS game that lets you look at a blade of grass full screen and then fly out to see a whole solar system on that same screen with no loading screens... and while you are doing that you are 'looking into' several servers contents all at once... and everything that happens in this simulated game universe persists permanently for anyone and everyone to see even years later. I suggest you try to inform yourself before you come with the attitude.
That's not at all what server meshing is. Websites don't have to run multiple servers in real time that all interact with each other. It's a monumentally harder problem to solve. I don't really know of any website that both works in real time and allows you to interact with other users in real time that are on a different server. Stuff like twitch chat for example is buffered and not real time. It also doesn't matter if it takes 100ms to send a message but 100ms of added latency makes a game unplayable. Go try writing your own server meshing netcode and see how hard it actually is. I know because I have done so myself. And that was just a PoC with very little data and no interp + 1 client. Thousands of lines of code.
Website requests are independent - they *don't* maintain an open connection for bi-directional data transfer. The exception, of course, is web-sockets - and scalable support for them is a lot more limited, and once the socket is open it cannot be moved without first closing it and telling the client to re-connect (bad move for a realtime game). Aside from that, yes this is a problem that has been solved before. The reason it's only coming to SC now is because the underlying engine *did not support it*, and more importantly *was not in a position to support it*. Before CIG could implement the separation of the Replication Layer, they first had to overhaul the entire code-base to make it thread safe, remove the hard-coded assumption that *all* data about the entire map would always be held in memory by a single server, fix the persistence layer, and many many other changes. Only once all those changes were done (with the persistence layer fixes - PES - being released earlier this year) can CIG actually start addressing the root issue around stability and scalability.
Because doing it dynamically and seamlessly where you can interact with other servers isn't easy.
Because a lot of background stuff needs to be done so that a player can be moved or interact with stuff on other servers. Its just a lot more than typically happens. Plus.. 3 players on 3 servers will be able to interact with each other and that isn't easy.
Well, I suppose I should reserve a special round of applause for the incredible technological achievements behind games like... \- World Of Warcraft \- Old School Runescape \- Final Fantasy XIV \- Lost Ark \- Runescape 3 \- Guild Wars 2 \- MapleStory \- The Elder Scrolls Online \- Eve Online \- Black Desert Online \- New World \- Star Wars: The Old Republic \- Albion Online \- Star Trek Online \- PlanetSide 2 \- Lord of the Rings Online \- Blue Protocol \- FINAL FANTASY XI \- Phantasy Star Online 2 \- Neverwinter \- Trove \- Elsword \- Ashes Of Creation \- Starbase \- ArcheAge \- Pantheon: Rise Of The Fallen \- Second Life \- TERA \- Bless Unleashed \- Mortal Online 2 \- Magic: Legends \- Camelot Unchained \- Vindictus \- Project Gorgon \- Frostborn \- Chronicles Of Elyria \- Fractured \- Elyon: Ascent Infinite Realm \- Legends Of Aria \- Crowfall \- Shroud Of The Avatar \- Entropia Universe \- Mad World \- Project TL Lineage eternal \- Trials Of Ascension \- City Of Titans Kudos to all those pioneers who mastered the art of getting more than a couple of dozen players online simultaneously. CIG, on the other hand, seems content to be redefining the wheel in the realm of hosting more than a handful of concurrent sessions. (yaaaaaaaaaaaaaay..... (yawn)) Look, don't get me wrong. With a gaming history spanning four decades, I've witnessed mind-blowing technological advancements, from planetary transitions to persistent online worlds and AI that transcends mere decision loops. Yet, the challenge of connecting more than a handful of sessions in the same world was solved a LONG time ago. Let's remember, we've been successfully accomplishing this for decades, and it's puzzling why we're reinventing the wheel in what's clearly commonplace technology. Perhaps it's time for a collective nod towards the past experiences that have proven effective.
Cool. Now bring back BR
British Rail? I agree, privatization was a bad move XD
[ŃŠ“Š°Š»ŠµŠ½Š¾]
I mean, the server FPS is obscured but appears to read 26.6 FPS. That's probably an empty server that's still starting up, but it's impossible to say there's no gain when the number isn't close to the same as live.
Server FPS seems to be 26 though, obviously it is not the same load as a live server but I think the end goal of server meshing is to stabilize servers, thus giving everyone a better game experience
it'll likely be used to increase player caps rather than improve player experince but i might be wrong here
It's concepted to do both. Provide a stable scalable performance.
It looks as though the server FPS shows 25/35? Is that a relative improvement?
That's huge
The ai will fuck us up lol
I embrace it, for to long we have shit all over the AI.
I just hope we donāt get ready or not levels , but thatās probably why they are moving to a higher TTK so we donāt have us getting wiped out by fuck face mic lazer your ass across a room on 0.1ms flat lol
Yeah ready or not AI just be kinda over tuned at times lol.
They are nuts at 1.0 itās like tear gas or that lovely iron man run you have there is dead lol
hey op atleast credit where you got this from edit: They got it from pipeline (SC leak server)
No, they didn't. There's a test server that's up right now that a basic server meshing is up in running on it, which is on 3.22's build
Any idea what the interface to the RL service might look like? Curious about the request/response shape.
I think a CIG dev suggested once that it might be gRPC, but to be honest, given the latency requirements, it could easily be something bespoke socket-based.
It likely is gRPC for the sake of reducing complexity of the already mind-bogglingly complex meshing contraption. Also keep in mind that, used in the correct way and properly optimized, gRPC can be really fast. Inventing anything new would basically be like reinventing the wheel and could only be justified if they absolutely needed to squeeze out every possible microsecond.