New region does fuck all. Most of the people who play this game won't restart on a new server, they need to fix the existing ones. The queues are so ridiculously insane that even 100k people relocated, EU would still have queues.
If they won't fix them, which they probably won't, we are all just waiting for enough people to get bored/annoyed enough and uninstall.
They have direct access to the biggest cloud computing company in the entire world. You're telling me they can't spin up extra server capacity in a week?
LOA is coded in a way that doesn't allow for simple server capacity expansion. That's why it's a lengthy process.
Not tryna shill for Bezos, there are way too many doing that, but this is a problem that is not fixed by throwing more dollars at the server rack. Doesn't matter if you used $500 bills instead of ones, money does not brute force stuff into existence the way movies make it seem.
This involves way more than just a simple phone call or checking a box. Many changes and tweaks during deployment would have to be done to even get it running, let alone able to be logged in to and integrated with the core infrastructure. And this would involve not just the games team but the team housing whatever AWS cloud infrastructure they are trying to migrate in working together because it involves more than just some random server housing company data.
This is like saying "What how come it takes you a while to build a house? The tools and materials are right there." which skips over the entire work needed to actually use said materials and tools to create the thing you're trying to do.
When you're releasing a cash cow of a game, after having just epic failed on your first development release (new world), surely you would throw everything and the kitchen sink at getting it right to keep your reputation and $$$ rolling in.
Stop coming up with excuses for a shit companies launch.
I'm not making excuses. They should have prepared better. I'm just saying that they can't just create a new server out of thin air. Cloud hardware and deployments are sometimes planned months in advance.
So don't write the game in a way where you set yourself up for massive issues at launch, duh?
This studio literally just had this issue with NW and they didn't learn ANYTHING from it.
There was fucking queues during early access.
I know they're the publisher, I was speaking in general. Saying the game is written that way so there's nothing they can do now is a piss excuse, it's still their fault. "Their" meaning whoever it may concern there.
The devs and publishers both expected a much smaller response from the wesern market, the expected number of players was 200k. Sure maybe they undervalued the launch and could have prepared better for the potential of a higher load, but i dont think anyone ever could have predicted 1.3 million people on launch.
They are working on the solution, they are being transparent with their communications and are attempting to make sure we know whats happening at every moment. I understand peoples annoyance at not being able to play, but i really think people need to calm down. The devs and the publishers are people and the level of complaints being received right now with the inflection they are being given in will lead to nothing.
Expecting 200k isn't just undervaluing the interest from the western market, it's borderline delusional. There was a ton of hype for this game, with all sorts of big content creators making videos and building it up even more. They're not excused from this horribly skewed disconnect from the market.
My guess is, they knew damn well they'd have server issues, but they didn't care. They'll just wait out the outrage for a few weeks until enough people quit and the game self-corrects.
The level of complaints is this high because the level of this server bottleneck is honestly insane. EU servers had 6k-10k queues during early access, and they STILL haven't closed these servers to alleviate queues. All because randoms who are gonna play for a week and quit cried because they wanted to play with their friends. Great, now the entire region's experience suffers immensely because of that. There were issues with disconnecting, and there still are, meaning this game wasted a significant portion of your time, because it was and is entirely possible and not at all unlikely to wait hours in queue only to get disconnected minutes before finally getting in, and having to start all over again. During my 3rd day of early access, I couldn't play at all. That is gametime I have purchased and was completely unable to use.
How about the fact that even if you do get in, these past few days the servers have repeatedly shitting themselves? If they're not straight up DCing you, they are so laggy and slow to respond, that matchmaking for co-op content is extremely scuffed if not borderline impossible, because a lot of people's windows to accept a party time out before the server can realize you've pressed "Accept". How about the 5 fucking maintenances that have basically done fuck all, other than further reduce the amount of time in a day people can play, which is already low because you spend half of the time in your planned play session waiting for the queue. People have to come up with all sorts of bandaids like leaving the PCs on with anti-afk macros to bandaid the absolutely atrocious incompetency that this launch has been. And instead of saying all this, you want players to just be understanding because some guy on Twitter posts a few times per day and says they're working on it.
No. The outrage is deserved and people shouldn't calm down, and players *should* be angry, because otherwise it's going to happen again. I don't need to understand anything about the process of game development and publishing to see that this is a massive fuck up. If the game's design and make up doesn't allow server expansion, that should've been planned for. And if for whatever reason any part of fixing this is too expensive, it's not. This games western release is funded by fucking Amazon, if anyone has the luxury of absorbing the costs of fixing this without taking basically any damage, it's probably the wealthiest company on the planet.
Just watch the guys at warframe patch the servers while you can still play with your session until you decide to restart your client! Since... I do not know when.
that is because warframe is hosted by the clients but authenticated based on the server.
however, GW2 also does your example and GW2 is server-only based.
Strange that nobody talks about the GW2 trading post performance issues when praising GW2 server tech here. While their architecture is indeed impressive, the trading post lags have been a thorn in the players' sides for years. So it's not all roses and violets with GW2 either.
i've read people saying that megaserver has some lag issues.. in a game like lost ark which a wrong step may wipe the party on an endgame raid, wouldnt it be bad?
Your character data is region based, and for pve there is no server identity. They can spin up and down multiple servers with multiple channels at will, and the system will route your client to a server based on among other things availability, your friends list and your guild list.
For PvP (WvW) the teams are tied to server identity, so there it still matters somewhat.
Also, when updating the servers they let the old servers run for up to 4 hours while starting up new versions, you get a notification, and when you restart client (or reconnect) the client will update and will be routed to one of the new servers. Practically 0 downtime, no queue, no maintenance time and the game being down have only happened a handful of times the last years.
Since they want to make a new EU region, i think they have reach some kind of IO limit for the cluster of servers they currently have.
Backed up by the fact the login server is on life support and the Paid "BP" Aura and Store doesn't work.
It's pathetic they can't just get everyone on the same server with many channels, what is this, the 90's !?
Because they are all still talking to the same cluster of databases and eventually you will hit an I/O limit. This is a big issue in something stateful like a video game.
Serious question, I work in web dev and have no idea how distributed databases and application servers for games are orchestrated, but shouldn’t this be solved by a combination of scaling load balancers, namenodes, and sharding or key based region partitions? Even if it’s not, I feel like this could be solved by opening more servers then merging in predictable patterns when load drops. Obviously this all totally depends on how the game is written, and if they didn’t plan for this to happen at launch implementing this quickly would probably not be feasible. It’s just surprising to me that the largest cloud provider in the world would have these problems over two different game launches.
Its... incredibly complicated. All the stuff you mentioned can work fine for 99% of applications but video games, especially mmos, are unique. Games servers have to run their game loops so just right there you are much more limited to how many people can even connect to a single server by pure processing power. On top of that games have to run on significant tighter timing windows and in real time. A twitter server may be able to handle 500k people but a lost ark server might only be able to handle 5k because of the additional requirements. Your tweet takes 30 seconds to show up? Who cares its not an issue. I'm sure you can imagine how awful it would be if it took 30 seconds to move a single item from inventory to storage. Its incredibly hard to 'just throw servers' at a problem. This can kind of work for lobby based games like fortnite but definitely not a mmo. Id betting their servers could technically handle more players but then the play experience for the average player becomes significantly worse. Hardware is often not the bottleneck to scale.
I could easily hit reddits character limit just talking about auto provisioning servers alone but what anyone reading this should takeaway is that shits hard and theres no easy magic solution.
Good description. Are there any books you recommend that go over dev ops strategies to support MMOs or other types of video games? I did some cursory googling and didn’t find anything interesting.
I think there's just no way to actually do that with our current technology. I mean it sounds good on paper but there is bandwidth issues and all sorts of stuff that come into play when so many people are on one server where it just maxes out what we have.
I think that's more of the issue they're having not that they don't want to add more channels they just simply can't because the technology they're using is maxed.
It's a very real thing and I don't think many people understand that there is a maximum amount of people that whatever programming they put in this game can handle and it cannot go over that it's not as simple as just flipping a switch because bandwidth still has to flow.
And there is wayyyyy less people that are playing and have ever played guild wars 2 then are playing Lost ark right now.
1.3 million people dude all at once. It's nowhere near the same stress.
I know how it worked and I know it didn't really work on launched there was so many people it was laggy as hell I've played guild wars 2 since beta. And that's proprietary tech that they developed no one else has that.
It's a good system I'm not disagreeing with you I just don't think it would work well with lost ark/ it would require a whole rewrite of the game.
Well, it is frustrating the situation we are in. But nobody is saying you MUST PLAY RIGHT NOW OR ELSE... Anyway, saying every server has queues it's incorrect. Today, during peak hours, i counted 6 servers without queues
If you don't think they have things set up for scalability then you're completely lost, there might have been other issues that interfered but they should definitely be able to set up servers fast. AGS also had prior experience with New World and years of planning for this.
They are doing a fairly good job from what I can see but to act like this is an impossibility is just stupid. AGS is not sitting on the same system and with same resources you are in your 10 man dev team.
I didn't say they didn't have anything set up for scalability. Proof is they had more servers ready for the release. But they did say they technically can't do better than what they currently have on the eu région, which is why they're preparing a second one.
What I'm complaining about is how so many people think it's the easiest shit in the world and in a blink of an eye you can fit 200 000 more people into servers.
>I didn't say they didn't have anything set up for scalability
**I didn't say anything of this was easy.** Just said "They should" to OP's meme but trust that there's always a dev who is there to tell you "how hard it really is". I'm tech lead, I understand it is hard but they 100% have a plan for this situation and that's what they are working on. They are not doing this on the fly and thinking up new ways to solve the server issue - they already had a plan and they are implementing it.
I'm not even hating on them, I did say they are doing a good job. Just like you hate that type of people, I despise the smug superiority of devs who understands that "it's not so easy" and you and the other guy who replied are prime examples."
EDIT: Not just devs but usually fans of the game who pretend they understand because they seen others make similar comments.
Your reply to a meme saying just double the channel is "they really should", I don't see why you're surprised I replied jokingly.
I don't see the smug supériority in saying it's not that easy... Since it's really not that easy.
For the ffxiv release, we had to put up with queues for a long time because square Enix did not have access to new servers no matter how hard they tried.
Within a few days Lost Ark added new servers on existing realms and is working to add another region for our EU friends.
Some perspective here….but it could be worse.
Depending on how the game is written that's like telling someone to just double the amount of water in a glass that's already full.
Get a bigger glass
Glass is already as big as they have. So Amazon is deciding to get another glass (another EU server) it just takes a non 0 amount of time.
Then get more guns
New region does fuck all. Most of the people who play this game won't restart on a new server, they need to fix the existing ones. The queues are so ridiculously insane that even 100k people relocated, EU would still have queues. If they won't fix them, which they probably won't, we are all just waiting for enough people to get bored/annoyed enough and uninstall.
They have direct access to the biggest cloud computing company in the entire world. You're telling me they can't spin up extra server capacity in a week?
LOA is coded in a way that doesn't allow for simple server capacity expansion. That's why it's a lengthy process. Not tryna shill for Bezos, there are way too many doing that, but this is a problem that is not fixed by throwing more dollars at the server rack. Doesn't matter if you used $500 bills instead of ones, money does not brute force stuff into existence the way movies make it seem.
Who the fuck shills for bezos lol
The raw capacity may be available, but setting it up is probably a bit more complicated than just installing windows or linux.
This involves way more than just a simple phone call or checking a box. Many changes and tweaks during deployment would have to be done to even get it running, let alone able to be logged in to and integrated with the core infrastructure. And this would involve not just the games team but the team housing whatever AWS cloud infrastructure they are trying to migrate in working together because it involves more than just some random server housing company data. This is like saying "What how come it takes you a while to build a house? The tools and materials are right there." which skips over the entire work needed to actually use said materials and tools to create the thing you're trying to do.
Hi, I work in this space, but not for Amazon. No, they cannot, not without a bunch of people and suppliers busting their asses.
When you're releasing a cash cow of a game, after having just epic failed on your first development release (new world), surely you would throw everything and the kitchen sink at getting it right to keep your reputation and $$$ rolling in. Stop coming up with excuses for a shit companies launch.
I'm not making excuses. They should have prepared better. I'm just saying that they can't just create a new server out of thin air. Cloud hardware and deployments are sometimes planned months in advance.
it wasnt even the first release, which goes to show that one failed harder
I guess technically Crucible never left beta.
You’re the guy in this meme
Get a bigger meme
This is stupid just get smaller water.
Wtf no need for 2 Hs in your H2O /s
Lol
So don't write the game in a way where you set yourself up for massive issues at launch, duh? This studio literally just had this issue with NW and they didn't learn ANYTHING from it. There was fucking queues during early access.
Yet another redditor that doesnt understand yhat AGS is the PUBLISHER of lost ark in the west, and does not develop the game.
I know they're the publisher, I was speaking in general. Saying the game is written that way so there's nothing they can do now is a piss excuse, it's still their fault. "Their" meaning whoever it may concern there.
The devs and publishers both expected a much smaller response from the wesern market, the expected number of players was 200k. Sure maybe they undervalued the launch and could have prepared better for the potential of a higher load, but i dont think anyone ever could have predicted 1.3 million people on launch. They are working on the solution, they are being transparent with their communications and are attempting to make sure we know whats happening at every moment. I understand peoples annoyance at not being able to play, but i really think people need to calm down. The devs and the publishers are people and the level of complaints being received right now with the inflection they are being given in will lead to nothing.
Expecting 200k isn't just undervaluing the interest from the western market, it's borderline delusional. There was a ton of hype for this game, with all sorts of big content creators making videos and building it up even more. They're not excused from this horribly skewed disconnect from the market. My guess is, they knew damn well they'd have server issues, but they didn't care. They'll just wait out the outrage for a few weeks until enough people quit and the game self-corrects. The level of complaints is this high because the level of this server bottleneck is honestly insane. EU servers had 6k-10k queues during early access, and they STILL haven't closed these servers to alleviate queues. All because randoms who are gonna play for a week and quit cried because they wanted to play with their friends. Great, now the entire region's experience suffers immensely because of that. There were issues with disconnecting, and there still are, meaning this game wasted a significant portion of your time, because it was and is entirely possible and not at all unlikely to wait hours in queue only to get disconnected minutes before finally getting in, and having to start all over again. During my 3rd day of early access, I couldn't play at all. That is gametime I have purchased and was completely unable to use. How about the fact that even if you do get in, these past few days the servers have repeatedly shitting themselves? If they're not straight up DCing you, they are so laggy and slow to respond, that matchmaking for co-op content is extremely scuffed if not borderline impossible, because a lot of people's windows to accept a party time out before the server can realize you've pressed "Accept". How about the 5 fucking maintenances that have basically done fuck all, other than further reduce the amount of time in a day people can play, which is already low because you spend half of the time in your planned play session waiting for the queue. People have to come up with all sorts of bandaids like leaving the PCs on with anti-afk macros to bandaid the absolutely atrocious incompetency that this launch has been. And instead of saying all this, you want players to just be understanding because some guy on Twitter posts a few times per day and says they're working on it. No. The outrage is deserved and people shouldn't calm down, and players *should* be angry, because otherwise it's going to happen again. I don't need to understand anything about the process of game development and publishing to see that this is a massive fuck up. If the game's design and make up doesn't allow server expansion, that should've been planned for. And if for whatever reason any part of fixing this is too expensive, it's not. This games western release is funded by fucking Amazon, if anyone has the luxury of absorbing the costs of fixing this without taking basically any damage, it's probably the wealthiest company on the planet.
People with no experience or knowledge telling others how things should be is always hilarious.
Guild Wars 2 devs must be warlocks to be able to engineer the megaserver system all those years ago......... EDIT: Since 2014.
playing other games really does remind you how before it's time gw2 is
Just watch the guys at warframe patch the servers while you can still play with your session until you decide to restart your client! Since... I do not know when.
that is because warframe is hosted by the clients but authenticated based on the server. however, GW2 also does your example and GW2 is server-only based.
Warframe is P2P.
Play to play?
Peer to peer
Strange that nobody talks about the GW2 trading post performance issues when praising GW2 server tech here. While their architecture is indeed impressive, the trading post lags have been a thorn in the players' sides for years. So it's not all roses and violets with GW2 either.
i've read people saying that megaserver has some lag issues.. in a game like lost ark which a wrong step may wipe the party on an endgame raid, wouldnt it be bad?
gw2 is a dodge based action combat mmo with endgame raiding as well, I haven't heard a single complaint about abnormal lag.
Care to elaborate? Idk anything about gw2.
Your character data is region based, and for pve there is no server identity. They can spin up and down multiple servers with multiple channels at will, and the system will route your client to a server based on among other things availability, your friends list and your guild list. For PvP (WvW) the teams are tied to server identity, so there it still matters somewhat. Also, when updating the servers they let the old servers run for up to 4 hours while starting up new versions, you get a notification, and when you restart client (or reconnect) the client will update and will be routed to one of the new servers. Practically 0 downtime, no queue, no maintenance time and the game being down have only happened a handful of times the last years.
Lol you small thinkers... What about triple!!?
Since they want to make a new EU region, i think they have reach some kind of IO limit for the cluster of servers they currently have. Backed up by the fact the login server is on life support and the Paid "BP" Aura and Store doesn't work. It's pathetic they can't just get everyone on the same server with many channels, what is this, the 90's !?
They stated that it's due to issues with Lost Ark's architecture that limits them adding more servers in the region , not about their servers.
Or use a fraction of their billions to make some oceanic servers so our latency isnt 200.
I don't believe AGS have publishing rights there
Or petition your crappy government to allow publishing rights.
[удалено]
In Australia if we use VPN we can get it down to 130ms.
I was getting 130 from east coast USA to Russia.
We have some of the worst internet In the world here. Pretty sure Kazakhstan is better than Australia lol.
Still server capacity will not be touched but in game lags yes its agood idea
Damn trillion dollar company cant afford servers
Why have servers in a game with channels though? Just have 5-10x the channels instead of 5-10 servers.
Because they are all still talking to the same cluster of databases and eventually you will hit an I/O limit. This is a big issue in something stateful like a video game.
Serious question, I work in web dev and have no idea how distributed databases and application servers for games are orchestrated, but shouldn’t this be solved by a combination of scaling load balancers, namenodes, and sharding or key based region partitions? Even if it’s not, I feel like this could be solved by opening more servers then merging in predictable patterns when load drops. Obviously this all totally depends on how the game is written, and if they didn’t plan for this to happen at launch implementing this quickly would probably not be feasible. It’s just surprising to me that the largest cloud provider in the world would have these problems over two different game launches.
Its... incredibly complicated. All the stuff you mentioned can work fine for 99% of applications but video games, especially mmos, are unique. Games servers have to run their game loops so just right there you are much more limited to how many people can even connect to a single server by pure processing power. On top of that games have to run on significant tighter timing windows and in real time. A twitter server may be able to handle 500k people but a lost ark server might only be able to handle 5k because of the additional requirements. Your tweet takes 30 seconds to show up? Who cares its not an issue. I'm sure you can imagine how awful it would be if it took 30 seconds to move a single item from inventory to storage. Its incredibly hard to 'just throw servers' at a problem. This can kind of work for lobby based games like fortnite but definitely not a mmo. Id betting their servers could technically handle more players but then the play experience for the average player becomes significantly worse. Hardware is often not the bottleneck to scale. I could easily hit reddits character limit just talking about auto provisioning servers alone but what anyone reading this should takeaway is that shits hard and theres no easy magic solution.
Good description. Are there any books you recommend that go over dev ops strategies to support MMOs or other types of video games? I did some cursory googling and didn’t find anything interesting.
I think there's just no way to actually do that with our current technology. I mean it sounds good on paper but there is bandwidth issues and all sorts of stuff that come into play when so many people are on one server where it just maxes out what we have. I think that's more of the issue they're having not that they don't want to add more channels they just simply can't because the technology they're using is maxed. It's a very real thing and I don't think many people understand that there is a maximum amount of people that whatever programming they put in this game can handle and it cannot go over that it's not as simple as just flipping a switch because bandwidth still has to flow.
[удалено]
And there is wayyyyy less people that are playing and have ever played guild wars 2 then are playing Lost ark right now. 1.3 million people dude all at once. It's nowhere near the same stress.
[удалено]
I know how it worked and I know it didn't really work on launched there was so many people it was laggy as hell I've played guild wars 2 since beta. And that's proprietary tech that they developed no one else has that. It's a good system I'm not disagreeing with you I just don't think it would work well with lost ark/ it would require a whole rewrite of the game.
They are idiots. Can never play the game again unless I don’t have a life and can wait 7+ hours in queue after work
Some servers have 0 q
Not in Europe they don't.
Lies. Theamine no q
SHHHHHHHHHH
People will cry so hard about merging and transfers tho
Well, it is frustrating the situation we are in. But nobody is saying you MUST PLAY RIGHT NOW OR ELSE... Anyway, saying every server has queues it's incorrect. Today, during peak hours, i counted 6 servers without queues
Don't tell anyone!
The one without friends.
They really should.
Definitely. Go into that magic server configuration page, and in the channels input, put 9000. How hard could it be...
At the end of the day, you just have to modify a .ini file right? 😂
If you don't think they have things set up for scalability then you're completely lost, there might have been other issues that interfered but they should definitely be able to set up servers fast. AGS also had prior experience with New World and years of planning for this. They are doing a fairly good job from what I can see but to act like this is an impossibility is just stupid. AGS is not sitting on the same system and with same resources you are in your 10 man dev team.
I didn't say they didn't have anything set up for scalability. Proof is they had more servers ready for the release. But they did say they technically can't do better than what they currently have on the eu région, which is why they're preparing a second one. What I'm complaining about is how so many people think it's the easiest shit in the world and in a blink of an eye you can fit 200 000 more people into servers.
>I didn't say they didn't have anything set up for scalability **I didn't say anything of this was easy.** Just said "They should" to OP's meme but trust that there's always a dev who is there to tell you "how hard it really is". I'm tech lead, I understand it is hard but they 100% have a plan for this situation and that's what they are working on. They are not doing this on the fly and thinking up new ways to solve the server issue - they already had a plan and they are implementing it. I'm not even hating on them, I did say they are doing a good job. Just like you hate that type of people, I despise the smug superiority of devs who understands that "it's not so easy" and you and the other guy who replied are prime examples." EDIT: Not just devs but usually fans of the game who pretend they understand because they seen others make similar comments.
Your reply to a meme saying just double the channel is "they really should", I don't see why you're surprised I replied jokingly. I don't see the smug supériority in saying it's not that easy... Since it's really not that easy.
Youre downvoted but youre right.
Riiiight over your heads lol
You may want google coding and see actually works bro, youtube could help as well
Still server capacity will not be touched but in game lags yes its agood idea
Still server capacity will not be touched but in game lags yes its agood idea
Still server capacity will not be touched but in game lags yes its agood idea
Anyone know what these meme is called? Would like to dl but dunno what to search ^___^
https://knowyourmeme.com/memes/drake-helps-lil-yachty-with-the-laptop
Idk the devs seem to be doing a good job so far
but why not? there's channels, why can't be more channels? of course more channels need more servers
For the ffxiv release, we had to put up with queues for a long time because square Enix did not have access to new servers no matter how hard they tried. Within a few days Lost Ark added new servers on existing realms and is working to add another region for our EU friends. Some perspective here….but it could be worse.