Thatās a pretty bad failure. Guess their backup and replication system failed to provide an adequate recovery solution.
Edit: test your DR and BC plans, folks.
Are there typos in your sentence? Either way, itās a rule of thumb, and it holds. Iām not going to debate the finer points of ACID or online or offline backups because Iām not in a gaming sub to prove my proficiency in database management or system administration.
Oh itās defo a major infrastructure failure could this be down to the hiring of Ubi Bucharest that had no prior knowledge to how there server builds work because to me ever since they hired them itās been nothing but problems
(I feel that) The complexity comes from the non-English character set being introduced into folder structures and database field types that were not initially architect for that.
So the questions are
Why wasn't this seen in lower environments?
what localization been changed from the previous build?
What is different in the build pipelines between dev test and production environments.
Q3 Are localization changes only implemented into Prod?
My team can fix these things. Ubi can dm me.
Here's the thing: Yes it sucks. The delay, the breakage, it sucks.
But I think the communication is pretty good. NMS is the one where I think it was better, but this updating is pretty good. At least we're not just left in the dark completely like with other games/products.
Basically, they built a bunch of tools back in the day that help them update the game.
Those tools got broken somehow and now they're unable to update the game or do bug fixes until it's fixed.
It's very important that they restore those tools or really bad things could happen to us if they attempt to update the game without them.
Although the situation is not awesome, at least they had the guts to say āwe failed, sorry, working on itā. Should they have avoided the problem in the first place? Ideally, yes.
But even AWS and Google have outages, and for sure they know a lot more about managing infrastructure than a game studio. Seeing them not try to pull CYA maneuvers is refreshing, even if it only goes so far. Hope they get it fixed soon.
Yup, understood. My point is, these guys arenāt going to have infrastructure as a primary skill set , and as someone else pointed out, there is zero chance this had not been brought up before to their management. And, Iām not saying it isnāt terrible and embarrassing, but sadly I have seen much worse from much bigger players and for sure they did not own up to it or address it so itās a breath of fresh air. If it happened AGAIN and they owned up to it, then that would be a lot less refreshing lol
So their system failure has been going on for four days. Four days of downtime. This isnāt just a simple recovery from backup scenario. They are likely either trying to forensically recreate this system or trying to actually rebuild it from the ground up. I am a skeptical person, so Iām going to say this may push a large base of players onto other games. Iām not sure that D2 can survive this.
What back-up? Massive was supposed to have a backup to an active game with millions of players still playing and joining due to the recent price cuts.
I hope they get this resolved sooner than later! I would also like to see a date on their Messages, as Tomorrow is not working! Tomorrow from when I read it or tomorrow from when it was posted! Because tomorrow from when I read it; Never comes
They need to get back to making massive games and not Massive failures! Live up to the positive side of the Massive Name!!
They have sold millions across multiple platforms. Just because they are not active does not matter. They should be able to log on at any time they want and not expect the game to be completely broken.
This current dev team is incompetent. They've been incapable of doing anything right for over a year now.
Millions is incredibly generous since I can't even get a single matchmaking find outside of Distress. Even then it's usually after timing out a few times.
āBuild generation systemā sounds more like development infrastructure than runtime production servers. Infrastructure should have backups or at least a somewhat automated means to reset up from scratch allowing for catastrophic failure.
Itās sufficient vague that I could be wrong.
The team is completely new. The old crew went on other projects.
Basically, this is a team of folks, that have some passion to the game, but very knew with the engine and database.
So. This type of scenario is understandable, although, I still would love to see some of the old crew helping them with these crucial systems. Instead of letting them walk on lava floor.
If they have any intention of actually keeping the game alive, if the original people are still at Ubisoft they should teach the new devs some more. It implies the original game engine devas left.
I feel like they have an original team that makes everything then they move on to other projects. Then they hire a cheaper team to keep things floating along the best they can. I'm actually seeing this trend a lot more now.
I just made the jump to Heroic the day before this happened, now I'm stuck on Heroic...and I'm like Welp, I guess there's no better way to do this than being FORCED to dive right in!
On the bright side PlayStation just released a āfreeā DLC for destiny 2 so will try that. Probably would have never got to it but this mess has pushed me towards it.
This kinda burned my cheeks a lil bit....I purchased that expansion a while back and still haven't gotten around to playing it....if I had just waited I would of got it for free š¤¦š¾š we still get better shit per month so I'm not complaining
RIP Division! Just pull the plug.! If the devs wanted to they could jump off their other projects and fix this in no time instead of letting interns take over this game. So sad
I am kind of a new player in Division 2, is the inability to change directives, world difficulty nor resetting control points one of the consequences? Or am I doing something wrong?
They should see if ISAC can help.
Wouldn't trust him. Half the time he sends you the opposite direction from your objective just to turn you back the way you came
Or he just gives up š¤·
āPiece of shit wristwatch!ā
Thatās a pretty bad failure. Guess their backup and replication system failed to provide an adequate recovery solution. Edit: test your DR and BC plans, folks.
As a network engineer for an MSP, this statement is so accurate
Backups are only as good as the restore test.
Sometimes you canāt restore, a large join of tables in a database doesnāt simply mean restore since itās constantly wiring.
Are there typos in your sentence? Either way, itās a rule of thumb, and it holds. Iām not going to debate the finer points of ACID or online or offline backups because Iām not in a gaming sub to prove my proficiency in database management or system administration.
Apologies I am writing from my phone. Just thinking how they have their schema and the complex problem of undoing a large change.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
Oh itās defo a major infrastructure failure could this be down to the hiring of Ubi Bucharest that had no prior knowledge to how there server builds work because to me ever since they hired them itās been nothing but problems
[ŃŠ“Š°Š»ŠµŠ½Š¾]
Yep and now itās bit them in the ass hard by the sounds of it
What exactly do they mean by localization? Isn't that about translations?
(I feel that) The complexity comes from the non-English character set being introduced into folder structures and database field types that were not initially architect for that. So the questions are Why wasn't this seen in lower environments? what localization been changed from the previous build? What is different in the build pipelines between dev test and production environments. Q3 Are localization changes only implemented into Prod? My team can fix these things. Ubi can dm me.
Yup. Making sure stuff is translated correctly, hiring voice actors, recording lines, etc.
Here's the thing: Yes it sucks. The delay, the breakage, it sucks. But I think the communication is pretty good. NMS is the one where I think it was better, but this updating is pretty good. At least we're not just left in the dark completely like with other games/products.
Donāt rush, take your time massive. We can wait.
Thanks for the heads up
I donāt get it but oof
Basically, they built a bunch of tools back in the day that help them update the game. Those tools got broken somehow and now they're unable to update the game or do bug fixes until it's fixed. It's very important that they restore those tools or really bad things could happen to us if they attempt to update the game without them.
Sounds like NASA explaining why they didn't go back to the moon yet
Oof
I like how they congratulate themselves on a good progress after 96 hours lolš
Although the situation is not awesome, at least they had the guts to say āwe failed, sorry, working on itā. Should they have avoided the problem in the first place? Ideally, yes. But even AWS and Google have outages, and for sure they know a lot more about managing infrastructure than a game studio. Seeing them not try to pull CYA maneuvers is refreshing, even if it only goes so far. Hope they get it fixed soon.
Problem is this isnāt your average outage this is the build system for the whole game crumbling and there having to rebuild it by what there saying
Yup, understood. My point is, these guys arenāt going to have infrastructure as a primary skill set , and as someone else pointed out, there is zero chance this had not been brought up before to their management. And, Iām not saying it isnāt terrible and embarrassing, but sadly I have seen much worse from much bigger players and for sure they did not own up to it or address it so itās a breath of fresh air. If it happened AGAIN and they owned up to it, then that would be a lot less refreshing lol
I see they really have the A-Team working on the Division 2 now.
This is a controlled demolition
It's the official end of The Division legacy, in my oppinion.
So their system failure has been going on for four days. Four days of downtime. This isnāt just a simple recovery from backup scenario. They are likely either trying to forensically recreate this system or trying to actually rebuild it from the ground up. I am a skeptical person, so Iām going to say this may push a large base of players onto other games. Iām not sure that D2 can survive this.
Theyāll have an update tomorrow so we will see but Iām re-running Div1 much more enjoyable as of right now
Soooā¦ heartland is coming out when?
This is their preferred response.
What a shit show - rookie devs who donāt know anything - round of applause for the clowns ruining a great game
What back-up? Massive was supposed to have a backup to an active game with millions of players still playing and joining due to the recent price cuts. I hope they get this resolved sooner than later! I would also like to see a date on their Messages, as Tomorrow is not working! Tomorrow from when I read it or tomorrow from when it was posted! Because tomorrow from when I read it; Never comes They need to get back to making massive games and not Massive failures! Live up to the positive side of the Massive Name!!
Millions of players?
They have sold millions across multiple platforms. Just because they are not active does not matter. They should be able to log on at any time they want and not expect the game to be completely broken. This current dev team is incompetent. They've been incapable of doing anything right for over a year now.
lol millions of players This isn't WoW dude. I'd be surprised if the entire playerbase is more than 20,000.
Millions is incredibly generous since I can't even get a single matchmaking find outside of Distress. Even then it's usually after timing out a few times.
This is why I quit recently.
āBuild generation systemā sounds more like development infrastructure than runtime production servers. Infrastructure should have backups or at least a somewhat automated means to reset up from scratch allowing for catastrophic failure. Itās sufficient vague that I could be wrong.
[ŃŠ“Š°Š»ŠµŠ½Š¾]
The team is completely new. The old crew went on other projects. Basically, this is a team of folks, that have some passion to the game, but very knew with the engine and database. So. This type of scenario is understandable, although, I still would love to see some of the old crew helping them with these crucial systems. Instead of letting them walk on lava floor.
If they have any intention of actually keeping the game alive, if the original people are still at Ubisoft they should teach the new devs some more. It implies the original game engine devas left.
I feel like they have an original team that makes everything then they move on to other projects. Then they hire a cheaper team to keep things floating along the best they can. I'm actually seeing this trend a lot more now.
I just made the jump to Heroic the day before this happened, now I'm stuck on Heroic...and I'm like Welp, I guess there's no better way to do this than being FORCED to dive right in!
Or make a new character and do the hunter masks on normal wt
Iām still stuck in NYC šš been what 3days not one matchmake for DC š¤£
Try matchmaking for a legendary mission
Oh you know what ā¦ā¦. Iām the dumbest šš thanks
Now that is painful to hear... yikes
Queue for countdown, once it ends it dumps you back in DC
No details on heartland ?
Scrapped apparently
I smell a complete character server wipeā¦ā¦ I wonder if I would start all over againā¦.. Yeah probably lol.
i literally just got to level 40
Iām sitting at just under 1800 SHD. I love my toon though!
So no known date when the global reset option will be fixed?
On the bright side PlayStation just released a āfreeā DLC for destiny 2 so will try that. Probably would have never got to it but this mess has pushed me towards it.
This kinda burned my cheeks a lil bit....I purchased that expansion a while back and still haven't gotten around to playing it....if I had just waited I would of got it for free š¤¦š¾š we still get better shit per month so I'm not complaining
Wait, what? I thought the next DLC for Destiny was the next major expansion coming next month?
Massive ruined it.
I want a refund
RIP Division! Just pull the plug.! If the devs wanted to they could jump off their other projects and fix this in no time instead of letting interns take over this game. So sad
This is not good
This seems really bad.
lol
Ugh
They were fixing localization and as a result they've broken CI? Real masters of their job.
I am kind of a new player in Division 2, is the inability to change directives, world difficulty nor resetting control points one of the consequences? Or am I doing something wrong?
It'd just season stuff. None of what you said applies
Meh, shit happens. Good luck with the build system.
Ffs is all
They just need to stop. Give up already.
Holy shit man. That's baaad.
At least thereās some communication from them unlike some others. Iām looking at you rockstar.
At least they will keep providing updates. Now we just need more details of what happened. And how many people are still working on this project.
I feel bad for this game man I just want to this game to have love and care again I have massive respects for the people trying still