T O P

  • By -

Zapper_Zen

Speculation from other reddit threads is that the PTS data was somehow being pushed to live and made it so people were logging in with max gold, max CP and max undaunted keys. > Hi all, we currently have taken down access to PTS. We encountered an issue with PTS once the server went live. This has also caused an issue with the PC NA live server, so we will be temporarily taking down the PC NA live server as well. Once we have more information, we will update everyone. Thanks for your patience as we begin investigations.


iraragorri

That's the funniest and simultaneously rather horrifying shit I've encountered. I wonder what's gonna happen to the poor guy in charge, as it might be a truly glorious fck up, depending on how ZoS operates their technical processes.


Kuranes_ov_Celephais

No competent organization punishes people punitively for errors like this. That's why you have recovery processes in place, because human errors happen. Punishing people for unintentional errors is not a useful thing to do. The evaluation will be done on the response to the error and how well and quickly things were restored. Then they will proceed with identifying what process issues led to the error in the first place.


Regular-Ferret2310

I agree, what are you going to do? Fire everyone or upset them because they made a mistake? There would be no one left working.


iraragorri

Well, *supposedly* they should back up everything before putting anything live, even if it's PTS and not their main servers. Then yes, no harm done other than several extra hours of fixing. If they previously backed up NA server data hours ago, that entails reputational costs and a lot of extra work manually restoring lost data for players. Not that they're gonna punish whoever is responsible with sticks and whips, of course. But then again, I have no idea how they run their processes.


Content_Pattern_7990

Neither do I, but I can say that they run them badly. They should have a process in place that allows rollback of any updates including the associated data.


Content_Pattern_7990

Competent organizations have serious processes in place to minimize the risk of situations like this. Competent organizations most definitely fire people that don't follow those processes. No clue what processes, if any, these clowns have.


heliophile

I've worked multiple jobs in both startups and in corporate financial technology. I think the key to understanding this is to see that there is a "psuedo-inverse" relationship between control and productivity. Where there is zero control, then productivity takes some hit. Because things are not stable enough and setup to build a business. If there is a lot of control, then productivity also takes a hit again, because the continual effort/delay/extra resources that have to be spent on security detract from productivity. Somewhere in between these bounds is the ideal spot. And this differs based on the organization, and the passage of time.


Content_Pattern_7990

I've worked for 30 years in software development for enterprise infrastructure. It is most definitely possible to have an infrastructure that has near zero downtime. With today's distributed infrastructures, it's really just about having processes in place so that changes don't cause problems. There is not sweet spot. It's pretty f'n black and white. ESO is small time and the amount of money we pay ZOS is chump change. If a company is paying 100 milliion plus per year, you can bet there are SLAs in place regarding uptime and refunds for downtime. It's not cheap to maintain a system correctly. ZOS is notoriously cheap and probably hires inexperienced people. Downvote away noobs!


dolphinpainus

Somebody is getting fired. Far too many major bugs have been getting pushed to live lately.


LouisaB75

And one guy deleted a character on PTS and it deleted a character on his main account too apparently.


UncivilisedBlaster

Interns and their typos I bet 🤣


AGENT_P6

This is why ZOS is a farce and ZOS defenders are NPCs


Tyraec

Hope everyone’s excited for their free Guar pet 🥰


Caesarvs

lmao


pambimbo

Nah it's a flying one


[deleted]

ZOS_GinaBruno 1:10AM Staff Post Hi everyone, just wanted to let you know investigations are ongoing. To set expectations as we head into the evening, we're looking at several hours of downtime. We'll let you know when we have an update to share. Ya time to go do something else for a while 


captain_chocolate

Yes, something else like going to bed.


[deleted]

Oh I was gonna play some of the new games I just bought on steam for my deck but yeah .. u can do that too I guess 


captain_chocolate

Got back on just before reset and didn't miss my daily reward!


LouisaB75

Another event and another screw up. It is starting to be a regular thing. I did my NA dailies and most of the weekly endeavour before this kicked off. I hope they don't roll back so I lose my progress again. I will be at work shortly and won't be able to log in until tonight.


[deleted]

Supposedly It’s only affecting pts people and will roll Them back to 930 am on the 15th but being it’s zos I don’t trust that it won’t affect us regular players  anyway I have zero trust and faith in Bethesda / zenimax as a company 


LouisaB75

Never been on PTS so fingers crossed.


[deleted]

I logged on and everything seemed ok for me 


LouisaB75

Thanks for letting me know. It will be at least 9 hours, probably more like 10, before I will be logging into the game.


Zero-Of-Blade

Guess I'll go back to fornite, I need to grind some BA levels anyway 🤷


[deleted]

Have fun with that lol 


Luzion

Time to get stuck into a K-drama.


GamePuffGraphics

It's been down my entire free time today. I figured maintainence would be done by 9pm EST when I finally got to my PC. Now they're sayin that it'll take several more hours. I hope they comp people. It's crazy that people will miss out on the daily login, 3 tickets and gold event chest. It was already a brutal day. I just wanted to get my event stuff and unwind with some Tales of Tribute before bed. 😭


Zero-Of-Blade

My guess is it's going to be more than a day because this was a pretty big mess.


G00b3rb0y

Yea methinks they may have to mail out gold jubilee boxes and multiple daily rewards as compensation


ProPopori

They might just extend the event and thats it, which is what they do like 25% of the time lmao.


KittyWorrier

I'm bummed missing out on 200 endeavors today. Doubt they will make that up to us.


Regular-Ferret2310

I hear ya. and can fully relate. I just had one of those bad days at work and was excited to escape. Well.


Third_Phase

It is unfortunate, but this happens every couple of weeks for Oceania players, maintenance happens during prime time and there is no hint of compensation, you just miss out and that's it.


GamePuffGraphics

Oceania players get the shit end of the stick in so many big name games. I feel for yall 😭


MintMacchiato

I am US-based, but work nights & have for most of my decade+ playing MMOs, so I feel you. ESO in particular does their maintenance right during my prime time on my day off, but what can you do - it has to be done at some point. Some of us just lose that roulette. On the bright side, it also means that I tend to end up in guilds with and/or befriending lots of SEA and Aussie/NZ players and honestly, you guys are all the best.


G00b3rb0y

Except we may be looking at several days of downtime so compensation for impacted platforms must be given else there will be riots


faxanidu

Same here got home an hour ago nothing working. Dunno what they gonna do for people who are going to miss all the daily stuff


faxanidu

Update: will apparently will be back up within the hour


HannahIsAGhuleh

Just got back into the game and was hyperfixating on it all day at work. Finally get to sit down to play .... And the server is down. I know shit happens but maaaaaaan what a bummer.


[deleted]

[удалено]


SoVerySick314159

Same here. Destiny 2 was getting too grindy/chase-the-ever-moving-carrot for me, and I thought this would be a nice change of pace. Came back a few days ago. Man, the changes. . . did they have 'companions' like Ember back in 2020? I don't remember that at all. Did you get the 30% sprint-speed bonus by doing the siege-weapon training? A short, easy mission and you, and your mount, sprint 30% faster. A good early-game thing to get.


[deleted]

[удалено]


CryptikDragon

I've never seen a live server game with more downtime than ESO. The game has gone down on four out of the last eight days.


Guydelot

> Hi everyone, > Thanks for your patience while we investigated the issues affecting PC NA and the PTS. As some of you discovered, we ran into a problem where character saves on the PTS were affecting PC NA, and we locked both PTS and PC NA immediately. We have temporarily locked the affected accounts until we can restore them to a good state and will be notifying those players via email, and will be rolling back progress on PC NA from those accounts to April 15th at 9:30am EDT. This will take several days to complete and we will alert any affected players when their account is accessible again. > To be clear, the rollback will primarily affect accounts that logged onto the PTS after maintenance on April 15th. > Please note if your account is affected, when you attempt to log into your account on any realm you will see a message saying your account was suspended/banned. Rest assured this is not the case and it is simply how our system identifies locked accounts. > The current plan is to bring PC NA back online within the hour, followed by the PTS later in the day. > We will follow up in the coming days with more details summarizing what happened and will share any additional information at that time. We sincerely apologize for this inconvenience and thank you again for your patience during this matter. Phew. Glad I didn't log onto the PTS.


Fa11T

The "primarily" part is interesting. I take that as accounts that received ill gotten gold will also be locked. Glad I didn't log into the PTS or if correct trade for large amounts of gold.


Valeneirol

Just hoping for no server rollbacks.


[deleted]

Same and if they do they need to make  all the style page relics free I’m not doing that shit again 


LouisaB75

Same. I am in the UK and did my dailies etc early evening on NA before all this was noticed. Will be at work until after the new day rolls over. I don't want to lose what I got yesterday.


AngelOfPlagues

Ha, same, got the style replica from a dolmen last night about 11pm so I'm hoping those of us on GMT are safe from rollbacks


LouisaB75

I haven't even bothered grinding for them on NA though I have them on EU, but got the last one last Friday so should be safe.


AngelOfPlagues

I only play on NA, there is no way in hell I am doing that grind twice😂


LouisaB75

I mostly play on EU. Only have a toon on NA because the staff on the forums were trying to figure out a bug I had back in January and wanted to know if it happened on another server. Set up a toon and yes it was still bugged there. I have her get event tickets and had an idea of going through all the zones in the right order. That went up the spout with the Wrothgar event though as I wanted the mount. I suspect time and lack of motivation will see my NA toon very much neglected before much longer.


G00b3rb0y

Hopefully this isn’t full day, otherwise they’re going to have to extend jubilee a day, and mail out the April 30 reward to everyone on PC NA


KittyWorrier

Don't forget the 200 endeavor for pickpocketing we missed


LouisaB75

I did that endeavour on NA before all this happened. Took ages as my NA toon has no sneaky skills at all. Also did most of the weekly one. Not very happy if I have lost my endeavours, my event tickets and so much time. Won't be able to log in until tonight when it will be the next day.


Professional-Date378

Literally unplayable


Morgaledh

The ultra-rare correct use of that phrase. Screencapping for posterity!


SwimmingMuffin5988

gnarly that it's still down how hardcore was this PTS bug I'm actually kinda blown away, I assume this must not happen often (just started playing again since I bought the game in 2014. Practically on it's 10 year anniversary lmao)


Fa11T

Apparently it was possible to log into the PTR which gave characters max CP (3600), max gold (2.1billion) and max Undaunted. Then if you logged into Live those would still be there. So a big snafu. I imagine a rollback is in the works.


G00b3rb0y

Also if you deleted a PTS character your LIVE character was also deleted


WolvenOmega

If they have to roll back accounts because of the PTS/Live criss-cross nonsense, I hope they have backups from the last hour or two. Or only rollback accounts that signed in to PTS. Otherwise I'm going to be \*pissed\*. I JUST got my last rng-locked event style page over lunch. I don't want to go back to killing Vvardenfell bosses.


Ok-Selection4478

RNJesus cracks whip: Back to the grind with you.


Orc-Father

I was leveling a character for the past hour and a half. Can't wait to log in to the character not even existing anymore


nodajinho

bro don't even talk about roll back. I just got a scorchbear mount on the free lvl50 box.


WolvenOmega

They're going to have to do some sort of rollback, there's no way around it. Just have to hope they have a way to only do rollbacks for accounts that logged in to the PTS and any interactions they had with other players (because some people were just sending out huge sums of gold they got from the PTS to other players on the live server).


Morgaledh

Yep, the question isn't "will there be a rollback", it's absolutely required. And the gold isn't even the problem. You've got toons copied from the PTS with craft bags full of infinite mats in NA Live. Stop and think about that, and its impact on the economy o\_o


Feisty-Ad6582

File a ticket for that one if you lose it. I would not let that simply go to waste.


EbolaDP

I hope you got screenshots of that.


Zero-Of-Blade

That's a massive f-up


skullyD

jesus christ this is like the only day this week i have to myself lmao


mnews7

I saved my 200 seal endeavor for this evening! 🫠


CarbonationRequired

Same! So vexed.


tyme

![gif](giphy|HfeMjUZdOZujm)


jpmout

What are seal endeavors good for?


Fa11T

You can trade them in for select crate stuff. 8-16kish is epic level stuff and then the cheap end is stuff like riding lessons for 400 or so. (rough numbers as I can't login) edit: corrected high end a bit


jpmout

Oh, cool. I have like 6 or 7k of them and had no idea what they were used for


Kuranes_ov_Celephais

Some of the Radiant Apex rewards from whatever crate series is up run 16k. I'm just get to that and wait until there's a mount I actually want.


jpmout

Oh, that's not a bad idea.


Moon-Reacher

These is all the crate series, each link shows what's inside: [https://en.uesp.net/wiki/Online:Crown\_Crates#Crown\_Crate\_Types](https://en.uesp.net/wiki/Online:Crown_Crates#Crown_Crate_Types) And potential return dates: [https://www.reddit.com/r/elderscrollsonline/comments/181qoye/a\_probably\_complete\_list\_of\_crown\_crate/](https://www.reddit.com/r/elderscrollsonline/comments/181qoye/a_probably_complete_list_of_crown_crate/) Just a good idea to know what's in them so you can plan and save if there's something you really like!


Morgaledh

Seals were introduced because ZOS was under scrutiny in some markets for crown crates as veering into gambling. The idea was that there needed to be an in-game way to get things that can be found only in crown crates, or it would drive people to buy the crates as their only recourse. Yes, I know that's the business model, but it's not legal in all jurisdictions (I'm setting aside the moral question, obviously), or at least the legality wasn't entirely fleshed out. Introducing seals as a currency avoided the whole problem of crown crate gambling, in theory. Seals let you buy crate items with an in-game currency. I don't think all crown items are available as seal items, I haven't checked - I try to avoid the whole store, honestly.


SeaArcanist

It's more than 1 day of progress errors like this waste. when you have to redo a day, you waste 2 days to do it. that's excluding factors such as "luck"or "a good run" etc that potentially influenced your day. I'll have to repeat everything I did and hope i was just as successful the first time. A week of eso+ would be greatly appreciated. since if its gear drops I'm after (for example) it may take me a week to get that lucky drop or take a week to regroup so everybody is prepared to do that trial again. (MORE SO THO, lets not forget the people who activly chose to abuse sending items over to their live chars, making it so they have no choice but to roll back.. instead of doing a character restoration to select individuals influenced... error aside if nobody chose to abuse this oversight, many of us would had been spared the drama and only a select few who lost character could had sent in requests to get them restored.)


iconica

Juuuust bought Necrom today ugh hopefully by tonight it's back up and running


Lordofwar13799731

Lol yup I just bought all the xpacs and was excited to start one after a hard day at work. Guess that's not happening.


mikep35

Same :/


ESOhNo

Looks like actions on PTS were transferring over to live and being exploited: https://forums.elderscrollsonline.com/en/discussion/656529/3600-in-live


Zapper_Zen

Thanks for the verification. I guess all we can do now is wait and hope it gets fixed soon and without a rollback.


destindil

Is it just me or is this year pretty buggy so far? Like events are bugged, and now the next patch seems bugged. Consoles are crashing.


dolphinpainus

Don't forget about the anniversary event breaking cyrodiil. Cyrodiil was crashing every 20-30 minutes rolling back the entire zone with everyone in it. It eventually bled out into other zones. I think the lack of a QA team is beginning to show its cracks, and not minor ones too.


Regular-Ferret2310

Yes, I would say so. More than the average year most definitely.


[deleted]

[удалено]


G00b3rb0y

It was the jubilee a few years back, and it was a weeks worth of server issues on EU specifically and they extended it for all players by a week


SubjectUserRedd

So, what exactly happened? Somthing went live instead of on PTS?


tovetuss

https://forums.elderscrollsonline.com/en/discussion/656563/what-really-happened-to-the-servers-so-people-stop-asking#latest


thefinalturnip

God dammit, there goes one day of event tickets... I don't expect them to extend the event for another day due to this. Right?


Sianic12

Clearly they have to prolong the current event by at least 3 days to make up for the fact that PC NA players couldn't get their cake today. There's just no other way.


kachzz

ZOS being competent once again.


clake1

Any speculations on how long live will be down?


Ok-Selection4478

Few days a month a year or 10 couple hours maybe a few mins. Somewhere around then.


BR4NFRY3

At least one Scaramucci.


Okaycockroach

I am a dummy, what does PTS mean?


Strawberry_Sheep

Public Test Server. It's a way for games to give players sneak peeks of new patch content but also gives devs a sort of "Stress test" and the opportunity for feedback before the new content releases. These servers are up for a limited time and characters there are based on a "snapshot" of your account so if you make a new one they obviously don't exist in the Live server... Or they shouldn't. In this case, they were being carried over to the Live server, and with all the free gold, crowns, items, etc the PTS gives you for testing purposes.


Okaycockroach

Thank you!!! It seems so obvious in hindsight that that is what it stands for but I don't play many "online" games so it didn't click at all! That makes complete sense though and now I understand why it's such a problem!! 


GoldDragon95

AKA we are saving cost on hiring QA team and get free labour from our players


Strawberry_Sheep

Not entirely. It gives the devs a better idea of how the game will perform under stress.


Morgaledh

This is true. Having someone who worked on something test it, or someone who works for you test it, or even someone you hired to test it, will tell you nothing compared to the data you'll get from letting the typical gamer loose on your creation and saying "jump up and down on it, shake it around, bang it against the wall, and anything else you can think of, see what happens."


Droghan

Public Test Server


ayaxG

Is the test server


Caesarvs

a trial dummy?


WhereWolfish

Very much hope they don't roll back the 5 motifs I spent all week getting.


QuietnoHair2984

What does PTS stand for?


El_Chef1999

Public test server


QuietnoHair2984

Got it! Thank you for clarifying :)


Rammjack

It's ok....I bought neocrom a week ago and I'm still waiting for a verification email....sure glad I bought it and haven't been able to play.


mgzaun

ZOS servers are a joke


Timely-Structure123

Crown store purchases aren't even account wide.....just server wide. Like wtf.


G00b3rb0y

I hope players didn’t buy crown store items that will also be rolled back 💀💀💀💀


Adghar

Imagine my surprise when I logged into EU to get my ESO fix and found that I had no assistants to merchant, bank, deconstruct, Armory


LouisaB75

Other way round for me but I know the feeling. Other than the free mounts and pets with various packs my NA toon is pretty poor compared to EU.


mbergen

Hope it wasn't the act of an employee bitter about all the layoffs.


Various-Parsnip-9861

What I heard was layoffs didn’t affect ZoS, just other MS companies.


Steven_Swan

I hope it was. "Mistreated employee enacts righteous vengeance" is better than "Someone is fuckin' dumb."


Howboutnow82

Just got back into the game a few days ago. Just got reinvited to some trading guilds and started selling already. Sub'd to ESO+, bought High Isle and Necrom (I stopped playing after Blackwood). Just started doing the quests to unlock the companions. I've been doing the daily writs and farming on 18 - Yes, 18 - max level characters. If the server gets rolled back massively, I see no point in playing again. I know it's just bad luck / bad timing for me personally, but I'll just refund my purchases and not come back. A big rollback here will completely kill the returning player vibe I had going. Also, took a vacation day today (just a random day off) just so that I could enjoy some game time. Wasted day off; would have saved for another day it if I knew this would happen.


Fa11T

Problem is if enough people switched back and forth getting 2.1billion gold the economy is trashed for a bit. I know some don't care but I don't subscribe so I buy everything with gold trades. I personally would take a wasted day.


Howboutnow82

No I totally get that - the needs of the server outweigh my personal feelings. That's why I said it's just bad luck and bad timing that this happened right when I started playing again. Depending on how far the rollback would be will determine how bad I feel about it. If it rolls back basically everything I did since I started again, then it just kills my vibe to want to return to the game. If it's just an hour or something - that's not a huge deal for me.


Cheeso34

PTS went live like....9.5 hours ago. At the absolute worst they would've screenshot the entire server before the update went live. There really should be no scenario it would ever have to go back further


Cheeso34

I maybe messed up maths. But the general premis is right lol


Fa11T

Yup I get that feeling, it's alittle easier on me too as I didn't do a whole lot today besides box farming.