When INNO makes an error

DeletedUser21774

Guest
People are asking if Inno is going to compensate people for the lost resources because of the Christmas peacetime screw up.

1. Players got sieged who shouldn't have been and lost resources and troops.
2. Players who launched legal conquests and lost resources and troops doing so, saw their sieges abruptly interrupted.

Does the council have any pull? I do have a suggestion. How about +50% resources or something like that for a time matching the hours of the christmas peace duration?
 

DeletedUser50451

Guest
I lost a lot of gold getting the city up and ready for conquest.
Both of my sieges were perfectly timed and within the rules of peacetime.
Resource boost would mean nothing.
I think Inno should give similarly built up cities for their mistakes.
 

TobiramaSenju

Phrourach
I lost a lot of gold getting the city up and ready for conquest.
Both of my sieges were perfectly timed and within the rules of peacetime.
Resource boost would mean nothing.
I think Inno should give similarly built up cities for their mistakes.

I dont think they would be able to give cities, due to location they can't pick a city for you and put on a rock near you , or oceans away.


I believe a resource boost or a recruitment boost on all cities for the time that peacetime was at.
 

Rachel.L

Phrourach
this is from the dev's blog, copy and pasted
doesn't seem there is any recompense planned
you can leave comments for the devs directly on the blog: https://devblog.grepolis.com/?p=2284

Why the Christmas peace time wasn’t so peaceful…
Posted on December 27, 2017 by Justus GrafNo Comments ↓
Dear players,

As probably a lot of you have experienced or at least got to know, our peace time didn’t really feel like bringing peace into the game this year. And unfortunately, this didn’t happen for the first time in Grepolis history, so this blog post aims to explain why peace times have been a constant issue in Grepolis, what went wrong this year and how we’re going to do better in the future.

Peace times going wrong? Again?
Unfortunately, Grepolis has quite a history regarding peace times – but this year, something was different! While we had problems with the logic and checks for those special occasions in the more distant past, the last peace times still had a lot of problems with them not being set up properly. And indeed, the system to set up such a “peace” was very sluggish, our Community Managers couldn’t really plan, and it had to be done manually on every single world, so this process was kind of destined to fail at some point (forgetting to tick a checkbox for one of the worlds, forgetting to paste one of the digits for a timestamp…). That is why we reworked the mechanic of setting up peace times to use the same proven mechanic we use for our ingame events (such as the recent Wheel of Fortune).

Why did we still have issues? Did we break it while refactoring? Didn’t we test enough?
While moving the system, we wrote a lot of automatically executed tests (known as Unit and Integration Tests) to ensure the functionality of the mechanic, especially also covering a lot of the scenarios we had issues with in the past, including the one which failed this year. And everything was passing. Of course, we also tested the peace time manually, basically simulated Christmas on our internal servers and there everything was working.

Wait, what? So, what did go wrong?
To explain this, I shortly need to explain how Grepolis works in the background. Grepolis is a game where our “Backend” (the “program” which is running on our servers, e.g. managing all the ongoing actions and fights and so on) is running under PHP, so it’s not actually “running” all the time but we have to build up the game with every request from your browser or app or our own system (e.g. if a command arrives). As you can imagine, Grepolis is a very big and complex game, so building it up could take quite some time. To improve this (and therefore have lower response times for your requests) we use a lot of different techniques, and one of them is caching. You might know caching from your web browser, where the browser remembers images or other files so you don’t have to load them again and again if they don’t change. We basically do the same in our backend, also especially with this “happening system” (meaning the system we use for our ingame events as it is kind of a special happening).

The idea is: Why should we ask the database (the place where we store all the information about the game) if we already know about something as asking the database is quite “expensive” (costs time), so we try to remember the answer we already got.

Okay, what does that have to do with the problems we experienced?
Just to recall the issue we had: 24 hours before the peace times, it should not be possible to send any more colony ships to other towns (so basically it shouldn’t be possible to conquer new cities if the coloships aren’t already on their way).

To ensure this and a lot of other rules, we have a list of checks to go through when a player tries to send an attack – any only if none of the rules are violated, we actually send the attack.

And here comes the problem: When sending an attack, we first check if there would be peace at the arrival time of an attack (lets call that timestamp1) OR 24 hours later if and only if a colony ship is being sent along with the attack (timestamp2).

And here comes the problem with our caching: We already loaded the peace data for timestamp1 and cached it. When doing the second check, our caching thought that it would already know everything about peace times – so if there would be a new peace time starting somewhen between timestamp1 and timestamp2, our game just wouldn’t see that.

Okay, got that – but how could our tests pass then?
This is actually the reason why solving this bug took so long (we started debugging at 9pm on the 23rd of December and it took us until 1am on the 24th to actually have a fix for that bug. The problem is: This caching is disabled on our test system as we don’t really need the performance boost there but on the other hand want to simplify our testing process (I won’t go deeper into that topic). So even though we were testing a lot, we weren’t testing exactly what was happening in the live game and this was definitely our fault, no argument about that – and we are really sorry about this!

Yay, bug fixed, but then we made another mistake ☹
In order to reduce the impact of this bug on you all, we wrote a script that would check all running attacks including a colony ship AND all running conquests to see when their attack has been launched.

Canceling the running attacks: worked like a charm. Canceling the running conquests: not so much – and again, we are very sorry about that. The logic of our repair script was basically like that: Get all conquests, check when they’ve been started, calculate the travel time from the origin town to the town under conquests and check if this could have been a legit attack. That was fine so far – besides one problem: When calculating when a conquest started (we don’t actually store that anywhere), we had to calculate that back from the moment in time when it will end (we store this) and subtract the hours a conquests takes on that world. Problem here was: On most worlds, this value is 0.

Why 0? There are no worlds with instant conquests?! 0 indicates a default behavior, which means that we should calculate 24 hours divided by the world speed. It was already 4am when we deployed that script (deploying takes some time and of course we again tested everything: first our fix under live conditions then our script, was working, go!) and by that time we just couldn’t remember this detail anymore – while on the other hand we were under sever time pressure as we wanted to minimize the impact of our bug as much as possible. On our test servers everything worked as we usually set this value there (we just don’t want to test something conquest related and then have to wait for 8 hours our more), so we also couldn’t catch this default case there.

This is the reason we then deployed another script on the 25th of December to compensate all players being affected by our faulty cancelings (this time not developed at 3am in the morning and with close to no time pressure) and executed it at 2:30pm.

Wrap-up
First of all, we owe you an apology – we are deeply sorry, you all deserve better and we promise to take this issue seriously, reflect on it and also take appropriate actions to prevent such issues in the future (and most important for now: prevent such an issue during the New Year’s peace time).

Second, this issue revealed some oddities and design problems with our existing code – which is absolutely normal on a game being as old as Grepolis, but of course we have to tackle them, so we don’t make the same mistakes again on just another case.

Third, thank you to everyone who kept up the Fair Play by not sending any more colony ships even though it would have been possible!
 

DeletedUser35069

Guest
Speaks volumes that a player has to go and find this information from somewhere else 2 days after it was posted and nobody from Inno thought to post a link or give the details here.
 

Rachel.L

Phrourach
Speaks volumes that a player has to go and find this information from somewhere else 2 days after it was posted and nobody from Inno thought to post a link or give the details here.

several players have requested a direct link between the dev blog and the forum but been ignored
I don't go there often but after the gpc snafu find that any explanation of things is there
 

DeletedUser36697

Guest
I truly like the fact: that a Player...NOT any of the Forum Gods, neither the Inner or Outer rings (Council) could provide the passage of this Info to the REST of the COMMUNITY wrt this matter..
So If I understand this correctly months and 100s of hours are spent electing a Players Council (elected by only those folks that are involved in the forum).
This Council then spends weeks trying to identify their roles and ways forward, communication systems and then subdivides into 2 parts of a whole (like the Protozoa I watched in grade 10 Biology) Once this has been done they are no longer in office or are running for re-election...

So what has been achieved through all this work and Hours of labour wrt to Peace-time Anger

Player Rachel.L posts info for us, the info and site that, NOT ONE OTHER PERSON OF RESPONSIBILITY SAW FIT To Act On..

2 things are coming to mind now.. Knowledge is power and it appears the paying players are the peons in this case.. and the other is; there are time folks spend so much time looking at the Flora and fauna they forget they are in a frigging forest and find them selves ...lost without direction

My Hats off to you Rachel.L for providing this info
smiley_emoticons_wallbash.gif
 

Baudin Toolan

Grepolis Team
Sorry guys I had meant to post this but due to the holiday I was away. Wasn't expecting a bug of this difficulty and impact to happen during that period of absence. As for a compensation that should have gone out the 26th or 27th I believe and the level of compensation was based upon how a player was affected. If you were affected by the peacetime bug (colony ship returned/city lost) and did not receive anything please let us know via support ticket.
 

Rachel.L

Phrourach
thanks @dadofwildthang for the shout out
afraid to comment on much of what you said cuz I don't want to be banned
let's just say I agree
think there was supposed to be another gpc call this week and was waiting to hear the results
thought i'd wait til after the first before bugging ppl
 

DeletedUser54192

Guest
Sorry guys I had meant to post this but due to the holiday I was away. Wasn't expecting a bug of this difficulty and impact to happen during that period of absence. As for a compensation that should have gone out the 26th or 27th I believe and the level of compensation was based upon how a player was affected. If you were affected by the peacetime bug (colony ship returned/city lost) and did not receive anything please let us know via support ticket.
What are you going to do to get this message out to the players who do not use the external forums? And how exactly are the team intending to compensate players who were rimmed or lost cities in key locations because they trusted the announcement and went offline to spend time with family?
 

Baudin Toolan

Grepolis Team
I do not honestly know if there is any way to compensate for the loss of a city based on location. Players who were affected have mostly sent in tickets detailing how they were affected already at the start of the bug and during it's duration. We are currently (Rich and I) going through and manually adding the compensation to those players who missed it but as for any further information spreading I can't say as I do not know yet. We're focusing on the cases we are aware of due to the players using the support system to notify us.
 

DeletedUser54192

Guest
Are we allowed to know what the compensation is? Even for those not affected, it would be good to know what is being done in situations like this.

For app players, the support system can be more of a foreign concept. Can I suggest sending out a mass mail (because Announcements don't appear to work on the app, last time I checked), to notify players of what to do if they were affected?
 

Baudin Toolan

Grepolis Team
I do not believe I can share the specific compensation but if you lost a CS during the conquest cancellation fix you should receive gold as well as spells to help with rebuilding. As for losing a city it is the same, gold and spells to help rebuild, but there are fewer instances of city losses.
 

DeletedUser35069

Guest
Sorry guys I had meant to post this but due to the holiday I was away. Wasn't expecting a bug of this difficulty and impact to happen during that period of absence. As for a compensation that should have gone out the 26th or 27th I believe and the level of compensation was based upon how a player was affected. If you were affected by the peacetime bug (colony ship returned/city lost) and did not receive anything please let us know via support ticket.

You seem to be stuck doing all communications. Not actually directing my comments at you as in my opinion you seem to be doing 90% of the work, passing things on the players, every admin post in the council section seems to be yours and apologising for things you had no real involvement in. Do we still have a community manager? Or does a cm not manage a community?
 

Baudin Toolan

Grepolis Team
I try to handle the forum so Rich can focus on the crazy amount of things he needs to do. One of the reasons the bigger servers have co-cm's is to help reduce the workload on the CM.
 

DeletedUser54360

Guest
Sorry guys I had meant to post this but due to the holiday I was away. Wasn't expecting a bug of this difficulty and impact to happen during that period of absence. As for a compensation that should have gone out the 26th or 27th I believe and the level of compensation was based upon how a player was affected. If you were affected by the peacetime bug (colony ship returned/city lost) and did not receive anything please let us know via support ticket.




I have to disagree.

Players like me removed spikes from cities that could not have been lost - an d then suddenly had to seige break attempt on seiges that should never have happpened. Precious Myth units were lost directly as a consequence and teh upshot is - that Basically INNO doe snot give a Rats butt about these losses. Even when the reports were provided quantifying the needless loss.

Thos e who participated in seiges that should not have happened and ergo were against the rules THEY GOT COMPENSATED..


Players like me who read understood and complied AND ACTED ACCORDINGLY got screwed over.

That is the long and the short of it. INNO mistake and yet I lose.

It is not right and it is not satisfactory.

I let you know copiously about how I was affected and got brushed off 5 times..

So I don't believe it.. INNO does not deal with these issues based upon fairnes but based upon expediency..
 
Top