Compensation Incoming!

Tuesday, October 31 marked the early-access launch of single-account servers, a historic day for the WAKFU project, which hadn't seen a server open since Dathura in 2012. Despite many improvements made following our two load tests (opening three servers instead of one, boosting server performance, technical improvements, etc.), we have had a number of issues, in particular because we had to accommodate many more players than expected.

Read on to learn what went on behind the scenes!

Recapping the Events

Early Access

The three single-account servers opened their doors at 2 p.m. (Paris time). Players gradually logged in. Soon enough, we noticed we were repeatedly getting three types of feedback:

• Some players reported a "beta unavailable" issue.
• Others encountered a multiple IP error message on login, even though they were the only one using the IP address.
• On Ogrest FR 1, significant lag was seen on some instances, and when this occurred, players' turns were automatically skipped.

For the first case, we noticed it was affecting players who didn't have the early-access key – but we soon determined that players who had been reimbursed so they could buy a bigger pack were also affected. We did what we needed to fix the issue for these players as quickly as possible.

The second case is more complicated, because that error message can appear in multiple situations. Our developer [Othan] managed to identify various occurrences, especially thanks to the "logs" files sent by players.
 

This message is for issues regarding the IP (your internet connection's ID) and MAC address (your computer's ID).
By investigating with players, we realized that: 
- Recovering the MAC address was a problem for those with multiple network cards or virtual cards.
- In some cases, players were not logged out properly and left behind a "ghost". Normally, on the next login, the ghost is kicked by the arrival of the real player. But with the IP restrictions, the ghost blocked the real player's arrival and so wasn't kicked.

Investigating the issue with players automatically skipping a turn was more difficult; that's why we started an Ankama Live in the early evening to announce we would be dispersing monsters in specific zones to spread out player load, reduce lag, and thereby stem the "skipped turn" phenomenon. It was also necessary to set up more restrictive queues.

This phenomenon occurs when there are temporary latency issues on a game server (an entity supporting several different instances) with heavy traffic. 

On Ogrest FR 1, we found that this phenomenon mainly happened on two game servers: the one supporting Astrub, plus a second one that supports dungeons in Astrub, the Tainela rift, and the Wild Estate. We've continued making more Astrub monsters and resources available on instances supported by other game servers (Brakmar, Bonta, Bilbiza, Ohwymi, and Sadida Kingdom), in order to reduce player load as much as possible on the problem instances.

The situation stabilized around 11:30 p.m. (Paris time), and we kept watch to make sure things were okay up to the day after.

On Wednesday, November 1, even though it was a holiday, we decided to perform unscheduled maintenance at 4 p.m. We distributed the Astrub dungeons across different game servers, fixed certain occurrences of the IP issue, and opened a new server: Ogrest FR 3.

On reopening at 5:31 p.m., a critical issue was detected on Ogrest INT and Ogrest FR 2. As a security measure, we immediately closed the servers at 5:52 p.m. to fix this issue and avoid a rollback. All servers were able to open back up at 7:11 p.m. Since traffic is high that time of day, we established queues to spare the servers and prevent further disruptions.

Free-to-Play Launch

On November 2, the official launch date for the single-account servers, we decided to delay the launch, initially planned for 2 p.m., for later in the day. This extension gave us enough time to prepare additional fixes that required us to perform unscheduled maintenance on all servers, as well as to draft and translate communications to provide key information.

At 3:30 p.m., this article was posted to convey the following:

• Maintenance would take place sometime that afternoon to resolve the persistent IP issues, improve the queue, and in particular, do something about the issue of players' turns skipping automatically.
• Give a reminder that free-to-play would launch later in the day.
• A new server, Ogrest FR 4, would open, not ruling out the possibility that others would be opened if necessary.
• The cosmetic elements of single-account packs, along with pre-registration rewards, would be given out again on Ogrest FR 3.
• The servers Ogrest INT, Ogrest FR 1, Ogrest FR 2, and Ogrest FR 3 would at first remain restricted to those with the early-access key, until things became stable.

• The rewards and contents of packs would ultimately be recredited to all single-account servers. We also included the amenities (bags, boxes, and croquettes).
• We removed the access restriction on Ogrest FR 3 and Ogrest INT.

Maintenance ended at 8 p.m., allowing us to do the awaited launch! We know the wait was long, because many players logged in at once. We monitored the situation on the servers at length, letting players in bit by bit to avoid destabilizing the servers.

The servers were still experiencing lag due to so much traffic, but thankfully, the turn-skipping issue was fixed.

When the servers have high latency, their internal clock slows down, which is not the case for the game client's clock. So after a while, the server is "in the past" in relation to the client. This disparity causes the client to think its turn is over, therefore instantly skipping it.

Mobs, however, are unaffected, as they share a clock with the server.

Due to a memory overload, we had to restart certain servers on Sunday and Monday to keep them from being cut off unexpectedly. To resolve this issue and perform other patches (fixing the campfire for the Amakna quest, fixing the XP bonus on secondary characters, improving the distribution of instances across the game servers, and various optimizations), we carried out a patch maintenance the morning of Wednesday, November 8.

Sometimes during peak traffic, ping spikes are observed on certain instances. We're keeping an eye on the situation, and our developers are studying ways to offer more optimizations and reach a certain level of stability that will allow us to open the FR 1 and FR 2 servers to everyone!

Compensating You

The launch of new single-account servers has been complicated, and we're so grateful for your patience and understanding. We're still working hard to find solutions in order to improve the game experience. The entire WAKFU team is sorry we had to go through this situation with you for the first time. We hope to continue sharing many thrilling adventures with you in the World of Twelve!

Thank you again for being so patient during these events, for which we're compensating you with:

For any player who logged in to a single-account server:

• the Demonstrate emote (brand-new!)
• 7-day Booster
• Zinit Haven Bag Kit
• Ogrest Finisher
• 5 Harvest Potions (increases the amount of resources harvested by 50% for 120 minutes)
• 2 Loot Potions (increases the loot drop rate by 50% for 120 minutes)
• 2 Experience Potions (increases combat XP by 50% for 120 minutes)

Demonstrate emote

Zinit Haven Bag Kit

For all players who have early access before the Free-to-Play release: 

• 10 Loot Potions (increases the loot drop rate by 50% for 120 minutes)
• 10 Experience Potions (increases combat XP by 50% for 120 minutes)

When can I get these?

The rewards will be distributed next week. A distribution of this size may take some time, so be patient if the compensation is a bit delayed – it's on its way!