Ankama Launcher: The Update Engine Is Speeding Up! 2022-11-03 14:00

Background

The launcher mainly serves to make the games available and to manage their updates. Updates are managed in the launcher by an internal framework known as Cytrus (the offspring of AGRUM (Ankama Games Release and Update Manager), which some will recall used to manage updates on the old DOFUS updater).

As we can all see, the current version (Cytrus 5) has its limitations: updates are always pretty big, even when the changes made to the game are fairly minor. And we stopped counting the number of times players have asked, "Uh… Is it normal for me to have to redownload the entire game every time there's an update?"

Let's look at why that is. To improve the game installation speed, many assets (items, maps, emotes, etc.) have been combined into larger files known as packs. This allows for a reasonable number of files to download and prevents pointless back-and-forths with the download server. For example, if there weren't a pack for them, you'd need to download 18,271 separate files just for the items in DOFUS 2! Instead you download a single 11.3 MB file.

But on the flip side, for an update in which only one item has been modified, the entire pack of items is re-downloaded. The same goes for translations where just a shifted comma results in a 43 MB download for the French pack! This largely explains why the size of updates doesn't exactly correspond to the changes applied to the games.

To resolve this, we're updating our Cytrus framework to version 6 – and we're going to explain what will be different!

Technical Improvements

Acknowledgments

To start off, we'd like to point out that the video game industry and the technologies that sustain it are constantly changing. Sharing best practices and innovations is a key driver of any progress. We, therefore, want to acknowledge the industry players sharing their inspirations and best practices that allow us to enhance our own tools. This devblog is a way of recognizing their contribution. We're so grateful to those who share what they come up with. Now it's our turn!

Splitting Files into Chunks

To improve the update process, we're going to avoid uploading files in their entirety as soon as they're modified. Instead, we will only upload the modified part. To do this, we start by separating each file, now split into several parts ranging from 16 to 256 KB; these are known as "chunks" in technical jargon. The changes in each update are then calculated using the chunks (instead of the files); this allows you to only download the modified chunks, which are then reassembled as files on your PC.

This allows us to reduce the download time for updates by using your RAM and CPU to perform the calculations and recreate the files on your PC.

It's an immediate benefit: if a small part of the file is modified, only the affected chunks will be downloaded.

To break up the files into chunks, we use a content detection algorithm called FastCDC that fragments the files deterministically. They're split up in a similar way from one update to another, even if it's the first part of the file that's been modified.

For DOFUS 2, once all the files are fragmented, we end up with about 123,000 chunks.

Pooling the Chunks

There may be two identical chunks in two different files. These can be recognized as they have the same identifier. So the chunk is only downloaded once to use it in reconstructing the two files, thus further reducing the amount of data to download.

This pooling saves us from downloading 2,000 chunks for DOFUS 2, leaving us with only about 121,000 chunks to download.

Bundles

All of these chunks are combined into bigger files known as bundles.

There are about 8,400 files in DOFUS 2. So with the old system, we had to do 8,400 individual file uploads.

To avoid having 121,000 separate downloads to retrieve all the chunks, we group them into bundles, amounting to 384 bundles for DOFUS 2. Then we'll be able to retrieve all the modified chunks from a bundle with just one request: a multipart ranges request. To put it simply, we specify what parts of the file we want to upload. This drastically limits the number of downloads during an update.

That means for a complete install of DOFUS 2, we'll upload no more than 384 files, down from 8,400 and representing the same volume of data.

The Manifest

In order for the launcher to calculate what files need to be updated, a file is made available by Cytrus. It indicates what files, chunks, and bundles are found in the game version.

We can then calculate the difference between what's available online and what's on your computer, and then download and reassemble the difference.

This type of file existed in Cytrus 5, but it only featured the status of available files without chunks or bundles.

Initial Comparative Figures

Here is a comparison of the sizes of download files for various recent game updates:

This is a small selection of updates to show that the development of Cytrus will have a big impact, with a few exceptions:

• When there are minor changes to big files, you'll notice a huge gain because only the updated chunk is downloaded.
• With big updates, you see a smaller gain: since the files have many changes, virtually all the corresponding chunks are downloaded.

We sincerely hope this update will improve your daily experience in the game!

– the Launcher team