How to Support Savefile Backwards Compatibility
If you're planning on making a hack and updating it with bugfixes and additional content in the future, you're going to have to deal with players wanting to migrate their save files forward to the new version of your hack. Which usually means dancing around making any major changes to the save file structures, and/or fretting when that does happen. This tutorial will provide advice on how to do save file versioning and forward migration, so you don't have to worry as much about making major changes to the save file.
If you wish to look at an example or follow along with code, you can check out this branch: https://github.com/tustin2121/pokeemerald/tree/examples/save-versioning
Adding a version to the save data is simple. If you've already released your hack and people's save files are out there, don't worry, it's not too late to add save versioning. This method is designed to handle the case where the old version of your game's save doesn't have any version information.
(If you haven't released your game yet and don't care about maintaining backwards compatibility with vanilla, you can skip this step.)
Before we do anything, we need to save off the current version of the save structure. Create a new file src/data/old_saves/save.v0.h. You don't have to add header guards because we will be including this in a .c file shortly, like the other files in the src/data folder.
Now open include/global.h and copy struct SaveBlock2 and struct SaveBlock1 into the new file. After you copy them, append a _v0 to their names, so that they're named struct SaveBlock2_v0 and struct SaveBlock1_v0. This indicates that these structures are for "version 0" of the save data. (Version 0 is the version before we implement save data versioning.)
The two save block structures are the minimum you will want to copy. If you're planning on editing any of the sub structures, you'll need to copy the old versions of those into this file as well, and replace the struct names in the save blocks with the _v0 versions of the sub structures.
For example, if you plan on changing how Pokemon are stored, you'll need to copy off all the PokemonSubstructs, the BoxPokemon and Pokemon structs and the PokemonStorage struct, and add _v0 to all their names.
If you want to be super safe with the save block preservation, you should save off all the structs like this and then replace all constants for array sizes with literal numbers, so that if you change the constants, the old save block does not change size. In my example, I've done the latter, but not the former.
In my example, I've also copied off the current version of the save blocks to its own file. You don't have to do this, but it may help later on if you wish to make a new version to just duplicate the file to the old_saves folder.
Now that the old version of the save structure is stored safely elsewhere, we can start adding version information to the save blocks:
struct SaveBlock2
{
+ u8 _saveSentinel; // 0xFF
+ // u8 unused;
+ u16 saveVersion;
u8 playerName[PLAYER_NAME_LENGTH + 1];
u8 playerGender; // MALE, FEMALE
u8 specialSaveWarpFlags;The first thing you'll likely balk at is that we're adding this version number to the front of the save block, causing all the fields to shift down by 4 bytes.
(Note: Any offset comments you may have before the fields will be outdated, and you may just want to remove them at this point; they're not needed.)
(Note 2: The unused byte in the middle there is due to struct padding rules. Feel free to, like, put a game id or something in there, maybe. Doesn't matter.)
Here's why we're doing this:
- In order to determine if a save needs to be updated, we need a version number. But the previous version of the save doesn't have a version number yet, so we go for the next best thing: would the save be invalid in the previous format?
- The very first data in the vanilla save file is the player's name. A valid save file cannot have a player name that is an empty string. (The game will choose a random name from a list if you attempt to move past the naming screen with an empty name.)
- The value 0xFF is the "end of string" terminator character, and if we find that at the beginning of a string, that means the string is empty.
- So in the future, when loading the save file, we check the first byte of the save structure, where normally the first letter of the player name would be. If it is NOT a 0xFF, then it is a version 0 save. If it IS a 0xFF, then we know there's a save version we can read.
Now that we have a field for save information, it's time to fill that in:
In include/constants/global.h:
#define LANGUAGE_SPANISH 7
#define NUM_LANGUAGES 7
+#define SAVE_VERSION_0 0
+#define SAVE_VERSION_1 1
#define GAME_VERSION (VERSION_EMERALD)
#define GAME_LANGUAGE (LANGUAGE_ENGLISH)
+#define SAVE_VERSION (SAVE_VERSION_1)(Feel free to come up with more descriptive names. My hack names the save versions after a yearly release.)
In new_game.c:
void NewGameInitData(void)
{
if (gSaveFileStatus == SAVE_STATUS_EMPTY || gSaveFileStatus == SAVE_STATUS_CORRUPT)
RtcReset();
gDifferentSaveFile = TRUE;
+ gSaveBlock2Ptr->_saveSentinel = 0xFF;
+ gSaveBlock2Ptr->saveVersion = SAVE_VERSION;
gSaveBlock2Ptr->encryptionKey = 0;
ZeroPlayerPartyMons();
ZeroEnemyPartyMons();If you compile and load up your game at this stage and load an old save, you may find a few problems while you play...
pokeemerald-quintywhat.mp4
That's expected right now. We're not done.
So now, we need to have the game read the save version and recognize when it is out of date. When the game loads, it shows the copyright screen and then reads and verifies the save data. It does this in CB2_InitCopyrightScreenAfterBootup (in intro.c). The function LoadGameSave(SAVE_NORMAL); loads the save game and puts the status of the save in gSaveFileStatus.
Then, once the main menu appears, the game checks gSaveFileStatus in Task_MainMenuCheckSaveFile (in main_menu.c) and shows a different kind of menu depending on what it found. We need to add a new kind of save file status for the save file being outdated and being updated.
In include/save.h:
#define SAVE_STATUS_EMPTY 0
#define SAVE_STATUS_OK 1
#define SAVE_STATUS_CORRUPT 2
#define SAVE_STATUS_NO_FLASH 4
+#define SAVE_STATUS_OUTDATED 10
+#define SAVE_STATUS_UPDATED 11
#define SAVE_STATUS_ERROR 0xFFIn src/save.c, the LoadGameSave function will call to TryLoadSaveSlot, and this is where we're going to check if the save is outdated:
static u8 TryLoadSaveSlot(u16 sectorId, struct SaveSectorLocation *locations)
{
u8 status;
gReadWriteSector = &gSaveDataBuffer;
if (sectorId != FULL_SAVE_SLOT)
{
// This function may not be used with a specific sector id
status = SAVE_STATUS_ERROR;
}
else
{
status = GetSaveValidStatus(locations);
CopySaveSlotData(FULL_SAVE_SLOT, locations);
}
+ if (status == SAVE_STATUS_OK) {
+ if (gSaveBlock2Ptr->_saveSentinel != 0xFF)
+ status = SAVE_STATUS_OUTDATED;
+ else if (gSaveBlock2Ptr->saveVersion != SAVE_VERSION)
+ status = SAVE_STATUS_OUTDATED;
+ }
+
return status;
}Just like explained above, we check to see if the save file is vanilla with the _saveSentinel and then if the sentinel is there, we check if saveVersion is set to the current version.
Now, let's head over to src/main_menu.c and make it so the game checks if the save version is wrong:
static void Task_MainMenuCheckSaveFile(u8 taskId)
{
@@ ...
switch (gSaveFileStatus)
{
case SAVE_STATUS_OK:
tMenuType = HAS_SAVED_GAME;
if (IsMysteryGiftEnabled())
tMenuType++;
gTasks[taskId].func = Task_MainMenuCheckBattery;
break;
case SAVE_STATUS_CORRUPT:
CreateMainMenuErrorWindow(gText_SaveFileErased);
tMenuType = HAS_NO_SAVED_GAME;
gTasks[taskId].func = Task_WaitForSaveFileErrorWindow;
break;
+ case SAVE_STATUS_UPDATED:
+ CreateMainMenuErrorWindow(gText_SaveFileOldUpdated);
+ tMenuType = HAS_SAVED_GAME;
+ gTasks[taskId].func = Task_WaitForSaveFileErrorWindow;
+ break;
+ case SAVE_STATUS_OUTDATED:
+ CreateMainMenuErrorWindow(gText_SaveFileOldErrored);
+ tMenuType = HAS_NO_SAVED_GAME;
+ gTasks[taskId].func = Task_WaitForSaveFileErrorWindow;
+ break;
case SAVE_STATUS_ERROR:
CreateMainMenuErrorWindow(gText_SaveFileCorrupted);
gTasks[taskId].func = Task_WaitForSaveFileErrorWindow;If the status is SAVE_STATUS_OUTDATED when it gets to the main menu, it means the save file is out of date and something went wrong updating it. (Which, since nothing happens to update the file right now, will always happen.) Once we get around to updating it, we'll eventually set the save status to SAVE_STATUS_UPDATED to make it so the player is shown a message saying it has been updated. Let's add those strings now:
In strings.c (and add externs for these in strings.h):
+const u8 gText_SaveFileOldUpdated[] = _("Your save file is for an older release\nof HACK NAME HERE.\pYour save will be updated. Please back\nup your old save if you wish to keep it.");
+const u8 gText_SaveFileOldErrored[] = _("Your save file is for an older release\nof HACK NAME HERE.\pThe attempt to update the save file\nhas failed.\pPlease report this to AUTHOR NAME.");If you want to have some error reporting, see my example where I also buffer a string reason to show to the player.
If you compile and run the game now, you'll find that the save file no longer appears, because it is out of date:
It's time to fix that!
So, let's head over to intro.c. Remember where the game loaded up the save game? We're gonna modify that now:
void CB2_InitCopyrightScreenAfterBootup(void)
@@ ...
LoadGameSave(SAVE_NORMAL);
+ if (gSaveFileStatus == SAVE_STATUS_OUTDATED)
+ {
+ if (UpdateSaveFile())
+ gSaveFileStatus = SAVE_STATUS_UPDATED;
+ }
if (gSaveFileStatus == SAVE_STATUS_EMPTY || gSaveFileStatus == SAVE_STATUS_CORRUPT)
Sav2_ClearSetDefault();If the normal game load determines that the save file is out of date, we will attempt to update the save file using the UpdateSaveFile function we're about to write.
So, let's go back to save.c and scroll to the bottom. It's time to add functions to update the save!
First, let's include the save structures we copied off earlier, and make a quick function to read off the save version from the save blocks:
#include "data/old_saves/save.v0.h"
u16 DetermineSaveVersion()
{
if (gSaveBlock2Ptr->_saveSentinel != 0xFF) return 0;
return gSaveBlock2Ptr->saveVersion;
}Next, we're gonna make the UpdateSaveFile function, which does the actual updating. (Make sure to add its declaration to the save.h header.)
bool8 UpdateSaveFile(void)
{
u16 version = DetermineSaveVersion();
u8* sOldSaveBlock;
bool8 result = TRUE;We grab the version from the already loaded save block. Next, we make some space on the heap for a workspace for the old save block:
// Load the old save file into the heap
sOldSaveBlock = AllocZeroed(SECTOR_DATA_SIZE * NUM_SECTORS_PER_SLOT);We're going to copy the way UpdateSaveAddresses loads the save data into RAM and use it to load the save block into the heap. (There's probably a better way to assign the data pointers, but this is good enough for our use right now.)
{
// Assign locations to load the old save block into the heap
u8* ptr1 = sOldSaveBlock; //pretend this is gSaveBlock2Ptr
u8* ptr2 = sOldSaveBlock; //pretend this is gSaveBlock1Ptr
u8* ptr3 = sOldSaveBlock; //pretend this is gPokemonStoragePtr
int i = SECTOR_ID_SAVEBLOCK2;
gRamSaveSectorLocations[i].data = (void *)(ptr1) + sSaveSlotLayout[i].offset;
gRamSaveSectorLocations[i].size = sSaveSlotLayout[i].size;
ptr3 = ptr2 = ptr1 + sSaveSlotLayout[i].size;
for (i = SECTOR_ID_SAVEBLOCK1_START; i <= SECTOR_ID_SAVEBLOCK1_END; i++)
{
gRamSaveSectorLocations[i].data = (void *)(ptr2) + sSaveSlotLayout[i].offset;
gRamSaveSectorLocations[i].size = sSaveSlotLayout[i].size;
ptr3 += sSaveSlotLayout[i].size;
}
for (i = SECTOR_ID_PKMN_STORAGE_START; i <= SECTOR_ID_PKMN_STORAGE_END; i++)
{
gRamSaveSectorLocations[i].data = (void *)(ptr3) + sSaveSlotLayout[i].offset;
gRamSaveSectorLocations[i].size = sSaveSlotLayout[i].size;
}
// Load the save from FLASH and onto the heap
CopySaveSlotData(FULL_SAVE_SLOT, gRamSaveSectorLocations);
}Now we've got the old save data on the heap, ready for us to reinterpret. But the old save is also currently loaded into the usual save blocks in memory, and it's the wrong data format, so we need to clean it out so we can have a clean slate to copy to:
// Zero out the data currently loaded into the save structs
CpuFill16(0, &gSaveblock2, sizeof(struct SaveBlock2ASLR));
CpuFill16(0, &gSaveblock1, sizeof(struct SaveBlock1ASLR));
CpuFill16(0, &gPokemonStorage, sizeof(struct PokemonStorageASLR));Now, we need to attempt to update the version! (We don't need a case for our current version, as we won't get here if the version matches.)
// Attempt to update the save
switch (version) {
case 0: // Upgrading from vanilla to version 1
result = UpdateSave_v0_v1(gRamSaveSectorLocations);
break;
default: // Unsupported version to upgrade
result = FALSE;
break;
}We'll get back to defining UpdateSave_v0_v1 in a little bit. First let's finish up this function:
// Clean up and perform post-load copying operations
Free(sOldSaveBlock);
CopyPartyAndObjectsFromSave();
// Note, the save is now up to date, but it won't be saved back to FLASH until the player saves the game.
return result;
}Alright! Everything is in place for us to support updating our game saves! Now to actually do it!
Let's head over to save.v0.h. We're gonna put the actual upgrading function in here to keep it together with the old data structures. Here's our method signature:
bool8 UpdateSave_v0_v1(const struct SaveSectorLocation *locations)The function we just wrote above is going to call this function, passing in the save sector locations set to the heap location where we've loaded the old data. We need to interpret that data as the old save file: (Note, these pointers are const so that we don't accidentally write to the old save data:)
const struct SaveBlock2_v0* sOldSaveBlock2Ptr = (struct SaveBlock2_v0*)(locations[SECTOR_ID_SAVEBLOCK2].data);
const struct SaveBlock1_v0* sOldSaveBlock1Ptr = (struct SaveBlock1_v0*)(locations[SECTOR_ID_SAVEBLOCK1_START].data);
const struct PokemonStorage* sOldPokemonStoragePtr = (struct PokemonStorage*)(locations[SECTOR_ID_PKMN_STORAGE_START].data);Now the rest of the function is entirely up to you and your needs. You can refer to my example for reference, but the gist of it is that you need to copy all the data over from the old file to the new file, updating it as needed.
Most fields can just be assigned to:
gSaveBlock2Ptr->playerGender = sOldSaveBlock2Ptr->playerGender;
gSaveBlock2Ptr->playerApprentice = sOldSaveBlock2Ptr->playerApprentice;
gSaveBlock1Ptr->pos = sOldSaveBlock1Ptr->pos;Arrays will need a bit more special handling. You can either copy as a block:
CpuCopy16(&sOldSaveBlock1Ptr->pokeblocks, &gSaveBlock1Ptr->pokeblocks, sizeof(gSaveBlock1Ptr->pokeblocks));
CpuCopy16(&sOldSaveBlock1Ptr->seen1, &gSaveBlock1Ptr->seen1, sizeof(gSaveBlock1Ptr->seen1));
CpuCopy16(&sOldSaveBlock1Ptr->flags, &gSaveBlock1Ptr->flags, sizeof(gSaveBlock1Ptr->flags));Or you can loop over the arrays for more complex upgrading:
for(i = 0; i < min(ARRAY_COUNT(gSaveBlock1Ptr->playerParty), ARRAY_COUNT(sOldSaveBlock1Ptr->playerParty)); i++)
{
// Possibly convert BoxPokemon_v0 to BoxPokemon, if those are different
gSaveBlock1Ptr->playerParty[i] = sOldSaveBlock1Ptr->playerParty[i];
}Structs that are not changed can be directly assigned:
*gPokemonStoragePtr = *sOldPokemonStoragePtr;Finally, one last consideration: One of the most common changes between versions is changes in maps. And the save game keeps a lot of information about the current map view so that the game can resume right where the player left off, including event object locations/templates and even map tiles around the player. If your player loads up in a changed map and starts walking around, they may find themselves stuck, with the location where they were loaded up a strange vestige of the last version of the game.
Because of this, it's best if we don't let the game use that data when it loads up, so it loads up a fresh copy of the map. But we can't guarantee that the player saved someplace that didn't change; we don't want them loading up into a newly built wall or out of bounds somewhere! So we can't just load up wherever the player saved.
So instead, we're going to use the game's already built-in "Continue Game Warp" that gets used after the player enters the Hall of Fame:
SetContinueGameWarpStatus();
gSaveBlock1Ptr->continueGameWarp = gSaveBlock1Ptr->lastHealLocation;This will put the player back at the last heal location they've used, as if they blacked out, loading up the map fresh when they continue the game next. This also means you don't have to worry about copying over the mapView, objectEvents, and objectEventTemplates arrays.
Remember to return TRUE at the bottom of the function, so that the UpdateSaveFile function knows the update worked! Or return FALSE at any point if the update fails for whatever reason (just know that that'll mean your players' save files won't survive the version update).
Everything is in place, but there's one last thing we need to fix before you go ripping out all your most hated and unwanted Emerald features [insert shaking fist at the Walda phrases].
You might note once you start deleting stuff from the save game that old versions of the save file won't load due to "being corrupt" instead of being out of date. This is a result of the game doing a checksum on the segments of the save data before even getting to the save block code.
This is because the game determines how much of the block to checksum based on how big the save block structs are. So when you cut down the save blocks, it will reduce how much of the save block it is summing up, resulting in false negative checksums.
There's two ways to fix this:
The easiest way to fix this is to go into save.c and find where SAVEBLOCK_CHUNK is defined and change the definition:
#define SAVEBLOCK_CHUNK(structure, chunkNum) \
{ \
- chunkNum * SECTOR_DATA_SIZE, \
- sizeof(structure) >= chunkNum * SECTOR_DATA_SIZE ? \
- min(sizeof(structure) - chunkNum * SECTOR_DATA_SIZE, SECTOR_DATA_SIZE) : 0 \
+ chunkNum * SECTOR_DATA_SIZE, SECTOR_DATA_SIZE \
}This will make it so that the entire save chunk is checked every time, which works out because the chunk that's not usually checked in vanilla is all 0s, which doesn't affect the checksum. However, this size is used for both checking the checksum and for copying the data from the in-memory save block to the write buffer. So if your save block is smaller than a save sector, garbage data past the end of the struct will get copied to and from the save block in memory. Things may still work, but it's messy. Especially with things like the ASLR on the save blocks leaving garbage behind in memory.
The more correct way to fix this is to make it so only the verification checksums do the whole sector size, but it's going to mean changes in more places and also not all the places where CalculateChecksum is used:
@@ -511 +511 @@ static u8 CopySaveSlotData(u16 sectorId, struct SaveSectorLocation *locations)
id = gReadWriteSector->id;
if (id == 0)
gLastWrittenSector = i;
- checksum = CalculateChecksum(gReadWriteSector->data, locations[id].size);
+ checksum = CalculateChecksum(gReadWriteSector->data, SECTOR_DATA_SIZE);
// Only copy data for sectors whose signature and checksum fields are correct
if (gReadWriteSector->signature == SECTOR_SIGNATURE && gReadWriteSector->checksum == checksum)
@@ -546 +546 @@ static u8 GetSaveValidStatus(const struct SaveSectorLocation *locations)
if (gReadWriteSector->signature == SECTOR_SIGNATURE)
{
signatureValid = TRUE;
- checksum = CalculateChecksum(gReadWriteSector->data, locations[gReadWriteSector->id].size);
+ checksum = CalculateChecksum(gReadWriteSector->data, SECTOR_DATA_SIZE);
if (gReadWriteSector->checksum == checksum)
{
saveSlot1Counter = gReadWriteSector->counter;
validSectorFlags |= 1 << gReadWriteSector->id;
}
}
@@ -578 +578 @@ static u8 GetSaveValidStatus(const struct SaveSectorLocation *locations)
if (gReadWriteSector->signature == SECTOR_SIGNATURE)
{
signatureValid = TRUE;
- checksum = CalculateChecksum(gReadWriteSector->data, locations[gReadWriteSector->id].size);
+ checksum = CalculateChecksum(gReadWriteSector->data, SECTOR_DATA_SIZE);
if (gReadWriteSector->checksum == checksum)
{
saveSlot2Counter = gReadWriteSector->counter;
validSectorFlags |= 1 << gReadWriteSector->id;
}
}
@@ -660 +660 @@ static u8 TryLoadSaveSector(u8 sectorId, u8 *data, u16 size)
if (sector->signature == SECTOR_SIGNATURE)
{
- u16 checksum = CalculateChecksum(sector->data, size);
+ u16 checksum = CalculateChecksum(sector->data, SECTOR_DATA_SIZE);
if (sector->id == checksum)
{And that's it! That's the basics of doing save versioning and backwards compatibility.
Things to remember:
- Going forward, you'll want to make a new save version at least every time you release with any changes to the save block.
- The functions which update the save block should be updated to always make the old version into the latest version, so someone with a save version 3 behind the current version can still migrate their save forward to the latest version. (You could set up incremental updates from one version to the next, but beware that that'll require more heap space and processor time to update things.)
- If you shuffle maps around in your ROM, remember that the save file saves the
lastHealLocationas a map warp and not a heal location id. You will probably want to keep a map of where the previous heal locations were so you can map the heal locations to the new map warps on upgrade. (And do this before you set the continue game warp to the last heal location.)- If all else fails, you can just set the heal location to the player's bedroom or something.
- If you decide to change the IDs of things, like pokemon species IDs or item IDs, between save versions, be ready to map all of those from save-to-save. I'd probably do it as a big look-up table, where you index into the table with the original id and retrieve the new ID.
- Be wary of how the meaning of variables or flags can change between saves.
- Remember that it's a pretty common practice in game design to make quest state IDs jump by 100 each step, so that it's easier to insert a state in the middle if you need to fix something.
Once again, if you want an example, my branch is there to peruse. Happy coding!
-- Tustin2121 (Feb 2023)