Development Guide
For anyone thinking of working on Smaghetti's codebase, this doc walks through everything there is to know. As of Jan 13, 2022, I am still writing this doc. There is a lot more to add.
You need
- Node (I am using 12 at time of writing, any new/recent Node should be fine)
- yarn 1, I am using 1.22.17. yarn 2, 3 and npm will all cause issues
clone the repo to your local machine then yarn to install dependencies
yarn develop will then start a local server on localhost:3000, it will also build storybook and start it on localhost:6006, it should pop open a browser to storybook once it is ready.
The app is a pretty typical nextjs app using redux and tailwind. src/pages is where all the pages are defined. I never put much code or logic there, these pages always call out to an actual page implementation. For example src/pages/editor/index.tsx basically just renders MakePage, which is defined over in src/components/editor/MakePage
Smaghetti always uses the containers/dummy approach but is admittedly a bit loose here. A decent example of a dumb component is DownloadButton and its corresponding container ConnectedDownloadButton
DownloadButton just renders DOM and has no logic. ConnectedDownloadButton is the opposite, no DOM and all logic. Both are placed in a DownloadButton/ folder where index.ts does export { ConnectedDownloadButton as DownloadButton }, so the rest of the app doesn't really know or care about connected vs dummy, the components are just black boxes.
The src/entities directory is where all entity code lives. An entity is something in SMA4 such as a Goomba, Brick, etc. The goal is to keep the entity source of truth entirely in one file (or directory) as much as possible. For example, Goomba.tsx contains everything that is needed to work with goombas:
- drawing them in the editor
- getting their binary data for adding into a SMA4 save file
- their resource and palette info for extracting out their image for display
- what object set and sprite graphic sets they are compatible with
- where they get placed in the palette (aka "item chooser")
Basically if you need to work on Goomba, it is found in this file. This has proven to work really well. It's very easy to say fix a Goomba bug, or add a new entity.
Entities are a tad strange in that their simpleRender and render functions return JSX. So they are kind of like React components with a bunch of extra "stuff". This strangeness has paid off though. Originally I was trying to handle details panes and all of the various things entities need in a detached, generic way. But it got so overwhelming and out of control. Entities in SMA4 behave in so many different ways, that just letting the Entity types handle their own rendering, their own detail panes, etc, has proven to be far more successful. I still extract out commonalities where it makes sense, such as GeneratorFrame which all generator entities use.
Smaghetti uses supabase as its backend. There is no local backend for development.
Setting up a supabase environment involves creating one in supabase, then with the db credentials, running all of the migrations. supabase.md shows how to do that.
All supabase calls are found in remoteData
There really isn't too much, basically just save levels, delete levels, vote on levels, and display published levels on the levels page. All in all, the backend is a minor part of Smaghetti.
This is possible, but you can't work on anything that requires supabase. So for example the editor will work just fine, but you can't log in or save a level.
Entities should be named with adjectives at the end and not the front. So an entity should be called "KoopaGreen", not "GreenKoopa". That way "KoopaGreen", "KoopaParaGreen", "KoopaRed", etc all stay near each other in the file system. Like in most editors that show a tree of all the files, all the koopas will be bunched together.
There are a lot of entities that don't follow this. Those are early entities I created before I realized this works better. You can't rename an entity as any level that has that entity will break. It's possible to rename an entity then write a "level version migration" for it, but that is so painful and awful I always avoid it unless absolutely necessary.
Smaghetti is very slowly adding binary parsing. Where you can take a SMA4 save file, and parse it out into a Smaghetti level. Once that is fully working (and it's a HUGE undertaking, and to be honest I don't work on it very much because it's so much work :( ), then we'd be able to rename entities all we want.
Smaghetti does not use static image files for entities. Instead, images are generated from the ROM itself. The first time someone comes to Smaghetti's editor, it will take a good while to extract all the images. But these images are saved locally in the browser, so on subsequent visits only new and changed images need to get generated.
Both tiles and palettes are needed for an entity's images to get extracted
For small entities, you can get their tiles manually. Let's pretend we are adding GreenKoopaTroopa as a new entity to the editor:
Edit Goomba.tsx so that its toSpriteBinary emits a koopa troopa instead of a goomba.
So change it from this
toSpriteBinary({ x, y }) {
return [0, this.objectId, x, y];
}
to this
toSpriteBinary({ x, y }) {
return [0, 0x6c, x, y];
}
Where 6c is green koopa troopa's object id.
Now in the editor, add a goomba to a level and play the level. When playing the level it will actually be a koopa due to the hack we just did.
Download this level and set it to be your mGBA save file for your sma4 rom.
Start up mGBA, load your level, pause the emulator (ctrl+p), then pull up the sprite viewer
Highlight one of the koopa's sprites, then one of the sprite's tiles. You are interested in the tile's vram address, and later you will also want the palette number (both boxed in red above).
Now pull up mGBA's memory viewer, and go to the vram address you found above, highlight the two rows, right click, and choose "copy selection"
The tile's data is now in your clipboard. Now head to the tile viewer click "dump compressed", then click "show indices", then in the far right text box, enter in the tile data you copied from mGBA and click "search". You will be taken to this tile's location in the ROM
From here you now have all the info you need to define the koopa's tiles in its entity file. You need the offset at the top, in this case 0x134104, and all of the tile indices that make up the Koopa. Go ahead and dump them into the koopa's resource in its entity, like this:
resource: {
palettes: [
[],
],
romOffset: 0x134104,
tiles: [
[258, 259],
[290, 291],
[322, 323],
[354, 355],
],
},Now the koopa's tiles are defined. Now let's grab the palette.
In the sprite viewer we saw we needed palette 2. So in the memory viewer, set the dropdown to palette memory. There are 32 palettes in total. The first 16 are for backgrounds, and the last 16 are for sprites (called objects in GBA terminology, which is confusing for SMA4).
Since this sprite uses palette 2, we actually want palette 18 (since this is a sprite and not a background, skip the first 16 palettes). This is a zero based index, so it is actually the nineteenth palette. Find the nineteenth palette (each palette is two rows), highlight it, right click and choose "copy selection". Now the palette data is in your clipboard
Head to a terminal in the smaghetti directory and run scripts/mgbaPaletteToArray.ts, using the copied palette data as the argument
12:09 $ yarn run-ts-node scripts/mgbaPaletteToArray.ts 967FFF7FC6186B02101BB413FD259E365F47BF1A1C003F253F46D17A2C6EA659
yarn run v1.22.17
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' scripts/mgbaPaletteToArray.ts 967FFF7FC6186B02101BB413FD259E365F47BF1A1C003F253F46D17A2C6EA659
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' scripts/mgbaPaletteToArray.ts 967FFF7FC6186B02101BB413FD259E365F47BF1A1C003F253F46D17A2C6EA659
[ 0x7f96, 0x7fff, 0x18c6, 0x26b, 0x1b10, 0x13b4, 0x25fd, 0x369e, 0x475f, 0x1abf, 0x1c, 0x253f, 0x463f, 0x7ad1, 0x6e2c, 0x59a6 ]
Done in 2.00s.The line just above "Done" is the palette data converted into a JavaScript array. Just copy it into the resource:
resource: {
palettes: [
[
0x7f96,
0x7fff,
0x18c6,
0x26b,
0x1b10,
0x13b4,
0x25fd,
0x369e,
0x475f,
0x1abf,
0x1c,
0x253f,
0x463f,
0x7ad1,
0x6e2c,
0x59a6,
],
],
romOffset: 0x134104,
tiles: [
[258, 259],
[290, 291],
[322, 323],
[354, 355],
],
}Smaghetti can now take this data and generate the koopa's image for use in the editor.
Smaller entities like a koopa only have one palette. More complex entities often have multiple palettes. Add each palette into the palettes array, then in the tiles array, a tile can indicate which palette it uses. The default is zero (the first palette). For example, here is ParaBobombGenerator
resource: {
romOffset: 0x18c914,
palettes: [
[
0x7f96,
0x7fff,
0x18c6,
0x26b,
0x1b10,
0x13b4,
0x25fd,
0x369e,
0x475f,
0x1abf,
0x1c,
0x253f,
0x463f,
0x7ad1,
0x6e2c,
0x59a6,
],
[
0x7fb4,
0x7fff,
0x0,
0x75ad,
0x7a94,
0x7f39,
0x25de,
0x273f,
0x1b1d,
0x2fbf,
0x53ff,
0x119,
0x167b,
0x6ab2,
0x7b98,
0x7bdd,
],
],
tiles: [
[
{ tileIndex: 64, palette: 1 },
{ tileIndex: 65, palette: 1 },
],
[
{ tileIndex: 96, palette: 1 },
{ tileIndex: 97, palette: 1 },
],
[69, { tileIndex: 69, flip: 'h' }],
[101, { tileIndex: 101, flip: 'h' }],
],
},Tiles can also be flipped either vertically or horizontally, above two tiles are flipped horizontally, flip: 'h'
The above approach is fine for small entities. It seems a bit tedious but you can get a small entity's resource in just a minute or two. But large resources are another story. Backgrounds are the worst offender here. Getting a background using the manual approach is an absolute nightmare. So instead, you can use the scripts/png2res.ts script. You can use png2res for small entities too.
Let's pretend the winter background isn't in Smaghetti yet and we'll add it.
Similar to how you hacked the goomba to emit a koopa, hack an existing background to emit the winter background. Head to levelData/constants and change plains from 0x5 to 0x8. 0x8 is the winter background value.
In the editor, create a level with the plains background, then save it and load it into mGBA. It will actually be the winter background since we hacked the value. Pause the emulator (ctrl+p) and pull up the map viewer, then select the correct background layer
In the lower left, click "export" and save the png somewhere. Load it into an image editor and crop it down as needed. For example, here is how I cropped the winter bg
Now open up the memory viewer, go to palette memory, and select all of the palette memory
Now click "save selection", and save it somewhere
Now run png2res
12:26 $ TS_NODE_FILES=true yarn run-ts-node scripts/png2res ../sma4/sma4.gba pngsForResources/winterWiki.png pngsForResources/winterWiki.palette > winterBg.tsRedirect the output to a file, and then that file will have your resource in it (plus some additional output junk you need to trim)
yarn run v1.22.17
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' scripts/png2res ../sma4/sma4.gba pngsForResources/winterWiki.png pngsForResources/winterWiki.palette
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' scripts/png2res ../sma4/sma4.gba pngsForResources/winterWiki.png pngsForResources/winterWiki.palette
// generated by png2res
{
"palettes": [
[
32662,
...
28439,
30585,
24373,
23316,
21211,
18007
],
...
],
"tiles": [
[
{
"tileIndex": 144,
"palette": 1
},
{
"tileIndex": 144,
"palette": 1
},
...Sometimes png2res fails to figure out a tile and there will be nulls. This is due to how the lossy conversion from 24bit to 16bit color space works. I've yet to find a good solution to this. But thankfully, nulls are rare and if they do happen, it's usually not too hard to manually patch the resource using the manual method.
The resource sha map has SHAs for every resource. This sha is calculated against the graphics that get created from a resource. This map is what tells Smaghetti which resources need to get regenerated whenever someone comes back to the editor again. Without this map, every resource would generate on every visit, which takes a few minutes.
If you add a new entity, or change the tiles and/or palettes of an existing entity or resource, you should regenerate this map by invoking TS_NODE_FILES=true yarn run-ts-node scripts/generateResourceShas.ts ./sma4.gba ./src/tiles/resourceShaMap.ts from smaghetti's root directory, where the first argument is the path to the sma4 gba rom. You can also do yarn generate-resource-sha-map but this shortcut assumes the rom is located at ../sma4/sma4.gba, which is where it is on my machine. If/when more people work on Smaghetti, I'll get rid of all this stuff that is specific to my machine.
You can confirm the map got updated by running git status, you should see src/tiles/resourceShaMap.ts has been changed. If you then do git diff src/tiles/resourceShaMap.ts, you should see the change is for the resource you edited.
A resource can either be defined inside an Entity, such as with GreenKoopaTroopa above. This is most common, almost all entities define their own resources. But there are also standalone resources in the resources directory. These are just like the resources in entities, but they are just on their own. This is where background resources live for example. Some resources are shared by several entities or used in other parts of smaghetti, so those resources are made standalone, such as GiantVegetable. GiantVegetable is standalone because its not an entity, it's just a payload for BuriedVegetable. Some resources are standalone because in the early days of Smaghetti I did that whenever an entity needed more than one resource. Now that there are better tools for extracting graphics, and now that entities can have more than one resource defined, those resources don't need to be standalone anymore but I've not had a chance to clean this up.
Objects in SMA4 terminology are things that go into a background layer, such as bricks, coins, floors, etc. The GBA calls sprites "objects", which is basically the opposite of SMA4 terminology.
Objects are divided into sets, and only one set can be loaded for a given level. Common objects like bricks and coins are found in pretty much every set. More unusual objects are sometimes only in one set. So when adding an object entity to Smaghetti, it is important to figure out which sets it is in.
Smaghetti uses brute force to determine this, using brute/getObjectSetBytes.ts. This script will pick an object set, set it on the level, load the level into gba.js, then take a screenshot. It does this for all object sets, and gathers a ton of screenshots. Then you can see the results using the brute/toHtmlResults.ts script.
Here is how to figure out what object sets PipeVerticalGiant is in.
Figure out what bytes are needed to emit the pipe into a level, in this case it is <bank and height>, y, x, 7, for example 44 16 5 7 Adds a giant pipe that is 5 tiles tall at (5, 16) use these bytes as the input to the getObjectSetBytes script. But you don't invoke this script directly, as we need to run several copies of it in parallel to speed up the dump
12:57 $ yarn dump-object-set-bytes '44 16 5 7'
yarn run v1.22.17
$ brute/getObjectSetBytes_parallel.sh '44 16 5 7'
Done in 0.04s.
✔ ~/dev/smaghetti [main|⚑ 1]
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 3 4
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 2 4
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 0 4
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 1 4
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 3 4
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 2 4
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 0 4
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getObjectSetBytes.ts '44 16 5 7' 1 4
[4,0] not-empty
[8,0] not-empty
[12,0] not-empty
[0,0] not-empty
...This creates four processes, each running the script. Four was chosen since my machine has 8 cores. If your machine has fewer cores, 4 might be really bad (you might find your machine becomes really unusable). If that is the case let me know, I'll look into making it so you can pick the number of processes.
On my machine, this dump takes about 6 minutes. When it is done, the brute/results_getObjectSetBytes_44_16_05_07 directory will have tons and tons of screenshots in it, here is one:
This particular screenshot is a miss, as we got garbled graphics and it doesn't look like the giant pipe at all.
Now run yarn run-ts-node brute/toHtmlResults.ts brute/results_getObjectSetBytes_44_16_05_07, and a browser window will open with the results deduped and gathered
You can see here the fifth result is almost correct. It is the pipe, but with the wrong graphic set. If you look through the results, eventually you find proper pipes
For example, here we see [5,11] is a valid pipe. Gather up all the valid values and place them in the entity
objectSets: encodeObjectSets([
[13, 11],
[11, 11],
[5, 11],
])NOTE! just because visually it looks correct, doesn't mean it actually is. Sometimes a bogus object set and a bogus graphic set happen to combine to look correct, but when you use them in a level, the entity does not behave correctly. Other times valid entities can look the same but actually be different, for example Coin and CoinWater look the same in the dump, but are totally different entities. This is a common source of bugs in Smaghetti :)
This is quite similar to determining object sets, but slightly different mostly because with sprites, sometimes Mario has to do something to see the sprite. For example, we can't know what is inside a question block until Mario hits it.
Let's do a dump for QuestionBlockGiant
First create a level that has a giant question block in it. You might need to hack an existing entity if it's not in the editor yet. Place the question block right above Mario's starting position, then download the level. When in dev mode, three files will get downloaded. One is new level.json, and that is what you want.
I typically take that json file and move it into the brute directory, named after the entity, ie mv 'new level.json' brute/giantQblock.json
Then invoke yarn dump-graphic-set-bytes brute/qiantQBlock.json jump
The "jump" parameter will cause Mario to continually jump while doing the dump. That way he hits the question block so we can see what is inside.
13:36 $ yarn dump-graphic-set-bytes brute/giantQblock.json jump
yarn run v1.22.17
$ brute/getSpriteGraphicSetBytes_parallel.sh brute/giantQblock.json jump
Done in 0.04s.
✔ ~/dev/smaghetti [main|⚑ 1]
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 1 2
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 0 2
$ yarn ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 2 2
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 1 2
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 0 2
$ /home/matt/dev/smaghetti/node_modules/.bin/ts-node --compiler-options '{ "module": "commonjs", "jsx": "react" }' brute/getSpriteGraphicSetBytes.ts brute/giantQblock.json jump 2 2
00_00_00_00_00_00 not-empty
00_00_00_00_00_00 not-empty
00_00_00_00_00_00 not-empty
01_00_00_00_00_00 not-empty
00_00_00_00_01_00 not-empty
00_00_01_00_00_00 not-empty
...Just like objects, this uses four cores and takes about 6 minutes on my machine. And just like with objects, yarn run-ts-node brute/toHtmlResults.ts brute/results_getSpriteGraphicSetBytes_giantQblock
Here we can see giant question block needs the first sprite graphic set to be 9, so its overall graphic set is [9, -1, -1, -1, -1, -1] -1 means it doesn't care what the other 5 slots are.
I am fudging this a tad, because sprites that have payloads sometimes are in different graphic sets depending on which payload it has.