CRDB: A binary database format for the OSCR #1029
Replies: 1 comment 6 replies
-
|
How does one edit and modify the database? Is it compiled from the original text format at build time, or are there authoring tools, or what's the plan on the authoring end of things? |
Beta Was this translation helpful? Give feedback.
-
|
It's also probably worth noting the file's format and type is part of the file's header, so someone could write a desktop application to read and edit CRDB files. The way the format is written it is very easy to generate with rudimentary code. So, this is the kind of app you might write to learn to work with the filesystem. I might write one eventually but I have other stuff I think is more important, but anyone is welcome to.
The format string is more or less the string to encode/decode it using PHP's pack and unpack functions which were ported from Perl. |
Beta Was this translation helpful? Give feedback.
-
|
I was originally gonna add this to the first post as an edit, but it ended up being way longer than I expected. So, to further add: As part of the last step of making this format final, I plan to make the CRDB generators take a JSON file instead of a No-Intro DB. This JSON file will be included in the repo much like the one for mappers shown in my screenshot in the first comment. The site will still be able to convert No-Intro DB files but it will convert them to a JSON file. The reason for this is to make a way to include arbitrary data without having 2 input files. When converting a No-Intro XML file to the JSON format it will use the current JSON database to fill in missing fields. There will be 1 or more checkboxes to select which data to prefer for fields that might be likely to be overwritten (i.e. you might want to take the iNES entry from No-Intro but keep the PRG/CHR/etc fields that would normally be extracted from it -- this would also be the most common use case for the NES as well). That should make it easy for someone to update the JSON and CRDBs with any updates No-Intro makes. I chose JSON because it's a standard format that, unlike XML, is easy to read and edit even if you aren't super code-savvy. As long as you understand how quotes work it should be pretty easy. I did toy around with the idea of a JSON-based DB for the firmware and there is a library for the ATmega to read JSON. It worked well, and much like the CRDB format it used about the same amount of PROGMEM or less than the TXT DBs do, however, it was noticeably slower than the TXT files. Like, painfully slow. The only DB it felt ok for was the one for NES mappers (the input JSON file for NES mappers is even the DB file I made for the JSON DB format). Eventually, I might start creating a database of our own, similar to the one run by No-Intro but more geared towards our purposes. No-Intro primarily focuses on storing checksums and cataloging cartridges where as the DB I'd write for us would be geared towards documenting how cartridges are dumped/laid out, the mappers and chips they use, etc. I'd make it have a login with GitHub option so anyone with write access to this repo could log in and make edits as needed. In this vein, it'd effectively be a nice-looking website-based GUI editor for the JSON and CRDB files. It would take the day's edits and commit all of them at a predetermined time (i.e. midnight) as a large batch (who made what changes would be part of the commit message). This should make it easier for people to collaborate on the same DB without worrying about stepping on each other's changes (remember that while JSON can be diffed/merged by Git fairly easily, the binary files cannot). It's probably worth mentioning that the new hardware versions that are based on the RP2040 and STM32MP1 architectures will likely use the JSON files directly rather than the CRDB ones simply because they have much more resources available to them so it just makes sense. As those new hardware versions can optionally access the Internet, I plan to make it so that, if the user wishes (opt-in), people can have the OSCR automatically submit the result of their dump (the name [if matched], the checksums, and any additional details like PRG/CHR or mapper info; it will not share the dumped data/the ROM itself) to OSCR.tools. This would be, of course, fully opt-in and up to the user to enable (it would also require an account). It's more to get confirmation that the existing data is correct than it is to discover the settings for unknown carts. I'd also be totally into sharing all this (non-user) data with No-Intro if they desire it. If anyone is concerned about an "online feature" or "account needed" I'll take this time to put your mind at ease: I hate always-online and account-required services that unnecessarily require an account. I also hate the immense data collection that happens these days. You don't need to just take my word for it either, simply head to my store or OSCR.tools and check your ad blocker/privacy addons and look at how many things were blocked/loaded from third parties. I'll save you the time: It's zero. The only scripts loaded from third-party domains on my store are on the checkout page because it's required for using PayPal. Even the by CC-BY-SA logo on OSCR.tools is rehosted locally, not linked from CC's domain. You can read more about my stance on privacy on my store's privacy policy page. |
Beta Was this translation helpful? Give feedback.
-
|
For the sake of git versioning, I think you'd want plain text files in the repo to be the source of truth for the data. Then as part of the build process, they can be converted to binary. BTW, about cataloguing the specifics of how cartridges are laid out/specific dump settings used - I'd like to store more of that data/store that data better in No-Intro. If you have any specific fields that need to be added or general ideas about that, it'd be great to discuss it (e.g. via DM if you want). |
Beta Was this translation helpful? Give feedback.
-
Yeah, that's why I said:
Unless you meant the existing DB files, in which case I must disagree. The TXT files are non-standard and could easily be broken without even realizing it. At least with JSON you usually get a specific error with a line number and can use a linter when editing it.
Well, for NES, PRG/CHR/RAM sizes stored as a field separate from the iNES header would be nice as I currently am parsing those out. For other systems, it could help to have ROM size (as in the size on the cartridge), mapper type, etc. As far as I'm aware, some ROMs used in the wild are over-dumped. Also, I absolutely loathe XML. It's difficult to work with in pretty much any language. There's a reason the industry is moving away from it and it's a good reason: XML sucks.
You can PM me on Discord if you want to. |
Beta Was this translation helpful? Give feedback.
-
|
Ah sorry I missed that part of your message. Those sound like good suggestions. And good idea for No-Intro to add JSON export too. Yeah lets discuss these more on discord. |
Beta Was this translation helpful? Give feedback.
-
For a while now I've been toying around with the idea of switching the databases to a binary format to increase the speed of searching/reading from them. It's a lot of work to get set up but once it's going it should provide a much better experience. The general concept is to store the data that is currently stored in text files in a format that is more suitable for being quickly read by a computer. This means using fixed-width fields and records.
Over the weekend I made some progress on porting the NES core to this new format, which I've named CRDB (Cartridge Reader Database). I've gotten the NES cartridge database converted over to a working state and am planning to move the mappers to a database as well. After this conversion, it will add the ability for NES cartridge records to break apart the iNES data from the data used by the firmware. That should mean fewer exceptions are needed. It also will add the much-requested sub-mapper support.
The new format doesn't just add new features. It is also lighter on code, easier to work with in-code, and much faster. See the video below and notice how quick the match for the cartridge is:
2024-09-16.15.31.09-1.mp4
Note that this is faster despite having more than double the cartridge records compared to the current database -- 6,602 vs. 3,114. I didn't change the order of the files in the database either, so it has to seek around 80% of the way into the database to find the match for the cartridge in the video.
It currently uses a little more RAM but a lot less PROGMEM. This is still fluctuating as I work on it but it's at least quite promising, especially considering I haven't moved the mappers yet.
Old TXT System
New CRDB System
Keep in mind the old TXT system's code is still in there, I've only converted the NES core. Once all systems are using it the resource usage should drop even further. It's worth noting that a good bit of the increased memory usage is coming from the added support for submappers. While I haven't moved the mappper table to a database yet, the cartridge database contains submapper info.
As I only have a few NES cartridges I am not able to fully test it. As a result, both systems will be supported via a flag in the
Config.hfile. This is only temporary so that it can be made public without potentially breaking master. To move on to the next step (supporting submappers), the old option will need to be removed. After that, those more familiar with the NES that actually have cartridges to test with will be able to easily add support for new mappers and submappers. It will actually be easier to add support once this is all done.Other systems beyond the NES will also be able to make use of the CRDB files, though mostly just for CRC32s and other checksums. If anyone has any questions, feedback, or feature requests regarding the new format, I welcome the input.
Beta Was this translation helpful? Give feedback.
All reactions