Simulate an SD card drive.

[leftCoast]

Need SD card for doing all sorts of stuff! I use one all the time for icons etc. My thought we'd need to be able to se up a folder online for the disk image. (Just a bunch of files and folders) and the SD "card" can point at that.

[urish]

Some points for discussion if/when we implement this:

Right now, projects consist of just text files (code + JSON + sometimes README). Supporting arbitrary binary files comes with some challenges:

1. You can upload them by pasting text to the code editor. We need some sort of file tree or way to manage files in a project.
2. Sharing binary files between different projects (so you don't have to upload the same files over and over again). We already support forking projects, so that could be a possible solution.
3. Where is the file data stored? Binary files can easily get to tens or 100s of megabytes. This can really pump up our storage requirements. If we store them on Wokwi, then for how long? How do we ensure we end up don't storing terabytes of data no-one really cares about? If we store them somewhere else (e.g. user-provided GitHub repo), is that a good user experience?
4. How do we handle large files when you open a project? We're already downloading a few MBs to your browser just for the simulator (depending on the type of project). If we had to download 10s of MBs more just to start the simulation, this would severely impact the user experience. Theoretically, we could implement a mechanism that would download blocks of storage as you read the data from the SD card, but that can be pretty complex to implement and it also means we'd need to pause the simulation whenever you read from the SD card, so not super ideal.
5. Do we permit all files types or do we have a whitelist? Do we scan uploaded files for viruses?

Most of these issues may be mitigated if we only store files locally, on the user computers, and not share them as part of the project. But then we lose the benefit of "share and click to play".

Anyway, it'd be useful to have some input on the use cases. How many files do you usually have in the SD card in your projects? How big? Which file types?

[leftCoast]

How about.. We take care of "hosting" the files ourselves and just give Wokwi (What is a Wokwi anyway?) a URL to the folder we want. I'd vote for just a file on a web server. That way we can easily share them when needed. And the user deals with how simple of complicated the file structure is. Worried about virus? Deal with trusted users.

[urish]

Hosting the files elsewhere will solve the storage concerns. There is still the project load time concern, and also, what do we do in case the files aren't found? Do we start the simulation with an empty SD card? Do we want the user?

Also, I just remembered there's another use case for the SD card, which is for data logger projects. In that case, there's no file storage needed (at least not always), but we do need to let the user download the files saved to the SD card.

Now the issue is at the top of the list (thank you!), so I'm starting to work on the implementation on the SD protocol as we figure out the storage and user experience details.

[urish]

References for the SD card SPI protocol:

1. http://www.rjhcoding.com/avrc-sd-interface-1.php
2. http://elm-chan.org/docs/mmc/mmc_e.html
3. https://www.fatalerrors.org/a/sd-card-initialization-and-command-details.html

[leftCoast]

Think of the SD card as just a different interface for a wifi board. The user gives you an initial URL (SD Card root) and you read from it or write to it using the File commands. Including directory listings etc. The simulator doesn't read it all in, It just does what the Arduino already does and read in a buffer. This way you don't have to know anything about what kind of file they are accessing, changing or creating. In my case its a lot of .bmp files. In many cases it would be text files. If there is no initial folder found? Do what the Arduino does with an unformatted SD card. It just fails. Allowing the user to host their own files makes it really nice because its just a drag and drop for most computers to put an SC card image onto a server. And its just another drag and drop to copy it back off once its been changed. A url alone gives read access. A URL with login give Read/Write access.

…

-jim lee

On Sat, September 18, 2021 2:13 am, Uri Shaked wrote: Hosting the files elsewhere will solve the storage concerns. There is still the project load time concern, and also, what do we do in case the files aren't found? Do we start the simulation with an empty SD card? Do we want the user? Also, I just remembered there's another use case for the SD card, which is for data logger projects. In that case, there's no file storage needed (at least not always), but we do need to let the user download the files saved to the SD card. Now the issue is at [the top of the list](https://wokwi.com/features) (thanks you), so I'm starting to work on the implementation on the SD protocol as we figure out the storage and user experience details. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #183 (comment)

[urish]

Thanks for sharing your thoughts!

Some thoughts about the bring-your-own URL approach:

1. The SD card needs to have a file system. This means that the user will need to know how to create a file system image with all the relevant files in it (e.g. FAT image). I'm not sure how many users actually know how to work with file system images.
2. There's no standard way for writing files to arbitrary URL on the web. If we wanted to support writing to a 3rd-party services, the implementation would require specific API integration for each 3rd party service we wanted to support (e.g. GitHub, Dropbox, Google Drive, etc.). Also, I'm not sure how many of the file hosting services actually support partial file writes, so this means appending a single line to a log in 128mb file system would require downloading then uploading the entire 128mb image.
3. Even if we go for a read-only solution (so not worrying about writes), we'd still need the user supplied URL to support CORS and HTTP partial request.
4. Operations on large file systems may cause a significant delay when running the simulation, so we'll need to indicate this somehow to the user (i.e. the simulation speed dropped to 0% because we're now downloading SD card data)

[leftCoast]

Thinking off the top of my head..

When you do the File.open(); thing, you copy the file, if found, to the user's computer. Then do all your stuff to it. When you .close() the file, you copy back teh resulting file from the user's PC/Mac. My thought would be to use good old fashioned ftp for the file transfers. As for the SD formatting, this shouldn't be an issue. The PC deals with all that. All the simulator "sees" is a URL pointing to a folder that it thinks of as the SD card's root. Maybe I'm simplifying this too much? I'm thinking in c++. I don't know what concerns you may have from the point of view of your toolset.

-jim lee

[urish]

Yes, there are a few challenging constraints when running in a browser environment:

1. FTP is not supported by modern browsers. We can only use HTTPS (secure http), and only to sites that support Cross-Origin Resource Sharing
2. Since the Arduino expects a file system on the SD card, the simulator doesn't really know what file Arduino is reading/writing from. It only knows which block on the physical SD card the code running on Arduino it tries to access. So we can either:
   a. download all the files in advance, and write them into a file system format that will be used during the simulation; or
   b. let the user prepare upload a file system image and work with that
   There might be some in-between path, but it'll be pretty complex to implement: the SD library implementation on Arduino (or another platform) can hold meta data in cache, so we must at least create all the file system structures (directory and file information blocks) in advance. Also, as files may be deleted, renamed, moved, etc., we'll have to keep track of the file system changes as the program is doing the modifications. I believe standard file system implementations (FAT, LittleFS, etc.) do not support this use case (monitoring third-party changes to the file system and converting them back into file operations), so we might have to come up with our own file system library to support this.

[urish]

Quick update: I got an initial working prototype of the protocol implementation, tested against Arduino's SD library. Now working on the graphics for the element. Will post a link to a demo project in a day or two...

[urish]

Graphics ready: https://elements.wokwi.com/?path=/story/microsd-card--default

[urish]

We have a prototype implementation!

https://wokwi.com/arduino/projects/310542489623724609

Some limitations:

• It takes about 1 second to initialize the filesystem on the SD card. Hence the delay() in the beginning of setup. Also, it may not work in the first attempt - if you get a message that the SD card failed to initialize, simply run the simulation again.
• The content of the file system is hard coded, a single file called "wokwi.txt" with some text.
• The file system size is hard coded to 8MB.
• The file system is not preserved between runs. We may make this configurable in the future, but for now, it starts from scratch whenever you start the simulation.

Thoughts about the next steps:

• The simulator should wait until the FS has been fully initialized, and only then start the simulation. This will fix the need to add a delay() before accessing the SD card.
• I'm planning to add file uploads to the simulator. This will allow users to upload arbitrary binary files (such as bitmaps, csv data, etc.) for storing on the SD card, and as a nice side benefit will as fix Option to upload the ino, .h and .cpp files  #64. We'll only allow uploads for registered users, and every free user will get a ~10MB file storage. I believe this should suffice for most project/users, and this also ensures that projects won't be slowed down / crash the project by downloading large files.
• We'll need to user interface to manage the files that you uploaded to the project. A directory tree could be one option, but I'm still thinking about easier alternatives for the short term.
• We may also need to create some interface that allows you to view/save the files from the SD card at the end of the simulation. This can be useful for data logger projects.

[leftCoast]

If you are going to do uploads, can you make sure you can accept folders containing folders? I have a entire system folder hierarchy I use to run most of my stuff.

[urish]

That's a good point. What is the average size of the filesystem and number of files in it?

[urish]

Progress updates:

1. The simulator now starts only after the SD card is fully initialized
2. The file system now contains all your project files (and not just a dummy file)
3. I made it work with the SdFat library too. It seems superior to the SD library in terms of functionality
4. There's an initial docs page, where you can also find link to usage examples (both with SD and SdFat).

[leftCoast]

Here's the SD card from my current project. Goto : http://leftcoast.biz/justStuff/ then the SD card IS the folder named... "NO NAME".

Now how would I get something like that into my projects?

[urish]

What are the TRK files that you have there?

[leftCoast]

I developed a way to incorporate and select different "apps" for my Arduino handhelds. One of the apps is a, ported to Arduino, 1970s text based Star Trek game. It was THE THING back in the day. I played it (earlier version) on HP 2000 as a kid and it was almost like begin there! (Chuckle) Anyway, those are saved games from doing testing with the Star Trek app. I"ll post a picture for you on Discord. Adafruit did a writeup on it about a year ago.

…

-jim

On Sun, September 26, 2021 1:03 pm, Uri Shaked wrote: What are the TRK files that you have there? -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #183 (comment)

[urish]

I've implemented a temporary solution so you can start testing the implementation without having to wait for the file upload functionality:

1. Create a project with a microSD card (or open an existing one)
2. In the code editor, press "F1" and select "Select directory for SD card".
3. Select the directory with the files you want to have in the SD card. It can also include sub directories.
4. The browser will ask you to confirm access to the site. Note: the files are not uploaded to Wokwi, they never leave your computer.
5. Start the simulation. Your files should be present in the simulated SD card.

Note: the files will be gone once you refresh the page. So you have to repeat this process whenever you open a simulation link.

[leftCoast]

As soon as I get the bloody TFT working, I'll attach an SD card reader to it and see if I can pull up some pictures.

-jim lee

[leftCoast]

Here's an idea for sharing projects with SD cards. Set it up so the creator of the simulation can put a URL on the simulator UI that the user can click to download the files somewhere. Then link that to the SD card. This way you are not doing the down load they are. Will that work?

[urish]

That could work, but I'm afraid most of the simulation creator will be too lazy to go ahead and upload their files. The spirit of Wokwi is to have everything just "work out of the box", so I think it makes sense that we take care of storage and download for the users, even if it means more work for us...

[leftCoast]

Help! I completely forgot how to do the file thing with the SD card. How do I get it to use my folder of files?

[urish]

Here are the instructions, Chrome only for the time being:

1. Press F1
2. Select "Select directory for SD card"
3. Choose the folder, confirm with the browser and you are good to go!

[leftCoast]

How about SD cards? Right now the SD drive is the card. But could we access something or create something on our list of items that is an SD card? Preloaded with files that we could possibly share or at least use when someone tried our simulation? As it sits now, there's only a tiny handful of people that can use the SD drive.

[urish]

So here's my thought:

If we add binary files and folder support to projects, you could create a subfolder in the project which will hold the content of the SD card.

Some form of folder support would probably be needed for #186, as we'd need some way to specify a different set of source code files for each MCU.

[leftCoast]

Yes! I was thinking that the two would probably merge at some point.

[leftCoast]

Lets get this finished up and useable (And sharable). IE Users can click on it and it just works.

[urish]

A'ight cap'n, putting it back in the oven

[urish]

Some progress: if you open a project with an SD Card, you'll see a new "SD Card" tab which lets you manage the files in the SD card. It doesn't actually upload the files yet, just stores them locally in your browser, that's coming up next.

[leftCoast]

Good progress! While we're at it, can you default to having libraries.txt as one of the defaults? I really hate typing it in each time.

[urish]

Thanks!

I'll note this in #229, which is related to libraries

[sutaburosu]

Just a quick note that a Discord user expressed the desire to be able to see/download files from the SD-card after the sketch has modified them.

[leftCoast]

So.. Is this getting close? I have my icon editor just about ready to try out on it.

This includes an entire document editing framework complete with file alerts and everything.

[urish]

Thanks for checking in!

There's still some infrastructure work that I need to finish. The user interface side of things is pretty much ready (but not very polished), you can preview it here:

https://sd-card-upload.preview.wokwi.com/arduino/projects/310542489623724609

Right now all the uploads will fail (due to the missing infrastructure - the data is sent to the server, but still not stored). Once that part is ready, this should be usable already. We're probably a couple of days away from that point...

[leftCoast]

Excellent!

[urish]

Good news - the storage part is done, and the feature has been merged into production!

Here's the first working example - SD Card with a custom BMP file:

https://wokwi.com/arduino/projects/319810932695892564

[urish]

Keeping this issue open as I still have to update the documentation :)

[urish]

And now we have some documentation as well: https://docs.wokwi.com/parts/wokwi-microsd-card#filesystem