Virtual file system

[milesrout]

Trillek needs a virtual file system (VFS). This is an abstraction layer over the real disk. By adopting a layered bind mount approach it would be very flexible allowing us to load assets from zip/tgz archives transparently, sandbox plugins' disk access to a directory and its children, load resources over the network transparently, load new versions of resources from different places hiding the previous version at runtime, etc.

With a file extension registration system, different systems will be able to register file extension handlers to load resources into the VFS in their own way. For example, client/resources/md5mesh.cpp can register a handler for the ".md5mesh" extension, and then whenever a file with extension ".md5mesh" is loaded, the handler will read the file, parse it, and load an MD5Mesh object into the VFS at that path. Or for example, a handler for the ".zip" extension will be able to load the zip archive and present it transparently to the rest of the system as a directory.

When directories are mounted, they would create virtual directories, the contents of which are lazily mounted to the filesystem as they are loaded. So if I mount (real filesystem path) ./assets.zip to (virtual filesystem path) /, then load /models/bob/bob.md5mesh, /models/bob/bob.md5mesh will be loaded from the archive and passed as a file stream (backed by a custom ZIP archive streambuf) to the ".md5mesh" file extension handler, which basically shouldn't need to know whether it's dealing with a real physical file or not. It just uses an iostream.

// archive.cpp
vfs.RegisterExtension(".zip", [] (VFS::Path mountpoint, std::iostream fileStream, VFS::Handle h) {
    h.MountDirectory(...);
});
// md5mesh.cpp
vfs.RegisterExtension(".md5mesh", [] (VFS::Path mountpoint, std::iostream fileStream, VFS::Handle h) {
   auto mesh = MD5Mesh::Create(...);
   h.MountFile(mesh);
});
// main.cpp
vfs.Mount("/", FilePath{"./assets.zip"});
vfs.Mount("/logs", FilePath::GetLogDirectory());
std::shared_ptr<MD5Mesh> mesh = vfs.Load("/models/bob/bob.md5mesh");

@adam4813 I would like to use std::filesystem, a C++17 feature, to replace the current FilePath which is pretty limited. Are you happy with that? Are we okay with requiring C++17?

[adam4813]

I think i already bumped it to 17 iirc in the cmake file. Also said file loaders are already registered for some extensions, so half the work is done.

…

On Thu, Nov 21, 2019, 7:57 PM Miles Rout ***@***.***> wrote: Trillek needs a virtual file system (VFS). This is an abstraction layer over the real disk. By adopting a layered bind mount approach it would be very flexible allowing us to load assets from zip/tgz archives transparently, sandbox plugins' disk access to a directory and its children, load resources over the network transparently, load new versions of resources from different places hiding the previous version at runtime, etc. With a file extension registration system, different systems will be able to register file extension handlers to load resources into the VFS in their own way. For example, client/resources/md5mesh.cpp can register a handler for the ".md5mesh" extension, and then whenever a file with extension ".md5mesh" is loaded, the handler will read the file, parse it, and load an MD5Mesh object into the VFS at that path. Or for example, a handler for the ".zip" extension will be able to load the zip archive and present it transparently to the rest of the system as a directory. When directories are mounted, they would create virtual directories, the contents of which are lazily mounted to the filesystem as they are loaded. So if I mount (real filesystem path) ./assets.zip to (virtual filesystem path) /, then load /models/bob/bob.md5mesh, /models/bob/bob.md5mesh will be loaded from the archive and passed as a file stream (backed by a custom ZIP archive streambuf) to the ".md5mesh" file extension handler, which basically shouldn't need to know whether it's dealing with a real physical file or not. It just uses an iostream. // archive.cpp vfs.RegisterExtension(".zip", [] (VFS::Path mountpoint, FilePath realpath, VFS::Handle h) { h.MountDirectory(...); }); // md5mesh.cpp vfs.RegisterExtension(".md5mesh", [] (VFS::Path mountpoint, /* type to be determined */ fileStream, VFS::Handle h) { auto mesh = MD5Mesh::Create(...); h.MountFile(mesh); }); // main.cpp vfs.Mount("/", FilePath{"./assets.zip"}); vfs.Mount("/logs", FilePath::GetLogDirectory()); std::shared_ptr<MD5Mesh> mesh = vfs.Load("/models/bob/bob.md5mesh"); @adam4813 <https://github.com/adam4813> I would like to use std::filesystem, a C++17 feature, to replace the current FilePath which is pretty limited. Are you happy with that? Are we okay with requiring C++17? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#148?email_source=notifications&email_token=AAW7KTI4Q2L5544TNZ6W4NLQU437TA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3IPQSA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW7KTJ3XDCIGPDIZAZ3XLLQU437TANCNFSM4JQKSWJA> .

[milesrout]

Yeah, if you have a look at my filesystem branch, I've started work on this.

[adam4813]

I will when i get a chance.

…

On Thu, Nov 21, 2019, 8:21 PM Miles Rout ***@***.***> wrote: Yeah, if you have a look at my filesystem branch, I've started work on this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#148?email_source=notifications&email_token=AAW7KTPJFIHEBSGWSR5IMGLQU462PA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE4JX3I#issuecomment-557358061>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW7KTPLFLOPXGUJPZO6GO3QU462PANCNFSM4JQKSWJA> .

[milesrout]

I don't think I pushed my latest changes to that branch, but essentially it makes it work as described above. Currently it's not quite architected right: it gives extension handlers the raw file path, assuming every resource they can access is a raw file on disk, which according to the above design wouldn't be true. Still, it basically works as of my latest unpushed changes. Just wanted some feedback before I put a whole lot more work into it.

[adam4813]

Though it will be nice to have, i am not sure how important it is to the 0.11 release, so maybe dont invest a lot of time right now?

…

On Thu, Nov 21, 2019, 8:34 PM Miles Rout ***@***.***> wrote: I don't think I pushed my latest changes to that branch, but essentially it makes it work like that. Currently it's not quite architected right: it gives extension handlers the raw file path, assuming every resource they can access is a raw file on disk, which according to the above design wouldn't be true. Still, it basically works as of my latest unpushed changes. Just wanted some feedback before I put a whole lot more work into it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#148?email_source=notifications&email_token=AAW7KTOIDCVZZ23YMEX4UR3QU5ALDA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE4KOOI#issuecomment-557360953>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW7KTN46PQLENPSMF5WRYTQU5ALDANCNFSM4JQKSWJA> .