Uncouple bundle information parsing from initialization

[Janfel]

The Problem

Currently, a bundles metadata (author, license, etc) can only be parsed by loading the bundle and therefore executing it. If a bundle requires a certain dependency, such as a library or another bundle, then even just parsing it's metadata might fail because of an unmet dependency. This makes it impossible for an external tool such as a package manager to parse and meet these dependencies before loading the bundle, without a separate package manager specific metadata file (e.g. aisu.moon).

The Proposal

To circumvent this issue, I propose a separate lua/moonscript file, which, if it exists, will be parsed before loading the bundle. The metadata will then be expected to be in this file.
When loading the bundle, the metadata table will be inserted into the exports table under the info key, thus keeping backwards compatibility and preventing naming conflicts between the tables. The resulting table will then be made accessible under _G.bundles as usual.

The Prospect

This would allow tool developers to specify tool-specific metadata to be placed in this file, maybe under a standardized location such as info.tools.[toolname], which can then be parsed without executing the bundle itself, making dependency resolution possible. Continuing with the aisu example, the build function would be placed under info.aisu.build or info.tools.aisu.build, making aisu.moon unnecessary.

Proposed names for the file:

• bundle.lua
• meta.lua
• info.lua
• project.lua (after pyproject.toml)
• package.lua (after package.json)

[refi64]

Tbh I think the main issue is just that you need to actually evaluate something to get the config, which makes it a bit harder to e.g. query for info or a search etc safely. In Aisu, I literally just parsed the MoonScript and extracted the data from there. It'd be kinda nice if we had something like jsonnet, just...some way to have a functional-ish subset of MoonScript syntax that can easily be evaluated.

…

On Mon, Aug 26, 2019, 5:34 PM Janfel ***@***.***> wrote: The Problem Currently, a bundles metadata (author, license, etc) can only be parsed by loading the bundle and therefore executing it. If a bundle requires a certain dependency, such as a library or another bundle, then even just parsing it's metadata might fail because of an unmet dependency. This makes it impossible for an external tool such as a package manager to parse and meet these dependencies before loading the bundle, without a separate package manager specific metadata file (e.g. aisu.moon). The Proposal To circumvent this issue, I propose a separate lua/moonscript file, which, if it exists, will be parsed before loading the bundle. The metadata will then be expected to be in this file. When loading the bundle, the metadata table will be inserted into the exports table under the info key, thus keeping backwards compatibility and preventing naming conflicts between the tables. The resulting table will then be made accessible under _G.bundles as usual. The Prospect This would allow tool developers to specify tool-specific metadata to be placed in this file, maybe under a standardized location such as info.tools.[toolname], which can then be parsed without executing the bundle itself, making dependency resolution possible. Continuing with the aisu example, the build function would be placed under info.aisu.build or info.tools.aisu.build, making aisu.moon unnecessary. Proposed names for the file: - bundle.lua - meta.lua - info.lua - project.lua (after pyproject.toml) - package.lua (after package.json) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#526?email_source=notifications&email_token=AAM4YSNEBVOSVRI3LDPISSTQGRK6VA5CNFSM4IPXKCN2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HHP53VA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAM4YSOKQIKI35VWTDY4L53QGRK6VANCNFSM4IPXKCNQ> .

[Janfel]

You can probably use something like the Rockspec Format used by LuaRocks. They are loading the files like regular Lua modules, just with an empty environment. This, however, would be rather declarative than functional and would not work with Moonscript out of the box (An issue with moonscripts loadstring not working properly). This sample project would probably be a better fit for your requirements.

[refi64]

Hmm, I'm still not entirely sure about the sandboxing approach. Whether bundles are directly grabbed off GitHub like now or go on some online bundle site, someone's going to have to run the config script without installing it for stuff like listing and searches to work.

Even with the example sandbox, a lot can go wrong: a malicious script could easily e.g. run forever or eat up a ton of RAM. A simple but more advanced sandbox from the Lua mailing list is a bit more thorough, but the memory part at minimum won't work on LuaJIT (which doesn't support __gc) and won't handle the case of high CPU usage.

If this is running on a server (e.g. for the package repository idea), it would be possible to use an external, more hardened sandbox, such as nsjail. However, if it's the client systems that are of concern, then it's really hard to rely on a sandbox being present on all platforms.

Python worked around this for a long time by auto generated EGG-INFO files as part of the package build process, but nowadays the preference is to define the package config in a basic config format via pyproject.toml / Pipfile.toml; any extra scripts go elsewhere.

Nimble works around this by maintaining a package list in JSON with separate metadata. IMO it's an awkward solution.

All this seems like a lot of effort to be able to use Lua for defining basic metadata...

Btw for the part about adding extensions, I personally just add new models that derive from the other ones, e.g.:

mode.register 
  name: 'bazel'
  extensions: {'bazel', 'bzl', 'BUILD'}
  patterns: { 'BUILD$', 'WORKSPACE$' }
  parent: 'python'
  create: -> {}

[Janfel]

One could always just use a weak sandbox on the client and expect the user to know what they are downloading (like Vim) or provide the majority of bundles in trustworthy repositories (like Emacs) that validate the scripts with a strong sandbox.
I am hesitant to use a language like YAML or TOML because this would add external dependencies and Lua's library ecosystem just isn't that great in that regard.

[refi64]

Oh yeah it's not perfect, I'm just saying that's it's a tad more complicated that it can appear. I just don't want to see a big bundle API change that we have to later redo for some other reason...

[Janfel]

I agree. Making up an API on the way is almost always a bad idea. Should this issue be closed then, or stay open as a reminder?

[shalabhc]

What's the downside of using json for something like this?

[refi64]

I mean json is kinda painful to write by hand. That being said...I just realized, can't we use GKeyFile for this? GLib is already a dependency so this requires nothing extra on our part, and it's imo flexible enough for this basic use case.

…

On Sun, Oct 20, 2019, 10:54 PM Shalabh Chaturvedi ***@***.***> wrote: What's the downside of using json for something like this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#526?email_source=notifications&email_token=AAM4YSPDHVZOK5T2MDTGPGTQPURXVA5CNFSM4IPXKCN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBY7HJI#issuecomment-544338853>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM4YSLO3ZR32VAEDULHVGDQPURXVANCNFSM4IPXKCNQ> .

[shalabhc]

I hadn't heard of GKeyFile before - looks like some kind of ini file format? I think in this case the ubiquity of json trumps other factors. I hope we're not writing gobs and gobs of this stuff - we write one small file for every package - so being a bit clunky isn't a big deal.

Maybe a good way forward to first nail down the exact metadata that will be stored in this file?

[shalabhc]

I was thinking another way to manage this is to have the bundle writer run a 'howl bundle' command that imports the bundle locally and introspects it to write out the metadata (then the format doesn't matter too much). The tool could even export things like the registered commands and config values, without having the bundle writer have to repeat those in the metadata config. WDYT?

[refi64]

I believe Arch does this with the AUR: you need to write a .SRCINFO file which contains metadata from the PKGBUILD script. The one con is that it can be *really* easy to accidentally push updates without updating .SRCINFO unless you set up some local Git hooks.

…

On Mon, Nov 4, 2019, 2:58 PM Shalabh Chaturvedi ***@***.***> wrote: I was thinking another way to manage this is to have the bundle writer run a 'howl bundle' command that imports the bundle locally and introspects it to write out the metadata (then the format doesn't matter too much). The tool could even export things like the registered commands and config values, without having the bundle writer have to repeat those in the metadata config. WDYT? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#526?email_source=notifications&email_token=AAM4YSKMPLLD7P4UW4JM33TQSCEJDA5CNFSM4IPXKCN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAWA7Q#issuecomment-549544062>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM4YSJKQH7V33I5DLXRUILQSCEJDANCNFSM4IPXKCNQ> .

[shalabhc]

Yeah that's a good point. What if the build command produced a zip file that you'd have to upload to a repo somewhere to make it available? Versioning of bundles is another issue we don't currently have a solution for. I'm imagining in the future we'd want some tools to build/upload/register the bundle anyway so might as well tie in the metadata generation to that.

Also for development on trunk, you could continue to git push the changes and even if the metadata wasn't regenerated I'd imagine it would still work in most cases.