-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compiling applications and assets to a single binary #10
Comments
/cc @bmeck |
@alexwhitman we have work underway w/ some oversight into the api and commitment from @trevnorris to help see this through. https://gist.github.com/bmeck/0deeefd070224c10566f We will never emulate a virtual file system, and most bundling applications (Windows EXE format, Bundler, Jar files, etc.) take the same stance on emulating file systems. A presentation will be done at Empire Node with more information. An old binary supporting this behavior is at http://bmeck.github.io/ , but the spec has changed since then. |
I know @creationix had an idea around this a few years ago |
Related: electron/electron#251 |
If you have asynchronous I say almost because of course native libraries or modules that have to be compiled or use C/C++ bindings are still an issue. |
@Mickael-van-der-Beek As stated in the gist linked, shared libraries will need to be dumped to disk in order to be loaded. Only Solaris supports loading a shared library from memory (Mac OS used to but deprecated the API). That is how jar files handle shared libraries in industry. @sindresorhus I think the only thing that will be slightly different from the thoughts in that issue are that we implemented a resources API rather than making a new protocol. |
I can't say I'm expert on this subject but we had lots of cases recently. At least 4 out of 10 tasks we have from customers mostly to show design, proof of concept, and those require to run on 'closed systems' during the presentations / POC etc. Node is a great way to combine things in-together but we had lots of troubles on client systems with packages, file system related differences.Before node, we were using PHP and things were even worse. I believe, there are many others like me would enjoy a packaging/compiling feature with a virtual file system (something similar to jxcore's) @bmeck why no for virtual file system? eventually it worked for jxcore (after several months of struggling though but it works smooth for most cases now) The only problem is .node files. If the package has a .node file, we just write it to the disk on initial run from the virtual file system. Thankfully we don't use native modules regularly. Some of our team members don't have cross platform development experience (difference in paths etc.), virtual file system helps us now to ease deployments especially when the customer test or server environment is unknown. +1 |
If a virtual file systems is an option, would Docker + Boot2Docker be a solution ? |
@martinies because filesystems work differently on different environments and bundled assets are read only. A bunch of things will have edge cases and randomly break, we tried that at the beginning of the year but too many problems with people needing to run stat() and expecting file system settings like case insensitivity to make it worth the problem. Take a look at the formats that are also in the same mindset:
If you want to mount a virtual file system via fuse or using self extracting executables that would be fine, but static in memory assets are not files and treating them as such leads to misunderstandings and leaky abstractions: see ( http://www.py2exe.org/index.cgi/WorkingWithVariousPackagesAndModules ). However, when we tried doing self extracting executables there were serious cleanup problems ( same as noted in electron/electron#251 ) We want the cases where those break to be obvious (so developers can learn to diagnose the actual problem), even if it means tweaking code a little. @Mickael-van-der-Beek docker is a bit heavy for distribution and still has some problems like how would you run 2 node apps in the same container. tl;dr We don't want to lie; lying makes fewer but much harder to deal w/ errors. |
Linux does too, indirectly. You start a thread, the thread creates a pipe or socket pair, then the parent thread calls |
Windoze supports it too. Nothing says you have to use the operating system's |
@bnoordhuis had not thought of that, would require some work for the setup and teardown of the pair. If the experience is the exact same everywhere either would work. Current implementation extracts to disk since that seems the normal solution across the board. |
It's quite straight-forward actually. Anyway, why reinvent the wheel: https://github.com/fancycode/MemoryModule |
I've been implementing the exact same feature for luvit recently. It's pretty trivial to append a filesystem to the main binary using zip format since most exe formats ignore extra data at the end and zip format ignores data at the beginning (hence how self-extracting zips work) I've been fighting the dlopen issue. If it's so easy to write code to load a module from memory, then why do jars write out to disk as was mentioned above? My new work is https://github.com/luvit/luvi |
Also, responding to the original post. I would not recommend patching fs to load from the vfs. It's quite a different beast and should have a different API I think. I do recommend patching require to look there in certain cases. For mine, that is for bootstrapping a single-binary app and for relative requires from files already in the vfs. |
Good to see that this has kicked off some discussion. @creationix The reason I mentioned |
@creationix from what I can tell it is that JAR file conventions rely on acting the same in many environments, and implementing code loaders do not get 100% parity with OS dlopen(). We were using archives as namespaces which is how @alexwhitman in the spec and implementation posted we do allow loading from disk or inside an archive via a concept called resources. These are present in most application programming environments with archive files. The important thing to note is that the concept of resources is not tied to archives themselves. If we emulate |
Re: Virtualized File System Not going to happen. Has far too large a footprint on core code, too many unknowns to deal with and an overall general PITA. I think @indutny properly stated it on IRC:
|
Hi, I want to share my experience on implementing app packaging in atom-shell, which I hope would be helpful for Node. IntroductionThe app packaging in atom-shell works by modifying node's Examples of uses: require('./test.asar/main.js');
fs.readFileSync('./test.asar/REAME.md');
fs.readdirSync('./test.asar');
child_process.fork('./test.asar/task.js'); Archive formatThe packages use asar archives, which is a custom archive format targets for fast random access. I didn't use Zip because it is both over complicated for our case and lack of some core features we need. I had listed our requirements and comparisons of different archive types here, and the conclusion was developing our own archive type was a better choice. Pros and consI don't know the number of users using the app packaging of atom-shell, it works for most current node apps without modifying one line of source code. Though there are still some limitations, and I had listed them in atom-shell's wiki. ImplementationIf you have skimmed the structure of asar archive format, you should know that the implementation would be very simple. You can find out how Node's APIs are overloaded in asar.coffee, which only has 300 lines, and the native Single binaryThe asar format doesn't support being concatenated to binaries like Zip, but it would be quite easy to do by putting the size of Thoughts for nodeI don't think adding new APIs for packaging is a solution. If Image a user who wants to compile an existing |
@zcbenz Zip archive's support symbolic links through the external file attributes. The fast lookup is generally only a minor problem since we do cache the central directory in memory. |
I think technically it could be possible to combine two ELF files (or two Mach files) into one single file. Making it load all symbols and relocate all data. |
Another kinda wacky thing, for "single binary" deployments, would be to This is just for code; non-code resources (data files) would need a We built something like this many years ago for IBM's J9 Java VM. Compiled We had a zip-based archive story that included the "snapshot" with a Would be an interesting area to explore, but obviously not a critical path On Tue, Oct 21, 2014 at 9:06 AM, Fedor Indutny [email protected]
Patrick Mueller |
@pmuellr I would have concerns about vendor lock in if we do use snapshotting, loading the code would be faster but would not be portable. |
ya, there's lots to worry about if you want to do this; platform You'll notice I didn't really paint this as a happy outcome of the work we Also true that as processor speeds increase, the benefits of this kind of Still, snapshots do seem to be an interesting idea, and perhaps there's On Sun, Oct 26, 2014 at 6:12 PM, Bradley Meck [email protected]
Patrick Mueller |
node-webkit does, I think. There is a provision for it in V8's It's not very amenable to general purpose code, however. For example, snapshots with "foreign" objects don't work; that means you can't use buffers or handles and those are rather pervasive in node. |
I also took a stab at this recently that might be helpful for other folks interested. methodUsing the löve/node-webkit self-extracting zip "single binary" approach, which is basically these 10 steps.
|
One of the talked about benefits of golang is that it compiles to a single binary. Projects such as nexe and jxcore allow compiling to a single binary but nexe isn't actively maintained and I've never made jxcore work consistently.
require
andfs
functions would need to be changed to know about packaged files and assets and I'm sure there's some other things that would need consideration.What are your thoughts about being able to do this in a maintained and supported manner?
The text was updated successfully, but these errors were encountered: