App packaging

[YurySolovyov]

It would be awesome to get packaging support with embedding resourses to executable file
Discussed a lot here: nwjs/nw.js#151 don't get confused with issue title btw

[zcbenz]

I'm not very positive on adding it, since anyway you would have to ship DLLs and resources files along with the exe file, and it won't prevent users unzip and change your source code.

[YurySolovyov]

At least it would be harder, for many people it's enough to just give up.

So, current packagin will stay the same? I mean is there a plans for improving it or adding features?

P.s. Personally, I'm ok with not embedding .dlls, main requerst is to embed resources and source code.

[zcbenz]

How about unzipping the .exe file to a temporary directory and then load the real app from it in the main.js, which is exactly what node-webkit does? The main.js is still visible to users though.

[YurySolovyov]

This might have problems with clean-up if main process crashes or closes in a way that makes clean-up hard or impossible. There is possible to workaround this, but it still not optimal IMO.

What are main problems with embedding resources in executable and accessing them with a pseudo-protocol like app://index.html ? This need some kind of internal routing though(for relative paths in css for example), but seems more reliable and probably faster (since you don't have to unpack resources).

[zcbenz]

Embedding resources into executable is only possible on Windows, to make it cross-platform we have to compress resources into zip and concatenate it to the executable, and then unzip them from the executable latter.

However packaging the resources into an external file and then use custom protocol to access the resources would possible, it also seems like a proper solution for app packaging to me.

[YurySolovyov]

When you saying " then unzip them from the executable latter", what will be destination of unzipping? If it is some temp folder, this is not far better then node-webkit approach.

Is it possible to concatenate resources without compressing so you don't have to unpack them anywhere?

[zcbenz]

If it is some temp folder, this is not far better then node-webkit approach.

The destination of unzipping is some temporary directory, and this is exactly the node-webkit approach, it is just in node-webkit the package system is implemented with C++ and the progress of unzipping is hidden from users.

Is it possible to concatenate resources without compressing so you don't have to unpack them anywhere?

In theory, yes, but with lots of efforts. And concatenating all resources to executable would make things slower because operating system need to read the whole executable into memory.

Support packaging resources into an external file is what I plan to implement, if what you want is to hide source code from users and read resources without unpacking, it should be enough.

[YurySolovyov]

Ok, thanks, better than nothing.

[YurySolovyov]

If I understood you correctly, there will be executable, and resource container, so when app starts, it will look for resources there, right? no packing/unpacking?

[zcbenz]

Almost yes, except that developers still need to pack the resources into one file.

[kevyworks]

I think the .pak file is fine imho, just don't forget to emit app.on("unpacking") so that we can have the ability to show something while it is doing so.

or is the hierarchy of the app events like this?
-> will-finish-launching
-> ready

[YurySolovyov]

I was trying to say that packing in terms off splitting runtime and resources is fine (e.g. external file with resources). Problem with packing is in terms of compressing (zip files, archives of any type). This is problematic because it requires some temporary directory to unpack to, and clean-up after app closes.

If that would be the case in future, events should not be changed I think, because there is no additional unpacking phase.

[YurySolovyov]

Bump
Any ETAs on this? or some status on when it will be considered for implementation?

[zcbenz]

I have done some research on the packaging format, here is the result:

Requirements of packaging format:

1. Support random access
2. Support filesystem features, like permission bits and symbol/hard links.
3. No need of decompression
4. Windows support
5. No third party dependencies
6. Lightweight

Candidates:

1. xar

https://code.google.com/p/xar/

Very ideal for our use cases but don't support Windows and require libxml

2. cdb and its variants

http://www.corpit.ru/mjt/tinycdb.html
https://github.com/gstrauss/mcdb/

Don't support Windows and file systems.

3. duplicity

http://duplicity.nongnu.org/index.html

No Windows support and relies on librsync, also targets for incremental backup.

4. gzip/bzip2 compressed formats

No random access support, users have to read the whole file to build index first.

5. zip archive

Locating a file is not fast, symbol links are not supported.

6. tarball

No random access support.

7. DAR

http://dar.linux.free.fr

Too heavy for our use case.

[zcbenz]

After the research I think developing our own package format is a better choice, the file format is similar to xar but uses JSON as header:

| uint32: header_size | string: json_header | bytes: file1 | ... | bytes: file64 |

The format of json header is something like:

{
  "name": "/",
  "files": [
    {
      "name": "tmp/",
      "files": []
    },
    {
      "name": "usr/",
      "files": [
        {
          "name": "bin/",
          "files": [
            {
              "name": "ls",
              "size": "512",
              "offset": "0",
              "executable": true
            },
            {
              "name": "cd",
              "size": "724",
              "offset": "512",
              "executable": true
            },
            {
              "name": "chrome",
              "symbolLink": "/opt/chrome/chrome-browser"
            }
          ]
        },
        {
          "name": "share/",
          "files": []
        }
      ]
    }
  ]
}

The size and offset are represented as string because JSON doesn't support int64 type.

[YurySolovyov]

Sounds great! Build tool should be pretty simple.
What about flat package header structure? I mean you can store full path to file so you don't have to nest json to emulate folders, and when app will request file, it should already know full path:

[{
    file:'/index.html'
    offset:64,
    size:512
    /* any other attrs can be here */
}, {
    file:'/css/style.css',
    offset: 576,
    size:1200
}]

Not sure how symlinks can apply, but I hope You got the idea.

[zcbenz]

Flat structure is fine for URL requests, but I also plan to make Node's fs module work in atom-shell's packages, fs has lots of filesystem APIs that using a tree structure can make implementation easier.

The utility to create packages has been created at https://github.com/atom/asar.

[YurySolovyov]

So, I assume node modules will work as if they were just in folder with atom executable, right?

[YurySolovyov]

One more question: are you going to use packaging in atom editor? I'm not sure but I think it can speed-up app launch since it will actually access only one file.

[thedaniel]

cc @kevinsawicki as he has some ideas about improving Atom's startup time through preprocessing the JS.

[YurySolovyov]

One possible problem is with installing/removing packages/themes. This will require partial updates of package and header, or if packaging is fast we can just recreate it each time.

[zcbenz]

One more question: are you going to use packaging in atom editor? I'm not sure but I think it can speed-up app launch since it will actually access only one file.

I'm not sure too but it definitely worths a try.

One possible problem is with installing/removing packages/themes. This will require partial updates of package and header, or if packaging is fast we can just recreate it each time.

Packages are installed under ~/.atom/ so it is not a problem.

[bmeck]

@zcbenz would love to have a chat about this as we are working on some stuff in node core that might cause conflicts

[zcbenz]

@bmeck My timezone is quite opposite to yours so I think it is hard for us to find a suitable time to chat 😸 , but I'll share some of my ideas in that thread.

Or you can wait until December when I'll move to San Francisco.

[bmeck]

@zcbenz its not landing in 0.12.x so we have time to discuss in December.

[frankhale]

@zcbenz, Congratulations on your upcoming move and I'd like to say THANK YOU for all of the extremely hard and technical work that you are doing to make Atom-Shell absolutely fantastically AWESOME! THANK YOU SO MUCH and safe travels to you!!!!!!!!

[reggi]

@zcbenz I don't understand the benefit of asar, or really understand the point. Is it supposed to prevent users from viewing the source code of an atom shell app? It doesn't have compression so it doesn't make the app smaller. Would love to understand the point. 😸

[bwin]

@reggi: the average person can't look into these files (don't confuse this with security of any kind) and it can speed up installation (because you have a lot less files). Also there are plans for an atom-shell-runtime and this would (edit: probably) be the app-format of choice.

[reggi]

@bwin Is there any way to hide my source code? I have a package.json with a github token in it that points to a private libs repo for my app, something like this. I'd have to remove stuff this before the app gets released publicly and thats a bummer.

"package-name": "git+https://<github_token>:[email protected]/<user>/<repo>.git"

[bwin]

First thing I can think of right now, would be a grunt (or gulp or whatever) task, that replaces this line before release. (You would have to make a copy of your working dir for releases to not overwrite your "original" package.json; I would anyway suggest a release-dir).

About encryption: since the encrypted file has to be decrypted on the user's machine, the key/password would also have to be on the users machine. So having this info even in an encrypted state would still be dangerous. So maybe stripping this info beforehand is the better solution.
(Also your "secret" file could end up in the users temp dir ...)
(edit: Also IMO this auth info doesn't belong on a users machine. Encrypted or not. You're never npm installing on a user's machine anyway.)

Edit: I'm not sure (never had private lib dependencies), but wouldn't ssh access to github with a key also solve your problem? (But maybe you just don't want that.)

[anaisbetts]

and it can speed up installation (because you have a lot less files)

On WIndows this is most likely a huge benefit, file operations on Windows are much slower than on Linux (because Windows supports way more features); writing 10000 1KB files is going to be orders of magnitude slower than writing one 10MB file, even though it's the same amount of data.

[thedaniel]

Doing a ton of file reads to load a large JS app is also a performance killer. One of our startup time speedup strategies in Atom involves concatting files for this reason.

[bmeck]

I'd love to get on a call about this as it is a different format than my archive loader and we can discuss things. Particularly I have concerns with implementing a fake fs module that is not fully featured/compatible.

I can be awake during Asia's daylight hours if that helps.

[ghost]

As my comments on asar were referenced above - I'll qualify them by saying a few things

1 - obfuscation/code-hiding is mostly pointless - if someone wants your code, they'll find a way to get it. If this is a problem, you're using the wrong tools (CSS/HTML/JS) entirely.

2 - runtime performance isn't a huge deal either - you shouldn't be storing a large amount of data inside a distribution archive, there are many, many better places to store data and code is generally small enough that read/write performance isn't an issue (and again, the tools aren't really "performance" focussed)??

3 - whatever archive format you use should support dynamic read/write from the app itself so that the app can download/update it's own archive (auto-update) as slickly as possible(*)

4 - it would help if it built on WIndows without a lot of dodgy dealing with age-old libraries ;0

(*) for Node Webkit I built a thing which downloads GitHub releases and re-packs them as package.nw files (without needing to write anything to disk other than the final result - it's all streamed/piped in-memory) The app then restarts itself and it's auto-updated!! - I'm not sure I could do that with an asar at this point?

[YurySolovyov]

1. It is already mentioned that it is enough to prevent just average person messing with code. it is just handy, and you should not rely on that for security. 100% agree with you here.
2. When somebody says asar gives performance increase, I think they refer to faster reads for large number of files, not faster read of one big resource. Though, I'm not I got it right. /cc @zcbenz
3. As docs states, asar archives are indeed read-only, so yeah, for auto-update you have to re-create them yourself.
4. not sure what you mean.

I mostly agree with you, asar could be better, but hey, we need to start somewhere.
Possibly related: #673

[ghost]

3 relies on there being an asar library which enables that sort of thing - right now, asar seems to be limited to archiving files already on-disk - it needs (and wouldn't be hard to give it I guess) pipe/streaming support?

4 relates to an issue I raised in the asar repo - it's not an easy thing to install on WIndows (I've yet to actually get it working properly without hackery) - it's also non-native but then so are most zip/tar/gzip tools I guess...

[YurySolovyov]

3 Looking at asar archive structure, I think yes, you need files to be on disk at archive creation time, so yes, if you want to convert github release .zip into .asar, you need to do it manually, which, I agree, is not great.

[YurySolovyov]

I am also thinking if monkey-patching node's fs module for transparent .asar support was a good idea, isn't it better to have separate asar-fs module to deal with asar-related tasks? It can be smaller and easier to maintain, and add features incrementally. ATM fs module can be treated as asar-fs (with known limitations), and node's fs is original-fs. Maybe it is just me, but it seems weird.

[smallscript]

Except that ZIP does support all the above, is extremely efficient for random access, does have symlinks, custom attrs, and zillions of tools that support that. I use it all the time for the exact use case you're describing including having it utilize a binary search to locate files, directories etc with NO requirement to pre-sort on load. ZIP can be located any position within a binary (front, middle, back). That, and sqlite together are components of EFS supporting versioning, replication, and single binary deployment.

Use with "https://github.com/richgel999/miniz" for efficient single file "C" (or has a header only) implementation.