Make this project simpler

[bsansouci]

This PR aims at making this project simpler for newcomers to understand.

Before this we used make to run ocamlbuild which built a topkg.ml file which we invoked and than ran ocamlfind to finally build the user's source code.
I think most of that isn't needed, so I stripped it down to having the make command calling rebuild directly.

I'm not sure why this project was setup the way it was, so please tell me if what I'm doing doesn't make sense. I'd love to skip learning topkg though ;)

[jordwalke]

I think that toppkg generates an opam install file. Is that correct? Maybe ocamlfind is sufficient.

[bordoley]

I think topkg makes building an opam package much easier. cc: @tekknolagi who set this up.

[chenglou]

Cc @yunxing too

[bsansouci]

@jordwalke There's an existent opam file here...

@bordoley I would love to know why. From looking at the difference with and without, topkg does nothing to help here. You can add opam deps in the _tags file, and there's an opam file.

EDIT: Indeed it seems like topkg generates a opam file and a ReasonNativeProject.install. How do those fit in the opam ecosystem?

[bordoley]

Crazy idea: Use package.json as the source of truth, and add a small js script that generates all the build artifacts for both bucklescript and native.

[bsansouci]

So topkg generates both META and .install file. It seems like the .install file is used by opam to figure out how to install that package, and META is used by ocamlfind to know where the build artifacts are (?).

How important is it to allow releasing on opam for the example project? (Guess answer: pretty important because otherwise people can't share their library lol).

My goal here is to avoid having to understand: ocamlfind (and META file), ocambuild, topkg (and topkg.ml and _tags), opam (and opam file, and .install file). Is there any way we can hide all that complexity? The two things users need to be able to do for now are to depend on opam things and publish to opam. Can we squish all the middle layers of abstractions to avoid having people learn that stuff?

@bordoley Do you think we could generate everything from package.json? That could be cool.

[jordwalke]

We can either build our own abstraction that squashes all the concepts or use an existing one. The only other tools I know if that do so are jBuilder and solvuu.

[tekknolagi]

topkg was used to simplify package generation and compile flow. It does generate META and .install files and allows the developer to specify a bunch of stuff in OCaml instead of arbitrary text files. It was the easiest solution that I found to packaging in my tenure as an intern -- though I only really spent a couple of days searching.

[bsansouci]

I've updated this PR. This is now using opam_of_packagejson a small utility that converts a package.json into the different files that you'll need. It's pushed on opam and built to native. It has very good defaults so most newcomers won't need to fiddle with any options and it uses the standard npm options which facilitates integration into Reason projects who are already using npm (this does not depend on npm though). Immutable-re is currently using it to simplify its release flow.

I'd like to emphasis that I think having so many technologies together here (ocamlbuild, topkg, opam, reason) is just too much for newcomers. With this PR we're eliminating a invasive dependency: topkg and gaining a trivial and noninvasive one opam_of_packagejson. If we use topkg here, we're forcing everyone to learn it. Opam_of_packagejson has good defaults, only acts during the release process, which is already made very tough because of opam (there are between 5 and 8 steps to release and if you miss any of them you gotta bump the version number) and works with the bsb-native flow (since it's not a build system merely a dumb tool). I'm hoping to tackle that problem next but I think this PR is a step forward in terms of reducing complexity.

[tekknolagi]

Are you committed to maintaining https://github.com/bsansouci/opam_of_packagejson as @dbuenzli is for topkg?

[tekknolagi]

Also it still requires people learn how to write and understand package.json.

[dbuenzli]

Don't know what is being discussed here. But for technical accuracy I would just like to point out that topkg does not generate a META file, you have to write it yourself. In the future I hope we can get rid of META files altogether.

I'd like to emphasis that I think having so many technologies together here (ocamlbuild, topkg, opam, reason) is just too much for newcomers.

Yes, make sure you understand the t of topkg.

[bsansouci]

@tekknolagi Yes. opam_of_packagejson is a very manageably small script. I don't intend it to become as powerful as @dbuenzli's topkg either. I think there is a place for topkg but not in the starter environment.

I can't imagine package.json being harder to understand than topkg's API. As @dbuenzli said you still need to understand the META file syntax, the topkg API and how opam works.
On top of the needing to learn a couple different kind of syntaxes for the different files, one is dependent on opam. Moving the package description into the package.json is a baby step towards being free to use other package manager than opam. I want to hide opam as an implementation detail rather than expose all of its power through topkg.

@dbuenzli Sorry I'm not sure I understand in what way topkg is transitory.

[bordoley]

@bsansouci, How will this work now that reason project uses jbuilder?

[jordwalke]

@bordoley It might still be cool to use opam_of_packagejson, and extend opam_of_packagejson to also generate jBuilder files. So we'd use a feature like package.json -> opam+jBuilder. I'm not sure if that would actually work out in practice, but it's worth a shot.

[bsansouci]

@bordoley I'm not sure I understand your question. This is tangential from how reason's built.

@jordwalke I think it's outside of the scope of opam_of_packagejson to relate to build system. The point is that it allows using a package.json to publish on opam. It doesn't concern itself with how you build.

[jordwalke]

That sounds like a neat little utility. Lol because Andrey has pretty much built the reverse direction (opam to package.json!) I do agree that the same utility that converts from package.json to opam file could be a great place to perform the opam publish workflow. But there's more to that than just creating an opam file - it needs a lot of work to rival npm's publish workflow IMHO.

[dbuenzli]

But there's more to that than just creating an opam file - it needs a lot of work to rival npm's publish workflow IMHO.

I don't know about this workflow. If you used it, what do you think is missing on the topkg/opam front ?

[jordwalke]

@dbuenzli I've only used opam-publish. Opam-publish was confusing because it creates this separate small metadata package apart from the actual source release.

But overall I think most of the issues are with opam's publish workflow, are related to opam's metadata system and publish workflow itself, and topkg wouldn't be any different right?
Among other issues your packages can get into this weird state where metadata gets out of sync with repos and the meta data that publishers thought they had published. And then there's the time lag time to accepting new releases. Npm's publish workflow is extremely lightweight, and is entirely in the hands of the publisher. There's no censor or gatekeeper directly in the workflow.

There's no way a human can intervene with the publishing of an npm package, or prevent it from being merged. Granted, there's reasons why human review benefits a compiled language ecosystem - but at this point most of those benefits are becoming automated as well for opam.

For npm it's just one command npm publish. That's it. Your package has the source, and it has an opam file, and all of this becomes the release. Any constraint solving is done on the package.json file within the release. You don't need to upload the source to a different release url than the metadata and keep them in sync, although you can upload a release to any URL you like - and even depend on URLs to .tar.gz package releases directly as dependencies. It's much more distributed. The newer npm even allows version range constraints on arbitrary git urls (where the release number is encoded in the URL).

I'd like to see opam become that lightweight, but from discussions so far it seems like this isn't what opam wants to be. That's okay by me as well, as there's benefits to having a more closed "app store" type model to package management.

Still, I would like to see the equivalent of npm but for opam packages. Perhaps it can even be npm! Having an open, instant release package management workflow and having a stable curation of packages aren't mutually exclusive. One is just a subset of the other.

[bsansouci]

@jordwalke I agree that there is a lot to the npm publish workflow.

Right now opam_of_packagejson just generates all the files you need to publish on opam based on your package.json. That means that you still have to go through all of the commands to publish
Maybe it should also do the publishing part to opam, bumping the version number and all. Those are the steps it would run.

That means that all you would need is a prepublish script and you run npm publish and your package gets published on opam and npm. What do you think?

@dbuenzli I don't know topkg enough to be able to tell you if anything's missing. What I can say is that writing a pkg.ml file and the associated mandatory files is more complicated than what it takes to publish on npm. There are two reasons for that:

• all you need in your package.json is a couple fields, no need for code
• the current Reason audience (JS devs mostly) are used to writing a package.json file (it's not like a makefile that nobody knows how to write, there isn't much to it)

So the reason I wrote opam_of_packagejson is because I want people to be able to just run one publish command and have their project published both on npm and opam. I could potentially generate a topkg ml file and run that but right now it was simpler to hardcode everything to get immutable-re working with it. It's very possible I'll come back begging to topkg though :p

So maybe I could go extend opam_of_packagejson to also run the appropriate publishing commands that you'd have to run manually, so that we truly have a one npm publish command. @jordwalke @bordoley What do you think?

[dbuenzli]

But overall I think most of the issues are with opam's publish workflow, are related to opam's metadata system and publish workflow itself. Your packages can get into this weird state where metadata gets out of sync with repos and the meta data that publishers thought they had published.

Note that there are two problems here which should really not be confused.

1. The fact that opam publish prepare is uterly confusing as to which metadata it will use. See e.g. the surrounding discussion here.
2. The fact that opam never uses metadata from release tarballs.

I won't comment on 2. since personally have the impression that it's a better model in the long term for repository quality.

Regarding 1. it is the reason I didn't use opam publish prepare in topkg, I generate the package myself based on what is in the repository and submit it via opam publish submit. But I do think that the obscurity of 1. leads to misunderstanding of 2.

@bsansouci Sure topkg is a hack made to work with existing tools which are oblivious of it, this is why you have to write a pkg.ml file. But I was not really asking about the details I was more asking about the high-level workflow you can read about in topkg help release.

[jordwalke]

@bsansouci

So maybe I could go extend opam_of_packagejson to also run the appropriate publishing commands that you'd have to run manually, so that we truly have a one npm publish command. @jordwalke @bordoley What do you think?

Yeah, perhaps - it might be an undertaking though. Basically, I wish we can create a cli that is nearly indistinguishable from npm's publish workflow.