Separate cue evaluator and cue cmd tool in separate projects

[shykes]

This is a thought experiment, I'm curious what others think.

TLDR: I'm starting to think cue cmd should be moved out of core cue, into a separate project - I will tentatively call it cuedo. This separation would be beneficial to both cue the language, and cuedo the tool.

The benefits would be:

• It's easier to answer the question "what is cue?". At the moment, the answer is "it's a language (Cue), an interpreter (cue and cuelang.org/go/cue) and a special automation tool that can be configured with cue files, and is bundled with the interpreter (but only the command-line interpreter, not the go library). That answer is too complicated, and confuses people. It would be much easier to just say "Cue is a language; it has an official interpreter available as command-line (cue) and a go library (cuelang.org/go/cue). It allows creating all sorts of amazing tools. A good example is cuedo. Check it out!

• It simplifies the cue file hierarchy. No more _tool.go files! cuedo, like any tool configured by cue, can be provided with a cue configuration. This could be a file, a directory or a package. It loads its configuration using cuelang.org/go/cue, then merges it against its own schema. Again, just like any other tool would. There's no need for a special class of files.

• It shows the ecosystem how to build good Cue tools. Instead of a baked-in special case which no third-party can reproduce, cuedo is just a regular tool. Whatever patterns it adopts, can (and should) be emulated by the broader ecosystem of Cue toolmakers.

• It gives cuedo a chance to succeed as a standalone tool. Just like cue is easier to explain separate from the tooling layer, cuedo will be easier to explain (and to adopt) separate from the language interpreter. There is a lot of demand for automation / scripting tools, and cuedo is a great addition to the toolbox, in great part thanks to its use of the Cue language. But as long as it remains bundled within cue, it cannot truly compete as an automation tool, because it does not clearly fit in the "automation tool" category.

• It makes cuedo configurations easier to share and reuse. Right now, there are restrictions to sharing my commands and tasks: the parts in _tool.cue files cannot be imported, for example. One can move as much as possible into a regular cue file, and import it from the _tool.cue file, but 1) it's optional so not everyone will do it, and 2) last time I tried this, I hit cue bugs which caused evaluation to fail.

• It shrinks the cue codebase, which makes it easier to maintain and contribute to. This is especially important at a time where the core team is still very small, and spread thin. By moving the tooling out, we can reduce the scope of the core project, and improve it faster.

Again, this is intended as the start of a conversation. I am eager to think what everyone else thinks!

[mvdan]

I like the idea. Here are two bonus points that I thought of while reading yours:

• cuedo can be released and versioned separately from the main cue interpreter and libraries. For example, a bugfix release for the tooling layer shouldn't require bumping the version for all of Cue.

• It forces the cue Go libraries to be powerful enough to build complex tools on top, such as cuedo. Right now, I imagine that it's possible (and even likely, given enough time) that the two will share internal code or undocumented behavior.

The biggest downside would perhaps be the added overhead in accomplishing this split, and maintaining these two separate projects in the long run. Perhaps cuedo could be more approachable for contributors to help.

[verdverm]

Some thoughts:

• When I speak about Cue, currently, I talk about configuration pains, the ecosystem, that we are at an inflection point, and where I think it is going. I tell them that I believe the Cue philosophy is the correct choice, and that the Pulumi (and now TF) direction and any of the other experiments are going to be popular, but wrong, here's why... ( separately, Dhall is an exception, but I prefer Cue's syntax)
• In general, I don't bring up Cue's tooling layer until later, if at all, when telling / teaching people about Cue. It needs more work before I can make use of it (Ctrl-C doesn't work iirc, but generally have to write too much for things that are one-liners)

Why do we need to remove cue cmd at all? If cuedo was to exists (It likely does in multiple forms across the community), and is separate from Cue, why does Cue need to change anything to support building tools? (maturing an exception)

Is it more about Cue's suggested default automation / scripting solution, and having that removed from the tool, so as not to confuse people? I guess I don't see a pressing enough reason to warrant the removal given that many of us are all already building various cuedos today. Are you wanting to limit this topic and discussion to scripting?

I think many of us are actively thinking about and working on solutions in this space. Personally, I have not felt that Cue has impeded my efforts while building hof. Certainly I look forward to a more mature and powerful Go API to help power the next generation of backend automation / developer tools. I guess I don't see the scripting layer as a priority either. Competing against bash / python with devops engineers is not easy, is that where you see cue cmd / cuedo existing? (we are working on a replacement as well... see hof/script if interested) Cue has been great for removing formerly Go code, and Cue has now taken a central role in our UX/DX. And while Cue is amazing for config and last statement, many have come looking for imperative Cue after discovering the wonders, and I don't think that's what cue cmd is trying to be. <- (@mpvl clarification?).

Getting back to cue cmd...

• It seems orthogonal to building tools on top of Cue code. It can stay or go, which could be decided by the recommended "default" (cuedo v cue cmd v none... here are options)
• I expect many of the sharing issues to go away once cue mod is implemented.
• I do like the idea of having one less file "class" to worry about, but we are going to have _test and more have been talked about in other discussions here.

Generally, I see the ecosystem experimenting with many ideas. I think we may slide the Pulumi / Terraform way for a bit before coming back to Cue and similarly philosophied ideas. As such, I'm not ready to be overly prescriptive about recommending best practices. I'd like to see more of what ideas and hear more about what users / devs think when presented many options. For Cue tool builders, they will likely all have their own DSLs and we all get a shared language for that, and I would say the "standard pattern" is to read Cue files in like the cue tool and do whatever you like from there. Yaml is not really more prescriptive than this, todays scripting solutions even less so.

A closing thought... Go has been consolidating the third party tools into go proper as it has matured. I like this pattern and have adopted it in my own creative works. I'd prefer a single tool to having to install and manage instead of N tools and M versions. I at least get to remove >= 1 of those under this philosophy.

---

I am eager to think what everyone else thinks!

Follow me on social, happy to help with that! :]

[mpvl]

CUE was set up as a monorepo in true Google tradition. This ensures that changes in any subsystem that cause errors elsewhere are exposed quickly. This may seem to slow down development, but the errors will just have to be dealt with down the road, often making it much more complex, not having the benefit of immediate breakage.

For instance, writing the new evaluator is indeed slowed down by the presence of the other components, but only because these components expose bugs.

I think other things can have velocity. One thing is splitting the core into more packages to better delineate responsibilities. The new evaluator splits the API/evaluator from one monolith package into about 10 so far. The compiler and evaluator are the most complex . These are now 1000 and 1800 lines respectively. They both used to be intertwined in a 28k-line package. This makes the dependencies of what may affect the core evaluator and what not a lot easier to see.

Another is to allow builtin development and testing within their own directories, without the need to generate them into core. This is not yet done, but paving the way for it.
Better documentation on how to contribute may help as well.

The cue command is seeing a separate development: more and more of the functionality is moving to the tools and encoding directory or the load package. It will be a fairly small shim in the end. This makes the monorepo quite convenient: refactoring across repos is more of a pain.

That said, as the core stabilizes, this benefit will diminish. Not for now, though, I suspect.

Having different versioning could already be very useful. For instance, it would be nice to version the language separately from the tool, but this can be done within the same repo using separate go.mod files for components. That also brings some pain, though.

[myitcv]

I'm not clear this is the right thing to do, not least because I'm not totally clear of the problems it seeks to address.

It's easier to answer the question "what is cue?"

By way of analogy: "Go" is the name of the project, language and cmd/go. In much the same way I think people will happily qualify what they are referring to: "the CUE project", "the CUE language", "the cue command". I think much of the haziness (which I have shared) has come from the evolution of cmd/cue as we collectively try to work out useful core functionality, UI/UX etc. Indeed the model of cmd/go is a pretty good example to follow IMHO - it could have many many more features, but that doesn't serve the community well. Far better is to make it solid and robust and allow others to build atop it, or as your are alluding, build other tools for specific jobs. Indeed, I actually think CUE is in better shape for those people looking to build tools like cmd/cue, simply because there is less of a cmd/go/internal/.... That said, until we reach v1 I think we should continue to optimise for a setup that allows code to most freely move between the different "parts" because what constitutes a "part" is, in many cases, still not totally well-defined (with some code even needing to move from cmd/cue-world to the loader).

But, post v1, I completely agree on the general point of not having any special internal backdoors that cmd/cue can use - otherwise you run into situations where other tools are immediately at a disadvantage, e.g. gopls and cmd/go via golang/go#39958. This can (and should) be enforced with CI checks to ensure this situation doesn't regress. As you said, such a state will also provides a live example of how to write great CUE tools for free.

Until and as part of establishing v1, I think it absolutely makes sense to raise discussions/issues like #448 that describe specific things that tool authors want/need that may or may not be possible in cmd/cue to establish a good set of tools packages.

It simplifies the cue file hierarchy

I don't understand this point. Can you expand on how this obviates the need for _tool.cue files? Because in essence this file naming methodology is just a brute force build tag of sorts.

But as long as it remains bundled within cue, it cannot truly compete as an automation tool, because it does not clearly fit in the "automation tool" category.

Can you explain a bit more on this? I don't understand how cmd/cue being bundled as part of the CUE distribution (just as cmd/go is bundled as part of the Go distribution) affects this.

Right now, there are restrictions to sharing my commands and tasks: the parts in _tool.cue files cannot be imported, for example

Definitely sounds like it would be worth opening up an issue/discussion if there are specific issues/suggestions here. My usage of _tool.cue files is markedly less that yours, so would be good to see in detail what you're running up against. I think we can assume CUE bugs will be fixed 😄 so I wouldn't want that to be part of a reason to split out cmd/cue.

It shrinks the cue codebase, which makes it easier to maintain and contribute to

What in particular do you think makes cmd/cue or indeed the CUE project hard to contribute to?

Also, what problems specifically would be solved by having cmd/cue versioned separately from the CUE distribution? I ask because I can see it creating some problems of documentation/communication at least.

And by "separate project" I assume you mean separate repo? What problems are solved by this?

[shykes]

@myitcv I think there's a misunderstanding on what I'm proposing. I am not proposing to unbundle cmd/cue from the CUE distribution. I think that bundling is perfectly fine, and agree that it works well for Go.

I'm talking about unbundling the "scripting layer" documented here: https://cuelang.org/docs/usecases/scripting/ . This feature is currently bundled with the CUE distribution via the cue cmd sub-command. There is no equivalent to this feature in the go tooling.

[myitcv]

Ah ok. I think a number of my questions remain though, if you substitute cuedo with cmd/cue?

In particular, I'm not clear how this allows us to reduce everything down to .cue files, i.e. doing away with _tool.cue files (and there being no delineation between non-tool and tool files).

[mpvl]

@shykes I understand your point a bit better now. The scripting/tooling layer is in my view core, as it is an essential part in completing CUE.

That said, I'm all in favor of hoisting out all tooling support into an API that the tool uses, allowing both tool and non-tool use of the same logic. More generally, things like dependency analysis and so on should be a separate package.

Also, I'm working on a draft design right now to replace cue cmd with cue run with simplified semantics, making tool files essentially not special and making it much more regular CUE.

[ibuildthecloud]

The points @shykes makes resonate with me as someone who is trying leverage cue in some new projects. I have three questions that bother me.

1. What exactly is cue?
2. Am I using cue correctly?
3. How do users of my tool learn cue?

All three of these questions I can't easily answer. Breaking the project up into language, go library, and tools might help to clarify these questions.

The use case I'm interested in is using cue as basically (using a dated reference) a better xml, xsd, and xslt. I realize this scope probably diminishes the power and vision of cue, but those are the problems I need solved at the moment. In this scenario most of the cue CLI become secondary tooling, but not first class for the users of my tool. Just as a most users of xml need not understand the full power of xsd or use tools like xmlint or xsltproc, more advanced users can learn and benefits from these things. For what I'm trying to accomplish putting cue and cue cmd in the foreground and entry point to cue is a bit confusing.

I realize what I may have just described could not align with the larger goals of cue, and I'm approaching this wrong, which brings me back to my 2nd question.

[Rots]

Do you happen to have any public projects/examples of converting between XML and cue? Would be very much appreciated.

[mpvl]

Note that I'm now slowly moving functionality from cue/cmd to the core APIs. For starters, the injection mechanism now fully lives in cue/load. Also loading of non-CUE files is slated to move to this package. Another part is to write a new, more thorough dependency analyzer as a separate API. The goal is that not much will remain that is specific to cue cmd.

[cueckoo]

This discussion has been migrated to cue-lang/cue#449.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.