Major syntax update

This release is characterized by a major syntax change. We don't take such changes lightly. The new syntax makes the language more regular (and simpler to parse), prepares it for some key features, and, we believe, makes it easier to understand. This is also a major step towards stabilizing the grammar of the language to the extent we can start guaranteeing backwards compatibility.

The old syntax is still supported. The cue fmt will rewrite your files from the old syntax to the new one.

Colons instead of spaces as separators for shorthands

CUE allows single-field structs to be written without curly braces if they are declared at another field. In the old syntax

a: {
    b: {
        c: 4
    }
}

could be written as

a b c: 4

In the new syntax, this is written as

a: b: c: 4

Aside from enabling some new features and simplifying the parser, this syntax also allows the shorthand to be used for both definitions and regular fields. For instance,

a: b:: c: int

previous had to be written as

a: {
    b:: { c: int }
}

Bulk optional fields

The new syntax now allows optional fields to be specified in bulk:

// Define a map from string key to numbers
numMap: [string]: number

// Apply constraint T to all fields with a name ending in "Test"
[=~"Test$"]: T

Using [string]: number is very similar to using a "bind label" previously. More on that later.

The old syntax for optional fields

foo?: bar

is now just syntactic sugar for

["foo"]: bar

Label filters are the equivalent of patternProperties in JSON schema.

Note that using colons as separators, which often have a connotation of defining something, interacts well with this language construct. For instance, a map of maps of ints is written as:

[string]: [string]: int

More aliasing

Shadowing of fields is a common issue in CUE (or similar languages, for that matter). CUE already supported various mechanisms to deal with this. For instance, in this example an alias is used to reach the outer a:

A=a
a: c: a: A.b
b: 4

Other tricks up CUE sleeves were quoted identifiers and allowing to make field non-referable using double quotes. Together these tricks proved both cumbersome and inadequate. The existing techniques also proved to be hard to handle for code generators and rewriters.

CUE now extends the aliasing model to field and label aliases. A field alias,

X=foo: bar

binds X to bar just as foo does. This allows the above to be written as

X=a: c: a: X.b
b: 4

This is not a great win, bit in many cases it will be. Moreover, it allows defining references to fields that can otherwise not be referenced, such as

X="foo-bar": baz
Y="\(x)Test": y

CUE now also allows label aliases, which are placed inside the square brackets:

[X=string]: { name: X }

These are only visible within the field's value. This replaces the "bind" labels, such as <Name>: { name: Name }. These are essentially the same, but explained differently and with a different syntax. The old syntax makes CUE hard to parse and is less flexible and will be phased out.

Packaging

Another change is the way the cue tool manages packaging. The cue.mod file and pkg directory are replaced with a cue.mod directory marking the root of a CUE module and with contents managed by the cue tool, analogously to how the git tool manages the .git directory.

See https://cuelang.org/docs/concepts/packages for more details.

Deprecations

Fields names starting with double underscore __

This was defined in the language specification, but previously not enforced. It now is. Field aliases make it easier to work around this restriction.

<X> syntax

Bind labels, or templates, are being replaced by the new optional field construct.
cue fmt will rewrite the old format to the new one.

Back-quoted identifiers

Back-quoted identifiers (i.e. foo-bar) were introduced to improve reachability of fields in case of shadowing. But they only solve half the problem. Field proved to be a more general solution, making back-quoted identifiers a redundant part of the language. As they are not trivial, they will be removed. As for now, the cue tool will rewrite programs using back-quoted identifiers using aliases.

Changelog

fe8ab34 Fix typo (#164)
2cd4162 README.md: bump supported go version to 1.12+
391e7ae Readme.md: add GolangCI badge
9707302 cmd/cue/cmd: added cue mod init command
fd127fe cmd/cue/cmd: clarify location for commands in error message
f979aa6 cmd/cue/cmd: correct module output
329cd5b cmd/cue/cmd: get go: convert Go block comments
29f70f2 cmd/cue/cmd: report error for certain deprecated syntax
72b36fb cmd/cue/cmd: update more examples to the new format
26a1a0e cmd/cue/cmd: update trim documentation to new syntax
b763fde cmd/cue: rewrite quoted identifiers
9752844 cue/ast: add NewList helper
4759dd2 cue/format: formatting of nested single-line fields
0b4f016 cue/load: allow cue.mod to be directory
b34bce7 cue/load: allow cue.mod to be directory
b13155b cue/load: make cue.mod the default desitination
60b298b cue/load: revert: report when "tool" pkg is used in non-tool file
312fca9 cue/parser: disallow declarations with __foo identifiers as per spec
7057dde cue/parser: fix comment attachement issue
4177df9 cue/parser: fix comment attachement issues
98b8d2f cue/parser: record position of deprecated feature
fa59c10 cue: generate builtins to update task documentation
8192b54 cue: generate new-style maps when exporting
ff34272 cue: implement key filters using new syntax
300af3e cue: introduce colon-separated labels
6d8c95d cue: more conversions to new style templates
467d7bc cue: move to square brackets
39df6c9 doc/ref/spec.md: bug fix: add solidus to escaped_char
c7791ac doc/ref/spec.md: introduce colons as separators
9ffcbbc doc/ref/spec.md: write out spec for aliases and optional field sets
2437f9d doc/tutorial/kubernetes: update formatting
23623fa doc/tutorial/kubernetes: update to new syntax

Streamlining embedding semantics

This release introduces two changes that may break existing configurations. One is really a bug fix relative to the spec.
These changes were not included in v0.0.12 to allow some of the bug fixes of the latter release
to be available before introducing these changes.

Top-level "emit" structs are embedded

Previously, a top-level embedded struct had different semantics from embedded structs within structs itself.
For instance, this configuration

{ 
  a: 3
}
b: 3

would previously yield { a: 3 }, but will now yield { a: 3, b: 3 }.
This semantics was a leftover from before CUE supported embedding.
But keeping this around was just too confusing.

Other values, like lists or strings retain their original semantics.
However, for consistency sake, these may only be accompanied by definitions.

So previously this was allowed

"Hello \(who)!"

who: "World"

but must now be

"Hello \(who)!"

who :: "World"

Failing Definitions are an error

Previously a definition was, unlike a normal field, not an error.
This was not according to the spec and this has now been rectified.
To allow a definition to fail, make it optional (using a `?).

Changelog

a3c7bef cue: change emit value semantics
3022ae9 cue: fail if non-optional definitions are bottom
0d0b9ad doc/ref/spec.md: add newline before comments

Relaxed closed structs

This release contains a host of bug fixes and improved conformance to the spec.
The most notable change, however, is the relaxing of the closed struct constructs.

Relaxed closed structs

With the previous release, each struct literal in a definition was treated strictly as closed.
This turned out to be both tedious and counter-intuitive.
This led to a spec change and corresponding different implementation where
a definition is only closed once it is referenced.
This means that merging literal structs defined within a definition works exactly
like unification of regular structs up to the point they are referenced.

Features

There are several builtins added, including in the list and regexp packages.
The latter added builtins to extract fields from strings into a struct using named groups.

Changelog

30a6c2b README.md: add Github build badge
75b9c7f all: remove large dependency
9e294ac cmd/cue/cmd: add MainTest
22e054b cmd/cue/cmd: always print newline at end of eval
0b043c9 cmd/cue/cmd: auto-detect test mode
e606e0d cmd/cue/cmd: don't use os.Exit and fix error output
91c1684 cmd/cue/cmd: fix import paths for go get
48b8f9a cmd/cue/cmd: fix race
374d395 cmd/cue/cmd: fix trim help
78043ab cmd/cue/cmd: fix: disable elimination of slices
bd64ebc cmd/cue/cmd: import: use astutil
a31e1ac cmd/cue/cmd: only allow dot files when specified explicitly
18dab4b cmd/cue/cmd: report instance-level errors
0489caa cmd/cue/cmd: use testscript for top level command testing
22a42d4 cmd/cue: fix tests under different platforms
c7dc930 cue/ast/astutil: allow recursive application
1a33709 cue/ast: reorganization of node types
929e71f cue/format: indent single line instead of 2 after embedding
de86f0f cue/load: be more permissive with file loading
ca66319 cue/load: report when "tool" pkg is used in non-tool file
a38f5bc cue/parser: add support for failing on legacy grammar
6eefcd0 cue/scanner: fix JSON compliance
e1656a1 cue: add Definitions option and deprecate Hidden
7d1cb13 cue: add Kind.String method
ffff7fa cue: add test for Issue #94
8776f21 cue: allow valid reference cycles in export
75d2c37 cue: bind reference as lookup in parent node
8927b63 cue: bug fix: retain references with interleaved embedding
f2654de cue: close struct only after referencing
081251e cue: document that zero value of Runtime is OK to use.
8d81be4 cue: enable recursive opening when embedding
d282553 cue: exclude definitions from value lookup
fbb38bc cue: fix comparing against bottom
d499c46 cue: improve dependency analysis for References
405a191 cue: improved error message
6deb0df cue: make users aware of the --list flag when encountering a list
4524da6 cue: move up processing of comprehension
c85bebe cue: pass on compile-time errors
fa7e3ce cue: relax rules for closed structs
827ebfb cue: remove IsValid
7068dea cue: workaround for empty "or" lists
21f6c44 doc/ref/spec.md: bring scoping in line with reality
b2703c6 doc/ref/spec: reflect reality w.r.t. octal numbers
ded0e1d doc/ref: fix typo in spec
9d53106 doc/tutorial/basic: number chapter directories
27b4ea5 doc/tutorial/basic: use definitions with "emit" examples
a9f25f8 doc/tutorial/basics/intro: disable test
00eb9d2 doc/tutorial/basics: add pieces on structs and definitions
934fec9 doc/tutorial/basics: correct name of section
f3cd848 doc/tutorial/basics: generate testscript cases
d4c1734 doc/tutorial/basics: order files
b0d1a07 doc/tutorial/basics: remove old docs
df08087 doc/tutorial/basics: reorganize and simplify intro
aeaf006 doc/tutorial/basics: update some sections
1ce0c51 doc/tutorial/kubernetes: tutorial fixes
d437bd5 doc/tutorial/kubernetes: update definitions
b0bba12 doc/tutorial/kubernetes: update definitions
78783f2 encoding/json: add JSON exporter
dd75ef6 encoding/json: belated fixes of review
9b70cf3 encoding/openapi: add paths:{}
7e4dc22 encoding/openapi: support Definitions
7213fa9 encoding/protobuf: don't encode default values for enums
e7abb20 encoding/protobuf: don't use disallowed "__" prefix
074cf5d pkg/list: adding flatten function as a builtin
a83af29 pkg/list: adding flattenN function as a builtin
5ac1fbf pkg/list: adding range function as a builtin
e783813 pkg/list: fix MinItems
20b0a0a pkg/list: remove Flatten and simpilfy FlattenN
68a054b pkg/regexp: add more regexp builtins
8dd325b pkg/tool/exec: fix null bug for stderr and stdout
67169f4 pkg: exclude some builtins from generation
c40e450 pkg: update documentation and run gofmt
8529d77 spec: fix some backtic-quotes for markdown rendering
a76cbed start using GitHub Actions as CI

v0.0.11

A few bug fixes

There were a few small bug fixes that, even though old, were serious enough to warrant a new release.

This also finalizes the implementation of allowing full expressions for embeddings.

Changelog

e53305e cue/parser: allow full expressions for embeddings
0ac550c cue: eliminate generation of block quotes
7c9b88c cue: fix bug in type inference
fdd9b71 cue: fixes number groundness inference

v0.0.10

Language touch ups

New-style Comprehensions

This release is characterized by the redesign of field comprehensions, now struct comprehensions. The old format is still supported, but cue fmt will rewrite it to the new format.

The new format looks like:

for k, v in a {
   "\(k)": v
}

instead of

"\(k)": v for k, v in a

This may seem like a gratuitous change, but it has many advantages:

• generated structs are syntactically and semantically like embeddings
• large comprehensions are clearer
• a single iteration can add multiple fields
• makes scoping clearer

Removal of block quotes

Because a comprehension may start with if, we can use if false { ... } to disable a large block. This, and various other considerations have led to the decision to remove block quotes from the language.

Block quotes are still supported, but they are no longer generated and cue fmt will translate them.

Quoted identifiers

CUE now allows identifiers to be quoted with back ticks:

`foo-bar`: 2

baz: `foo-bar`

This allows fields with odd names to still be referenced using identifiers. More importantly, the allowed space of identifiers should ideally be limited. For instance, it is a good idea to require them to be in Unicode normal form NFC. Allowing arbitrary strings as identifiers seemed like a bad idea. CUE will soon disallow this. For now, cue fmt rewrites strings used labels is they are referenced by some nodes.

Quotes also have another benefit. A disadvantage of the new style comprehensions is that it is harder and less clear to allow the keywords of comprehensions as field names. Consider for instance:

for x in src: 2

CUE will now only allow one such keyword as a field name, but requires quoting otherwise. A future cue fmt may rewrite this as:

`for` x `in` src: 2

Backwards incompatible changes

The language changes introduced do not break anything, as the old formats are still supported. However, at some point they won't be. So be sure to run cue fmt at some point.

• The ast.LabelName has been changed to have proper semantics in light of the upcoming changes. The signature has been change to reflect the changes properly and to assure code is properly investigated in light of these changes.
• Comprehensions now iterate only over non-optional regular fields, as would perhaps be expected.
• Top-level emit values are printed in eval output
• Top-level struct embeddings are merged in the API

Changelog

b50aecd cmd/cue: make fmt rewrite deprecated cue
21fa535 cue/ast/astutil: export Resolve functionality
0dcf17b cue/ast/astutil: implement Apply
e4523e2 cue/ast/astutil: pick name based on import path
530467a cue/ast/astutil: support adding imports
24d14c5 cue/ast/astutil: use unlikely name for import
40c76b7 cue/ast: add API for setting a position
f0bcfb9 cue/ast: add API to modify comments
92408e3 cue/ast: add back LabelName
749a667 cue/ast: convenience constructors for common types
fe5f886 cue/ast: first step to new comprehension format.
57d6751 cue/format: expressions.input
9af9a90 cue/format: output new comprehension format
982b194 cue/format: tweaks to support embeddings
71eafc4 cue/parser: parse new comprehension syntax
3c38bfc cue: add test for package name override
4245fb4 cue: for structs, iterate only on non-optional regular fields
835ed58 cue: implement quoted identifiers
da00c8b cue: implementation of new comprehensions
7fc421b cue: remove block comments and slices from the language
1fa0552 cue: tighten API of comprehensions
b36d81a cue: various cleanup
1f5a903 doc/ref: allow embeding to be Expression
d44b092 doc/tutorial/basics: some minor fixes
48f82f1 doc/tutorial/basics: update w.r.t. spec.
15aeace pkg/list: add Haskell-style Take and Drop
1ffb95f pkg/list: adding slice function

v0.0.9

Bug fixes and builtins

The central theme of this release is bug fixes, most notably in the cue command line utility.
It also adds and improves on some of the builtins in the pkg directory.

Backwards incompatible changes

The crypto hash builtins now return a value of type bytes, instead of a list of bytes.

Changelog

cb84024 cmd/cue/cmd: fix --dryrun flag description of import command
fe695b1 cmd/cue: give access to cue Command in run functions
9818761 cmd/cue: update get to use definitions
507d6cc cmd/cue: use concrete mode for vet on data files
f3df637 cmd/cue: use stderr and non-zero exit for errors
330999b cue/pkg/crypto: make return types bytes, instead of list
1c4d597 cue/scanner: don't allow repeated _ in numbers
18637db cue: closedness only applies to regular fields
5e8c391 cue: exempt aliases and embeddings from recursive closing
34988fa cue: fix 0/0
4017875 doc/ref: spec changes to comprehensions and identifiers
da39f0f doc/tutorial/basic: Unified TOC, next/prev, titles
980e018 doc/tutorials/kubernetes: update comments
8b4cab6 doc: fixed small errors
d2563c1 go.mod: update xerrors to a985d3407aa7
41e30c6 pkg/list: adding aggregation for decimal lists
9d5bc94 pkg/strings: Remove fmt.Println
b36c351 pkg/strings: fix MinRunes
061bde1 spec: corrections, clarifications, comments

v0.0.8

Changelog

f40ecd9 cue: flx bug for len on non-concrete values
7f9d026 doc: updating slack invitation link
8f3347f go.mod: update proto package

v0.0.7

Changelog

6817c99 cue/parser: bug fix: allow embeddings on one line
d99e32e cue: define BottomKind as 0
b7e44a9 cue: fix builtin type signature
8089092 cue: fix premature fixing of some incomplete values

v0.0.6

Changelog

3056594 : add Homebrew install instructions to README.md
aa35a9f cmd/cue: allow help to work without cue files
28a9191 cmd/cue: allow vet to check CUE against JSON and YAML
90bca2f cmd/cue: don't break help if not commands are defined
29a11be cmd/cue: fix linter warnings
3908dac cue/ast: make package clause a declaration
ed08085 cue/ast: split off Ellipsis from ListLit
9471633 cue/format: fix linter warnings
6a9badd cue/load: add Dependencies, rename Root and improve test
96ef9aa cue/load: implement package name qualifiers
d208ff1 cue: add another example
80b7904 cue: allow bottom for optional fields
cc5aaa6 cue: changes ahead of unchecked unification
f1b87bf cue: disallow string and bytes indexing and slicing
e47dda7 cue: don't print definitions in concrete mode
7c1b39e cue: fix Windows breakage
6351fc2 cue: fix Windows build breakage
828a643 cue: fix crash related to package name
2e7c882 cue: fix error formatting issues
bc432d5 cue: fix issues related to '_'
6659da2 cue: fix linter warnings and divide by zero error
400f621 cue: implement closed structs for subsumption
5134dee cue: implement embedding and close structs
cd81fa9 cue: implement parsing and formatting of definitions
bfbcd50 cue: support multierror return from Unify
94b3193 cue: unknown reference are incomplete
6c35af6 doc/ref: change meaning of numbers and lists, remove %
62658a8 doc/ref: define defintions, closed structs and embedding
7414fae doc/ref: propose restrictions on package paths
fbab65d doc/ref: remove dot import
4108f80 doc/ref: simplify string model
ed718e2 doc/tutorials/basic: update hidden.md
2e9193f doc/tutorials/basic: update numbers.md
483afc5 doc/tutorials/basic: update regexp.md
6dc6d63 doc: adding user-defined package import example
2b56efa encoding/openapi: add support for deprecated field from protobuf
ceb003d encoding/openapi: allow empty string from ReferenceFunc
35abfa7 encoding/openapi: define strict order on fields
9fad62f encoding/openapi: implement structural schema
97a56d7 encoding/openapi: remove deprecation notice of old api
743365e encoding/protobuf: add more proto to JSON conversions
c1d1f1a encoding/protobuf: ensure predeclared identifiers are not masked
5f47167 encoding/protobuf: make map field as not required by default
84ffa3c encoding/protobuf: make oneOf fields required
13c9718 encoding/protobuf: provide default import path and short name
b1d4439 encoding/protobuf: remove extra space in comment
41cce52 pkg/encoding: add {yaml.json}.Validate
ff51e07 pkg/strings: add ByteAt, ByteSlice, and Runes

v0.0.5

Changelog

74c19ff .goreleaser.yml: set commit_author info for homebrew config
2463e2d cue/ast: rename EmitDecl
b7d24b8 cue/errors: always add error
3bc7dd3 cue/load: Unix-related fixes related to 5a3cce0
55b58eb cue/load: fix comment
9811321 cue/load: fix module loading bug
dd9bff0 cue/parser: added fuzzing code and fix one issue
f03161d cue/parser: attach attributes to correct field
9467491 cue/scanner: added fuzzing code
9c96be3 cue: add Raw option
56f120c cue: add Struct type
5ba7174 cue: add Value.Equals method
ed81a47 cue: allow getting Runtime from Instance
b6baed4 cue: cleanup before implementing close structs
c3b171d cue: fix attribute parsing bugs
0be096f cue: fix bug when using multiple custom validators
1372f3e cue: fix import shortname mapping
29236f2 cue: fix possible drop of doc comment in unification
60ae081 cue: fix top shortcut
9501233 cue: handle builtins used as values
e2e2e63 cue: ignore fields of Go types unmappable to CUE
9934915 cue: implement marshaling and restoring instances
5a3cce0 cue: implement recursive marshalling
332ea47 cue: improve coverage of Go value conversion
f06f781 cue: improve error messages
42cfa68 cue: improve naming of Runtime methods
4fe2fb5 cue: include package name in debug output for builtins
48d8732 cue: maintain type cache per index (runtime)
d8f4450 cue: observe JSON mappings for google.protobuf.Struct
99e7b6d cue: preserve path info in unification errors
0265ac2 cue: remove unify function
222d96c cue: tighten type
eac8f9a cue: treat cycle errors as normal incomplete errors
c8e131d cue: treat disjunctions as normal non-concrete values
259bdfe cue: unexport NewInstance from api
dc30df1 cue: update builtins
d6ec5cf cue: use more realistic arguments for TestDecode
5608a83 cuego: use Runtime to simplify code
c3d0b48 doc/tutorial/kubernetes: fix typo
66ec959 doc/tutorial/kubernetes: fix typos
15bb998 doc/tutorial/kubernetes: support for gnu sed
26d8901 doc/tutorial/kubernetes: update to minimize diff
980b30c doc: adding cue community contact information
240a995 encoding/gocode: add Go code generator
73e3f6b encoding/openapi: bug fixes in expandReference
6cea136 encoding/openapi: introduce support validator methods
6bf3697 encoding/openapi: support time types
afd1cc5 encoding/protobuf: fix linter issues
64cb20a encoding/protobuf: implement time type conversions
4e5b69a encoding/yaml: expose internal yaml functionality
1233594 internal/third_party/yaml: decode bool and null to BasicLit
4697b78 internal/third_party/yaml: improve yaml positining
1ceb046 pkg/list: add list builtins for OpenAPI features
b05c768 pkg/math: added MultipleOf
1588fe0 pkg/net: added package
730165c pkg/strconv: remove CanBackquote
ce195d2 pkg/strings: add MinRunes and MaxRunes
3086ea6 pkg/struct: add Min/MaxFields builtins
8090639 pkg/time: define time types and builtins
ca05b5f pkg/time: use UTC time zone for Unix and Parse
55039f3 pkg: adding base64 encoding for strings