v1.0.0-rc.1

[expede]

I may get pilloried for this version. WIP, obviously

Preview 📚

Okay, this version switches to IPLD. This makes it much easier to not need a IPLD version off to the side, and many teams that have adopted UCAN already use IPLD somewhere in their stack. There is a contingent of people that feel strongly in favour of JWT for a variety of reasons. I also defended the JWT strategy for a long time, beacuse I had many first-hand converstaions of "I need to sell this to management, please tell me it's a JWT and not some inscrutable binary format".

A few things have changed:

• While Embedded IPLD hasn't merged yet, we know exactly how it works
  • Mostly quibbling about format right now, but I hope to close it soon
  • This makes IPLD it WAY friendlier for those not deeply familiar with IPLD
• Batch signatures needed some upgrading
  • Noteworthy that W3C folks are working on this, but no major JWT implementation has this yet
• We understand now Invocations can take over some of the heavy lifting that Delegations were doing previously

I believe that this proposal makes writing both UCAN Delegations and libraries much easier and more comprehensible. It also lowers our maintenance burden between multiple formats.

Changelog

Metadata

• Bump version to 1.0.0-rc.1

Structure

• Remove revocation section
• Point at ucan-wg/revocation
• Point at ucan-wg/invocation
• Remove ucan/* (moving to ucan-wg/ucan-uri)
• Point at ucan-wg/ucan-uri
• Move prf field to ucan-wg/invocation + add to top-level ucan-wg/spec
• Use DNF + compat form

Time

• Restrict time bounds to 53-bits (because JS)
• Explicit time bounds checking logic
• Make exp nullable

Prose

• Remove signer role (confusing to some people)
• Remove confusing analogy about lanyards in section 1.2
• Change term "discharge" to something clearer
• Clarify outer/inner terminology (or better: change it)
• Clarify in section 6 [now] 5, that you only invalidate single capabilities; not everything
• Remove old session ID recommendation now that we have content addressing
• Move FAQ to the high level spec
• Rename "top" to "wildcard" because it's a more familiar term for normies
• Remove bottom case entirely in new syntax

[Gozala]

Thank you @expede for capturing all of the discussions that have occured in side channels in this spec, I am super excited to see it in current state.

I made couple of suggestions and notes inline but here is the list of things that I'd like to align on before we call this done.

• Support for powerlines
• I think we were an agreement that like was a better operator name than match, but spec uses later. I have some reservations, but mostly want to make sure that there is a good rational for switching names.
• Restricting negation to single cause is unnecessarily restricting and leaves undefined behavior when multiple clause may be present. I suggest we do what datomic does here and allow multiple clauses. Or alternatively we define what behaviour should be if more than one clause is present.
• I was under assumption that we had an agreement on selector behavior captured by the table here selector syntax #5 specifically in regards to errors and ? operator but some of the spec seems to be in contradiction.
• Looks like slice selection has being omitted, unless this is deliberate I'd like to include it.
• Iteration over unary relations are defined as false conditions, which I find unnecessarily restrictive and making a case a case that they should be treated as collections of single item instead (details inline).
• Clarification of selectors over bytes would be really appreciated.

[Gozala]

This seems out of sync with the rest of the spec perhaps something like this ?

Suggested change

	Capabilities are the semantically-relevant claims of a delegation. They MUST be presented as a map under the `cap` field as a map. This map is REQUIRED but MAY be empty. This MUST take the following form:
	Capability is the semantically-relevant claim of a delegation. It is represented by subset of the fields of the delegation map. Capability MUST take the following form:

[Gozala]

Imbalanced parens and I think you meant following

Suggested change

	| `and` | `[Statement]` | `["and", [[">", ".a", 1], [">", ".b", 2]]` |
	| `or` | `[Statement]` | `["or", [[">", ".a", 1], [">", ".b", 2]]` |
	| `and` | `[Statement]` | `["and", [">", ".a", 1], [">", ".b", 2]]` |
	| `or` | `[Statement]` | `["or", [">", ".a", 1], [">", ".b", 2]]` |

[expede]

@Gozala LOL yeah, I think I blended the the two options here 😵‍💫 The other way is to always force an array, but I'm happy either way. Will update per your comment.

// Alternative
["and", [[">", ".a", 1], [">", ".b", 2]]] // I like this less

[expede]

(Will merge, but want to check the prose to make sure it's also clear about this case first)

[Gozala]

Should not be able to enclose multiple statements ? Unless you have objections I propose that it should, for what it's worth in datomic not clause expresses that all of the nested predicates MUST be false.

[expede]

be able to enclose multiple statements

That may be confusing, no? Does it add an implicit and? Datomic likely does an implicit or due to how DNF tuple selection works (the policy language is not a query language)

[expede]

@Gozala

Restricting negation to single cause is unnecessarily restricting and leaves undefined behavior when multiple clause may be present.

Also I disagree about undefined behaviour when multiple clauses are present: that case is disallowed since multiple clauses after a not matches no forms for the rationale given earlier.

Despite the parens and use of operators in prefix, I really think we should stay away from Scheme idioms since they're very foreign to normie devs. Arguably we should switch the operators to words for this reason, but it's not a hill that I'm going to die on.

[Gozala]

This contradicts some of the prior agreements that made here #5 specifically I made case that:

1. reference to missing key
2. out of bound index reference
3. Iteration over non collection

All must be errors which could be trapped using try ? suffix to opt-in into return null behavior. My rational had been that optionality could be handled explicitly.

[expede]

Yeah I think is a case of me working from the Rust version where we had diverged while iterating quickly. You're right and the rationale makes sense to me; will fix 👍

[expede]

Ah, it's also how jq behaves:

> echo '{"a": 1}' | jq '.b.c.d.e.f'
null

> echo '[1]' | jq '.b'
jq: error (at <stdin>:1): Cannot index array with string "b"

When writing this, I had jq running on the side to check edge cases. Anywhere that we diverge from jq will need to be highlighted in big neon lights (or we need to do whatever they do since it's easier to manually test against / principle of least surprise)

[Gozala]

Attempting to quantify over a non-collection MUST return false and MUST NOT throw an exception.

What is the rational for this decision ? I have being advocating that quantifying over non collection should behave as quantifying over single element collection because it is helpful in schema evolution when things 1:1 relations can change to 1:n relations.

In my experience with schema-on-read systems like datomic are a lot more flexible because they follow this approach.

I should clarify that my position implies that iteration over null should be equivalent to iteration over [] as opposed to [null].

[Gozala]

I do not believe iteration over map had being covered and if it is supported it needs to be clarified what . gets bound to for the maps.

[expede]

The rationale is that it breaks the policy assertion. Quantification implies that there is some collection, so IMO the other option is just throwing an error. Much like using ? elsewhere, you can always recover by wrapping in an or to say "or if it's not a collection, do this instead".

I should clarify that my position implies that iteration over null should be equivalent to iteration over [] as opposed to [null].

And I guess that iteration over 1 should be [1] and over false would be [false]? If so, that seems fiddly to me, like how truthy values lead to weird edge cases. But that could be the strongly typed person in me talking... but this is how jq treats this situation:

$ echo '[1,2,3]' | jq "all < 5"
true

$ echo '1' | jq "all < 5"
jq: error (at <stdin>:1): Cannot iterate over number (1)

$ echo '[null]' | jq "all == 5"
false

$ echo 'null' | jq "all == 5"
jq: error (at <stdin>:1): Cannot iterate over null (null)

and somewhat infuriatingly, null is the smallest number

$ echo '[null]' | jq "all < -999999"
true

$ echo '[null]' | jq "all > -999999"
false

Also, that they have any and all that behaves like this makes me want to change every and some back to all and any for consistency

In my experience with schema-on-read systems like datomic are a lot more flexible because they follow this approach.

Datalogs are query languages. This is a policy language. I'm increasingly convinced that they're duals of each other: querying is finding "some" matching data in a larger set, and a policy language enforces that all of the data you have conforms to expectation.

[smoyer64]

Selector analysis/implementation

I'm doing preliminary work on a Go implementation of UCAN v1.0.0-rc1
and have a few recommentations based on attempting to implement policy
validation. This comment in particular will focus on the selector
element as this seems to have the most discontinuity within the policy
portion of the v1_ipld delegation` specification.

Methodology

The following methodology was employed to develop the policy
validation (not all these steps are fully complete pending this
discussion):

1. Copy and slightly modify the policy ABNF from the delegation
   specification. I've submitted a PR
   with just enough changes to make the ABNF valid.

2. Copy and normalize the examples in the Selectors
   section of the delegation specification for use as interop
   testcases.

3. Generate Go Examples (tests with output) for each testcase. For
   each testcase, a jq library is used to evaluate the example args
   against the selector.

4. Execute each testcase with the same input using jq 1.6 and verify
   the the output is expected. For each testcase, a jq library is used
   to evaluate the example args against the selector.

Result

1. There are many inputs which are valid selectors per the ABNF but
   that are invalid or non-working filters in jq.

2. There are examples in the specification that don't behave per the
   specification.

Recommendations

1. The specification includes the following line:

   Any selection MAY begin and/or end with a single dot. Multiple dots
   (e.g. .., ...) MUST NOT be used anywhere in a selector.

   Any jq filter that doesn't begin with a . is effectively a constant
   and applying that filter to ANY input value will result in the filter
   constant being sent to the output. This behavior can be simply
   demonstrated by the following shell command:

   $ echo -n '{"field":"value"}' | jq --raw-output '"filter"'
   filter

   Since the third element of an equality or inequality is also
   effectively a constant, statements composed this way will yield a value
   that can be computed without the invocation's args field and should
   therefore be excluded from the predicate tree evaluation. This

   Therefore, I'd recommend that the word "MAY" in the above quoted text
   be replaced with the word "MUST".

2. The delegation specification includes .to[99]? as an example of
   using a try operator. The results of a jq command with an
   unknown object or array index is null, so stating that the ?
   operator turns what would otherwise be an error into a null
   isn't demonstrated by this example. This behavior can be demonstrated
   by the following shell commands:

   $ echo -n '{"field":"value"}' | jq --raw-output '.["nope"]'
   null
   $ echo -n '{"field":"value"}' | jq --raw-output '.["nope"]?'
   null

   We should choose a try operator example that effectively demonstrates
   the intended utility. Note that the examples in the jq manual don't
   show the utility of the Optional operator ? either.

3. The delegation specification makes the following statements:

   UCAN Delegation uses predicate logic statements extended with jq-style selectors as a policy language.

   and

   Selector syntax is closely based on jq's "filters".

   but then further explains the differences between UCAN selectors and
   jq filters in the "Differences from jq"
   section. After reading through this section, it seems like we should
   be able to say that the selector grammar is a proper subset of the
   jq filter grammar.

   I have mixed feelings about the name selector as we're actually
   filtering the invocation's args and then applying the predicate.
   I can however see some utility in "naming" the subset. Using the word
   try to describe the ? operator is a bit harder to justify - that
   word has connotations in many other languages (how do I catch the result
   of a selector with a try?) In jq, this operator is described as
   an Optional and it's very similar to optional chaining in both
   Typescript and Groovy. If Optional isn't the correct word,
   perhaps we can pick another without "baggage"?

4. The delegation specification states that the ABNF included in the
   Policy language
   section is the "formal syntax". In my opinion, this ABNF should be
   suitable for use when generating a parser in your language of choice.
   In its current state, the selector syntax is too permissive and it
   should be strengthened to limit selectors to those that are intended
   throughout the rest of the document. This is obviously impacted by
   the decisions about recommendations v1.0.0-rc.1 #1 and chore: sketch of the power-line concept #3 above.

   Since I'm already generating a Go parser from this grammar, I have
   a vested interest in the ABNF being syntactically strict and would
   be happy to complete this work. If recommendation chore: sketch of the power-line concept #3 is adopted,
   I'd also fuzz the selector by generating a corpus from the ABNF
   and executing it in jq. The output of jq should then provide
   the test results expected from the generated Go code.

Notes

1. It's entirely possible that I'm reading the specification incorrectly -
   if so, please don't take the above input personally as I'm simply
   trying to adopt the awesome ideas presented in the UCAN specifications.

2. I was going to compare my prototype policy code the the Rust
   implementation that was linked at the end of Brooklyn's IPFS Camp
   presentation (https://github.com/expede/rs-ucan.) Can someone
   provide the link to this code? Is it suitable to think of the
   Rust implementation as the RI?

[Gozala]

@smoyer64 thanks for taking a stab at it and sharing the feedback. It may be worth pointing out that selector syntax while was mostly inspired by jq, compatibility with it was not a goal. Bunch of jq features aren't supported and there were set of cases where diverging from jq seemed like a better choice in our context, you can view some of the details here #5

I'm not sure about the rust implementation but I had a draft of JS implementation that can be viewed here https://github.com/storacha-network/ucanto/pull/344/files

[expede]

Thanks for the feedback @smoyer64 ! Responses inline:

Quickly jumping to the end first

1. It's entirely possible that I'm reading the specification incorrectly -
   if so, please don't take the above input personally as I'm simply
   trying to adopt the awesome ideas presented in the UCAN specifications.

Thank you for the deep engagement with the material! I think other than the question about how strictly we adhere to jq, I'd like to implement the rest of your comments! 🙏

[...]

1. Copy and slightly modify the policy ABNF from the delegation
   specification. I've submitted a PR
   with just enough changes to make the ABNF valid.

PR merged! Thanks :)

[...]

Result

1. There are many inputs which are valid selectors per the ABNF but
   that are invalid or non-working filters in jq.

Yes, as Irakli responded earlier, we do have some divergences from jq. We think that they're well motivated ( #5 ) , but they have already bitten both you and me, so perhaps we should rethink that.

Recommendations

[...]

Therefore, I'd recommend that the word "MAY" in the above quoted text
be replaced with the word "MUST".

Agreed ✅ Will fix (DONE)

2. The delegation specification includes .to[99]? as an example of
   using a try operator. The results of a jq command with an
   unknown object or array index is null, so stating that the ?
   operator turns what would otherwise be an error into a null
   isn't demonstrated by this example. This behavior can be demonstrated
   by the following shell commands:

This is one of the examples that we chose to diverge on. In short: jq is (largely) a parser that processes streams, and UCAN policies are a policy language that operate on static data. The security implications of permissive behaviour may be significant, and we can always get the null behaviour by using a try (?).

There's also a few cases of expressivity, e.g. distinguishing between a null value and a missing key.

Note that the examples in the jq manual don't show the utility of the Optional operator ? either.

Interesting! Can you expand on this?

3. The delegation specification makes the following statements:

UCAN Delegation uses predicate logic statements extended with jq-style selectors as a policy language.

and

Selector syntax is closely based on jq's "filters".

After reading through this section, it seems like we should be able to say that the selector grammar is a proper subset of the jq filter grammar.

🤔 Perhaps. jq operates on streams, and is more permissive in how it treats things like assertion violation (e.g. when a field is not present), which may not be appropriate for an access control context.

I have mixed feelings about the name selector as we're actually filtering the invocation's args and then applying the predicate.

Could you expand? I'm not certain that ours is filtering in the same sense as jq filters on streams (or in the sense that you filter on arrays or collections). We have things lilke nested quantification to handle these cases instead. Arguably ours are "lenses" or "coarse parsers", but in these relatively simple cases that's often used interchangeably with "selector". I'm happy to call them something like IPFS Lenses if that's helpful, but it may get confused with the more common bidirectional lenses or profunctor lenses (with setters).

If Optional isn't the correct word, perhaps we can pick another without "baggage"?

Sure, let's switch it to "Optional" ✅ (DONE)

4. The delegation specification states that the ABNF included in the
   Policy language
   section is the "formal syntax". In my opinion, this ABNF should be
   suitable for use when generating a parser in your language of choice.

If helpful, we could also switch to IPLD Schema if that's better for this purpose than ANBF. I was trying to avoid IPFS Schema since it has its own set of design issues.

In its current state, the selector syntax is too permissive and it
should be strengthened to limit selectors to those that are intended
throughout the rest of the document. This is obviously impacted by
the decisions about recommendations #1 and #3 above.

Yes, it does depend on the other design decisions for certain. I'm not as confident in the strategy of generating a parser directly since you'll need to reserialise the CBOR to JSON for this to work, right? That could be expensive on every request.

2. I was going to compare my prototype policy code the the Rust
   implementation that was linked at the end of Brooklyn's IPFS Camp
   presentation (https://github.com/expede/rs-ucan.) Can someone
   provide the link to this code?

I'll flip the switch on the Rust implementation momentarily (DONE). It needs a few tweaks to come in line with your and Irakli's comments, but a version of it was being used right at the end of Fission.

Is it suitable to think of the Rust implementation as the RI?

We have two parallel implementations: JS and Rust.

[expede]

Any jq filter that doesn't begin with a . is effectively a constant and applying that filter to ANY input value will result in the filter constant being sent to the output.

Indeed, that's how the Rust code works.

#[test_log::test]
fn test_fail_missing_leading_dot() -> TestResult {
    let got = Selector::from_str("[22]");
    assert!(got.is_err());
    Ok(())
}

Will update the spec ✅

[expede]

@smoyer64 I pushed the branch directly to the ucan-wg fork, which is world-visible: https://github.com/ucan-wg/rs-ucan/tree/v1.0-rc.1 Wasm doesn't work yet, and some tweaks are needed to bring up to the above changes in the spec

[smoyer64]

I'm happy to concede that the deviations from strict compatibility with jq perhaps led the discussion in the wrong direction ... I hadn't seen #5 but that's exceedingly helpful in understanding the spec. There are examples of selectors that both pass the ABNF and that are invalid. One example that I can remember without consulting my notes is that, while the text states that .. is not allowed, code generated from the ABNF doesn't reject it. I'm still happy to do some fuzzing while considering #5.

Interesting! Can you expand on this?

This example from the jq manual works with and without the optional operator: https://jqplay.org/?q=.foo%3F&j=%7B%22notfoo%22%3A+true%2C+%22alsonotfoo%22%3A+false%7D

I'm not as confident in the strategy of generating a parser directly since you'll need to reserialise the CBOR to JSON for this to work.

Yes and I'm happy to see that @alanshaw has written a tokenizer/parser/matcher that works on the IPLD nodes.

I'll flip the switch on the Rust implementation momentarily (DONE).

Awesome! This will make it easier to decipher the spec - should I feed anything I think are ambiguities back into PRs?

[MichaelMure]

Any chance to make that cmd a [String], so that multiple capability can be delegated with the same payload?

Unless I missed something, if a (larger) service wants to delegate a set of capabilities to a user, it might not be possible to express that with a nice wildcard. Instead, you'd have a list of command: /foo/*, /bar/baz, /bar/boz, ....

As I understand, with cmd being a String (and not allowing comma-separated value I assume), this has the following consequences:

• a separate payload need to be issued and sign for each of the capabilities
• the receiver (audience) need to handle multiple delegation: much more complex UX/DX and logic necessary on the client side
• much larger token overall, especially as the signatures are duplicated.

[MichaelMure]

Or maybe better, have it be a [Capability], so that one policy remains attached to one command neatly.