Ability to use links as the source for 'options'

[evert]

We use HAL because we like using links. Aside from the 'discovery' benefit, we heavily use links to express relationships between entities. For example, a blog article might have many category links.

The majority of the cases where we will be using the new options feature is to let users specify these relationships (let a user select a category when writing an article).

The ability to use a link to specify the source for the dropdown is helpful but has some drawbacks. We would like to be able to select the 'source' for a property by specifying:

1. A resource uri
2. A rel.

For example, in the article category case we'd want to point to a 'categories' collection and the 'item' rel.

{
  name: 'category',
  type: 'uri',
  options: {
    link: { href: '/category' },
    rel: 'item
  }
}

What does this give us:

1. We can re-use endpoints that exist all over the place, instead of creating new endpoints specifically for emitting the array structure HAL forms wants.
2. We can be format-agnostic. You can point to a Siren, Collection+JSON or a HTML document, it all magically works. (provided ofc your client also has an abstraction layer for different formats).
3. We can make a much richer interface. For example, each category might have a 'color' property, which we can use for the background. Each category could have an 'image' property. The idea here is that the format tells you how to get the list of options, but doesn't prescribe how to render each option. How each option is rendered is dependent on the media type, and what media types a client can support/render.

I feel that this is a very natural fit for HAL Forms and HATEOAS in general.

[mamund]

can you expand on this a bit? possibly write up a short example that includes the HTTP request/response that you want to support?

[evert]

Yes, given a field:

{
  name: 'category',
  type: 'uri',
  options: {
    link: { href: '/category' },
    rel: 'item'
  }
}

A client might do the following request:

GET /category HTTP/1.1

Which results in the following response:

HTTP/1.1 200 OK
Content-Type: application/hal+json

{
  "_links": {
    "self": { "href": "/category" },
    "item": [
      { "href": "/category/1", "title": "Gardening"},
      { "href": "/category/2", "title": "Cooking"},
    ]
}

This might ultimately render:

<select name="category">
  <option value="/category/1">Gardening</option>
  <option value="/category/2">Cooking</option>
</select>

Important though is that the use of HAL is not prescribed, if our response to GET /category returned Siren, Collection+JSON or HTTP Link headers, it should work in an identical manner:

HTTP/1.1 200 OK
Content-Length: 0
Link: </category/1>; rel=item title=Gardening
Link: </category/2>; rel=item title=Cooking

Does that make sense?

[evert]

To expand on that, if our list of HAL categories returns something like this instead:

HTTP/1.1 200 OK
Content-Type: application/hal+json

{
  "_links": {
    "self": { "href": "/category" },
    "item": [
      { "href": "/category/1", "title": "Gardening"},
      { "href": "/category/2", "title": "Cooking"},
    ],
  },
  "_embedded": {
    "item": [
      {
        "_links": {
          "self": { "href": "/category/1"},
        },
        "title": "Gardening",
        "color": "green"
      },
      {
        "_links": {
          "self": { "href": "/category/2"},
        },
        "title": "Cooking",
        "color": "red"
      }
    ]
  }
}

A client could choose to use the 'color' to further enhance the UI. Important note though, that should not be prescribed in HAL-Forms. HAL Forms should just talk about the fact that the list of options is a list of links, and not say anything about how they might be rendered; that's up to the application. HAL Forms just requires that the target resource uses something that uses Weblinks and leave it up to the application to try and interpret that, send the right Accept headers, etc.

[mamund]

OK, i think i am getting it.

the current spec lays out "link" interactions for text/csv and application/json. you are suggesting we need to add support for more media types (HAL directly, Cj Siren, even headers).

if i have it about right -- that means we need to come up w/ the rules for other media types, too.

for starters, i'd like to see worked up versions of options.link support for HAL and maybe one other format. I get the feeling you already have a HAL solution in mind, right?

[evert]

the current spec lays out "link" interactions for text/csv and application/json. you are suggesting we need to add support for more media types (HAL directly, Cj Siren, even headers).

I would want to suggest to completely leave out media-type specific information. All of these formats and HTML all use the same semantic concept of a web link. It's up to the client to figure out (and negotiate) what they can support. A comparison is the HTML <img> tag. Nothing is explicitly said about supported formats, and specifications didn't need to change when browsers started adopting .webp as an image format.

This is not unlike regular links in HAL, HAL doesn't require your links to point to other HAL documents. The client is in charge of navigating the web and knows what it can and cannot interpret. If a client only understands HAL for this purpose, they would include application/hal+json in the Accept header.

The only requirement in HAL Forms is that the target URI should return something that's conceptually a RFC8288 Web Link.

[evert]

Another comparison I have is the Prefer-Push header I've worked on (and now abandoned).

https://tools.ietf.org/html/draft-pot-prefer-push-01

This spec was supposed to inform servers what resources to push via HTTP/2, but the client specified these resources via their link relationship (rel), while being opinionated about the format those links appear in.

[mamund]

OK. color me a skeptic but that might be due to my not "getting it" quite yet.

here's what i suggest. show us a working version of what you'd like to see. essentially, extend HAL-FORMS locally to do what you want and show a client working as you wish.

let's see where that leads us.

[evert]

I will!

[mamund]

thanks. looking forward to it. ping me directly if you need anything at all.

[carlobeltrame]

I completely agree we need something like this. The biggest problem I see currently is the rel. Assuming we have distinct clients, one of which understands HAL-FORMS and application/json, one of which understands HAL-FORMS and application/hal+json, and others that understand yet different formats. Also assume our API server is able to serve all of those formats.

Requesting the categories with Accept: application/json might yield the following response:

[{
  "title": "Gardening", "color": "green"
},{
  "title": "Cooking", "color": "red"
}]

Requesting the same categories with Accept: application/hal+json might return the following:

{
  "_links": {
    "self": { "href": "/category" },
    "item": [
      { "href": "/category/1", "title": "Gardening"},
      { "href": "/category/2", "title": "Cooking"},
    ],
  },
  "_embedded": {
    "item": [
      {
        "_links": {
          "self": { "href": "/category/1"},
        },
        "title": "Gardening",
        "color": "green"
      },
      {
        "_links": {
          "self": { "href": "/category/2"},
        },
        "title": "Cooking",
        "color": "red"
      }
    ]
  }
}

Now, what is the correct, common value of rel to use for all of these formats? @evert used item in his example, but that is specific to the HAL version of the response. If we want to allow arbitrary formats, the response could even be XML, or YAML, or CSV.
This is also where the <img> analogy fails. In an <img> tag, there is no need to specify the path to data inside the image - the full image is rendered in any case.

I think we either need to leave out the rel, forcing clients to have some knowledge (in the form of configuration) about where the specific API "usually" locates the items in a given format used by the client, which violates the intent of HAL-FORMS that the client should be mostly API-agnostic. Or we need to limit the supported formats (at least for now), which hurts the interoperability of HAL-FORMS.

[evert]

@carlobeltrame

Now, what is the correct, common value of rel to use for all of these formats? @evert used item in his example, but that is specific to the HAL version of the response. If we want to allow arbitrary formats, the response could even be XML, or YAML, or CSV.

I have 2 notes here that might help with this:

1. We could add an option type so the client can send a good (suggested) Accept header.
2. If your API returns application/json, it could still encode the same links in HTTP Link headers. This would at least make the HTTP response mostly semantically equivalent.

I agree with the general premise of though that not every content-type will have a concept of a weblink, but perhaps adding a type parameter could solve that.

[odrotbohm]

I think we either need to leave out the rel, forcing clients to have some knowledge (in the form of configuration) about where the specific API "usually" locates the items in a given format used by the client, which violates the intent of HAL-FORMS that the client should be mostly API-agnostic.

Isn't that actually just stating that the client has to understand the media type returned by the server and the media types' way of expressing collections? 🤔 If that media type doesn't have a very precise way of describing that, then it might not be a good candidate to return remote options, unless you're willing to live with some wiggle room for client implementations. That said, I think HAL in particular is well enough defined to support that case (see my comment on the other ticket).

I think a smart client could pick up all elements of the _embedded clause, identify that they're previews of a resource (by the presence of a self link) and use that as value for the options. That said, I think we're moving out of the scope of both options and HAL FORMS. I don't think it's the spec's responsibility to define how to identify prompts and values from arbitrary media types. If we do that we open up a pandoras box with every other week someone showing up asking for the processing model of values and prompts for another media type.

I think we're very well set up by the spec defining a prompt and a value and primarily rely on API authors to find ways to prepare responses in a way that they're easily processable for a HAL FORMS client without disambiguation. I don't think we should go ahead and try to make every existing resource representation – even if it's HAL based – that someone made up, work.

[odrotbohm]

What strikes me as odd about the requested semantics for rel is that it has a very specific assumption about the way the remote resource is organized. It seems to cater HAL very explicitly and I think that's something we should avoid. Process HAL according to the spec (accepting the vague parts) and collect all "collection items" you can find. If an existing resource is not suited for that (e.g. if you have multiple entries in _embedded but only want to use the values of one entry) then just expose a dedicated resource for that. I don't think we should overload the spec with knobs to make existing application level semantics (see what I did here, Mike? ;) ) work with it.

[evert]

Process HAL according to the spec (accepting the vague parts) and collect all "collection items"

Following your wording, and how I consider the HAL spec wants this, I think the appropriate way to find all 'collection items' is not to look in _embedded, but to look at all item link relationships it defines.

There is nothing else in the HAL drafts that suggest that the inclusion of anything in _embedded implies that it's a collection member somehow.

[evert]

Using _embedded for anything other than a transport-level optimization would make it incompatible with any HAL implementation I've seen & worked on. Usually _embedded is stripped out before the response reaches the user. And again I'm not trying to convince you to do anything different, but overloading _embedded to mean anything else would for us simply mean we can't reasonably implement this.

[evert]

For a while we actually had a system that would do HTTP/2 Pushes instead of HAL embeds for connections that supported it because to me they have the same purpose. =)

[odrotbohm]

I am not sure, why you think we're "overloading" anything here. The most exhaustive example clearly shows entries in _embedded to exactly work that way: ideally, you find exactly one entry because HAL FORMS currently doesn't know anything about the requirement of the target resource to expose elements under a key. If that key is an array, take that. If it's not, the object is the sole element of the collection.

If the document contains multiple entries we have two options (haha!). Either teach HAL FORMS which of the keys to use (which I'd like to avoid because it comes with a lot of HAL specific implicit knowledge and: do we want to do this for other media types as well?), or the client can fall back to pick up all keys' values (because it doesn't care about rels and grouping elements by rels), which might lead to not perfect results because there's no guarantee about ordering, duplicates etc. but is a decent fallback. If you want to avoid the problems of multiple key, again, you can expose a dedicated resource to arrange the results in a less ambiguous structure. I think it's a pretty cool example how tow media types can nicely link together without actually baking too much knowledge about one into the other.

There is nothing else in the HAL drafts that suggest that the inclusion of anything in _embedded implies that it's a collection member somehow.

Well, the HAL FORMS' link attribute defines the remote resource to represent a collection, i.e. we're looking for one in the media type we receive by resolving the link. If that is a HAL document, we need to work with what we know about HAL. As a HAL document is always a document (never an array), the only top-level field we know about that contains application data is _embedded. It is defined to map from rels to Resource Objects. As we're not interested in the rels, it seems like a natural step to me to collect all available ones.

[carlobeltrame]

I agree that a dedicated resource with just one rel in _embedded is the best way to solve this. I'd even be okay with having to configure globally in my client, which rel from _embedded to use, I've seen item, items or a dynamic name based on the collection.

But when that is not a possibility for a given API server, collecting objects from multiple embedded arrays with different rels makes no sense to me - objects in different rels might not even have the same data structure and fields. Consider the following response:

{
  "title": "Moby Dick",
  "_links": { "self": { "href": "/books/1" }, "author": { "href": "/authors/1" } },
  "_embedded": {
    "chapters": [
      { ... }, { ... }
    ]
  }
}

Assume I use this resource for the options of a dropdown select field that allows to choose one chapter of the given book. Some time later, another developer decides that a book might be written by multiple authors, and changes the response on this endpoint to this:

{
  "title": "Moby Dick",
  "_links": { "self": { "href": "/books/1" }, "authors": { "href": "/authors?book=1" } },
  "_embedded": {
    "chapters": [
      { ... }, { ... }
    ],
    "authors": [
      { ... }, { ... }
    ]
  }
}

Suddenly, because of a completely unrelated change, my chapter selection starts bugging out (in case the id and name fields are not called the same for chapters and authors), or even worse, starts offering authors for selection.

As stated above, using the entries returned from /chapters?book=1 or /books/1/chapters instead of /books/1 is of course the superior solution, when it is available. The question for me is, do we even need the spec to specify what happens in case a server is not able to serve these more specialized resources?

[evert]

@odrotbohm Although I think you are wrong, it's not my goal here to convince you of this ;) We're talking a bit in circles now, but the reality is your usage of HAL / _embedded is hardly unique. I'm hoping we can come to a middle-ground that makes sense for the people that strictly use _embedded as the 'caching pattern', as well as the folks that add more semantic meaning to it such as yourself.

Another nice benefit of targeting web links by rel, is that we can now also support HTML Links, HTTP Links, JSON:API, Siren, C+J as sources, because they all share this concept. I don't see targeting a list of web-links by rel a HAL-specific choice, because a Weblink is such a central component to every HATEOAS format and the web in general.

Another thing that desperately needs to be discussed is: what are the values that would be sent? In my proposal in this issue, the value would end up being the href of the link (or self link if you use embedded), and the prompt for each option inferred by the implementor as best as possible.

How did you and @toedter envision the value and prompt to be inferred if you did primarily use _embedded as a source?

[odrotbohm]

@carlobeltrame – Aren't we discussing a straw man here? How likely is it that the resource objects contained in these collections, that are obviously designed for completely different purposes contain value and prompt attributes? Even considering that you could tweak the pointers using valueField/promptField, wouldn't you rather just provide a resource that only returns the necessary attributes?

@evert - I am not "adding more semantic meaning". I am citing the spec. 🙃

The representations looked like this:

{
  "_embedded" : {
    "doesn't-really-matter" : [
      { "value" : "…", "prompt" : "…" },
      { "value" : "…", "prompt" : "…" }
    ]
  }
}

That seemed to be in line with "we remove the HAL-induced envelope" and then apply the semantics required by HAL FORMS.

I guess we need to decide whether the focus of this ticket is on _links or _embedded. It seems you started with the former and we have diverted into the latter. Sorry, if this was my fault. The examples seemed to indicate that the latter was at the center of the discussion.

[evert]

The representations looked like this:

{
"_embedded" : {
"doesn't-really-matter" : [
{ "value" : "…", "prompt" : "…" },
{ "value" : "…", "prompt" : "…" }
]
}
}

That seemed to be in line with "we remove the HAL-induced envelope" and then apply the semantics required by HAL FORMS.

This example makes sense, if you want to allow embedding arbitrary data that are not HAL resources (don't have a URI). I don't see a reasonable way to support this; but if this makes it in the spec we simply won't implement it.

There's at least some support that this is weird from the spec author (editorialized):

Q: Is there any consensus on how to deal with resources that may be contained in an _embedded collection, but aren’t actually addressable on their own with their own self link?
Andrej: If it is not "addressable" then it is not a resource (in the REST sense of the word) in my opinion. And thus it should not be in _links and neither in _embedded.
Mike Kelly: I tend to agree with Andrej [...]

An alternative for you might just be to define a media-type that explicitly means: "This is a HAL-flavored response, that uses a specific format in _embedded as a data-source for options lists, e.g.: "application/prs.options-source.hal+json". The HAL-Forms spec should allow this kind of extension with new media-types.