K9-Mail Compatibility: list special-use type error

[iot-resister]

To get K-9 Mail working, I changed the follwing

 s := make([]string, len(fields))
 for i, v := range fields {
    s[i] = fmt.Sprint(v)
  }
 
   if mailbox, err := imap.ParseString(s[0]); err != nil {
...

to line: https://github.com/emersion/go-imap/blob/master/commands/list.go#L41

Please excuse my go. I've only been writing it for one day.

references: https://stackoverflow.com/questions/44027826/convert-interface-to-string-in-golang

[emersion]

This sounds like a bug with K9 mail. The RFC specifies:

list            = "LIST" SP mailbox SP list-mailbox
list-mailbox    = 1*list-char / string
list-char       = ATOM-CHAR / list-wildcards / resp-specials

[iot-resister]

This my first time every hearing about ABNF, but shouldn't mailbox be a string? And isn't that SPECIAL-USE ? Mbsync sends []string, but k9 logs out an []interface.... So they are sending the wrong type?

mailbox         = "INBOX" / astring
                    ; INBOX is case-insensitive.  All case variants of
                    ; INBOX (e.g., "iNbOx") MUST be interpreted as INBOX
                    ; not as an astring.  An astring which consists of
                    ; the case-insensitive sequence "I" "N" "B" "O" "X"
                    ; is considered to be INBOX and not an astring.
                    ;  Refer to section 5.1 for further
                    ; semantic details of mailbox names.

I created an issue for k9 mail.

thunderbird/thunderbird-android#4729

Feel free to ignore my questions if they are grossly misinformed.

[cketti]

K-9 Mail 5.600 will only send the extended LIST command when the SPECIAL-USE extension is advertised. My reading of the spec was (and really still is), that the SPECIAL-USE capability may only be advertised if the server also supports LIST-EXTENDED. K-9 Mail versions 5.7xx+ will check for both capabilities before sending the extended LIST command.

See also emersion/hydroxide#41

[emersion]

Ah, right. Sorry, I didn't remember about this.

[exander77]

This should not be closed as it is not fixed. It relates to this: emersion/go-imap-specialuse#1
And by extension, this is not fixed as well: #21

As per https://tools.ietf.org/html/rfc6154#page-7

When SPECIAL-USE capability is advertised, LIST (SPECIAL-USE) "" "*" must be handled. This is not an optional feature and it means, that it go-imap-specialuse breaks RFC6154. I would extend: https://github.com/emersion/go-imap-specialuse
But, I don't see the ability to extend command and partially handle it in extension, it seems to be all or nothing:

// Command gets a command handler factory for the provided command name.
func (s *Server) Command(name string) HandlerFactory {
  // Extensions can override builtin commands
  for _, ext := range s.extensions {
    if h := ext.Command(name); h != nil {
      return h
    }
  }

  return s.commands[name]
}

SPECIAL-USE is tightly intervened with the LIST and maybe it should be merged into go-imap instead of being an extension.

I did a hackish solution here: exander77@f85eae3

[emersion]

When SPECIAL-USE capability is advertised, LIST (SPECIAL-USE) "" "*" must be handled.

Your link doesn't justify this.

[exander77]

@emersion Why do you think that?

[cketti]

@emersion: I tried to illustrate in this comment that announcing the SPECIAL-USE capability doesn't make sense if all you support is the regular LIST command with special-use attributes. The spec specifically states that this feature is not associated with a capability string:

RFC 6154, section 2:

There is no capability string related to the support of special-use attributes on the non-extended LIST command.

[exander77]

For the extended list command [RFC5258], this extension adds a new
capability string, a new selection option, and a new return option,
all called "SPECIAL-USE". Supporting implementations MUST include
the "SPECIAL-USE" capability string in response to an IMAP CAPABILITY
command.

If the client specifies the "SPECIAL-USE"
return option, the LIST command MUST return the new special-use
attributes on those mailboxes that have them set.

@cketti You covered it well here: emersion/hydroxide#41

I do think it is way worse than it doesn't make sense to advertise it, it breaks RFC6154. The capability SPECIAL-USE is there only and only if the extended LIST command supports SPECIAL-USE. Having special flags for special use folders is no capability at all. Flags are already supported.

Uf, this is going on from 2018? The RFC is extremely clear.

[exander77]

@emersion So... what are we going to do about it? I have a kind of dirty patch (mentioned above) as I don't see an easy way how to implemented using extensions. And I think We can all agree that this is wrong and breaking RFC6154.

Received some comments from Barry Leiba:

I believe that Section 2 makes it clear that it works both ways: the
first paragraph says how special-use behave with non-extended LIST,
and the second paragraph says what it does with extended LIST.

Ah, wait, I see. The example in Section 5.2 says that it's an example of use of extended LIST, but the server doesn't advertise extended LIST... that's the problem. Yes, that's an error in Section 5.2, and this report should be marked "Verified".

(Sorry for the noise: one further comment.) When it's marked "Verified", a comment should be added that it is OK to support special-use without extended list, as specified in the first paragraph of Section 2, but that the example in 5.2 says it's an example of use with extended list, so the example is wrong.

The RFC guy actually confirms it. He is saying that it is ok to support special-use without an extended list and We all agree with that. Special-use folders are just flags. What is wrong is announcing SPECIAL-USE capability which is exclusive to extended LIST supporting SPECIAL-USE selection and return option.

There are 4 different meanings of special-use in RFC6154:

1. there are SPECIAL-USE Metadata for Special-Use Attributes, this relates to METADATA extension in RFC5464, see 4 and 8.6,
2. there is SPECIAL-USE IMAP Capability, see 8.3,
3. there is SPECIAL-USE Selection Option, see 8.4,
4. there is SPECIAL-USE Return Option, see 8.5.

Case 1. is just an extension of RFC5464 and is not announced (these are just flags):

There is no capability string related to the support of special-use attributes on the non-extended LIST command.

Cases 2. and 3. and 4. are bound together:

For the extended list command [RFC5258], this extension adds a new capability string, a new selection option, and a new return option, all called "SPECIAL-USE".

[exander77]

I have been reading more in: emersion/hydroxide#41

And there are misconceptions there:

So the "bug" in the second example is the CAPABILITY list is missing LIST-EXTENDED.
Yes, this is what I reported.

No, there is no bug in the example. RFC5258 clearly states:

Note that extensions that don't require support for an extended LIST command, but use extended LIST responses (see below), don't need to advertise the "LIST-EXTENDED" capability string.

This is exactly the case of SPECIAL-USE as it works without LIST-EXTENDED, but is using extended LIST responses. Also, I would note here, that LIST-EXTENDED does not need to be advertised at all based on RFC5258 (it only extends LIST command).

And:

Seems like K-9 is assuming the server supports RFC 5258 without checking it.

No, SPECIAL-USE implies extended LIST responses as RFC5258 notes. You are confusing extended LIST responses with extended LIST command and its options, but so does 5.2. which text is quite imprecise:

5.2.  Example of an Extended IMAP LIST Command

   This example shows an IMAP LIST response from a server that supports
   this extension.  The client uses the extended IMAP LIST command.

What they mean here is:

5.2.  Example of an Extended IMAP LIST Command 
   or IMAP LIST Command Extended by This Extension Which Is Using Extended IMAP LIST Command Response

   This example shows an IMAP LIST response from a server that supports
   this extension.  The client uses the extended (either RFC5258 or by this extension) IMAP LIST command.

It confuses the reader with Extended LIST Command as defined in RFC5258 and LIST command extended by this extension which is using Extended LIST Command responses.

K-9 does not assume EXTENDED-LIST support but extended LIST responses from a LIST command with SPECIAL-USE selection and return options.

There are commands and there are responses, there are selection options, there are return options and there are capabilities.

[cketti]

K-9 does not assume EXTENDED-LIST support but extended LIST responses from a LIST command with SPECIAL-USE selection and return options.

That's a stretch. K-9 Mail 5.600 does assume the server supports the extended LIST command when the SPECIAL-USE capability is present. It then uses the command LIST (SPECIAL-USE) "" "*". In 5.7xx beta versions the code was changed to check for both the SPECIAL-USE and LIST-EXTENDED capabilities. Due to other changes the issued command itself changed to LIST "" "*" RETURN (SPECIAL-USE).

RFCs are often not great at mentioning error cases. They state what you should do and hope you make the correct assumptions about what not to do. Technically, the behavior of go-imap falls into undefined behavior territory.

RFC 6154, section 2:

An IMAP server that supports this extension MAY include any or all of
the following attributes in responses to the non-extended IMAP LIST
command. The new attributes are included along with existing
attributes, such as "\Marked" and "\Noselect". A given mailbox may
have none, one, or more than one of these attributes. In some cases,
a special use is advice to a client about what to put in that
mailbox. In other cases, it's advice to a client about what to
expect to find there. There is no capability string related to the
support of special-use attributes on the non-extended LIST command.

For the extended list command [RFC5258], this extension adds a new
capability string, a new selection option, and a new return option,
all called "SPECIAL-USE". Supporting implementations MUST include
the "SPECIAL-USE" capability string in response to an IMAP CAPABILITY
command. If the client specifies the "SPECIAL-USE" selection option,
the LIST command MUST return only those mailboxes that have a
special-use attribute set. If the client specifies the "SPECIAL-USE"
return option, the LIST command MUST return the new special-use
attributes on those mailboxes that have them set. The "SPECIAL-USE"
return option is implied by the "SPECIAL-USE" selection option. The
extended LIST command MAY return SPECIAL-USE attributes even if the
client does not specify the return option.

Paragraph 1 states that there's no capability associated with including the attributes in the regular list response. It does not say you MUST NOT use the SPECIAL-USE capability if you only support this mechanism, but not the extended LIST command feature.

Paragraph 2 deals with an extension to the extended LIST command. It doesn't apply to go-imap because go-imap doesn't support the extended LIST command.

Unfortunately, the spec doesn't explicitly spell out what servers have to do when the SPECIAL-USE capability is announced. But I hope there's no doubt that announcing the SPECIAL-USE capability without supporting the extended LIST command (and announcing the LIST-EXTENDED capability) is not a useful thing to do and is just asking for this kind of compatibility issues.