Improve handling of 8bit message content and message/global
#4926
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rsto
force-pushed
the
cyrus_eai
branch
4 times, most recently
from
135b679
to
41c3c3e
Compare
|
@rsto I tried to give this a good review while on a train to Rome, but its been a long day and I couldn't focus enough to give it justice. I probably can't get back to this until my flight home on June 8. If you need it reviewed sooner, maybe someone else on the team can take a look. |
elliefm
reviewed
May 31, 2024
cassandane/tiny-tests/JMAPEmail/email_set_encode_plain_text_attachment
Outdated
Show resolved
Hide resolved
elliefm
approved these changes
Jun 7, 2024
rsto
force-pushed
the
cyrus_eai
branch
2 times, most recently
from
ebc62f7
to
e8de054
Compare
|
This now also changes the Xapian backend to normalize any text using NFKC with CaseFolding before indexing and querying. It bumps the Xapian index version to 17. The default stemmer keeps using lowercase Cyrus Search Form. If any of the existing indexes contains version 16 or less, then queries search for any terms that either match the NFKC-normalized term or the unnormalized term. |
|
This now also includes #4977 |
rsto
force-pushed
the
cyrus_eai
branch
2 times, most recently
from
f635f45
to
0116bdf
Compare
Turns out that most uses of this function force a newline before the parameter anyway, so don't require the caller to provide the header prefix length. Signed-off-by: Robert Stepanek <[email protected]>
Signed-off-by: Robert Stepanek <[email protected]>
It passed a non-ascii blob as text/plain content.
Up until now the Content-Transfer-Encoding header for message/rfc822 attachments was ignored. That is fine for rfc822 but we want to treat message/rfc822 and message/global equally, and the global subtype does allow its content to be encoded. This patch changes the parse code to not throw away the encoding identifier of a message/rfc822 body part before storing it in cache. It bumps the mailbox minor cache version to indicate this and handles previous versions on read.
The User-Agent header is not defined for MIME messages, it only is defined for NetNews and HTTP. Use "X-Mailer" instead which at least is mentioned in RFC 2076.
This changes the Xapian backend to normalize any text using NFKC with CaseFolding before indexing and querying. It bumps the Xapian index version to 17. The default stemmer keeps using lowercase Cyrus Search Form. If any of the existing indexes contains version 16 or less, then queries search for any terms that either match the NFKC-normalized term or the unnormalized term.
It really didn't belong to global.h in the first place.
Up until now, Cyrus did not convert the contents of header fields and a couple of iCalendar properties to search form. This required us to convert those part contents to lower case in the Xapian wrapper, but we rather want this all proper NFKC-casefolded. Calling charset_convert on these values before sending them to the indexer makes sure all indexed text is in proper form.
The query runner might rewrite the search query before running it, so report the query costs after that.
brong
approved these changes
Jul 25, 2024
|
Let's go for it. Merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates how Cyrus handles messages with 8bit content, both in headers and body parts values. This PR consists of a number of commits, some of which are refactoring only. The main functional changes are listed below:
header:Xxx:asTextproperty values. JMAP Email/set{create} still q-encodesheader:Xxx:asTextproperty values but NFC-normalizes the value before encoding.message/rfc822ormessage/global. The cache version is updated to13.Of all changes, the ones done to the MIME serialization of JMAP Emails is the most critical: once the MIME message got generated its JMAP Email input as supplied by the client gets thrown away. Accordingly, this PR contains a bunch of test cases to trigger each special condition in the JMAP Email to MIME logic.