Highlighted Snippets/Excerpts via API?

[sbw]

I notice the searchmysite.net results page includes snippets or excerpts of each page with the search terms highlighted.

But I don't see that included in the API response. Would it be difficult to add?

In the API response, what would be the best way to mark up the highlighted terms? I'd be fine with Markdown, if the excerpt itself had Markdown characters escaped. But maybe there's a better way to do it.

[m-i-l]

Solr uses an hl=on parameter to switch the highlighting on, hl.fl to define the list of fields which are highlighted, and hl.simple.pre and hl.simple.post parameters for the string that appears before and after the highlighted text respectively. See the Solr docs at https://solr.apache.org/guide/8_11/highlighting.html for more information.

I could do something similar to Solr with the search API, providing an hl=on, hl.fl, and hl.simple.pre and hl.simple.post parameters.

FYI, in the search results script at https://github.com/searchmysite/searchmysite.net/blob/main/src/web/content/dynamic/searchmysite/search/search.py, I couldn't simply set hl.simple.pre and hl.simple.post to <b> and </b> respectively and show the highlighting content in unescaped HTML because that led to all sorts of weird issues when HTML fragments appeared in the highlighting content, e.g. open tags without matching close tags, so I had to do some processing to ensure the <b> and </b> tags were the only unescaped HTML.

I'm not sure about doing any of that extra processing on the API results though, because I wouldn't know how the API was going to be used.

[sbw]

Thanks. For my needs, a simple API argument like snippet=yes would be sufficient. You might return the snippet as a JSON array, like the array you produce in search.py. Then I would do the highlighting similarly to search/results.html. You might omit the ellipses from the API response, again leaving that to the caller.

Looking at the Solr doc, I don't need the full highlighting object from the Solr response. I just need one or more snippets from the body of the page. Plus maybe the title and description. If each snippet is an array, then more than one snippet would be be an array of arrays. But a single snippet would make me happy.

I believe hl.fragsize determines the number of characters in the snippet? Might be nice to expose that in the Search My Site API. Even something simple like snippet=100.

Other API users may have more sophisticated requirements, up to the full set of Solr args. If so, I would be happy to work with that, if you prefer to go that way. (I haven't yet dug into the source to learn the names of the fields your store in Solr.)

[m-i-l]

Actually, in the spirit of the super-simple API, I think it may be best just to include all fields, including the fragment field, by default.

So how about returning this where the term appears in the content field:
fragment = ["text before the search ", "query", " and text after"]

There would also be cases of:
fragment = ["text before the search ", "query", " and text after the search ", "query", "."]

And some cases where the fragment field wouldn't appear at all because the query didn't appear in the content field (e.g. was only in the title).

If someone wants to optimise their API and remove the fields that they don't use, I could always add a fields parameter where you specify what fields you want.

[sbw]

Super simple sounds good to me. I'd be glad to see the snippets as you suggest.

[m-i-l]

@sbw Now implemented. Thanks for your suggestion. I do think it makes the API more useful.

[sbw]

Perfect, thanks! Example: https://sbw.org/search/?q=santa+fe