Search results page fails to render if search string includes special characters

[tfmorris]

Evidence / Screenshot (if possible)

Relevant url?

https://openlibrary.org/search?q=Cornell+%2777%3A+The+Music%2C+the+Myth+and+the+Magnificence+of+the+Grateful+Dead+Show+at+Barton+Hall&mode=everything

Steps to Reproduce

1. Go to ...
2. Do ...

• Actual:
• Expected:

Details

• Logged in (Y/N)?
• Browser type/version?
• Operating system?
• Environment (prod/dev/local)? prod

Proposal & Constraints

Related files

Stakeholders

[JaydenTeoh]

@cdrini Hi, I just pushed a fix for this issue, do let me know if I missed out on anything!

[cdrini]

This is an issue with the library we're using to parse solr queries. I've created issues there:

• Parser fails at single quote character (eg "hi 'bye'") jurismarches/luqum#86
• Non-decimal boost throws TypeError instead of ParseError (eg "hello^foo") jurismarches/luqum#87

Their community did in the past fix another issue I reported: jurismarches/luqum#79 . Maybe you could try to fix these issues on that repo @JaydenTeoh and open a PR? Using their commit jurismarches/luqum@0cc7767 as a template?

In the meantime, we can try adding these to the escape flow. Currently all user-submitted queries pass through this function:

• openlibrary/openlibrary/plugins/worksearch/schemes/__init__.py

  Line 66 in 14f8029

  def process_user_query(self, q_param: str) -> str:

This function will try to parse the query, and if parsing errors, try to escape every character to convert the query to a plain solr string with no special features. This is done by this function:

openlibrary/openlibrary/solr/query_utils.py

Lines 120 to 140 in ed47d57

	def fully_escape_query(query: str) -> str:
	"""
	Try to convert a query to basically a plain lucene string.
	>>> fully_escape_query('title:foo')
	'title\\\\:foo'
	>>> fully_escape_query('title:foo bar')
	'title\\\\:foo bar'
	>>> fully_escape_query('title:foo (bar baz:boo)')
	'title\\\\:foo \\\\(bar baz\\\\:boo\\\\)'
	>>> fully_escape_query('x:[A TO Z}')
	'x\\\\:\\\\[A TO Z\\\\}'
	>>> fully_escape_query('foo AND bar')
	'foo and bar'
	"""
	escaped = query
	# Escape special characters
	escaped = re.sub(r'[\[\]\(\)\{\}:"\-+?~^/\\,]', r'\\\g<0>', escaped)
	# Remove boolean operators by making them lowercase
	escaped = re.sub(r'AND|OR|NOT', lambda _1: _1.group(0).lower(), escaped)
	return escaped

We likely want to expand this function to include the single quote.

[JaydenTeoh]

@cdrini Noted on that, I have expanded the fully_escaped_query function in #7931 for now, I will take a look at the luqum parser repo to make a fix!