Migrate entity search to Haystack #711

hancush · 2021-01-15T22:10:37Z

Overview

This PR:

Replaces the custom Solr search back end that powers entity search with a conventional implementation of Haystack. N.b., it does not migrate the logic to populate and query the composition documents to form the org charts from Use Solr to build org charts #416.
Updates the startup instructions in the README and makes some small adjustments to the Docker setup.

Connects #674.

Demo

Notes

There are a few classes of changes here:

Adds ${ENTITY_MODULE}.search_indexes for each search entity: person, unit, violation, and source.
Updates schema.xml to include Haystack fields. N.b., the template into which these fields are generated lives in templates/search_configs/schema.xml and is a lightly edited version of the original schema.
Replaces the custom search view with a Haystack FacetedSearchView and adjusts the search templates to work with the new context.
Renames make_search_index to update_composition_index and removes code to add documents related to the base search entities.
Updates the encrypted configs to include HAYSTACK_* variables (as exemplified in local_settings.example.py).
Updates the tests to remove tests for index updates on model changes. (We'll be using Haystack's RealtimeSignalProcessor for this behavior, i.e., we do not need to test third-party code.)

Testing Instructions

If you do not have a local instance, follow the updated README instructions to retrieve a dump of the staging database and load it into your database.
Build the entity search index: docker-compose run --rm app python manage.py update_index -v 3
- Note that this will take about half an hour.
Build the composition search index: docker-compose run --rm app python manage.py make_search_index
Create a superuser and log in so you can access source search: docker-compose run --rm app python manage.py createsuperuser
Navigate to your local homepage and begin to execute a variety of searches. Confirm that load times are reasonable, and that the results generally match those on https://back.securityforcemonitor.org.
Continue to make test searches from the search results page. Change sorting and paging parameters, and confirm the expected results resemble staging.
Use the personnel search to find a commander. Navigate to their detail page and confirm the organization charts render as expected.
Use unit search to find a unit with at least one parent. Navigate to its detail page and confirm it renders as expected.

…aystack

solr_configs/conf/schema.xml

sfm_pc/views.py

fgregg

much better! what a great improvement. let's take the opportunity to address some smaller perplexities

Dockerfile-solr-jts

fgregg · 2021-01-27T17:05:36Z

Makefile

@@ -28,7 +28,8 @@ update_db : import_directory import_db auth_models.json flush_db link_locations
 	python manage.py loaddata auth_models.json
 	python manage.py make_materialized_views --recreate
 	python manage.py update_countries_plus
-	python manage.py make_search_index --recreate
+	python manage.py rebuild_index --noinput


do we need a command to build the schema?

also, it would seem like these search index commands maybe belong under a different PHONY target?

fgregg · 2021-01-27T17:06:45Z

README.md

@@ -75,7 +83,8 @@ Build and start the Docker image for the Solr server:

 Open up another shell and create the search index:

-    docker-compose run --rm app ./manage.py make_search_index
+    docker-compose run --rm app ./manage.py update_index


you do not have to do address this in this PR, but we should use the makefile commands here, imo.

I'm not sure what the application of this Makefile is, to be honest. There's one reference to the import_google_docs target in the README, but I don't see other references to the recipes in the code base (including deployment scripts). I think I'd like to update the Makefile in our import refactor so it has some useful targets for local development and eventual deployment on Heroku. I'll plan to make those changes in a separate PR.

fgregg · 2021-01-27T17:08:00Z

organization/search_indexes.py

+
+class OrganizationIndex(SearchEntity, indexes.Indexable):
+
+    '''


what is this comment doing for us?

Ah, I included these as a reference for what needs to go into each index. I think it's safe to remove!

fgregg · 2021-01-27T17:10:51Z

organization/search_indexes.py

+        countries = set()
+
+        for division in prepared_data['division_ids']:
+            countries.update([country_name(division)])


i understand this is copy paste, but this seems like it should be

countries.add(country_name(division))

or even better

countries = {country_name(division) for division in prepared_data['division_ids']}

fgregg · 2021-01-27T17:19:09Z

person/search_indexes.py

+        countries = set()
+
+        for division in prepared_data['division_ids']:
+            countries.update([country_name(division)])


another weird use of update

fgregg · 2021-01-27T17:19:21Z

person/search_indexes.py

+        person_division_id = object.division_id.get_value()
+
+        if person_division_id:
+            division_ids.update([person_division_id.value])


another weird use of update

fgregg · 2021-01-27T17:20:30Z

person/search_indexes.py

+                org_division_id = org.division_id.get_value()
+
+                if org_division_id:
+                    division_ids.update([org_division_id.value])


another weird use of update

fgregg · 2021-01-27T17:20:54Z

person/search_indexes.py

+
+        memberships = [mem.object_ref for mem in object.memberships]
+
+        if memberships:


don't need this conditional

fgregg · 2021-01-27T17:22:43Z

search/management/commands/update_composition_index.py

+            name = organization.name.get_value()
+
+            parents = organization.parent_organization.all()
+            published = all([p.value.published for p in parents])


this can be all(p.value...), don't need the list comprehension

hancush · 2021-01-27T20:14:28Z

Okey doke, @fgregg, I think I responded to and/or addressed all of your comments. Do you want to take another look?

hancush · 2021-01-28T19:37:26Z

TODO:

Retain filters when executing keyword search (currently removes when submitting form)
Investigate poor results (see below)

if I search "22 Batallón de Infantería" with a filter set for "Units" and "Mexico" I expect it would narrow down the results, but it doesn't do that:

https://back.securityforcemonitor.org/en/search/?q=22%20Batall%C3%B3n%20de%20Infanter%C3%ADa&end_date=&start_date=&entity_type=Organization&selected_facets=countries_exact%3AMexico

It returns 1006 results, and "22 Batallón de Infantería" does not appear at or near the top of those results.

hancush · 2021-03-05T15:00:19Z

Superseded by #746 (rebase, plus additional work to patch bugs).

hancush added 9 commits January 13, 2021 15:16

Update setup instructions

b072a65

Exclude dump files from VCS

106c78a

Stash Haystack progress

40810ea

Translate Organization index to Haystack

cbcb2a3

Set up /search/haystack view and update template clone to work with H…

612e146

…aystack

Remove page type from paginator

d910aa1

Fix typo in template, add missing prepare method to Org index

69ba238

Wrap Organization migration, simplify search UI

c5a1647

Migrate Person index to Haystack

89dc832

hancush force-pushed the hcg/haystack branch from 7443d05 to 89dc832 Compare January 21, 2021 21:24

hancush added 8 commits January 21, 2021 15:28

Fix icon toggle

731fcfb

Add violation index

0c62a1f

Continue Violation migration

9fd67a4

Finish Violation migration

985bda1

Add Source index, use plural for multi-value fields

ed9ecb6

Continue use of plural

cdc02e8

Add Source search

77ad571

Start cleaning up unused code

f9967a3