Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use admin IDs to resolve entities during import, add validation for conflicts in entity maps #777

Merged
merged 19 commits into from
Sep 27, 2021
Merged

Use admin IDs to resolve entities during import, add validation for conflicts in entity maps #777

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
5493b89
Use admin IDs provided in sheet instead of entity mapping to resolve …
hancush Sep 15, 2021
edb5f8e
Update test fixtures to handle differences between UUIDs and entity n…
Sep 15, 2021
ad40887
Merge pull request #778 from security-force-monitor/update-importer-t…
hancush Sep 15, 2021
4a6aea9
Debug failing source test, add scaffolding for new tests
hancush Sep 15, 2021
f72d610
Add test for entities sharing a name
hancush Sep 16, 2021
d22f52a
Add test that all data is sourced, docs for source assignment
hancush Sep 20, 2021
b0f23a8
Log an error if two records sharing a UUID have different names
hancush Sep 20, 2021
d82c30a
Skip failing violation tests, pending response
hancush Sep 20, 2021
0cc5707
Clean up implementation and comments
hancush Sep 21, 2021
e182853
Add V1 of EntityMap
hancush Sep 22, 2021
645b231
Add V2 of EntityMap
hancush Sep 22, 2021
1e54ea2
Add docstrings, break logging into its own method
hancush Sep 22, 2021
9f65c4f
Expand logging test to cover cases where name has more than one UUID
hancush Sep 22, 2021
f8ac153
Please run CI
hancush Sep 22, 2021
b2509fd
Merge branch 'hcg/import-patches' into hcg/entity-resolution
hancush Sep 22, 2021
c8b9813
Format from kwargs
hancush Sep 22, 2021
ad9d10b
Use nested structure in entity map
hancush Sep 27, 2021
4d0d98b
Minor copyedit to docstring
hancush Sep 27, 2021
1a13346
Tighten up values check
hancush Sep 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on:
pull_request:
branches:
- master
- hcg/translation

jobs:
test:
Expand Down
237 changes: 143 additions & 94 deletions sfm_pc/management/commands/import_google_doc.py
View file
Open in desktop

Large diffs are not rendered by default.

20 changes: 12 additions & 8 deletions tests/fixtures/importer/persons.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
"person:owner:admin","person:status:admin","person:comments:admin","person:id:admin","person:name","person:name:source","person:name:confidence","person:other_names","person:other_names:source","person:other_names:confidence","person:country","person:country:source","person:country:confidence","person:posting_unit_id:admin","person:posting_unit_name","person:posting_unit:source","person:posting_unit:confidence","person:posting_role","person:posting_role:source","person:posting_role:confidence","person:posting_title","person:posting_title:source","person:posting_title:confidence","person:posting_rank","person:posting_rank:source","person:posting_rank:confidence","person:posting_first_cited_date","person:posting_first_cited_date_month","person:posting_first_cited_date_day","person:posting_first_cited_date_year","person:posting_first_cited_date:source","person:posting_first_cited_date:confidence","person:posting_first_cited_date_start","person:posting_first_cited_date_start_context","person:posting_first_cited_date_start_context:source","person:posting_first_cited_date_start_context:confidence","person:posting_last_cited_date","person:posting_last_cited_date_month","person:posting_last_cited_date_day","person:posting_last_cited_date_year","person:posting_last_cited_date:source","person:posting_last_cited_date:confidence","person:posting_last_cited_date_end","person:posting_last_cited_date_end_context","person:posting_last_cited_date_end_context:source","person:posting_last_cited_date_end_context:confidence","person:notes:admin"
"Importer Test owner (Importer Test Person Alpha Name)",3,"Importer Test Comment (Importer Test Person Alpha Name)","3d55875b-a7db-46e3-83ad-f4feea4c74cc","Importer Test Person Alpha Name","c71b062c-63a3-47b8-9168-905004c66efd","Low","Importer Test Person Alpha Other Name 1; Importer Test Person Alpha Other Name 2","02d44aee-d4d5-4fe0-86a0-8871bf9f4ef7;b3d423d1-c1fb-40b8-864c-ee4219d9dcbe","Medium","tc","896b9eda-17ed-4df0-83c0-515bf9b101d9","Low","494a58d2-93e5-4454-9c08-74ac97c184da","Importer Test Organization Alpha Name ","a4a80e20-647f-4d39-9bf3-b5ae359afb20","Low","Test Role (Importer Test Person Alpha Name in Importer Test Organization Alpha Name)","50b7bb32-3f82-46c7-ad75-7d9f36588ff4","Medium",,,,,,,"2018-02-01",2,1,2018,"99cf742a-a39e-4796-b1a6-31abd276952c","Low","Y",,,,"2019-05-01",5,1,2019,"decf3e18-744e-4c38-a066-aec12aea4d03","High","N",,,,
"Importer Test owner (Importer Test Person Beta Name)",3,"Importer Test Comment (Importer Test Person Beta Name)","5fdd09e9-27f1-4dc3-a92f-a022e6730543","Importer Test Person Beta Name","92d97d71-86d4-4993-8e70-037b96960d25","Medium","Importer Test Person Beta Other Name 1; Importer Test Person Beta Other Name 2","6a2f641a-22a9-46f5-8046-f39f2bd0a093;94abc8a4-39ef-45f9-94f9-12c0fbc29d12","Low","tc","10c24fa2-9190-4eec-92ea-cd929fd108bc","Medium","217f93b8-bf71-4c97-a1a5-d5b3aa6c4896","Importer Test Organization Beta Name ","c59e16ab-8db1-45e8-b8ba-45b24dd19908","Medium","Test Role (Importer Test Person Beta Name in Importer Test Organization Beta Name)","d49bb448-cc54-4a27-b600-4154415a3f6d","Low",,,,,,,"2018-02-01",2,1,2018,"c7bfc896-3eae-4bf8-836a-83bba7ed5cd1","Medium","N",,,,"2019-05-01",5,1,2019,"174148da-2c72-4dca-abaa-88fc835c38c0","Medium","Y",,,,
"Importer Test owner (Importer Test Person Gamma Name)",3,"Importer Test Comment (Importer Test Person Gamma Name)","45cbd39d-0011-42b6-b46b-eb6ab0a9006e","Importer Test Person Gamma Name","23f333e4-9009-411c-879e-4180d95f147c","High","Importer Test Person Gamma Other Name 1; Importer Test Person Gamma Other Name 2","3f7a78cf-d44d-4581-9383-91fa01fc112a;4027e2d8-e608-47ec-9637-a8a64bf0fe00","High","tc","75df9010-f2e7-4bc0-891c-9d8e64f6cb83","High","60da42d0-8111-4518-80df-3fedec91cd90","Importer Test Organization Gamma Name ","45c60ce6-278a-4f97-befe-5b1d84b1d0fa","Medium","Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name)","52b1a585-7f80-4193-a93b-da5c971a3e49","Medium",,,,,,,"2018-02-01",2,1,2018,"dd5589b5-713a-4850-bfdd-87abf2610fb9","High","Y",,,,"2019-05-01",5,1,2019,"c124b8b9-a049-428c-96f2-b48998e5def2","Low","N",,,,
"Importer Test owner (Importer Test Person Delta Name)",3,"Importer Test Comment (Importer Test Person Delta Name)","08700499-9b00-4f92-81e2-50c1438bf073","Importer Test Person Delta Name","fda47d05-7ede-4cf4-a5ea-de67f746d27a","Low","Importer Test Person Delta Other Name 1; Importer Test Person Delta Other Name 2","79aed6f6-92c2-4e2b-a223-ad8fa1031383;d8324c27-bc69-4a61-9270-de0772d26433","Low","tc","bf89efcf-2b43-4097-a1c3-19524937ad31","Low","acb6ea98-97b3-41e1-8df2-903be407e907","Importer Test Organization Delta Name ","801f1e85-6ae8-45ed-901b-cee6bba3443f","High","Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name)","1001d2d1-62e9-46a8-9383-8026b95031b7","High",,,,,,,"2018-02-01",2,1,2018,"473f10de-463b-4997-9aab-f89251c746d3","Low","N",,,,"2019-05-01",5,1,2019,"d2b34e76-fcab-47ee-96a0-417dcaf51f33","High","Y",,,,
"Importer Test owner (Importer Test Person Not Ready Name)",1,"This persons does not have a status of 3 so it should not be imported","73f7fa8c-fcc6-437d-bbec-9d1c30bf47b0","Importer Test Person Not Ready Name",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"Importer Test owner (Importer Test Person Duplicate Posting)",3,"This person is a duplicate posting for Delta and should update the existing one","08700499-9b00-4f92-81e2-50c1438bf073","Importer Test Person Delta Name","fda47d05-7ede-4cf4-a5ea-de67f746d27a","Low","Importer Test Person Delta Other Name 1; Importer Test Person Delta Other Name 2","79aed6f6-92c2-4e2b-a223-ad8fa1031383;d8324c27-bc69-4a61-9270-de0772d26433","Low","tc","bf89efcf-2b43-4097-a1c3-19524937ad31","Low","acb6ea98-97b3-41e1-8df2-903be407e907","Importer Test Organization Delta Name ","801f1e85-6ae8-45ed-901b-cee6bba3443f","High","Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name)","1001d2d1-62e9-46a8-9383-8026b95031b7","High",,,,,,,"2018-02-01",2,1,2018,"473f10de-463b-4997-9aab-f89251c746d3","Low","N",,,,"2019-05-01",5,1,2019,"d2b34e76-fcab-47ee-96a0-417dcaf51f33","High","Y",,,,
"Importer Test owner (Importer Test Person Named Perpetrator)",3,"This person is a named perpetrator in an incident","83d1e012-8770-4885-a18e-1d78ecde6a51","Test Incident perpetrator (Test Incident Alpha)","fda47d05-7ede-4cf4-a5ea-de67f746d27a","Low",,,,"tc","bf89efcf-2b43-4097-a1c3-19524937ad31","Low","494a58d2-93e5-4454-9c08-74ac97c184da","Importer Test Organization Alpha Name ","a4a80e20-647f-4d39-9bf3-b5ae359afb20","Low","Test Role (Test Incident perpetrator (Test Incident Alpha)","1001d2d1-62e9-46a8-9383-8026b95031b7","High",,,,,,,"2018-02-01",2,1,2018,"473f10de-463b-4997-9aab-f89251c746d3","Low","N",,,,"2019-05-01",5,1,2019,"d2b34e76-fcab-47ee-96a0-417dcaf51f33","High","Y",,,,
person:owner:admin,person:status:admin,person:comments:admin,person:id:admin,person:name,person:name:source,person:name:confidence,person:other_names,person:other_names:source,person:other_names:confidence,person:country,person:country:source,person:country:confidence,person:posting_unit_id:admin,person:posting_unit_name,person:posting_unit:source,person:posting_unit:confidence,person:posting_role,person:posting_role:source,person:posting_role:confidence,person:posting_title,person:posting_title:source,person:posting_title:confidence,person:posting_rank,person:posting_rank:source,person:posting_rank:confidence,person:posting_first_cited_date,person:posting_first_cited_date_month,person:posting_first_cited_date_day,person:posting_first_cited_date_year,person:posting_first_cited_date:source,person:posting_first_cited_date:confidence,person:posting_first_cited_date_start,person:posting_first_cited_date_start_context,person:posting_first_cited_date_start_context:source,person:posting_first_cited_date_start_context:confidence,person:posting_last_cited_date,person:posting_last_cited_date_month,person:posting_last_cited_date_day,person:posting_last_cited_date_year,person:posting_last_cited_date:source,person:posting_last_cited_date:confidence,person:posting_last_cited_date_end,person:posting_last_cited_date_end_context,person:posting_last_cited_date_end_context:source,person:posting_last_cited_date_end_context:confidence,person:notes:admin
Importer Test owner (Importer Test Person Alpha Name),3,Importer Test Comment (Importer Test Person Alpha Name),3d55875b-a7db-46e3-83ad-f4feea4c74cc,Importer Test Person Alpha Name,c71b062c-63a3-47b8-9168-905004c66efd,Low,Importer Test Person Alpha Other Name 1; Importer Test Person Alpha Other Name 2,02d44aee-d4d5-4fe0-86a0-8871bf9f4ef7;b3d423d1-c1fb-40b8-864c-ee4219d9dcbe,Medium,tc,896b9eda-17ed-4df0-83c0-515bf9b101d9,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,a4a80e20-647f-4d39-9bf3-b5ae359afb20,Low,Test Role (Importer Test Person Alpha Name in Importer Test Organization Alpha Name),50b7bb32-3f82-46c7-ad75-7d9f36588ff4,Medium,,,,,,,2018-02-01,2,1,2018,99cf742a-a39e-4796-b1a6-31abd276952c,Low,Y,,,,2019-05-01,5,1,2019,decf3e18-744e-4c38-a066-aec12aea4d03,High,N,,,,
Importer Test owner (Importer Test Person Beta Name),3,Importer Test Comment (Importer Test Person Beta Name),5fdd09e9-27f1-4dc3-a92f-a022e6730543,Importer Test Person Beta Name,92d97d71-86d4-4993-8e70-037b96960d25,Medium,Importer Test Person Beta Other Name 1; Importer Test Person Beta Other Name 2,6a2f641a-22a9-46f5-8046-f39f2bd0a093;94abc8a4-39ef-45f9-94f9-12c0fbc29d12,Low,tc,10c24fa2-9190-4eec-92ea-cd929fd108bc,Medium,217f93b8-bf71-4c97-a1a5-d5b3aa6c4896,Importer Test Organization Beta Name ,c59e16ab-8db1-45e8-b8ba-45b24dd19908,Medium,Test Role (Importer Test Person Beta Name in Importer Test Organization Beta Name),d49bb448-cc54-4a27-b600-4154415a3f6d,Low,,,,,,,2018-02-01,2,1,2018,c7bfc896-3eae-4bf8-836a-83bba7ed5cd1,Medium,N,,,,2019-05-01,5,1,2019,174148da-2c72-4dca-abaa-88fc835c38c0,Medium,Y,,,,
Importer Test owner (Importer Test Person Gamma Name),3,Importer Test Comment (Importer Test Person Gamma Name),45cbd39d-0011-42b6-b46b-eb6ab0a9006e,Importer Test Person Gamma Name,23f333e4-9009-411c-879e-4180d95f147c,High,Importer Test Person Gamma Other Name 1; Importer Test Person Gamma Other Name 2,3f7a78cf-d44d-4581-9383-91fa01fc112a;4027e2d8-e608-47ec-9637-a8a64bf0fe00,High,tc,75df9010-f2e7-4bc0-891c-9d8e64f6cb83,High,60da42d0-8111-4518-80df-3fedec91cd90,Importer Test Organization Gamma Name ,45c60ce6-278a-4f97-befe-5b1d84b1d0fa,Medium,Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name),52b1a585-7f80-4193-a93b-da5c971a3e49,Medium,,,,,,,2018-02-01,2,1,2018,dd5589b5-713a-4850-bfdd-87abf2610fb9,High,Y,,,,2019-05-01,5,1,2019,c124b8b9-a049-428c-96f2-b48998e5def2,Low,N,,,,
Importer Test owner (Importer Test Person Delta Name),3,Importer Test Comment (Importer Test Person Delta Name),08700499-9b00-4f92-81e2-50c1438bf073,Importer Test Person Delta Name,fda47d05-7ede-4cf4-a5ea-de67f746d27a,Low,Importer Test Person Delta Other Name 1; Importer Test Person Delta Other Name 2,79aed6f6-92c2-4e2b-a223-ad8fa1031383;d8324c27-bc69-4a61-9270-de0772d26433,Low,tc,bf89efcf-2b43-4097-a1c3-19524937ad31,Low,acb6ea98-97b3-41e1-8df2-903be407e907,Importer Test Organization Delta Name ,801f1e85-6ae8-45ed-901b-cee6bba3443f,High,Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name),1001d2d1-62e9-46a8-9383-8026b95031b7,High,,,,,,,2018-02-01,2,1,2018,473f10de-463b-4997-9aab-f89251c746d3,Low,N,,,,2019-05-01,5,1,2019,d2b34e76-fcab-47ee-96a0-417dcaf51f33,High,Y,,,,
Importer Test owner (Importer Test Person Not Ready Name),1,This persons does not have a status of 3 so it should not be imported,73f7fa8c-fcc6-437d-bbec-9d1c30bf47b0,Importer Test Person Not Ready Name,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Importer Test owner (Importer Test Person Duplicate Posting),3,This person is a duplicate posting for Delta and should update the existing one,08700499-9b00-4f92-81e2-50c1438bf073,Importer Test Person Delta Name,fda47d05-7ede-4cf4-a5ea-de67f746d27a,Low,Importer Test Person Delta Other Name 1; Importer Test Person Delta Other Name 2,79aed6f6-92c2-4e2b-a223-ad8fa1031383;d8324c27-bc69-4a61-9270-de0772d26433,Low,tc,bf89efcf-2b43-4097-a1c3-19524937ad31,Low,acb6ea98-97b3-41e1-8df2-903be407e907,Importer Test Organization Delta Name ,801f1e85-6ae8-45ed-901b-cee6bba3443f,High,Test Role (Importer Test Person Alpha name in Importer Test Organization Alpha Name),1001d2d1-62e9-46a8-9383-8026b95031b7,High,,,,,,,2018-02-01,2,1,2018,473f10de-463b-4997-9aab-f89251c746d3,Low,N,,,,2019-05-01,5,1,2019,d2b34e76-fcab-47ee-96a0-417dcaf51f33,High,Y,,,,
Importer Test owner (Importer Test Person Named Perpetrator),3,This person is a named perpetrator in an incident,83d1e012-8770-4885-a18e-1d78ecde6a51,Test Incident perpetrator (Test Incident Alpha),fda47d05-7ede-4cf4-a5ea-de67f746d27a,Low,,,,tc,bf89efcf-2b43-4097-a1c3-19524937ad31,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,a4a80e20-647f-4d39-9bf3-b5ae359afb20,Low,Test Role (Test Incident perpetrator (Test Incident Alpha),1001d2d1-62e9-46a8-9383-8026b95031b7,High,,,,,,,2018-02-01,2,1,2018,473f10de-463b-4997-9aab-f89251c746d3,Low,N,,,,2019-05-01,5,1,2019,d2b34e76-fcab-47ee-96a0-417dcaf51f33,High,Y,,,,
Importer Test owner,3,This person has a different name but the same UUID,cc4f157c-c863-49fc-87cf-beecfe0366ee,Importer Test Different Names for Same UUID,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,,,,tc,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,Test Role,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,,,,,,,,,,,,,,,,,,,,,,,,,,,
Importer Test owner,3,This person has a different name but the same UUID,cc4f157c-c863-49fc-87cf-beecfe0366ee,Importer Test Different Names for Same UUID,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,,,,tc,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,Test Role,134e8598-6fb3-4d61-a7f7-504b3f6221c4,Low,,,,,,,,,,,,,,,,,,,,,,,,,,,
Importer Test owner,3,This person has the same UUID as another person,d5567101-7101-4abc-ba69-cd73e17682ef,Importer Test Different UUIDs for Same Name,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,,,,tc,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,Test Role,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,,,,,,,,,,,,,,,,,,,,,,,,,,,
Importer Test owner,3,This person has the same UUID as another person,8db5c1a3-54cf-4daa-8698-fb2bd4479216,Importer Test Different UUIDs for Same Name,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,,,,tc,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,494a58d2-93e5-4454-9c08-74ac97c184da,Importer Test Organization Alpha Name ,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,Test Role,19c0f7cc-b242-40c5-aa16-0f3916c3b0ba,Low,,,,,,,,,,,,,,,,,,,,,,,,,,,
Loading