-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Merge Software by ID #55
Open
RickiJay-WMDE
wants to merge
11
commits into
main
Choose a base branch
from
merge-software
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
80c5c14
Merge Software by ID
RickiJay-WMDE 2544e15
Merge Data
RickiJay-WMDE 0a15fa9
Light Refactor
RickiJay-WMDE f0e15eb
Test
RickiJay-WMDE 16ade0c
Fix IDs
RickiJay-WMDE 19c8d55
Tags
RickiJay-WMDE b49d423
error
RickiJay-WMDE 1d06715
Rigor
RickiJay-WMDE e8223d8
Test Data
RickiJay-WMDE 04cea71
Organize
RickiJay-WMDE 6886918
Test Fail
RickiJay-WMDE File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
"""Update Data""" | ||
|
||
from fetch_data.update_data.merge_software import merge_software_by_id |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
"""Merge Software""" | ||
|
||
from sqlalchemy import Select, Update, and_, delete, select, update | ||
from data.database_connection import get_async_session | ||
from model.database import ( | ||
WikibaseSoftwareVersionModel, | ||
) | ||
from model.database import WikibaseSoftwareModel | ||
from model.database.wikibase_software.software_tag_xref_model import ( | ||
software_tag_xref_table, | ||
) | ||
|
||
|
||
async def merge_software_by_id(base_id: int, additional_id: int) -> bool: | ||
"""Merge Software by ID""" | ||
|
||
software_query = get_select_software_query([base_id, additional_id]) | ||
update_software_version_query = get_update_software_version_query( | ||
base_id, additional_id | ||
) | ||
update_software_tags_query = get_update_software_tags_query(base_id, additional_id) | ||
delete_additional_tags_query = software_tag_xref_table.delete().where( | ||
software_tag_xref_table.c.wikibase_software_id == additional_id | ||
) | ||
delete_software_query = delete(WikibaseSoftwareModel).where( | ||
WikibaseSoftwareModel.id == additional_id | ||
) | ||
|
||
async with get_async_session() as async_session: | ||
software_list = (await async_session.scalars(software_query)).all() | ||
assert len({s.software_type for s in software_list}) == 1 | ||
|
||
await async_session.execute(update_software_version_query) | ||
await async_session.execute(update_software_tags_query) | ||
await async_session.execute(delete_additional_tags_query) | ||
await async_session.flush() | ||
|
||
await async_session.execute(delete_software_query) | ||
await async_session.commit() | ||
|
||
async with get_async_session() as async_session: | ||
remaining = (await async_session.scalars(software_query)).all() | ||
return len(remaining) == 1 | ||
|
||
|
||
def get_select_software_query(id_list: list[int]) -> Select[WikibaseSoftwareModel]: | ||
"""Select WikibaseSoftwareModel in ID list""" | ||
|
||
software_query = select(WikibaseSoftwareModel).where( | ||
WikibaseSoftwareModel.id.in_(id_list) | ||
) | ||
|
||
return software_query | ||
|
||
|
||
def get_update_software_tags_query(base_id: int, additional_id: int) -> Update: | ||
"""Add Additional Software Tags to Base""" | ||
|
||
update_software_tags_query = software_tag_xref_table.insert().from_select( | ||
[ | ||
software_tag_xref_table.c.wikibase_software_id, | ||
software_tag_xref_table.c.wikibase_software_tag_id, | ||
], | ||
select(base_id, software_tag_xref_table.c.wikibase_software_tag_id).where( | ||
and_( | ||
software_tag_xref_table.c.wikibase_software_id == additional_id, | ||
software_tag_xref_table.c.wikibase_software_tag_id.not_in( | ||
select(software_tag_xref_table.c.wikibase_software_tag_id).where( | ||
software_tag_xref_table.c.wikibase_software_id == base_id | ||
) | ||
), | ||
) | ||
), | ||
) | ||
|
||
return update_software_tags_query | ||
|
||
|
||
def get_update_software_version_query(base_id: int, additional_id: int) -> Update: | ||
"""Update Software Version from Additional ID to Base ID""" | ||
|
||
update_software_version_query = ( | ||
update(WikibaseSoftwareVersionModel) | ||
.where(WikibaseSoftwareVersionModel.software_id == additional_id) | ||
.values(software_id=base_id) | ||
) | ||
|
||
return update_software_version_query |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
"""Test Merge Software""" | ||
|
||
import pytest | ||
|
||
from tests.test_schema import test_schema | ||
|
||
|
||
MERGE_SOFTWARE_QUERY = """ | ||
mutation MyMutation($baseId: Int!, $additionalId: Int!) { | ||
mergeSoftwareById(baseId: $baseId, additionalId: $additionalId) | ||
}""" | ||
|
||
|
||
@pytest.mark.asyncio | ||
@pytest.mark.mutation | ||
@pytest.mark.dependency(name="merge-software-by-id") | ||
async def test_merge_software_by_id_mutation(): | ||
"""Test Add Wikibase""" | ||
|
||
result = await test_schema.execute( | ||
MERGE_SOFTWARE_QUERY, variable_values={"baseId": 1, "additionalId": 3} | ||
) | ||
assert result.errors is None | ||
assert result.data is not None | ||
assert result.data.get("mergeSoftwareById") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do it see it correctly, that the merging is something that the user has to do manually?
Could you please shortly describe the workflow this feature would allow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct. This is strictly a manual process.
On identification of a duplicate extension - such as
Discord Notifications
with and without space, orMiraheze Magic
and<extensionname-mirahezemagic>
, both of which we've actually encountered in this project - this would allow us to merge the two records. We call the first record "base" and the second "additional", and pass in thebaseId
andadditionalId
.All of the software version records collected from wikibases that referred to the additional software would be shifted over to the base software. Any tags collected from Mediawiki for the additional would also be shifted to the base (avoiding duplicates, of course).