Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AC-1682] Flexible collections: data migrations for deprecated permissions #3437

Merged
merged 48 commits into from
Jan 25, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
e14ba4d
[AC-1682] Data migrations for OrgUsers or Groups with AccessAll enabled
r-tome Nov 10, 2023
1527af7
[AC-1682] Added script to update [dbo].[CollectionUser] with [Manage]…
r-tome Nov 10, 2023
577e591
[AC-1682] Updated sql data migration procedures with performance reco…
r-tome Nov 13, 2023
23a3e5c
[AC-1682] Moved data migration scripts to DbScripts_transition folder
r-tome Nov 13, 2023
d693511
Apply suggestions from code review: Remove Manage permission from Col…
r-tome Dec 6, 2023
0475b93
[AC-1682] Removed unnecessary Collection table join on ManagersEditAs…
r-tome Dec 6, 2023
3e7e3a0
[AC-1682] Change JOIN to INNER JOIN in SQL scripts
r-tome Dec 6, 2023
be35a2b
[AC-1682] Renamed sql script to recent date and added correct order t…
r-tome Dec 6, 2023
b7773b6
Merge branch 'master' into ac/ac-1682/data-migrations-for-deprecated-…
r-tome Dec 6, 2023
7d718b6
[AC-1682] Add new rows to CollectionUser for Managers and users with …
r-tome Dec 18, 2023
69c0997
Merge branch 'main' into ac/ac-1682/data-migrations-for-deprecated-pe…
r-tome Dec 18, 2023
e7ce15f
Merge branch 'main' into ac/ac-1682/data-migrations-for-deprecated-pe…
r-tome Jan 9, 2024
fcd2dd3
[AC-1682] Update FC data migration scripts to clear AccessAll flags a…
r-tome Jan 10, 2024
8bf17eb
[AC-1682] Updated data migration scripts to bump the account revision…
r-tome Jan 10, 2024
dfc4032
[AC-1682] Created Organization_EnableCollectionEnhancements to migrat…
r-tome Jan 10, 2024
0d8609a
[AC-1682] Added script to migrate all organization data for flexible …
r-tome Jan 10, 2024
54cc6fa
[AC-1682] Deleted old data migration scripts
r-tome Jan 10, 2024
a214c60
Revert "[AC-1682] Deleted old data migration scripts"
r-tome Jan 11, 2024
ee43767
[AC-1682] Modified AccessAllCollectionUsers script to bump revision d…
r-tome Jan 11, 2024
71978fe
Merge branch 'main' into ac/ac-1682/data-migrations-for-deprecated-pe…
r-tome Jan 11, 2024
f4450c0
[AC-1682] Update data migration script to only enable collection enha…
r-tome Jan 11, 2024
8cc8895
[AC-1682] Updated AccessAllCollectionGroups migration script to use U…
r-tome Jan 11, 2024
8a5ea76
[AC-1682] Bumped up the date on data migration scripts
r-tome Jan 11, 2024
68c586f
[AC-1682] Added back batching system to AccessAllCollectionUsers data…
r-tome Jan 11, 2024
ba06076
[AC-1682] Added data migration script to set FlexibleCollections = 1 …
r-tome Jan 11, 2024
f78e28f
[AC-1682] Modified data migration script to contain multiple transact…
r-tome Jan 11, 2024
3bf1b53
[AC-1682] Deleted old data migration scripts
r-tome Jan 11, 2024
c4ad7d7
[AC-1682] Placed temp tables outside transactions
r-tome Jan 11, 2024
96b5278
[AC-1682] Removed batching from AllOrgsEnableCollectionEnhancements s…
r-tome Jan 11, 2024
667f3cd
Merge branch 'main' into ac/ac-1682/data-migrations-for-deprecated-pe…
r-tome Jan 12, 2024
7a8cbd6
Merge branch 'main' into ac/ac-1682/data-migrations-for-deprecated-pe…
r-tome Jan 17, 2024
d1a8720
[AC-1682] Removed bulk data migration script
r-tome Jan 17, 2024
e651d1f
[AC-1682] Refactor stored procedure to enable collection enhancements
r-tome Jan 17, 2024
95977b1
[AC-1682] Added missing where clause
r-tome Jan 17, 2024
160fc3e
[AC-1682] Modified data migration script to have just one big transac…
r-tome Jan 18, 2024
6a519b9
[AC-1682] Combining all updated OrganizationUserIds to bump all revis…
r-tome Jan 18, 2024
6579ede
Update src/Sql/dbo/Stored Procedures/Organization_EnableCollectionEnh…
r-tome Jan 22, 2024
ea244df
[AC-1682] Renamed aliases
r-tome Jan 22, 2024
7dbb9e6
Merge branch 'ac/ac-1682/data-migrations-for-deprecated-permissions' …
r-tome Jan 22, 2024
0144c25
[AC-1682] Simplified inner queries
r-tome Jan 23, 2024
1e2a15d
[AC-1682] Bumping each modified groups RevisionDate
r-tome Jan 23, 2024
086c88f
[AC-1682] Removed updating CollectionUser existing records with [Read…
r-tome Jan 23, 2024
6fa4687
[AC-1682] Updating OrganizationUser RevisionDate
r-tome Jan 23, 2024
f1fb994
[AC-1682] Updated the stored procedure file
r-tome Jan 23, 2024
ebecc69
[AC-1682] Selecting distinct values to insert into CollectionUser table
r-tome Jan 24, 2024
737b81f
Revert "[AC-1682] Removed updating CollectionUser existing records wi…
r-tome Jan 25, 2024
5cc8912
[AC-1682] Bumped up the date on the migration script
r-tome Jan 25, 2024
f14e04c
[AC-1682] Updating OrganizationUser RevisionDate
r-tome Jan 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
-- Create a temporary table to store the groups with AccessAll = 1
SELECT [Id] AS [GroupId], [OrganizationId]
INTO #TempGroup
FROM [dbo].[Group]
WHERE [AccessAll] = 1;

-- Update existing rows in [dbo].[CollectionGroup]
UPDATE CG
SET
CG.[ReadOnly] = 0,
CG.[HidePasswords] = 0,
CG.[Manage] = 1
r-tome marked this conversation as resolved.
Show resolved Hide resolved
FROM [dbo].[CollectionGroup] CG
INNER JOIN [dbo].[Collection] C ON CG.[CollectionId] = C.[Id]
INNER JOIN #TempGroup TG ON CG.[GroupId] = TG.[GroupId]
WHERE C.[OrganizationId] = TG.[OrganizationId];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure, but is this WHERE clause doing anything?

The INNER JOIN ensures that records exist in both tables, so if we INNER JOIN CollectionGroup, Collection and TempGroup, we should only end up with CollectionGroup records for Groups where AccessAll = 1. Then we can update the values without reference to the orgId. I think.

That said, I'm happy to leave it if that's safer or if there's another reason.

@rkac-bw

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, it also seems redundant to me, but perhaps its for performance reasons.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No performance gain with where clause, if does not add anything then it can be removed


-- Insert new rows into [dbo].[CollectionGroup]
INSERT INTO [dbo].[CollectionGroup] ([CollectionId], [GroupId], [ReadOnly], [HidePasswords], [Manage])
SELECT C.[Id], TG.[GroupId], 0, 0, 1
r-tome marked this conversation as resolved.
Show resolved Hide resolved
FROM [dbo].[Collection] C
JOIN #TempGroup TG ON C.[OrganizationId] = TG.[OrganizationId]
r-tome marked this conversation as resolved.
Show resolved Hide resolved
LEFT JOIN [dbo].[CollectionGroup] CG ON CG.[CollectionId] = C.[Id] AND CG.[GroupId] = TG.[GroupId]
WHERE CG.[CollectionId] IS NULL;

-- Drop the temporary table
DROP TABLE #TempGroup;
r-tome marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
-- Step 1: Insert into a temporary table with an additional column for batch processing, update 50 k at a time
SELECT [Id] AS [OrganizationUserId], [OrganizationId], CAST(ROW_NUMBER() OVER(ORDER BY [Id]) / 50000 AS INT) AS Batch
INTO #TempOrgUser
FROM [dbo].[OrganizationUser]
WHERE [AccessAll] = 1;

-- Step 2: Get the maximum batch number
DECLARE @MaxBatch INT = (SELECT MAX(Batch) FROM #TempOrgUser);
DECLARE @CurrentBatch INT = 0;

-- Step 3: Process each batch
WHILE @CurrentBatch <= @MaxBatch
BEGIN
-- Update existing rows in [dbo].[CollectionUser]
UPDATE target
SET
target.[ReadOnly] = 0,
target.[HidePasswords] = 0,
target.[Manage] = 1
r-tome marked this conversation as resolved.
Show resolved Hide resolved
FROM [dbo].[CollectionUser] AS target
JOIN (
SELECT C.[Id] AS [CollectionId], T.[OrganizationUserId]
FROM [dbo].[Collection] C
JOIN #TempOrgUser T ON C.[OrganizationId] = T.[OrganizationId] AND T.Batch = @CurrentBatch
) AS source
ON target.[CollectionId] = source.[CollectionId] AND target.[OrganizationUserId] = source.[OrganizationUserId];

-- Insert new rows into [dbo].[CollectionUser]
INSERT INTO [dbo].[CollectionUser] ([CollectionId], [OrganizationUserId], [ReadOnly], [HidePasswords], [Manage])
SELECT source.[CollectionId], source.[OrganizationUserId], 0, 0, 1
r-tome marked this conversation as resolved.
Show resolved Hide resolved
FROM (
SELECT C.[Id] AS [CollectionId], T.[OrganizationUserId]
FROM [dbo].[Collection] C
JOIN #TempOrgUser T ON C.[OrganizationId] = T.[OrganizationId] AND T.Batch = @CurrentBatch
) AS source
LEFT JOIN [dbo].[CollectionUser] AS target
ON target.[CollectionId] = source.[CollectionId] AND target.[OrganizationUserId] = source.[OrganizationUserId]
WHERE target.[CollectionId] IS NULL;

-- Move to the next batch
SET @CurrentBatch = @CurrentBatch + 1;
END;

-- Step 4: Drop the temporary table
DROP TABLE #TempOrgUser;
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
-- Update [dbo].[CollectionUser] with [Manage] = 1 for all users with Manager role or 'EditAssignedCollections' permission
UPDATE cu
SET cu.[ReadOnly] = 0,
cu.[HidePasswords] = 0,
cu.[Manage] = 1
FROM [dbo].[CollectionUser] cu
JOIN [dbo].[Collection] c
ON cu.[CollectionId] = c.[Id]
r-tome marked this conversation as resolved.
Show resolved Hide resolved
JOIN [dbo].[OrganizationUser] ou
ON cu.[OrganizationUserId] = ou.[Id]
WHERE (ou.[Type] = 3 OR (ou.[Permissions] IS NOT NULL AND
ISJSON(ou.[Permissions]) > 0 AND JSON_VALUE(ou.[Permissions], '$.editAssignedCollections') = 'true'))
Loading