Skip to content

Commit e5ee20f

Browse files
committed
Improve import/export: batching, OptionSet, perf, CI/CD
- Add ExecuteMultipleRequest batching for import (configurable BatchSize, UI field, schema support) - Support Multi-Select OptionSet (OptionSetValueCollection) in export/import - Ensure deterministic XML export ordering (alphabetical attributes) - Optimize attribute filtering and deduplication for performance - Hoist metadata lookups out of inner loops - Fix CSV/text export off-by-one error and improve update error logging - Add GitHub Actions for CI build and release automation - Update solution, README, and nuspecs for new features and copyright
1 parent 87846da commit e5ee20f

12 files changed

Lines changed: 643 additions & 65 deletions

.github/workflows/build.yml

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: CI Build
2+
3+
on:
4+
push:
5+
branches:
6+
- '**'
7+
pull_request:
8+
branches:
9+
- master
10+
11+
jobs:
12+
build:
13+
runs-on: windows-latest
14+
15+
steps:
16+
- name: Checkout repository
17+
uses: actions/checkout@v4
18+
with:
19+
submodules: recursive
20+
21+
- name: Calculate version
22+
id: version
23+
shell: pwsh
24+
env:
25+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
26+
run: |
27+
$year = (Get-Date).Year
28+
$month = (Get-Date).Month
29+
$startOfMonth = (Get-Date -Day 1 -Hour 0 -Minute 0 -Second 0).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ")
30+
31+
$headers = @{
32+
Authorization = "Bearer $env:GITHUB_TOKEN"
33+
Accept = "application/vnd.github.v3+json"
34+
}
35+
$url = "https://api.github.com/repos/$env:GITHUB_REPOSITORY/actions/workflows/build.yml/runs?created=>=$startOfMonth&per_page=1"
36+
$response = Invoke-RestMethod -Uri $url -Headers $headers -ErrorAction Stop
37+
$rev = $response.total_count
38+
39+
$version = "1.$year.$month.$rev"
40+
"VERSION=$version" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
41+
Write-Host "Computed version: $version"
42+
43+
- name: Patch AssemblyInfo.cs
44+
shell: pwsh
45+
run: |
46+
$version = "${{ steps.version.outputs.VERSION }}"
47+
$file = "XTB\Properties\AssemblyInfo.cs"
48+
(Get-Content $file) `
49+
-replace 'AssemblyVersion\("[^"]*"\)', "AssemblyVersion(`"$version`")" `
50+
-replace 'AssemblyFileVersion\("[^"]*"\)', "AssemblyFileVersion(`"$version`")" |
51+
Set-Content $file
52+
Write-Host "Patched $file to version $version"
53+
54+
- name: Patch nuspec versions
55+
shell: pwsh
56+
run: |
57+
$version = "${{ steps.version.outputs.VERSION }}"
58+
$nuspecs = @(
59+
"XTB\ShuffleRunner.nuspec",
60+
"XTB\ShuffleBuilder.nuspec",
61+
"XTB\ShuffleDeployer.nuspec"
62+
)
63+
foreach ($nuspec in $nuspecs) {
64+
$path = Resolve-Path $nuspec
65+
[xml]$xml = Get-Content $path
66+
$xml.package.metadata.version = $version
67+
$xml.Save($path)
68+
Write-Host "Patched $nuspec to version $version"
69+
}
70+
71+
- name: Setup NuGet
72+
uses: NuGet/setup-nuget@v2
73+
74+
- name: NuGet restore
75+
run: nuget restore Rappen.XTB.Shuffle.sln
76+
77+
- name: Setup MSBuild
78+
uses: microsoft/setup-msbuild@v2
79+
80+
- name: Build solution
81+
run: msbuild Rappen.XTB.Shuffle.sln /p:Configuration=Release /p:Platform="Any CPU" /m
82+
83+
- name: NuGet pack
84+
shell: pwsh
85+
run: |
86+
New-Item -ItemType Directory -Force -Path nupkg | Out-Null
87+
nuget pack "XTB\ShuffleRunner.nuspec" -OutputDirectory nupkg
88+
nuget pack "XTB\ShuffleBuilder.nuspec" -OutputDirectory nupkg
89+
nuget pack "XTB\ShuffleDeployer.nuspec" -OutputDirectory nupkg
90+
91+
- name: Upload nupkg artifacts
92+
uses: actions/upload-artifact@v4
93+
with:
94+
name: nupkg-${{ steps.version.outputs.VERSION }}
95+
path: nupkg/*.nupkg

.github/workflows/release.yml

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: Release
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
run_id:
7+
description: 'CI run ID whose nupkg artifact to publish (find it in the Actions tab URL)'
8+
required: true
9+
10+
jobs:
11+
release:
12+
runs-on: windows-latest
13+
14+
steps:
15+
- name: Download nupkg artifact from CI run
16+
uses: actions/download-artifact@v4
17+
with:
18+
run-id: ${{ inputs.run_id }}
19+
github-token: ${{ secrets.GITHUB_TOKEN }}
20+
pattern: 'nupkg-*'
21+
path: nupkg
22+
merge-multiple: true
23+
24+
- name: Resolve version from nupkg filename
25+
id: version
26+
shell: pwsh
27+
run: |
28+
$pkg = Get-ChildItem nupkg\*.nupkg | Select-Object -First 1
29+
if (-not $pkg) { throw "No .nupkg found in artifact" }
30+
# filename example: Rappen.XrmToolBox.Shuffle.Runner.1.2026.4.5.nupkg
31+
$version = $pkg.BaseName -replace '^Rappen\.XrmToolBox\.Shuffle\.\w+\.', ''
32+
"VERSION=$version" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
33+
Write-Host "Releasing version: $version ($($pkg.Name))"
34+
35+
- name: Publish to NuGet.org
36+
run: nuget push nupkg\*.nupkg -ApiKey ${{ secrets.NUGET_API_KEY }} -Source https://api.nuget.org/v3/index.json -NonInteractive
37+
38+
- name: Create GitHub release
39+
uses: softprops/action-gh-release@v2
40+
with:
41+
tag_name: v${{ steps.version.outputs.VERSION }}
42+
name: Release ${{ steps.version.outputs.VERSION }}
43+
generate_release_notes: true
44+
files: nupkg/*.nupkg

README.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,24 @@ Improving by [@imranakram](https://github.com/imranakram) & [@rappen](https://gi
1414
### *Shuffle tools are now available in the XrmToolBox Tool Library!* 🥳
1515
---
1616

17+
## Recent Changes
18+
19+
### Multi-Select OptionSet support
20+
Export and import of Multi-Select OptionSet (OptionSetValueCollection) fields now works correctly. Previously, exported data.xml contained the literal string "OptionSetValueCollection" instead of actual values.
21+
22+
### ExecuteMultipleRequest batching
23+
Import operations (Create, Update, Delete) are now batched using `ExecuteMultipleRequest` for significantly improved performance on large datasets. Configurable via the `BatchSize` attribute on the Import element (default: 200, max: 1000). Set to 1 to disable batching. The Shuffle Builder UI includes a new "Batch size" field.
24+
25+
### Deterministic XML export ordering
26+
Entity attributes are now sorted alphabetically during export, eliminating spurious diffs in version control when re-exporting unchanged data.
27+
28+
### Bug fixes and performance improvements
29+
- Fixed off-by-one error in CSV/text export that could cause an IndexOutOfRangeException
30+
- Metadata lookups (PrimaryIdAttribute) hoisted out of inner loops to reduce overhead
31+
- Replaced O(n) list searches with HashSet for attribute deduplication during import
32+
- Replaced O(n^2) attribute filtering in SelectAttributes with single-pass LINQ approach
33+
- Update failures now log the exception message for easier diagnostics
34+
1735
## Home page
1836
https://jonasr.app/shuffle/
1937

Rappen.XTB.Shuffle.sln

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11

22
Microsoft Visual Studio Solution File, Format Version 12.00
3-
# Visual Studio Version 17
4-
VisualStudioVersion = 17.5.33516.290
3+
# Visual Studio Version 18
4+
VisualStudioVersion = 18.4.11626.88 stable
55
MinimumVisualStudioVersion = 10.0.40219.1
66
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Rappen.XTB.Shuffle", "XTB\Rappen.XTB.Shuffle.csproj", "{13AE5564-5C72-4A70-8AC5-00D227E8200A}"
77
EndProject
@@ -20,6 +20,14 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
2020
XTB\ShuffleRunner.nuspec = XTB\ShuffleRunner.nuspec
2121
EndProjectSection
2222
EndProject
23+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = ".github", ".github", "{D9752343-576F-48AD-A576-4CFC3499C7F4}"
24+
EndProject
25+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "workflows", "workflows", "{7F7BC1BA-C6C6-47B9-8D60-268822E0A334}"
26+
ProjectSection(SolutionItems) = preProject
27+
.github\workflows\build.yml = .github\workflows\build.yml
28+
.github\workflows\release.yml = .github\workflows\release.yml
29+
EndProjectSection
30+
EndProject
2331
Global
2432
GlobalSection(SolutionConfigurationPlatforms) = preSolution
2533
Debug|Any CPU = Debug|Any CPU
@@ -37,6 +45,7 @@ Global
3745
GlobalSection(NestedProjects) = preSolution
3846
{5C10BFE2-AFAA-4B01-A570-B30EF41DE1F0} = {44DE50B5-DA2B-4BD6-9D10-8BB345F68226}
3947
{A939CF3B-672A-4F68-8E6A-89EFE8C8CFBB} = {17CCEFED-E37B-47CF-BD9F-E7596E65FCE9}
48+
{7F7BC1BA-C6C6-47B9-8D60-268822E0A334} = {D9752343-576F-48AD-A576-4CFC3499C7F4}
4049
EndGlobalSection
4150
GlobalSection(ExtensibilityGlobals) = postSolution
4251
SolutionGuid = {ED295346-E5B5-4006-855E-1100CEB0F456}

XTB/Builder/Controls/DataBlockImportControl.Designer.cs

Lines changed: 41 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

XTB/ShuffleBuilder.nuspec

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,14 @@
1919
</description>
2020
<summary>Build schema files for the Shuffle. Empower yourself to achieve more.</summary>
2121
<releaseNotes>
22-
Updated dependencies and bug fixes.
22+
- New Batch size field on Import configuration for ExecuteMultipleRequest batching
23+
- Multi-Select OptionSet (OptionSetValueCollection) export/import support
24+
- Deterministic attribute ordering in XML export for clean version control diffs
25+
- Fixed off-by-one error in CSV/text export
26+
- Performance improvements: metadata lookup hoisting, HashSet deduplication, optimized attribute filtering
27+
- Update failure messages now include exception details
2328
</releaseNotes>
24-
<copyright>Copyright 2023-2025 Jonas Rapp, Imran Akram</copyright>
29+
<copyright>Copyright 2023-2026 Jonas Rapp, Imran Akram</copyright>
2530
<tags>XrmToolBox Shuffle</tags>
2631
<dependencies>
2732
<dependency id="XrmToolBox" version="1.2025.10.74" />

XTB/ShuffleDeployer.nuspec

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,12 @@
1919
</description>
2020
<summary>Deploy solutions and datas with the Shuffle. Empower yourself to achieve more.</summary>
2121
<releaseNotes>
22-
Updated dependencies and bug fixes.
22+
- ExecuteMultipleRequest batching for import operations (configurable BatchSize, default 200)
23+
- Multi-Select OptionSet (OptionSetValueCollection) export/import support
24+
- Deterministic attribute ordering in XML export for clean version control diffs
25+
- Fixed off-by-one error in CSV/text export
26+
- Performance improvements: metadata lookup hoisting, HashSet deduplication, optimized attribute filtering
27+
- Update failure messages now include exception details
2328
</releaseNotes>
2429
<copyright>Copyright 2023-2026 Jonas Rapp, Imran Akram</copyright>
2530
<tags>XrmToolBox Shuffle</tags>

XTB/ShuffleRunner.nuspec

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,12 @@
1919
</description>
2020
<summary>Export and Import with the Shuffle. Empower yourself to achieve more.</summary>
2121
<releaseNotes>
22-
Updated dependencies and bug fixes.
22+
- ExecuteMultipleRequest batching for import operations (configurable BatchSize, default 200)
23+
- Multi-Select OptionSet (OptionSetValueCollection) export/import support
24+
- Deterministic attribute ordering in XML export for clean version control diffs
25+
- Fixed off-by-one error in CSV/text export
26+
- Performance improvements: metadata lookup hoisting, HashSet deduplication, optimized attribute filtering
27+
- Update failure messages now include exception details
2328
</releaseNotes>
2429
<copyright>Copyright 2023-2026 Jonas Rapp, Imran Akram</copyright>
2530
<tags>XrmToolBox Shuffle</tags>

shared/Xrm.Shuffle.Core/Resources/ShuffleDefinition.cs

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,17 +87,23 @@ public partial class DataBlockImport {
8787
/// <remarks/>
8888
[System.Xml.Serialization.XmlAttributeAttribute()]
8989
public bool Overwrite;
90-
90+
9191
/// <remarks/>
9292
[System.Xml.Serialization.XmlIgnoreAttribute()]
9393
public bool OverwriteSpecified;
94-
94+
95+
/// <summary>Number of records per ExecuteMultipleRequest batch. Set to 1 to disable batching. Max 1000.</summary>
96+
[System.Xml.Serialization.XmlAttributeAttribute()]
97+
[System.ComponentModel.DefaultValueAttribute(200)]
98+
public int BatchSize;
99+
95100
public DataBlockImport() {
96101
this.CreateWithId = false;
97102
this.Save = SaveTypes.CreateUpdate;
98103
this.Delete = DeleteTypes.None;
99104
this.UpdateInactive = false;
100105
this.UpdateIdentical = false;
106+
this.BatchSize = 200;
101107
}
102108
}
103109

shared/Xrm.Shuffle.Core/Resources/ShuffleDefinition.xsd

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,11 @@
263263
<xs:documentation>DEPRECATED. Use Save attribute instead.</xs:documentation>
264264
</xs:annotation>
265265
</xs:attribute>
266-
<!--<xs:attribute name="ExecuteMultiple" type="xs:boolean" use="optional" default="false" />-->
266+
<xs:attribute name="BatchSize" type="xs:int" use="optional" default="200">
267+
<xs:annotation>
268+
<xs:documentation>Number of records per ExecuteMultipleRequest batch. Set to 1 to disable batching. Max 1000.</xs:documentation>
269+
</xs:annotation>
270+
</xs:attribute>
267271
</xs:complexType>
268272

269273

0 commit comments

Comments
 (0)