feat(hubs): add Azure Resource Manager ingestion - #2277
Closed
Brett Wilson (MSBrett) wants to merge 29 commits into
Closed
feat(hubs): add Azure Resource Manager ingestion#2277Brett Wilson (MSBrett) wants to merge 29 commits into
Brett Wilson (MSBrett) wants to merge 29 commits into
Conversation
Add generic ARM query ingestion with quota and Savings Plan definitions, normalized KQL surfaces, catalog queries, documentation, and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 24, 2026 00:28 — with 
GitHub Actions
Error
GitHub Actions
Error
microsoft-github-policy-service
Bot
added
the
Needs: Review 👀
Contributor
|
Hubs + ADX (managed) deployment failed. View logs. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds reusable Azure Resource Manager ingestion to FinOps hubs, supporting savings-plan recommendations and quota/capacity datasets.
Changes:
- Adds manifest-driven ARM GET ingestion and deployment wiring.
- Adds normalized quota and savings-plan schemas and KQL queries.
- Extends portal configuration, documentation, and regression tests.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/templates/finops-hub/test/main.test.bicep |
Adds feature deployment combinations. |
.../Recommendations/schemas/recommendations_1.1.json |
Defines savings-plan translation. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P3Y.json |
Adds three-year recommendations. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P1Y.json |
Adds one-year recommendations. |
.../Recommendations/app.bicep |
Registers new recommendation assets. |
.../Quota/schemas/quota_1.0-usage.json |
Defines usage translation. |
.../Quota/schemas/quota_1.0-sql.json |
Defines SQL quota translation. |
.../Quota/schemas/quota_1.0-disk.json |
Defines disk translation. |
.../Quota/schemas/quota_1.0-capacity-reservation.json |
Defines capacity translation. |
.../Quota/queries/Quota-Microsoft-StorageUsage.json |
Adds Storage usage ingestion. |
.../Quota/queries/Quota-Microsoft-SqlSubscriptionUsage.json |
Adds SQL usage ingestion. |
.../Quota/queries/Quota-Microsoft-PremiumSSDv2Disk.json |
Adds disk inventory ingestion. |
.../Quota/queries/Quota-Microsoft-ComputeUsage.json |
Adds compute usage ingestion. |
.../Quota/queries/Quota-Microsoft-CognitiveServicesUsage.json |
Adds AI quota ingestion. |
.../Quota/queries/Quota-Microsoft-CapacityReservation.json |
Adds capacity inventory ingestion. |
.../Quota/queries/Quota-Microsoft-AppServiceUsage.json |
Adds App Service usage ingestion. |
.../Quota/app.bicep |
Deploys quota queries and schemas. |
.../IngestionQueries/app.bicep |
Integrates ARM staging and finalization. |
.../AzureResourceManager/README.md |
Documents the ARM engine. |
.../AzureResourceManager/metadata.bicep |
Defines ARM app metadata. |
.../AzureResourceManager/app.bicep |
Implements ARM request pipelines. |
.../AzureResourceGraph/app.bicep |
Extends the shared engine contract. |
.../Analytics/scripts/IngestionSetup_v1_2.kql |
Normalizes savings-plan recommendations. |
.../Analytics/scripts/IngestionSetup_v1_0.kql |
Adds quota transforms and tables. |
.../Analytics/scripts/IngestionSetup_RawTables.kql |
Adds the quota raw table. |
.../Analytics/scripts/HubSetup_v1_0.kql |
Adds the versioned quota function. |
.../Analytics/scripts/HubSetup_Latest.kql |
Adds the latest quota alias. |
src/templates/finops-hub/modules/hub.bicep |
Wires ARM and quota apps. |
.../fx/hub-deploymentScript.bicep |
Forces deployment-script refreshes. |
src/templates/finops-hub/main.bicep |
Exposes quota configuration. |
src/templates/finops-hub/createUiDefinition.json |
Adds portal controls and guidance. |
src/scripts/Build-HubIngestionQueries.ps1 |
Generates manifests for both apps. |
src/queries/INDEX.md |
Catalogs the new queries. |
src/queries/finops-hub-database-guide.md |
Documents the quota contract. |
src/queries/catalog/savings-plan-recommendation-breakdown.kql |
Summarizes savings-plan recommendations. |
src/queries/catalog/quota-storage-usage.kql |
Queries Storage quota state. |
src/queries/catalog/quota-sql-subscription-usage.kql |
Queries SQL quota state. |
src/queries/catalog/quota-premium-ssd-v2-disks.kql |
Queries Premium SSD v2 inventory. |
src/queries/catalog/quota-headroom.kql |
Calculates quota headroom. |
src/queries/catalog/quota-current-usage.kql |
Queries current quota state. |
src/queries/catalog/quota-compute-usage.kql |
Queries compute quota state. |
src/queries/catalog/quota-cognitive-services-usage.kql |
Queries AI quota state. |
src/queries/catalog/quota-capacity-reservations.kql |
Queries capacity reservations. |
src/queries/catalog/quota-app-service-usage.kql |
Queries App Service quota state. |
src/powershell/Tests/Unit/HubsIngestionQueries.Tests.ps1 |
Extends ingestion regression coverage. |
.gitignore |
Ignores Copilot tracking artifacts. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Adds an eighth view to the FinOps hub dashboard canvas extension, porting the AI and emerging workloads page from the Azure Data Explorer dashboard into the extension's own query and rendering pipeline. Query layer (kusto.mjs): - getAi() supplies 16 named queries across estate composition, token and model economics, workload detail, allocation, and rate posture. - Month-over-month comparisons anchor to the last closed month. The newest month in a window is normally still ingesting, so comparing it against a full month reported a spend collapse that was not real. - Token direction uses word-boundary regexes rather than substring matching, so meters such as "Cd Wr Std" classify correctly instead of falling into an Other bucket, and a future "Throughput" meter cannot be read as output. - Owner tags fold case-insensitively while retaining the highest-cost casing for display, so one owner cannot appear as several rows. - Follows repo KQL guidance: no bare joins, and contains reserved for genuinely fused substrings. Rendering (public/): - Six KPI cards, two new chart primitives, and fifteen panels. - Formatters gain fmtRate, fmtQty, fmtShare, moneyColumn and rateColumn. Column helpers hold one precision per column, because a reader compares cells down a column rather than against each value's own magnitude, and mark sub-unit values rather than rounding real spend to zero. - Axis ceilings round up to readable steps and never clip the series. - Every table is wrapped so it scrolls within its panel. An unwrapped table previously forced horizontal scrolling on the whole page at narrow widths. - The active tab is scrolled into view, including on deep links. - Panels are paired by content height so no card renders mostly empty. Also fixes two latent defects reachable from existing views: table swatches were not displayed, and threshold colours were defined only under .kpi so they had no effect inside tables. Verified against a live hub across 1.2M rows, with no horizontal overflow at 1600, 1280, 950, 768 and 480 CSS pixels. Test suite covers the new behaviour and passes 34/34. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Quota answered only the demand side of capacity: how much of my limit have I used. It could not answer the supply side: can a deployment land in this region and zone at all. Two Azure Resource Manager quota queries close that gap. - Quota-Microsoft-AvailabilityZoneMapping records each subscription's logical-to-physical availability zone mapping, so zonal capacity is comparable across subscriptions. - Quota-Microsoft-ComputeResourceSku records VM SKU family, zone, and the region and zone restrictions that block placement regardless of quota. Both queries get a schema, a raw table, an ingestion transform, a Hub function, and a catalog query for rehydration. Fix rehydration filters that returned the wrong rows. Each catalog query now filters on x_SourceType, ResourceType, and ResourceName together instead of x_SourceType alone. Premium SSD v2 and disk quota come from the Compute provider but describe storage, so they belong to the Storage view, not the Compute view. Compute usage now keeps only core and VM family metrics. Raise the ingestion query schedule from daily to every four hours. Quota and restriction state changes faster than a day. Publish a manifest only when the query's own parquet file is present. Checking folder existence alone published manifests for queries that wrote nothing, because a sibling query had already created the folder. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A repository-scoped Copilot canvas that reads a FinOps hub cluster directly, so a capacity manager can inspect quota and capacity without building a report first. It carries seven capacity views: App Service, Azure AI, Compute, Azure SQL, Storage, capacity reservations, and Premium SSD v2. The Compute view adds an estate-wide VM family by region heatmap. Supply and demand are separate channels, because they are separate questions: - Supply is a bar on the left edge of the cell. Red means a deployment cannot land here, whether the whole region is restricted or every zone is. Amber means some zones are restricted. Green means open. - Demand is the colour of the percentage itself, measured against a selectable high-water mark that defaults to 70 percent. Colour never carries the alarm alone. Each cell also states its condition in words, so the heatmap reads without colour vision. At estate scale the matrix is a lookup grid, not a picture. It keeps frozen row and column headers, always-visible scrollbars, a family search, region chips, and four condition lenses. Add npm run test-dashboard and run it in the development workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
graphify-out/ is generated locally and must not enter the repository. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brings in the AI and emerging workloads view. Both branches had first committed the dashboard independently while it was untracked, so git saw every dashboard file as an add/add conflict with no common ancestor. Resolved as a real three-way merge against checkpoint 3370dbac5, the last shared state of the untracked working copy. Six of the eight files split cleanly: the capacity work and the AI work touched disjoint regions. Two needed a decision: - extension.mjs: both sides rewrote the canvas description. Kept the new view and kept "capacity views" rather than "capacity evidence views". - The test file: both sides appended to the same tail. Rebuilt as the union, 31 capacity tests plus 12 AI tests, with no name collisions. The merged AI code reintroduced the word "evidence", which this dashboard no longer uses. Renamed the "Evidence" table column to "Detail", reworded the rate-posture panel, and reworded four comments, so the vocabulary test passes on the union rather than by suspending the rule. Verified live against the hub: nine tabs, the AI view rendering all 15 panels with data and no horizontal overflow, and the Compute heatmap still reporting 135 region-restricted and 73 all-zones-restricted cells with red supply bars. 43 of 43 tests pass and the design detector is clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- add filter-aware family and subscription detail tabs - page subscription results with search and focus restoration - align Kusto filters and preserve unknown headroom states - add contract and accessibility regression coverage 🗺️ - Generated by Copilot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 24, 2026 19:55 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 63 changed files in this pull request and generated 8 comments.
Suppressed comments (3)
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:1005
- Native
AbsoluteUrlpagination follows the response'snextLinkwithout checking its authority. A compromised or malformed ARM response can redirect the REST linked service to another host while it is authenticated with the managed identity, which violates the linked requirement to keep continuation requests on the current Resource Manager authority. Validate every continuation URL before issuing it; native unvalidated paging is not safe for this engine.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:1030 - The request URL is built directly from manifest-controlled
queryScopeandqueryvalues and sent through the authenticated ARM linked service. There is no runtime rejection of absolute/scheme-relative URLs, fragments, backslashes, malformed resource IDs, or unsupported scopes, so a modified query manifest can escape the intended ARM-relative request contract. Add validation before this dataset is invoked and attach authentication only after the value is proven ARM-relative.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:400 - This prefix check accepts every Microsoft.Billing resource ID, including unsupported nested resources such as invoice sections. The Savings Plan queries are only compatible with billing accounts and billing profiles, so one unsupported configured scope can make the parallel query pipeline fail. Validate the exact resource-ID shape and dispatch only scopes whose derived resource type is declared compatible by the query.
Contributor
|
Hubs + ADX (managed) deployed to |
- validate ARM scopes and continuations using one stored response per page - filter configured scopes and propagate source metadata - preserve quota and recommendation account identities - enforce deployment order and least-privilege workflow access 🔒 - Generated by Copilot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 24, 2026 21:04 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Hubs + ADX (managed) deployed to |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 26, 2026 19:45 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Hubs + ADX (managed) deployed to |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 26, 2026 20:10 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Hubs + ADX (managed) deployed to |
Restore page-wise subscription and location traversal, lifecycle-owned ARM execution, explicit concurrency boundaries, selective transient retries, expected no-data handling, and typed quota usage mappings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This reverts commit ae6d96d.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add gitignore entries for graphify and codebase-memory-mcp local state so personal code-intelligence tooling does not leak into commits. - .graphifyignore: local corpus scoping (graphify-out/ already ignored) - .codebase-memory/: generated index state (graph.db.zst, .semantic, adr.md) - .codebase-memory.json, .cbmignore, .cbmpathwhitelist: local scoping config Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 31, 2026 15:01 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Hubs + ADX (managed) deployed to |
…itignore - Update ftk-local-dashboard extension (UI split into ui.css/ui.js, kusto.mjs and app.js changes, updated test suite and docs) - Add new quota catalog queries: compute collection coverage, compute family offer status, compute family usage, and quota schema - Update queries INDEX.md - Ignore local .azure/ azd deployment artifacts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 31, 2026 23:20 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 72 out of 75 changed files in this pull request and generated 14 comments.
Suppressed comments (1)
docs-mslearn/toolkit/changelog.md:33
- These changelog entries describe the unrelated local dashboard and link #2277, while this PR adds ARM ingestion, quota, and savings-plan recommendations. The described feature still has no changelog entry, contrary to the PR's stated pending requirement; move the dashboard entries to their PR and add this PR's feature entry.
- Added a repository-scoped FinOps hub dashboard canvas with separate capacity views for App Service, Azure AI, Compute, Azure SQL, Storage, capacity reservations, and Premium SSD v2. The Compute view adds an estate-wide VM family by region heatmap that reads supply restrictions and quota utilization as separate signals ([#2277](https://github.com/microsoft/finops-toolkit/pull/2277)).
- Added an AI and emerging workloads view to the FinOps hub dashboard canvas, covering estate composition, token and model economics, workload detail, allocation coverage, and commitment and rate posture. Month-over-month comparisons anchor to the last closed month so a partially ingested month cannot report a false decline ([#2277](https://github.com/microsoft/finops-toolkit/pull/2277)).
Contributor
|
Hubs + ADX (managed) deployed to |
- Match LocationInfo entries by region in quota-compute-family-offer-status.kql
instead of blindly taking index 0, so multi-region SKUs report zone data for
the correct region.
- Add explicit Fail activities after the Copy Query and Query Probe error
handlers in AzureResourceManager/app.bicep so ADF surfaces pipeline failures
instead of reporting Succeeded when a scope's data wasn't written.
- Fix x_SourceVersion coalesce ordering in IngestionSetup_v1_2.kql so
SavingsPlan rows aren't clobbered by a later case branch.
- Register the AzureResourceManager app as a Recommendations dependency and
use "path" instead of "name" in the recommendations schema so field mapping
resolves correctly.
- Correct finops-hub-database-guide.md's Quota() table (drop stale
CapacityReservation/PremiumSSDv2Disk rows, add AvailabilityZoneMapping) and
refresh INDEX.md's query count.
- Loosen the PowerShell query-file naming regex to allow compound {Name}
segments (e.g. SavingsPlan-P1Y).
- Add the new Bicep/schema paths to dev.yml's docs-only path filter and bump
the changelog ms.date.
- Fix DEFAULT_QUERY placeholder in the dashboard extension (Costs -> Costs()).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bring the Azure AI sub-view up to parity with the Compute and Azure SQL capacity matrix views, following the App Service pattern (dynamic, unregistered model/tier quota pools rather than a fixed metric registry, since each Azure AI model/tier quota pool is independent and summing across models would misrepresent headroom). - kusto.mjs: load the quota-cognitive-services-usage.kql catalog query, add azure-ai to CAPACITY_MATRIX_CLASS_IDS (which also correctly disables demand/cost-schema handling for this class), add buildAzureAiModelQuery/buildAzureAiSubscriptionQuery and azureAiModelCell/annotateAzureAiModelRows, and wire the azureAiHeatmap payload into getCapacity()/getCapacitySubscriptionPage. - app.js: add the azure-ai CAPACITY_MATRIX_CONFIG entry, azureAiMatrixCell, matrix/subscription detail columns, and Supply-tab KPIs (model-region pairs, in use, at limit, no usable quota). - test/ftk-local-dashboard.test.mjs: add dedicated query-builder, cell-semantics, and payload/paging tests for azure-ai; update the two pre-existing tests that assumed azure-ai still used cost/demand fields; extend the shared matrix filter test with an azure-ai fixture. Verified against live remote Kusto data via the dashboard's own get_view canvas action: 5,265 model/region rows with correct utilization and open/none-quota classification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This reverts commit d5ae204.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… query loop ExecuteSubscription was a non-looping IfCondition router (direct vs regional), not a real expansion axis, so it added a pipeline hop the rest of the repo's patterns don't have (msexports, ingestion, and the existing query engines are all exactly 2 layers: outer loop pipeline -> inner worker pipeline). Per ADF docs, a ForEach cannot nest another ForEach, and the sanctioned workaround for two real iteration dimensions is "outer pipeline with the outer loop invokes an inner pipeline with the nested loop" - which is exactly ExecuteTenant's ForEach Subscription -> ExecuteRegional's ForEach Location already. Inlined ExecuteSubscription's two IfCondition branches directly into ForEach Subscription's activities and deleted the now-redundant pipeline. Separately, Loop Thru Queries (shared by every query provider in IngestionQueries) was capped at 4 instead of 30 for public routing: up to 11 Azure Resource Manager tenant-scope query types can run concurrently there, each fanning out through Subscription's batchCount (30/4) into the shared CopyQuery pipeline. 11 x 30 = 330 would exceed CopyQuery's capacity (concurrency 30 + 100 queued = 130); 11 x 4 = 44 stays inside it for both routing modes. Verified live against the Trey R&D hub: patched queries_AzureResourceManager_ExecuteTenant and queries_ExecuteETL directly via the ADF pipeline API, deleted the orphaned queries_AzureResourceManager_ExecuteSubscription pipeline, then triggered queries_ExecuteETL end to end. 28+ pipeline runs observed across ExecuteQuery/ExecuteTenant/ExecuteRegional/CopyQuery/queries_ETL_ingestion with zero failures before the run was cancelled (already proven correct; full completion across ~120 subscriptions x 63 regions would take hours given Region's batchCount=1 serialization - a separate throughput concern to resolve next, not a correctness issue). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CopyQuery had its own pipeline-level concurrency cap (30 public / 4 private routing), and was invoked from three independent fan-out points (ForEach Scope, ForEach Subscription's direct-query branch, ForEach Location). Their batchCounts multiplied against that one shared budget, which forced ForEach Location down to 1 and serialized region-level parallelism. Tried inlining CopyQuery's activities directly at each call site first (matching ccm_datafactory/AzureResourceGraph literally), but ADF rejects it: an IfCondition activity cannot contain another IfCondition activity, which the probe+copy pattern requires. Instead, removed CopyQuery's concurrency property entirely, matching AzureResourceGraph's proven pattern (no concurrency setting, no shared ceiling). ADF's own default admission ceiling (~10,000 concurrent pipeline runs) and real ARM/IR throughput become the actual constraints instead of an artificial 130-slot budget. Raised ForEach Location's batchCount from 1 to 8 (matching ccm_datafactory's proven ForEach batchCount) now that it's no longer capped by CopyQuery's shared budget. Left ForEach Scope at 1 since there are no configured billing scopes on any live hub to validate a higher value against. Proven live against the Trey hub: 100+ CopyQuery invocations across Region-scoped query types (ComputeUsage, HDInsightUsage, MachineLearningUsage) with zero throttling or concurrency-related failures. One unrelated pre-existing failure was discovered and left unfixed (out of scope): the empty-result probe hits ADF's ~4MB WebActivity response-size limit for Microsoft.Compute/skus, documented in README.md. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Subscriptions - List API paginates via nextLink once a tenant has
enough subscriptions; the previous single WebActivity call only
returned the first page and would silently under-count subscriptions
at scale (surfaced while stress-testing the design against a
5,000-subscription target). ExecuteTenant now loops via Set
Subscriptions Url / Get All Subscription Pages (Until) / Get
Subscriptions Page (WebActivity) until the response stops returning a
nextLink, then feeds the accumulated array into Filter Enabled
Subscriptions.
Two ADF expression-language constraints shaped the loop, both found
and fixed by testing live against Trey:
- A SetVariable cannot read the same variable it writes ("self
reference is not supported"), so the array is merged into a scratch
variable (subscriptionsArrayNext) and copied back in a second,
non-self-referencing SetVariable.
- ARM omits nextLink entirely (not null) on the last page, and ADF
throws evaluating a property path that doesn't exist at all -
coalesce() does not protect against that. The next-page URL is now
set via an IfCondition gated on contains(activity(...).output,
'nextLink') so the possibly-absent property is only read once its
presence is confirmed.
Also documents the 5,000-subscription/4-hour scale investigation:
researched ARM throttling scope, ADF's factory-level concurrency
ceilings, and Azure Optimization Engine's architecture as prior art.
ARM throttling is scoped per subscription/service-principal/operation-
type and, for resource-provider endpoints, per subscription per
region - this design's per-subscription/per-region fan-out does not
contend on a shared bucket. ADF's binding ceiling (3,000 concurrent
External activity runs per Data Factory subscription per Azure IR
region, default = maximum) comfortably exceeds the concurrency this
design needs at 5,000 subscriptions. Azure Optimization Engine's
ARG-based bulk query does not transfer to this workload: ARG has no
resource type for subscription quota/usage data, only for actual
resources with a resourceId.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Closing per author decision: sprint rejected. The ARM ingestion redesign (subscription/region concurrency, pagination, pipeline collapse) was committed and deployed to a live test environment but never validated end-to-end — a live run surfaced both a real bug (ComputeResourceSku query exceeding ADF's 4 MB WebActivity response limit) and a stale concurrency cap left over from an earlier design iteration. Author has rejected the PR on that basis. |
Contributor
|
Test environments cleaned up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠️ Description
This PR added a reusable Azure Resource Manager ingestion path for GET-only APIs and connected its output to the existing FinOps hub ingestion flow.
Managed Azure Resource Manager ingestion
Quota and recommendation data
Quota()functions and query-catalog entries for current usage, headroom, source-specific quota views, and savings plan recommendation breakdowns.Documentation and tests
.copilot-tracking/workflow artifacts.This work provides shared ingestion groundwork for MACC consumption. MACC Lots and MACC Events ingestion are not included in this PR. Quota and capacity support extends beyond the current iteration described in #1721.
Refs #1721
Related to #2120 and #2129
📷 Screenshots
Not applicable.
📋 Checklist
🔬 How did you test this change?
The full unit suite passed with 2,453 tests and no failures. The full lint suite passed with 3,439 tests and no failures. App Service ingestion completed through Data Factory and Azure Data Explorer, and all seven quota catalog projections returned the expected schema and key behavior.
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?A changelog entry is pending before this draft is ready for review.
📖 Did you update documentation?
Public documentation remains pending before this draft is ready for review.