[Optimization Engine] Keep reservations visible when their SKU has no ISF ratio - #2310
Conversation
The reservation workbooks join the instance size flexibility open data with kind=inner. The Catalogs API that feeds it only returns SKUs that can still be purchased, so an active reservation on a retired size has no row to match and is dropped from the report entirely -- not shown at zero, absent. The customer keeps paying for a reservation the tool stops mentioning. Reservations run one or three years and outlive the sellability of their size, so this is reachable in normal use (#2300). All seven ISF joins in reservations-usage.json become leftouter, with the group and ratio falling back to the SKU itself: | extend ISFGroup = coalesce(ISFGroup, SKUName_s), Ratio = coalesce(Ratio, 1.0) Without that fallback a plain leftouter is worse than the current behaviour: Ratio and ISFGroup come back null, every unmatched reservation collapses into one null bucket, and the utilization column reads empty. Ratio 1 is correct for these rows. Util7Days_s comes from Azure and already accounts for flexibility, so the percentage is right; the ratio only converts between group units, which is a no-op for a single-SKU group. reservations-potential.json and benefits-simulation.json keep kind=inner. They model what to buy, where restricting to purchasable SKUs is correct. Note the workbook KQL is not executed by any test in this repo, so this change is reasoned rather than verified. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Some ISFGroups leftouter joins elsewhere in the same workbook still lack the required coalesce fallback, which can leave null group names or null/incorrect calculations for SKUs missing from the ISF data.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the Optimization Engine Reservations usage workbook KQL so reservations remain visible even when their SKU no longer appears in the instance size flexibility (ISF) open-data feed (e.g., retired / no-longer-purchasable SKUs), preventing active reservations from disappearing from utilization reporting.
Changes:
- Switch key ISF enrichment joins from
kind=innertokind=leftouterinreservations-usage.json. - Add
coalesce(...)fallbacks forISFGroupandRatio(defaulting to the reserved SKU / 1.0) in the updated queries to avoid null-group bucketing and null math.
File summaries
| File | Description |
|---|---|
| src/optimization-engine/views/workbooks/reservations-usage.json | Adjusts workbook KQL join semantics and introduces coalesce fallbacks so reservations with missing ISF rows are still included and computed correctly. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| "content": { | ||
| "version": "KqlItem/1.0", | ||
| "query": "let ISFGroups = externaldata(ISFGroup:string, ArmSKUName:string, Ratio:double) [@\"https://raw.githubusercontent.com/microsoft/finops-toolkit/dev/src/open-data/InstanceSizeFlexibility.csv\"] with(ignoreFirstRecord=true) | extend ArmSKUName=tolower(ArmSKUName);\r\nlet LinuxOnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s == 'Virtual Machines' and PriceType_s == 'Consumption' and MeterSubCategory_s !endswith \"Windows\"\r\n| extend MeterSubCategory_s = substring(MeterSubCategory_s, 0, indexof(MeterSubCategory_s, ' Linux'))\r\n| summarize LinuxUnitPrice=max(todouble(UnitPrice_s)) by LinuxMeterId=MeterID_g, MeterName_s, MeterSubCategory_s, MeterRegion_s, LinuxUnitOfMeasure=UnitOfMeasure_s;\r\nlet VMOnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s == 'Virtual Machines' and PriceType_s == 'Consumption'\r\n| extend NonWindowsMeterSubcategory = substring(MeterSubCategory_s, 0, indexof(MeterSubCategory_s, ' Windows'))\r\n| extend WindowsMeterSubCategory = MeterSubCategory_s\r\n| extend NonWindowsMeterSubcategory = substring(NonWindowsMeterSubcategory, 0, indexof(NonWindowsMeterSubcategory, ' Linux'))\r\n| summarize UnitPrice_s=max(todouble(UnitPrice_s)) by MeterID_g, MeterName_s, NonWindowsMeterSubcategory, WindowsMeterSubCategory, MeterRegion_s, UnitOfMeasure_s\r\n| join kind=leftouter ( LinuxOnDemandPriceSheet ) on MeterName_s, MeterRegion_s, $left.NonWindowsMeterSubcategory == $right.MeterSubCategory_s\r\n| extend PricesheetPrice = iif(isnotempty(LinuxUnitPrice), LinuxUnitPrice, UnitPrice_s)\r\n| extend PricesheetUnitOfMeasure = iif(isnotempty(LinuxUnitOfMeasure), LinuxUnitOfMeasure, UnitOfMeasure_s)\r\n| extend UnitHrs = toint(substring(PricesheetUnitOfMeasure, 0, indexof(PricesheetUnitOfMeasure, 'Hour')-1))\r\n| extend OnDemandUnitPrice = PricesheetPrice/UnitHrs\r\n| distinct MeterID_g, OnDemandUnitPrice;\r\nlet OnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s != 'Virtual Machines' and PriceType_s == 'Consumption'\r\n| summarize OnDemandUnitPrice = max(todouble(UnitPrice_s)) by MeterID_g\r\n| distinct MeterID_g, OnDemandUnitPrice\r\n| union (VMOnDemandPriceSheet)\r\n| summarize OnDemandUnitPrice=min(OnDemandUnitPrice) by MeterID_g;\r\nlet ReservationPricesheet = AzureOptimizationReservationsPriceV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| extend Term_s = iif(reservationTerm_s == '3 Years', 'P3Y', 'P1Y')\r\n| extend TermDivider = iif(reservationTerm_s == '3 Years', 3, 1)\r\n| extend ReservationPrice = todouble(replace_string(unitPrice_s,',','.'))/TermDivider/12/730\r\n| summarize arg_max(TimeGenerated, ReservationPrice) by Location_s=tolower(armRegionName_s), SkuName=tolower(armSkuName_s), Term_s;\r\nlet ReservationOnDemandMeters = AzureOptimizationConsumptionV1_CL\r\n| where TimeGenerated > todatetime('{LookbackPeriod:startISO}') and TimeGenerated < todatetime('{LookbackPeriod:endISO}') and ChargeType_s == 'Usage' and isnotempty(ReservationName_s)\r\n| extend SkuName = tolower(parse_json(AdditionalInfo_s).ServiceType)\r\n| distinct ReservationId_g, MeterId_g, SkuName;\r\nAzureOptimizationConsumptionV1_CL\r\n| where TimeGenerated > todatetime('{LookbackPeriod:startISO}') and TimeGenerated < todatetime('{LookbackPeriod:endISO}') and ChargeType_s == 'Usage' and isnotempty(ReservationName_s)\r\n| where ReservationId_g in ({Reservation:value})\r\n| extend RINormalizationRatio = tostring(parse_json(AdditionalInfo_s).RINormalizationRatio)\r\n| extend UsedRIs = todouble(Quantity_s) * todouble(RINormalizationRatio) / 24\r\n| summarize UsedRIsDaily=round(sum(UsedRIs),2) by Date_s, ReservationId_g\r\n| summarize AvgRIsUsedDaily=round(avg(UsedRIsDaily),2) by ReservationId_g\r\n| join kind=rightouter (\r\n AzureOptimizationReservationsUsageV1_CL\r\n | where TimeGenerated > ago(1d)\r\n | where ReservationId_g in ({Reservation:value})\r\n | summarize arg_max(TimeGenerated, *) by ReservationId_g\r\n | where ProvisioningState_s in ('Succeeded','Expiring')\r\n | extend UsedQuantity = todouble(TotalReservedQuantity_s) * todouble(Util7Days_s) / 100\r\n | extend UsedQuantity30d = todouble(TotalReservedQuantity_s) * todouble(Util30Days_s) / 100\r\n | extend SKUName_s=tolower(SKUName_s)\r\n | project ReservationId_g, ReservationName_s=DisplayName_s, SKUName_s, Location_s, UsedQuantity, UsedQuantity30d, TotalReservedQuantity_s, Term_s, AppliedScopeType_s\r\n) on ReservationId_g\r\n| project ReservationId_g=ReservationId_g1, ReservationName_s, TotalReservedQuantity_s, SKUName_s, Location_s, AvgRIsUsedDaily=iif(isempty(AvgRIsUsedDaily), 0.0, AvgRIsUsedDaily), UsedQuantity, UsedQuantity30d, Term_s, AppliedScopeType_s\r\n| join kind=inner ( ISFGroups ) on $left.SKUName_s == $right.ArmSKUName\r\n| join kind=leftouter ( ReservationOnDemandMeters ) on ReservationId_g\r\n| summarize arg_max(MeterId_g, *) by ReservationId_g\r\n| join kind=leftouter ( ReservationPricesheet ) on SkuName and Location_s and Term_s\r\n| join kind=leftouter ( OnDemandPriceSheet ) on $left.MeterId_g == $right.MeterID_g\r\n| extend DiscountPercent = (1 - ReservationPrice/OnDemandUnitPrice) * 100\r\n| extend AvgRIsUsedInSmallestRatio = Ratio * AvgRIsUsedDaily\r\n| summarize TotalReservedQuantity_s=sum(todouble(TotalReservedQuantity_s)*Ratio), AvgRIsUsedDaily=sum(AvgRIsUsedInSmallestRatio), UsedQuantity=sum(UsedQuantity*Ratio), UsedQuantity30d=sum(UsedQuantity30d*Ratio), AvgDiscountPercent=avg(DiscountPercent) by ISFGroup, Location_s, Term_s, AppliedScopeType_s\r\n| extend Util7Days_s = UsedQuantity/TotalReservedQuantity_s*100, Util30Days_s = UsedQuantity30d/TotalReservedQuantity_s*100\r\n| extend AvgRIUsagePercentInSmallestRatio = round(AvgRIsUsedDaily / TotalReservedQuantity_s * 100, 1)\r\n| extend AvgDiscountPercent=iif(AvgDiscountPercent > 0.0, AvgDiscountPercent, 0.0)\r\n| extend SavingsMargin=round(todouble(Util7Days_s))-100.0+AvgDiscountPercent \r\n| project-away AvgRIUsagePercentInSmallestRatio, AvgRIsUsedDaily\r\n| project-reorder ISFGroup, Location_s, Term_s, AppliedScopeType_s, TotalReservedQuantity_s, Util7Days_s, UsedQuantity, Util30Days_s, UsedQuantity30d\r\n| order by Util7Days_s asc", | ||
| "query": "let ISFGroups = externaldata(ISFGroup:string, ArmSKUName:string, Ratio:double) [@\"https://raw.githubusercontent.com/microsoft/finops-toolkit/dev/src/open-data/InstanceSizeFlexibility.csv\"] with(ignoreFirstRecord=true) | extend ArmSKUName=tolower(ArmSKUName);\r\nlet LinuxOnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s == 'Virtual Machines' and PriceType_s == 'Consumption' and MeterSubCategory_s !endswith \"Windows\"\r\n| extend MeterSubCategory_s = substring(MeterSubCategory_s, 0, indexof(MeterSubCategory_s, ' Linux'))\r\n| summarize LinuxUnitPrice=max(todouble(UnitPrice_s)) by LinuxMeterId=MeterID_g, MeterName_s, MeterSubCategory_s, MeterRegion_s, LinuxUnitOfMeasure=UnitOfMeasure_s;\r\nlet VMOnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s == 'Virtual Machines' and PriceType_s == 'Consumption'\r\n| extend NonWindowsMeterSubcategory = substring(MeterSubCategory_s, 0, indexof(MeterSubCategory_s, ' Windows'))\r\n| extend WindowsMeterSubCategory = MeterSubCategory_s\r\n| extend NonWindowsMeterSubcategory = substring(NonWindowsMeterSubcategory, 0, indexof(NonWindowsMeterSubcategory, ' Linux'))\r\n| summarize UnitPrice_s=max(todouble(UnitPrice_s)) by MeterID_g, MeterName_s, NonWindowsMeterSubcategory, WindowsMeterSubCategory, MeterRegion_s, UnitOfMeasure_s\r\n| join kind=leftouter ( LinuxOnDemandPriceSheet ) on MeterName_s, MeterRegion_s, $left.NonWindowsMeterSubcategory == $right.MeterSubCategory_s\r\n| extend PricesheetPrice = iif(isnotempty(LinuxUnitPrice), LinuxUnitPrice, UnitPrice_s)\r\n| extend PricesheetUnitOfMeasure = iif(isnotempty(LinuxUnitOfMeasure), LinuxUnitOfMeasure, UnitOfMeasure_s)\r\n| extend UnitHrs = toint(substring(PricesheetUnitOfMeasure, 0, indexof(PricesheetUnitOfMeasure, 'Hour')-1))\r\n| extend OnDemandUnitPrice = PricesheetPrice/UnitHrs\r\n| distinct MeterID_g, OnDemandUnitPrice;\r\nlet OnDemandPriceSheet = AzureOptimizationPricesheetV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| where MeterCategory_s != 'Virtual Machines' and PriceType_s == 'Consumption'\r\n| summarize OnDemandUnitPrice = max(todouble(UnitPrice_s)) by MeterID_g\r\n| distinct MeterID_g, OnDemandUnitPrice\r\n| union (VMOnDemandPriceSheet)\r\n| summarize OnDemandUnitPrice=min(OnDemandUnitPrice) by MeterID_g;\r\nlet ReservationPricesheet = AzureOptimizationReservationsPriceV1_CL\r\n| where TimeGenerated > ago(14d)\r\n| extend Term_s = iif(reservationTerm_s == '3 Years', 'P3Y', 'P1Y')\r\n| extend TermDivider = iif(reservationTerm_s == '3 Years', 3, 1)\r\n| extend ReservationPrice = todouble(replace_string(unitPrice_s,',','.'))/TermDivider/12/730\r\n| summarize arg_max(TimeGenerated, ReservationPrice) by Location_s=tolower(armRegionName_s), SkuName=tolower(armSkuName_s), Term_s;\r\nlet ReservationOnDemandMeters = AzureOptimizationConsumptionV1_CL\r\n| where TimeGenerated > todatetime('{LookbackPeriod:startISO}') and TimeGenerated < todatetime('{LookbackPeriod:endISO}') and ChargeType_s == 'Usage' and isnotempty(ReservationName_s)\r\n| extend SkuName = tolower(parse_json(AdditionalInfo_s).ServiceType)\r\n| distinct ReservationId_g, MeterId_g, SkuName;\r\nAzureOptimizationConsumptionV1_CL\r\n| where TimeGenerated > todatetime('{LookbackPeriod:startISO}') and TimeGenerated < todatetime('{LookbackPeriod:endISO}') and ChargeType_s == 'Usage' and isnotempty(ReservationName_s)\r\n| where ReservationId_g in ({Reservation:value})\r\n| extend RINormalizationRatio = tostring(parse_json(AdditionalInfo_s).RINormalizationRatio)\r\n| extend UsedRIs = todouble(Quantity_s) * todouble(RINormalizationRatio) / 24\r\n| summarize UsedRIsDaily=round(sum(UsedRIs),2) by Date_s, ReservationId_g\r\n| summarize AvgRIsUsedDaily=round(avg(UsedRIsDaily),2) by ReservationId_g\r\n| join kind=rightouter (\r\n AzureOptimizationReservationsUsageV1_CL\r\n | where TimeGenerated > ago(1d)\r\n | where ReservationId_g in ({Reservation:value})\r\n | summarize arg_max(TimeGenerated, *) by ReservationId_g\r\n | where ProvisioningState_s in ('Succeeded','Expiring')\r\n | extend UsedQuantity = todouble(TotalReservedQuantity_s) * todouble(Util7Days_s) / 100\r\n | extend UsedQuantity30d = todouble(TotalReservedQuantity_s) * todouble(Util30Days_s) / 100\r\n | extend SKUName_s=tolower(SKUName_s)\r\n | project ReservationId_g, ReservationName_s=DisplayName_s, SKUName_s, Location_s, UsedQuantity, UsedQuantity30d, TotalReservedQuantity_s, Term_s, AppliedScopeType_s\r\n) on ReservationId_g\r\n| project ReservationId_g=ReservationId_g1, ReservationName_s, TotalReservedQuantity_s, SKUName_s, Location_s, AvgRIsUsedDaily=iif(isempty(AvgRIsUsedDaily), 0.0, AvgRIsUsedDaily), UsedQuantity, UsedQuantity30d, Term_s, AppliedScopeType_s\r\n| join kind=leftouter ( ISFGroups ) on $left.SKUName_s == $right.ArmSKUName\r\n| extend ISFGroup = coalesce(ISFGroup, SKUName_s), Ratio = coalesce(Ratio, 1.0)\r\n| join kind=leftouter ( ReservationOnDemandMeters ) on ReservationId_g\r\n| summarize arg_max(MeterId_g, *) by ReservationId_g\r\n| join kind=leftouter ( ReservationPricesheet ) on SkuName and Location_s and Term_s\r\n| join kind=leftouter ( OnDemandPriceSheet ) on $left.MeterId_g == $right.MeterID_g\r\n| extend DiscountPercent = (1 - ReservationPrice/OnDemandUnitPrice) * 100\r\n| extend AvgRIsUsedInSmallestRatio = Ratio * AvgRIsUsedDaily\r\n| summarize TotalReservedQuantity_s=sum(todouble(TotalReservedQuantity_s)*Ratio), AvgRIsUsedDaily=sum(AvgRIsUsedInSmallestRatio), UsedQuantity=sum(UsedQuantity*Ratio), UsedQuantity30d=sum(UsedQuantity30d*Ratio), AvgDiscountPercent=avg(DiscountPercent) by ISFGroup, Location_s, Term_s, AppliedScopeType_s\r\n| extend Util7Days_s = UsedQuantity/TotalReservedQuantity_s*100, Util30Days_s = UsedQuantity30d/TotalReservedQuantity_s*100\r\n| extend AvgRIUsagePercentInSmallestRatio = round(AvgRIsUsedDaily / TotalReservedQuantity_s * 100, 1)\r\n| extend AvgDiscountPercent=iif(AvgDiscountPercent > 0.0, AvgDiscountPercent, 0.0)\r\n| extend SavingsMargin=round(todouble(Util7Days_s))-100.0+AvgDiscountPercent \r\n| project-away AvgRIUsagePercentInSmallestRatio, AvgRIsUsedDaily\r\n| project-reorder ISFGroup, Location_s, Term_s, AppliedScopeType_s, TotalReservedQuantity_s, Util7Days_s, UsedQuantity, Util30Days_s, UsedQuantity30d\r\n| order by Util7Days_s asc", |
|
Closing — folded into #2308 instead. Splitting this out was my call and the wrong one; the change belongs in the same PR as the rest of the ISF work. Same commit, same content, now on `RolandKrummenacher/isf-legacy-floor`. |
|
All Contributors (@all-contributors) please add Roland Krummenacher (@RolandKrummenacher) for code |
|
@microsoft-github-policy-service[bot] Roland Krummenacher (@RolandKrummenacher) already contributed before to code |
Summary
Fixes #2300. The reservation workbooks join the instance size flexibility open data with
kind=inner. That data comes from the Azure Catalogs API, which only returns SKUs you can still purchase — so an active reservation on a size Azure no longer sells has no row to match and is dropped from the report entirely. Not shown at zero utilization: absent. The customer keeps paying for a reservation the tool stops mentioning.Reservations run one or three years and routinely outlive the sellability of their size, so this is reachable in ordinary use. It is what the reporter of #2300 hit.
Change
All seven ISF joins in
reservations-usage.jsonbecomeleftouter, with the group and ratio falling back to the SKU itself:The
coalesceis not optional. A plainleftouteris worse than today:RatioandISFGroupcome back null,TotalReservedQuantity_s * Ratiobecomes null, andsummarize ... by ISFGroupcollapses every unmatched reservation into a single null bucket with an empty utilization column.Ratio = 1is correct for these rows.Util7Days_scomes from Azure and already accounts for instance size flexibility, so the percentage is right as-is. The ratio only converts between group units, which is a no-op for a group of one.reservations-potential.jsonandbenefits-simulation.jsonkeepkind=inner. They model what to buy, and there restricting to purchasable SKUs is correct.Known limitation
Drilldowns fall back to
coalesce(ISFGroup, VMSize)— the consumed size — while the overview falls back toSKUName_s, the reserved size. For a reservation onStandard_DS4_v2runningDS3_v2underneath, those don't meet: which sizes belong to a retired group is exactly the information the ISF data no longer carries. So a drilldown on an unmatched reservation shows usage of the identical SKU only, not the group's flexibility.Better than showing nothing, but not a full restoration. Recovering it would need the retired ratios, which is what #2300 originally proposed and #2308 explicitly stopped short of.
Why not fix the data instead
An earlier revision of #2308 backfilled 1,220 retired SKUs from the ratio file the API replaced. It was dropped: those rows would be frozen forever, since nothing can refresh or correct them once the source is gone, inside a dataset whose contract is a weekly refresh from the authoritative API. Fixing the consumer avoids carrying dead reference data.
Test plan
Av2,Dv2,Dv3,Ev3,F,G,H,NC,NV, …) and confirm it now appears with a correct utilization percentageThis is reasoned, not verified. No test in this repo executes the workbook KQL, and I have no Optimization Engine workspace with reservation data to run it against. The join semantics and the null behaviour that motivates the
coalesceshould be confirmed by someone who does before merging.🤖 Generated with Claude Code