[Property] Many fields incorrectly described as percentage instead of proportion

[carlfischerjba]

Description

Several columns are described as percentages which supposes values between 0 and 100 when they are in fact proportions between 0 and 1.

Reasons for change

It's essential for creators and users of OED to use the right values and for the spec to be unambiguous. 0.5% is not the same as a proportion of 0.5 which would be equivalent to 50%, however, 0.5 would be accepted in either case and therefore not flagged by validation tools.

Scope of change

• Location File
• Accounts File
• Reinsurance Scope
• Reinsurance Info

Impact of change

Renaming fields to replace Percent with Proportion would be the clearest but would be a breaking change.

In some cases, the field name is correct but the description mentions a percentage. Any change to the description would be fully backwards compatible and would be a good first step.

Data type, default values, are blanks allowed, list valid values

The following fields have Percent in their name but should ideally have Proportion:

• PercentComplete
• PercentSprinklered
• CededPercent
• PlacedPercent
• DeemedPercentPlaced

The following fields incorrectly mention "percentage" in their description instead of "proportion".

• ScaleFactor
• PercentComplete
• LocParticipation
• SurgeLeakage
• PercentSprinklered
• BrickVeneer
• BuildingExteriorOpening
• CededPercent
• PlacedPercent
• DeemedPercentPlaced
• TreatyShare

Additionally, some of the above do not contain valid values in the spec [0, 1] which means that errors will never be caught.

[aiste-kalinauskaite]

Making changes to the headers would have a significant impact on all parties involved. In this regard, I want to refer to an issue that I have highlighted this in (as it's as relevant here): #215. Additionally, such changes would necessitate further operational rewrites on the Oasis LMF side, including adding additional code to divide by 100 wherever any of the fields are utilized.

On a more positive note, making descriptions clearer is certainly a beneficial change. The rationale behind expressing these values as proportions makes all downstream processes smoother, as it eliminates the need to constantly divide by 100 during calculations. It would be advantageous to include clear descriptions for all relevant fields, stating something along the lines of: "All percentages are expressed as proportions to streamline downstream calculations. For example, 12.5% should be expressed as 0.125". This approach would add clarity and avoid the disruption caused by changing the headers.

It is important to remember that the specifications should always be used in conjunction with descriptions and other available information, rather than relying solely on field names.

The only instances I have found where the [0,1] range is omitted are as follows, and these are intentional:

• The ScaleFactor was increased in OED3 to accommodate values above 100%, thus the [0,1] requirement was removed. Therefore, 150% would be entered as 1.5.
• For BrickVeneer and BuildingExteriorOpening, these are secondary modifiers where percentage ranges are defined in discrete, pre-defined categories and their ranges reflect the categorisation defined.

[carlfischerjba]

@aiste-kalinauskaite I can see that in practice it's too late to correct the headers now but it's worth considering if more fields were to be added and worth improving the descriptions of the existing fields.

such changes would necessitate further operational rewrites on the Oasis LMF side, including adding additional code to divide by 100 wherever any of the fields are utilized.

Not as far as I understand it. All those fields are already proportions. I don't see any that take percentage values (even though the names might suggest so). It would therefore just be a renaming exercise, but point taken that represents a breaking change.

It is important to remember that the specifications should always be used in conjunction with descriptions and other available information, rather than relying solely on field names.

Ideally yes, but it shouldn't be used as an excuse for accepting misleading naming. The names should give strong and accurate hints about the meaning of the values. I'm raising it now because I've seen smart people caught out by it recently.

stating something along the lines of: "All percentages are expressed as proportions to streamline downstream calculations. For example, 12.5% should be expressed as 0.125".

A percentage is just one way of representing a proportion and it's not the one that's used here, so I would suggest only mentioning percentage as an example: "Proportion sprinklered. Values between 0 and 1 inclusive (0.5 = 50%); -999 means unknown." An additional explanation could be added: "Note that despite the name this is not a percentage value between 0 and 100 but a value between 0 and 1 representing a proportion." but maybe that's excessive.

For BrickVeneer and BuildingExteriorOpening, these are secondary modifiers where percentage ranges are defined in discrete, pre-defined categories and their ranges reflect the categorisation defined.

OK, I see now that they link to the Other values sheet and are described as "Code that represents...".

[johcarter]

For OED v4 I'll go through the descriptions for the fields and make sure it is clear where values should be entered as proportions, but won't rename the Percent fields as it would be too disruptive.

[aiste-kalinauskaite]

@carlfischerjba the comment below was made in relation to if your proposal were to go ahead:

such changes would necessitate further operational rewrites on the Oasis LMF side, including adding additional code to divide by 100 wherever any of the fields are utilized.

As it is not, then it is not relevant.
I am always happy with better descriptions though. Thank you @johcarter !