Measurability

[AlexDawsonUK]

Issue: We need to improve how the impact, effort, GRI, and other variables of our guidelines are calculated. This is something to be undertaken by a new measurability Task Force.

Considerations: We could choose to create a new model for calculating emissions (that can filter into the tooling to build impact / implementation assessments and reports, which in turn can guide the WSG regarding best practices), we could utilize digital carbon ratings (or another system) to apply a user-friendly method of grading guidelines, we could conduct research, or do other things besides, we have to account for time considerations and the necessity to provide accurate data and recommendations.

Related:

• System Boundaries: We need to describe where we start and stop measuring data and what our scope is for measurability (this can be placed within the introduction) so that people's expectations of our work can be managed and their awareness of any limitations we have can be easily identified.
• Tradeoffs: When providing calculations, it's important people can see not only what the net benefit of something is (we have a benefits section) but also the potential downsides. Because many guidelines have similar potential consequences of implementation we didn't include such a section up to this point but it could be worth re-examining and including in any measurability.
• Rating Sort: It is currently difficult for implementors to identify which guidelines are the most impactful or the ones they should focus on first, therefore, once scoring of guidelines through measurability data has been established we can better re-organize and potentially allow the filtering / sorting or highlight the guidelines of most critical importance within the spec.
• Implementation Report: Interoperability is something that we need to work on if we want to set our standards at the highest they can be. If we can increase our measurability and providers can put our work into action (and showcase this with data), we can build a report that can also feed back into the guidance we give (based on what works and doesn't).

Source: CG #98, CG #99, CG #101, CG #104, CG #112.

[mgifford]

I am a bit worried about the idea of creating a new model. This is really such a rapidly evolving space, and the internet is continuing to change over time. 

Maybe what we need is a commitment to proving progress over perfection. Every model will have its advantages and disadvantages. I keep going back to Ceci n'est pas une pipe. A model is always going to be imperfect. The map isn't the terrain. 

As long as you're aware of the advantages/disadvantages of the model you are using, the goal should be the same. How do we reduce our impact using this model? How do we ensure we are finding ways to make our technology have a reduced environmental footprint than it did before. 

The model doesn't need to be fixed, but we could provide a few recommendations.

[airbr]

I think continued efforts into improving the effort calculation either in its calculation or presentation/communication about it could improve the guidelines. I think it could promote adoption of the guidelines and promote creative works by others who have accessed the JSON API to allow for more kinds of calculations based on inputs that might seem out of our scope to allow for all kinds of creative sorting.

I revisited https://w3c.github.io/sustyweb/#take-a-more-sustainable-approach-to-image-assets 2.16 Take a More Sustainable Approach to Media Assets today. Id like more people to know about apps and techniques for image optimization generally, even though I am only regularly familiar with using one particular manual GUI app. The current guideline ranks this as effort medium but this was the easiest thing to adopt in my limited experience of using a GUI optimizer for images that required particular attention. Knowing how to image optimize gives me a sense of connection to the process and feels practical.

The Guidelines already do a good job by de-lineating some areas by topic: User-Experience Design, Web Development and others. I think it would be great to either help calculate effort based on occupation or industry, or at least do what we can to enable others to work with our data we are drawing from so that they can also do it, somewhere in there is a good approach I think!

Thank you for this proposal/efforts 👍 . Just adding a comment to encourage basically.

[ldevernay]

Measurability is indeed a huge topic (and we just this week started a similar group in France for the RCP of the ADEME).
Creating a new model would be a waste of time in my opinion.
We need to set boundaries, metrics and indicators as a wishlist then see what is currently covered by existing models and tools and where to go from here while embracing continuous improvement (some things do not seem to be even technically feasible right now because of the limitations of the tools and data sources).

[hanopcan]

Perhaps a way forward to break this big task down is to cherry-pick five guidelines to attempt to measure. Run that as a pilot and see what comes out. Those first five can be chosen for perceived ease to measure, just to get the ball rolling so we start learning how this works in practice. This would support with following through on the suggestion of "progress over perfection" suggested by @mgifford earlier on.

Then see what happens and hopefully pick the next five to measure and so on... 😁

[ldevernay]

I think, there are two distinct topics to be considered here :

• How do we estimare environmental impacts as accurately as possible (eg : how much greenhouse gas emissions / abiotic resources needed / water consumed / etc for my website in a year/month/etc) ? Which requires a list of required metrics, environmental indicators, a model, data sources and a measurement methodology. Might be one of the toughest topics we will be facing (but also one of the most essentials).
• How do we check conformance to a given guideline (eg : 3.2 for code minification) ? Which is in my opinion more relevant to this issue : Useful Tooling #11

In the end, I think we need 2 different task forces to tackle this :

• A measurability task force focused on assessing environmental impacts
• A "Tooling for compliance" task force

(both interest me a lot, btw)