OWD project: Fix errors in BCD for API, CSS and JS going back to at least 2020 #85
Comments
|
@queengooborg @robnyman do you have permissions to label this as a proposal for Q3 planning? |
|
I have a feeling that this will span across both Q3 and Q4 for how big the project is. Maybe we should separate this into three projects, one for each category? |
Yes, I think having smaller projects is a good thing here. This helps get focused and claiming victories is always a good feeling and a good way to communicate progress. |
|
I think an important first step here is to list all of the contradictions between BCD and collector data for the past two years' releases. There is no script for this currently. |
|
Thanks for filing this OWD project proposal! The OWD prioritization criteria guides us to make decisions. EffortIt seem like there is some scripting needed to determine the contradictions between BCD and the collector. Dependencies
Community enablementCorrect me if I'm wrong but dealing with the contradictions is hard. Not sure how much we will be able to involve the community with this project. MomentumSeems like correct compat data is of interest for current initiatives like Interop? Enabling learnersn/a Enabling professionalsn/a Underrepresented topics / ethical webn/a Operational necessitiesn/a Addressing needs of the Web industryInterop? |
Elchi3
added
the
proposal (actionable)
Elchi3
changed the title
|
In yesterday's planning call, @queengooborg said:
Assigning this to @queengooborg to post an initial assessment of what work is required here and whether we want this project to be split up which would then allow us to prioritize say CSS data over other data. |
|
To quote myself, I think we need a new approach to judge if this is done, and how much remains:
Just running the We can detect contradictions without being able to fix them automatically. |
|
foolip/mdn-bcd-collector#2317 demonstrates many problems in |
|
In today's BCD call I proposed to a have "preparation" project for this issue. I'd like to propose to do a project that creates a continuously updated BCD issue very similar to mdn/browser-compat-data#3555. In that issue, the collector reports something like this after every BCD release (percentages are fiction):
It only takes into account data for browser releases after 2020-01-01. We can see for which folders it should run. (api, js, css, mathml, svg) @queengooborg what do you think about doing that in q4? It would involve work on the collector to spit out such metrics. Once we have this, we follow up with projects to decrease the number of cases where the collector is broken and where the collector reports wrong data that probably needs fixing in BCD. @foolip I hope this makes sense. Let me know if you have anything to add to this idea. |
|
@Elchi3 that approach to burning down errors makes sense to me, it was very motivating in past efforts to see the numbers slowly approach 100%. I think there are two numbers we can track:
I don't think we can track where the collector is wrong in any meaningful way, we update custom tests continuously when we discover this. |
Elchi3
added
q4 2022
and removed
proposal (actionable)
|
In our planning call yesterday, @queengooborg said that this is done. |
|
I think we should move it to a separate issue, yes -- perhaps open one up in the collector repository? |
|
It looks like there's quite a lot of differences between BCD and the collector remaining for post-2020 data. Here's what I did. First, list the browser releases that happened in 2020-2022 (ignoring 2023 to not make this a moving target):
Then, get the results for those from https://github.com/GooborgStudios/mdn-bcd-results/tree/01f943d878fc1d51364a2791906aefb69ade3692: 140 JSON files7.1.2-chrome-100.0.4896.127-mac-os-10.15.7-ffe2e9031b.json Then, running mdn/browser-compat-data#18539 shows changes to 220 files. After differences for Chrome, Firefox and Safari desktop are sorted out, the next step would be to see what additional errors exist for Edge (likely many) and mobile browsers. Is this project continuing in Q1? My definition of done would be that |
After fixing a bug in the
(Note: the These numbers don't include changes that set the browser to By the way, may we break this project up into smaller chunks? Perhaps, divided into each category and each year (ex. API 2020, JS 2022, etc.)? |
|
Breaking this into smaller chunks sounds great, it's a lot of work, and for reviewers it's probably easier to review if there's a theme of the day, rather than simply going A-Z. I think the highest priority should probably be where a truthy value changes to false or vice versa, because that means BCD (or the collector) is wrong about current support. If No doubt there will be a lot of bugs to fix in the tests and some in @queengooborg do you think "changed files" is an OK way to track this, or how would you ideally want to define the burndown list and split into manageable chunks? |
|
I pushed a commit that adds the ability to specify I opted to specify "changed files" as the metric in my report above as it was the easiest for me to quickly pull numbers for, but the best metric would probably be the diff count, since some files may have many more changes than others (especially bigger interfaces like Document and Element). I'm also thinking that the best way to split this would be per category and browser (ex. API: Chrome), with project tasks for each grouping of releases (ex. 2022 releases, 2021, falsy...). It would look something like this:
|
|
That breakdown looks pretty good to me! My assumption is that after Falsy + 2022 Releases + 2021 Releases, the kind of exercise I did in #85 (comment) would produce no changes at all given the post-2020 results. In other words, the only remaining changes that the collector could make should be ones about pre-2020 differences, that it would need pre-2020 data to know about. I point this out because there's a difference between "update BCD to reflect all additions/removal in release X" and "update BCD to be consistent with release X", and it's the latter I have in mind for this project. |
|
fwiw mdn/browser-compat-data#19188 is the specific request to generate statistics for this goal. |
|
We have been using mdn/browser-compat-data#19585 as a tracker and now reached 100 files of discrepancies. Working on getting these resolved, too. |
|
At 60 files now. Aiming to finish this by the end of the quarter. |
|
We have finished this goal as of today! Whoo! There are a few interfaces that were skipped: PushManager and PushSubscription (these two require an overhaul of the collector's code, which isn't worth it at this stage), as well as RTCStatsReport (which requires a WebRTC expert to verify our tests). |
|
This is fantastic news, thank you @queengooborg for all your hard work on this! |
|
See #167 (comment) for how large the efforts would be if we would go further into the past. |
#66 and similar efforts in the past have eliminated all true and null values in BCD for API, CSS and JS data. @queengooborg and I used https://mdn-bcd-collector.gooborg.com/ to finish the API data over the course of 2021, and the collector also covers a lot of CSS and JS.
We have found that comparing BCD to feature detection results invariably finds a lot of errors in BCD. By comparing BCD with the collector results, we can find many false positives and false negatives. This will likely lead to many features looking "better" (more green) as support has shipped in the past 2 years but BCD was never updated.
Such cross-checking can in principle go back over 10 years, but I suggest limiting it to browser releases after 2020-01-01, and updating earlier data only when it's really easy and does not require much digging to verify.
For reviewers, here's a link to the guide on how to review collector PRs: https://github.com/GooborgStudios/mdn-bcd-collector#reviewing-bcd-changes
The text was updated successfully, but these errors were encountered: