Skip to content

Conversation

@Dinnerbone
Copy link
Contributor

Example:

    -- Color Differences --
    Total > 0: 97201
    difference |     amount | percent | withPriors | remaining
             1 |      95994 |  98.76% |     98.76% |      1207
             2 |        413 |   0.42% |     99.18% |       794
             3 |        185 |   0.19% |     99.37% |       609
             4 |         93 |   0.10% |     99.47% |       516
             5 |         78 |   0.08% |     99.55% |       438
             6 |         57 |   0.06% |     99.61% |       381
             7 |         44 |   0.05% |     99.65% |       337
             8 |         28 |   0.03% |     99.68% |       309
             9 |         33 |   0.03% |     99.72% |       276
            10 |         22 |   0.02% |     99.74% |       254
            11 |         24 |   0.02% |     99.76% |       230
            12 |         21 |   0.02% |     99.78% |       209
            13 |          4 |   0.00% |     99.79% |       205
            14 |          2 |   0.00% |     99.79% |       203
            15 |          5 |   0.01% |     99.80% |       198
            16 |          8 |   0.01% |     99.80% |       190
            17 |          5 |   0.01% |     99.81% |       185
            18 |          8 |   0.01% |     99.82% |       177
            19 |          4 |   0.00% |     99.82% |       173
            20 |          6 |   0.01% |     99.83% |       167
            21 |          5 |   0.01% |     99.83% |       162
            22 |          4 |   0.00% |     99.84% |       158
            23 |          3 |   0.00% |     99.84% |       155
            24 |          1 |   0.00% |     99.84% |       154
            25 |         23 |   0.02% |     99.87% |       131
            26 |          2 |   0.00% |     99.87% |       129
            27 |          3 |   0.00% |     99.87% |       126
            28 |          1 |   0.00% |     99.87% |       125
            30 |          2 |   0.00% |     99.87% |       123
            40 |          2 |   0.00% |     99.88% |       121
            41 |          3 |   0.00% |     99.88% |       118
            42 |          1 |   0.00% |     99.88% |       117
            43 |          2 |   0.00% |     99.88% |       115
            44 |          1 |   0.00% |     99.88% |       114
            45 |          2 |   0.00% |     99.88% |       112
            46 |          1 |   0.00% |     99.89% |       111
            47 |          1 |   0.00% |     99.89% |       110
            48 |          2 |   0.00% |     99.89% |       108
            49 |          3 |   0.00% |     99.89% |       105
            50 |          2 |   0.00% |     99.89% |       103
            51 |          1 |   0.00% |     99.90% |       102
            53 |          1 |   0.00% |     99.90% |       101
            54 |          1 |   0.00% |     99.90% |       100
            55 |          1 |   0.00% |     99.90% |        99
            56 |          1 |   0.00% |     99.90% |        98
            60 |          1 |   0.00% |     99.90% |        97
            69 |          1 |   0.00% |     99.90% |        96
            83 |          1 |   0.00% |     99.90% |        95
            91 |          1 |   0.00% |     99.90% |        94
           115 |          1 |   0.00% |     99.90% |        93
           126 |          1 |   0.00% |     99.91% |        92
           130 |          1 |   0.00% |     99.91% |        91
           141 |          1 |   0.00% |     99.91% |        90
           145 |          1 |   0.00% |     99.91% |        89
           173 |          1 |   0.00% |     99.91% |        88
           199 |          1 |   0.00% |     99.91% |        87
           218 |          1 |   0.00% |     99.91% |        86
           227 |          1 |   0.00% |     99.91% |        85
           238 |          1 |   0.00% |     99.91% |        84
           239 |          1 |   0.00% |     99.91% |        83
           241 |          5 |   0.01% |     99.92% |        78
           242 |          7 |   0.01% |     99.93% |        71
           243 |         16 |   0.02% |     99.94% |        55
           244 |         12 |   0.01% |     99.96% |        43
           245 |         17 |   0.02% |     99.97% |        26
           246 |          7 |   0.01% |     99.98% |        19
           247 |          9 |   0.01% |     99.99% |        10
           248 |          5 |   0.01% |     99.99% |         5
           249 |          3 |   0.00% |    100.00% |         2
           250 |          2 |   0.00% |    100.00% |         0
    -----
    test avm2/away3d_advanced_shallow_water_demo ... FAILED

In this case, we know that we could e.g. set tolerance = 20, max_outliers = 167 without needing to back-and-forth test difference tolerances to find something that works

(bonus: gitignore the actual/difference files because I keep accidentally checking those in...)

@Dinnerbone Dinnerbone added A-tests Area: Tests & Test Framework T-feature Type: New Feature (that Flash doesn't have) labels Oct 23, 2025
@Dinnerbone
Copy link
Contributor Author

(@torokati44 I think we briefly discussed doing something like this a year or two ago, if you have a better idea to display this please let me know :D)

@kjarosh
Copy link
Member

kjarosh commented Oct 23, 2025

I implemented printing a histogram once; tolerance vs outliers (but I didn't create a PR because I didn't fine-tune it and then forgot :D). It was definitely more condensed than the table.

@Dinnerbone
Copy link
Contributor Author

I started aiming for a histogram but I've tried that a few times before, ended up never finishing it, and decided to just do a quick and dirty thing to get something in for today 😅

Sounds like you had a similar experience. Please feel free to do something better, I just want better info :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tests Area: Tests & Test Framework T-feature Type: New Feature (that Flash doesn't have)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants