Skip to content

Conversation

@Dinnerbone
Copy link
Contributor

Unfortunately there's too many failures here that I cannot list the details. We should make a tool to visualize all our known failures.

This is every test from misc-ming.all with some exceptions:

  • There's a SharedObjectTest which is useful, it has a couple of failures for us, but we need support for .sol loading and writing in our test framework
  • There's a few which test fscommand, external process launching, IPC between processes with LocalConnection, etc. I opted not to include those because we just don't support those - we can revisit them if we ever want to
  • A couple of tests are purely demos for users to interact with, and don't provide us with much gain
  • There's an interval test which we're flaky on because of imprecise timings. I decided to leave that for now but we can revisit if we want to see if it's an actual failure or just a bad test
  • There's a FlashVars test which is pretty incompatible with Ruffle's test runner, we neither have a way to specify the swf query parameters nor the specific FlashVars parameters. In fact, Ruffle largely has no distinction between those - this is possibly something we should investigate later and fix, but I can't include it as a known_failure test if there's no way to express the test
  • Some tests just don't seem to be built, so they aren't being included

All tests were ran in Windows FP 9 through 32, and split into folders such as fp11 when FP 11 changed the output of the test compared to a previous version. I suspect one or two may be different in FP 7 or below, but I have no debug builds of those to test with.

Expected image output is mostly from FP but some are from Ruffle when I know they're correct and it's just text differences.

     Summary [   2.705s] 120 tests run: 62 passed, 58 failed, 4020 (unrelated tests) skipped
        FAIL [   1.698s] tests::tests from_gnash/misc-ming.all/BeginBitmapFill
        FAIL [   0.415s] tests::tests from_gnash/misc-ming.all/BitmapDataDraw/fp11
        FAIL [   0.432s] tests::tests from_gnash/misc-ming.all/BitmapDataDraw/fp15
        FAIL [   0.430s] tests::tests from_gnash/misc-ming.all/BitmapDataDraw/fp9
        FAIL [   0.831s] tests::tests from_gnash/misc-ming.all/ButtonEventsTest
        FAIL [   0.419s] tests::tests from_gnash/misc-ming.all/ButtonPropertiesTest
        FAIL [   0.345s] tests::tests from_gnash/misc-ming.all/DefineTextTest
        FAIL [   0.455s] tests::tests from_gnash/misc-ming.all/DragDropTest
        FAIL [   0.422s] tests::tests from_gnash/misc-ming.all/DrawingApiTest
        FAIL [   0.440s] tests::tests from_gnash/misc-ming.all/EmbeddedFontTest
        FAIL [   1.265s] tests::tests from_gnash/misc-ming.all/GradientFillTest/fp10
        FAIL [   1.252s] tests::tests from_gnash/misc-ming.all/GradientFillTest/fp9
        FAIL [   0.296s] tests::tests from_gnash/misc-ming.all/KeyEventOrder
        FAIL [   0.846s] tests::tests from_gnash/misc-ming.all/NetStream-SquareTest
        FAIL [   0.397s] tests::tests from_gnash/misc-ming.all/PrototypeEventListeners/fp10
        FAIL [   0.370s] tests::tests from_gnash/misc-ming.all/PrototypeEventListeners/fp11
        FAIL [   0.402s] tests::tests from_gnash/misc-ming.all/ResolveEventsTest
        FAIL [   1.141s] tests::tests from_gnash/misc-ming.all/TextSnapshotTest/fp10
        FAIL [   1.153s] tests::tests from_gnash/misc-ming.all/TextSnapshotTest/fp9
        FAIL [   0.367s] tests::tests from_gnash/misc-ming.all/action_order/ActionOrderTest3
        FAIL [   0.346s] tests::tests from_gnash/misc-ming.all/action_order/ActionOrderTest4
        FAIL [   0.374s] tests::tests from_gnash/misc-ming.all/action_order/ActionOrderTest5
        FAIL [   0.347s] tests::tests from_gnash/misc-ming.all/action_order/PlaceAndRemove/fp10
        FAIL [   0.319s] tests::tests from_gnash/misc-ming.all/action_order/PlaceAndRemove/fp9
        FAIL [   0.369s] tests::tests from_gnash/misc-ming.all/action_order/action_execution_order_extend_test/fp10
        FAIL [   0.349s] tests::tests from_gnash/misc-ming.all/action_order/action_execution_order_extend_test/fp9
        FAIL [   0.341s] tests::tests from_gnash/misc-ming.all/action_order/action_execution_order_test/fp10
        FAIL [   0.352s] tests::tests from_gnash/misc-ming.all/action_order/action_execution_order_test4
        FAIL [   0.342s] tests::tests from_gnash/misc-ming.all/action_order/action_execution_order_test6
        FAIL [   0.377s] tests::tests from_gnash/misc-ming.all/attachExtImported
        FAIL [   0.330s] tests::tests from_gnash/misc-ming.all/callFunction_test
        FAIL [   0.420s] tests::tests from_gnash/misc-ming.all/displaylist_depths/displaylist_depths_test
        FAIL [   0.356s] tests::tests from_gnash/misc-ming.all/displaylist_depths/displaylist_depths_test2
        FAIL [   0.350s] tests::tests from_gnash/misc-ming.all/displaylist_depths/displaylist_depths_test3
        FAIL [   0.375s] tests::tests from_gnash/misc-ming.all/duplicate_movie_clip_test
        FAIL [   0.377s] tests::tests from_gnash/misc-ming.all/get_frame_number_test
        FAIL [   0.363s] tests::tests from_gnash/misc-ming.all/goto_frame_test
        FAIL [   0.343s] tests::tests from_gnash/misc-ming.all/init_action/InitActionTest
        FAIL [   0.331s] tests::tests from_gnash/misc-ming.all/init_action/InitActionTest2
        FAIL [   0.527s] tests::tests from_gnash/misc-ming.all/key_event_test
        FAIL [   0.412s] tests::tests from_gnash/misc-ming.all/loading/LoadBitmapTest/fp11
        FAIL [   0.403s] tests::tests from_gnash/misc-ming.all/loading/LoadBitmapTest/fp9
        FAIL [   0.451s] tests::tests from_gnash/misc-ming.all/loading/loadMovieTest
        FAIL [   0.381s] tests::tests from_gnash/misc-ming.all/loop/loop_test10
        FAIL [   0.345s] tests::tests from_gnash/misc-ming.all/loop/loop_test6
        FAIL [   0.335s] tests::tests from_gnash/misc-ming.all/loop/loop_test7
        FAIL [   0.016s] tests::tests from_gnash/misc-ming.all/masks_test
        FAIL [   0.712s] tests::tests from_gnash/misc-ming.all/matrix_test
        FAIL [   0.344s] tests::tests from_gnash/misc-ming.all/opcode_guard_test
        FAIL [   0.331s] tests::tests from_gnash/misc-ming.all/path_format_test
        FAIL [   1.209s] tests::tests from_gnash/misc-ming.all/place_object_test
        FAIL [   0.278s] tests::tests from_gnash/misc-ming.all/place_object_test2
        FAIL [   0.306s] tests::tests from_gnash/misc-ming.all/register_class/RegisterClassTest4
        FAIL [   0.328s] tests::tests from_gnash/misc-ming.all/register_class/registerClassTest2
        FAIL [   0.283s] tests::tests from_gnash/misc-ming.all/replace_buttons1test
        FAIL [   0.273s] tests::tests from_gnash/misc-ming.all/reverse_execute_PlaceObject2_test2
        FAIL [   0.284s] tests::tests from_gnash/misc-ming.all/runtime_vm_stack_test
        FAIL [   1.071s] tests::tests from_gnash/misc-ming.all/shape_test

@Dinnerbone Dinnerbone added A-tests Area: Tests & Test Framework T-chore Type: Chore (like updating a dependency, it's gotta be done) labels Oct 22, 2025
@Dinnerbone Dinnerbone force-pushed the gnash/ming branch 2 times, most recently from 7d46031 to 8335958 Compare October 22, 2025 23:14
@Dinnerbone
Copy link
Contributor Author

(this was so much work... and it's not their biggest testsuite... 😭)

@kjarosh kjarosh self-requested a review October 23, 2025 05:51
@kjarosh
Copy link
Member

kjarosh commented Oct 23, 2025

Which tests use output from Ruffle? The test framework generates nonoptimal images and usually they can be several times smaller when compressed properly.

@Dinnerbone
Copy link
Contributor Author

I don't recall which (I'd hazard a guess at it's "any passing test"), but that reminds me to put oxipng on my jj fix :D I'll amend the commit with compressed pngs

(I once suggested we oxipng all our generated pngs but nobody else was enthusiastic about that idea, fwiw)

@Dinnerbone
Copy link
Contributor Author

Copy link
Member

@kjarosh kjarosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't like having output not from FP, because I know for a fact that I'll be the one replacing it with the real output, but let's go :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tests Area: Tests & Test Framework T-chore Type: Chore (like updating a dependency, it's gotta be done)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants