Outputs both TAP and default PHPUnit formatting

[jcandan]

Description:

When using the nikeee\PhpunitTap\TapExtension configured via the <extensions> block in phpunit.xml, both TAP formatted output and the default PHPUnit text output are shown. The expected behavior is to have only the TAP output printed to stdout.

Environment:

• PHPUnit Version: 11.5.7
• PHP Version: 8.4.3
• phpunit-tap : v2.0.1

Steps to Reproduce:

1. Create a project with the phpunit.xml configuration.

<?xml version="1.0" encoding="UTF-8"?>
<phpunit>
  <extensions>
    <bootstrap class="nikeee\PhpunitTap\TapExtension" />
  </extensions>
  <testsuites>
    <testsuite name="Example Test Suite">
      <directory suffix="Test.php">./tests</directory>
    </testsuite>
  </testsuites>
</phpunit>

2. Add a test file (e.g., tests/FunctionsTest.php) containing both passing tests and a failing test. For example:

   <?php
   // tests/FunctionsTest.php
   
   use PHPUnit\Framework\TestCase;
   
   // Include the procedural functions
   require_once __DIR__ . '/../src/functions.php';
   
   final class FunctionsTest extends TestCase
   {
       public function testAdd(): void
       {
           $this->assertEquals(5, add(2, 3));
       }
   
       public function testAddWithNegativeNumbers(): void
       {
           $this->assertEquals(-2, add(-1, -1));
       }
   
       public function testAddWillFail(): void
       {
           // Intentionally failing test: Expect 2 but add(1, 5) returns 6
           $this->assertEquals(2, add(1, 5), '1 + 5 should equal 6, but we fail on purpose');
       }
   }

3. Run PHPUnit from the command line:

   ./vendor/bin/phpunit

Actual Output:

./vendor/bin/phpunit 
PHPUnit 11.5.7 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.3
Configuration: /path/to/projects/phpunit-tap-sandbox/phpunit.xml

..F                                                                 3 / 3 (100%)TAP version 13
1..3
ok - FunctionsTest::testAdd # time=4.389ms
ok - FunctionsTest::testAddWithNegativeNumbers # time=0.038ms
not ok - FunctionsTest::testAddWillFail # time=37.694ms
  ---
  message: 'Comparison Failure'
  severity: fail
  thrown: false
  actual: '6'
  expected: '2'
  ...

Time: 00:00.099, Memory: 8.00 MB

There was 1 failure:

1) FunctionsTest::testAddWillFail
1 + 5 should equal 6, but we fail on purpose here.
Failed asserting that 6 matches expected 2.

/path/to/projects/phpunit-tap-sandbox/tests/FunctionsTest.php:31

FAILURES!
Tests: 3, Assertions: 3, Failures: 1.

Expected Output:

Only the TAP formatted output should be printed (for example):

TAP version 13
1..3
ok - FunctionsTest::testAdd # time=...ms
ok - FunctionsTest::testAddWithNegativeNumbers # time=...ms
not ok - FunctionsTest::testAddWillFail # time=...ms
  ---
  ...

There should be no additional default PHPUnit summary output (header, progress dots, and final summary).

Please advise if this is an intended behavior, a configuration issue, or a bug. Any assistance or a workaround would be greatly appreciated.

[nikeee]

In general, having non-TAP-lines between the TAP output is not a problem according to the TAP spec. However, depending on the consumer implementation, it might become an issue that the TAP version header (TAP version 13) is not on a new line.

I don't know if its possible to suppress the original output, as php 10+ removed custom result printers.

You can use a designated output file as a workaround:

    <bootstrap class="nikeee\PhpunitTap\TapExtension">
      <parameter name="fileName" value="output.tap"/>
    </bootstrap>

[jcandan]

1. It seems to me that the output stream must be entirely valid TAP; any additional information must be formatted as comments (i.e., lines beginning with #). Arbitrary non-TAP output (like standard PHPUnit output) is not permitted.

2. I believe that the TAP version header must be the first line:

   To indicate that this is TAP14 the first line must be...

   Though, it would be possible to interpret that loosely since it is not explicitly stated for version 13 (which is unfortunate).

3. Pest is able to override the output. I imagine it might be possible to remove the original output.

   Though I've not yet dug into how that might work for your extension.

As an aside, perhaps you'd consider chiming in at pestphp/pest/issues/1350.

[nikeee]

It seems to me that the output stream must be entirely valid TAP; any additional information must be formatted as comments (i.e., lines beginning with #). Arbitrary non-TAP output (like standard PHPUnit output) is not permitted.

Quote form TAP 13 spec:

A harness must only read TAP output from standard output and not from standard error. Lines written to standard output matching /^(not )?ok\b/ must be interpreted as test lines. After a test line a block of lines starting with ‘—’ and ending with ‘…’ will be interpreted as an inline YAML document providing extended diagnostic information about the preceding test. All other lines must not be considered test output.

I believe that the TAP version header must be the first line:
[...]
Pest is able to override the output. I imagine it might be possible to remove the original output.

Open to PR (keep in mind that I'll reject AI-generated code or PRs).

[jcandan]

PR #32 produces the following result.

Thanks for taking the time to look this over for me.