OneTimeTeardown failures don't show up in results (formerly: Include TearDown errors on stdout)

[emwl]

Our Renovate bot just updated our test adapter use to 5.1.0, and some of our projects started to fail on CI with for no apparant reason.

Turns out that we had a masked NullReferenceException in a tear down method, caused by running tests in parallel and accidentally clearing a non-thread-safe shared resource there. With #1247 this now pops up on stderr, but I couldn't find any way to view the stderr output in a TRX file from a VsTest run on Azure DevOps server. (At least short of downloading the TRX and opening it with a text editor.)

The VsTest task echoes everything from stdout (which is also part of the TRX, as a <StdOut> element), but anything stderr just...vanishes in a seemingly ignored <RunInfo> element.

Would it be possible to (also) write those setup/teardown errors to stdout (in addition to stderr)?

It would make analyzing such failures a lot less painful, since they're currently not visible in AzDO. If anything fails there, our devs will look at the job and review the step that failed, only to see that all tests are green but the run failed anyways.

[OsirisTerje]

Good to hear you actually found a hidden bug in your code :-)

Now, I tried to repro your case:

I tried this with the code here : https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1290 and running it with dotnet test --logger "trx"

The results in the TRX file is:

You write " but anything stderr just...vanishes in a seemingly ignored element.", but looking at the trx file I got out above it seems it is where it should be.

I have not tried VSTest, but might be it does something weird there with StdErr. It doesn't really make sense, since StdErr should be more important than StdOut.

PS: Is this because you're using the classic (aka Old) pipeline, instead of using the new yaml based pipelines? If you changed to that and using dotnet test, you should be good.

@nohwnd @Evangelink Do you guys have any comments on this ?

UPDATE:

Tried it with vstest.console, which is what the VSTest task is using (afaik), and it looks good there too:

[emwl]

TL;DR: It's a OneTimeTearDown that fails for me, which apparently isn't associated with any of the tests inside of the fixture when you look at the TRX.

---

You write " but anything stderr just...vanishes in a seemingly ignored element.", but looking at the trx file I got out above it seems it is where it should be.

Well, it is in the TRX, but AzDO wont show it anywhere.

The job log looks like this (and that's the only error or warning in there; nothing else is red or orange):

And the test result section looks like this:

(Yes, there is a filter on the outcome; which is "Aborted" and "Failed" by default, but removing it just shows all green.)

I have not tried VSTest, but might be it does something weird there with StdErr. It doesn't really make sense, since StdErr should be more important than StdOut.

I agree. And it does show up on console, it just never makes its way into the output log of AzDO. As far as I can tell, all it includes is this in the TRX:

    <RunInfos>
      <RunInfo computerName="AGENT02" outcome="Error" timestamp="2025-08-11T07:41:35.7860564+02:00">
        <Text>TearDown failed for test fixture Project1.General.SomeLogicTests</Text>
      </RunInfo>
      <RunInfo computerName="AGENT02" outcome="Error" timestamp="2025-08-11T07:41:35.7863371+02:00">
        <Text>TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
StackTrace: --TearDown&#xD;
   at Project1.Logic.Fancy.UpdateSomething() in E:\Agent\_work\61\s\Project1\General\Logic\Fancy.cs:line 67&#xD;
   at Project1.Logic.Fancy.Reset() in E:\Agent\_work\61\s\Project1\General\Logic\Fancy.cs:line 51&#xD;
   at Project1.General.SomeLogicTests.TearDown() in E:\Agent\_work\61\s\Project1.Tests\General\SomeLogicTests.cs:line 71</Text>
      </RunInfo>
    </RunInfos>

...but nothing as part of the actual test case. It simply looks like this (wrapped for readability):

<UnitTestResult executionId="0c68742d-36fc-48c3-8ba1-dc06f5416fb9"
     testId="cbd45323-abb3-85ea-588c-28cc8b3df690"
     testName="IsThisFancyOrNot(42)"
     computerName="AGENT02"
     duration="00:00:00.0180000"
     startTime="2025-08-11T07:41:35.6867715+02:00"
     endTime="2025-08-11T07:41:35.7042821+02:00"
     testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b"
     outcome="Passed"
     testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d"
     relativeResultsDirectory="0c68742d-36fc-48c3-8ba1-dc06f5416fb9" />

And now that I stare at this...it's a parameterized test using TestCaseSource. Yours is just a regular Test. It's also a OneTimeTearDown (for the whole fixture), not just a TearDown (once per test).
Guess that was something I overlooked since the TRX only says "TearDown" (but I never really looked at the actual test until now).
Maybe that's the difference there (and something else making this not work as I'd expect it)?

It looks a bit like this:

namespace Project1.General;

public class SomeLogicTests
{
    private Fancy _fancy;
    [OneTimeSetUp]
    public void SetUp() { _fancy = new(); } // this is a bit more involved in terms of setup, so it's only done once for the fixture
    [OneTimeTearDown]
    public void TearDown() { _fancy.Reset(); } // this throws somewhere internally

    private static object[] GetTestCases() => [ 23, 42, 69 ];
    [TestCaseSource(typeof(SomeLogicTests), nameof(GetTestCases)]
    public void IsThisFancyOrNot(int theNumber) => Assert.That(_fancy.Check(theNumber), Is.True);
}

PS: Is this because you're using the classic (aka Old) pipeline, instead of using the new yaml based pipelines? If you changed to that and using dotnet test, you should be good.

This is a pretty old project that's still on .NET Framework 4.8; and while we have a lot of modernizations done (SDK-style project file, more recent C# version with PolySharp), but we're not ready for moving to .NET 9+ just yet. I can see if dotnet test changes anything, but I vaguely remember that it had some other drawbacks a few years back when we first looked at this. (Then again, that was quite a few VS and AzDO releases ago.)

Also, yes, classic pipeline. There's a lot going on, and we never had the time to really look into yaml based pipelines.

[OsirisTerje]

It's a OneTimeTearDown that fails for me, which apparently isn't associated with any of the tests inside of the fixture when you look at the TRX

OneTimeTearDown belong to the Test Fixture, and we send results back for each test case. We can't add anything to those.
This has been up several times earlier, and there are a few ways to handle it. I believe MSTest added another ghost testcase to cover this possibility. We could do the same.

The result comes out in the Trx in the RunInfo as you mentioned above, which is because there is no test to attach it to, and the Trx doesnt have any TestFixture concept.
Since the info is there (in the Trx), the VSTest task should react to it. It may be wise to add this is as an issue to the VSTest Task guys - @nohwnd @Evangelink - is that you guys ? Or can you push this issue to the right people.

[OsirisTerje]

Just for info, the same behavior can be seen in Visual Studio:

[nohwnd]

Unfortunately no, we don't own that task, VSTest task and the surrounding UI is managed by @skmanoj

[OsirisTerje]

@nohwnd But do you own the TestExplorer / testhost ? It is the same there.

[emwl]

More of a side question, but was the stacktrace prefix "TearDown" chosen intentionally over something like "Fixture TearDown" or "OneTimeTearDown"? I feel like that would've pointed me towards the right direction more quickly. (And, obviously: Is that part of the test adapter, or is it part of the host?)

[nohwnd]

yes, testhost is owned by me.

[manfred-brands]

@emwl "--TearDown" comes from nunit itself.