[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021

DarshitChanpura · 2023-07-17T18:35:01Z

What is the bug?

Windows CI has been flaky with recent runs for tasks, specifically citest, dlicRestApiTest.

See these runs for example:

https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/1 <-- 3 tasks failed
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/2 <-- 2 tasks failed (1 prev failed task passed)
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/3 <-- 1 task failed
https://github.com/opensearch-project/security/actions/runs/5577190458/attempts/5 <-- all tasks passed

Failure Details

	(citest, windows-latest, 11)	(citest, windows-latest, 17)
1st run	- org.opensearch.security.SecurityAdminTests.testSecurityAdmin - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled - org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.httpclient.HttpClientTest.testPlainConnection - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled	- com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticatorTest.initialConnectionFailureTest - com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testIntegLdapAuthenticationSSL - org.opensearch.security.SecurityAdminTests.testIsLegacySecurityIndexOnV7Index - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled - org.opensearch.security.multitenancy.test.TenancyPrivateTenantEnabledTests.testPrivateTenantDisabled_Update_EndToEnd - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testSensitiveMethodRedaction - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
2nd run	- org.opensearch.security.SecurityAdminTests.testSecurityAdminRegularUpdate - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery	- com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testAttributesWithImpersonation - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testWriteHistory - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction - org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution - org.opensearch.security.multitenancy.test.MultitenancyTests.testMt - org.opensearch.security.multitenancy.test.MultitenancyTests.testMtMulti - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
3rd run	- org.opensearch.security.SecurityAdminTests.testSecurityAdmin - org.opensearch.security.SecurityAdminTests.testSecurityAdminInvalidCert - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.MultitenancyTests.testMt - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate

Test Methods with failures across all mentioned runs

- org.opensearch.security.SecurityAdminTests.testSecurityAdmin
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesEnabled
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiRolesDisabled
- org.opensearch.security.multitenancy.test.MultitenancyTests.testTenantParametersSubstitution
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testDeleteByQuery
- org.opensearch.security.httpclient.HttpClientTest.testPlainConnection
- com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticatorTest.initialConnectionFailureTest
- com.amazon.dlic.auth.ldap2.LdapBackendIntegTest2.testIntegLdapAuthenticationSSL
- org.opensearch.security.SecurityAdminTests.testIsLegacySecurityIndexOnV7Index
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
- org.opensearch.security.multitenancy.test.TenancyPrivateTenantEnabledTests.testPrivateTenantDisabled_Update_EndToEnd
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testSensitiveMethodRedaction
- org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
- org.opensearch.security.multitenancy.test.MultitenancyTests.testMt
- org.opensearch.security.multitenancy.test.MultitenancyTests.testMtMulti
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testWriteHistory
- org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser
- org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate
- org.opensearch.security.SecurityAdminTests.testSecurityAdminInvalidCert
- org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest

What is the expected behavior?
No flakiness.

The text was updated successfully, but these errors were encountered:

peternied · 2023-07-17T19:04:15Z

[Triage] Thanks for filing, could we get a list of tests that are impacted?

cwperks · 2023-07-19T15:47:06Z

@DarshitChanpura it looks like there were a few fixes put into main for windows support that may not have been backported to 2.x.

Particularly this one: #2180

There is a list of issues that @peternied added to this PR description: #2291 - it may be best to backport the one's of those that can be backported

peternied · 2023-07-19T16:22:29Z

From #3032 (review)

CI / test (citest, windows-latest, 11) (pull_request) Failing after 29m

Tests with failures:
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testAutoInit
 - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testUpdate
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testAutoInit
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testComplianceEnable

DarshitChanpura · 2023-07-21T16:44:37Z

[Triage] Thanks for filing, could we get a list of tests that are impacted?

Updated the issue description to include two sections: 1. Run Details. 2. Test Methods with failure

DarshitChanpura · 2023-07-21T16:48:51Z

Seems like there were no failures in #3032 and #3037 after #3035 was merged which suggests this issue might have been fixed.

cwperks · 2023-07-21T21:55:52Z

I've seen these failures in a couple of places now:

Tests with failures:
 - org.opensearch.security.SecurityAdminTests.testSecurityAdmin
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testRestApiNewUser
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction
 - org.opensearch.security.auditlog.compliance.ComplianceAuditlogTest.testInternalConfig
 - org.opensearch.security.multitenancy.test.MultitenancyTests.testMultitenancyAnonymousUser
 - org.opensearch.security.multitenancy.test.MultitenancyTests.testMt
 - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll
 - org.opensearch.security.auditlog.integration.BasicAuditlogTest.testScroll
 - org.opensearch.security.httpclient.HttpClientTest.testSslConnectionPKIAuth
 - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest
 - org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testBCryptHashRedaction

When trying a single test suite locally like this:

./gradlew -x opensslCITest test --tests ComplianceAuditlogTest -i

I see that it fails in blocks like this and I can see the following error in the logs:

    [2023-07-21T17:52:31,538][WARN ][rest.suppressed] path: /humanresources/_doc/100, params: {pretty=, index=humanresources, id=100}
    org.opensearch.OpenSearchSecurityException: No user found for indices:admin/mapping/auto_put
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:356) ~[main/:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[main/:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1537) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.cluster.action.index.MappingUpdatedAction.sendUpdateMapping(MappingUpdatedAction.java:161) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.cluster.action.index.MappingUpdatedAction.updateMappingOnClusterManager(MappingUpdatedAction.java:126) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnPrimary$1(TransportShardBulkAction.java:415) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:646) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:467) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:531) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:412) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:124) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:223) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]

which I had seen before when testing out this PR: #2765

I created a branch to revert that change in 2.x locally to check if it would help with CI stability (https://github.com/cwperks/security/tree/check-2.x-build) and the tests did pass after the first try.

Its puzzling why that error is platform specific according to CI.

cwperks · 2023-07-24T13:18:59Z

I think #2765 may need to be reverted. After seeing the error on both main and 2.x on Friday I did more sanity testing on the RC build to try to produce the error outside of CI, but I was not able to.

To see examples of the error you can check a recent build on main and look for

org.opensearch.OpenSearchSecurityException: No user found for indices:admin/mapping/auto_put

or another on 2.x here

From what I have put together the issue is reproducible on many of the ComplianceAuditlogTest like this one.

and fails when writing the first document to the humanresources where under the hood its doing a PutMappingRequest (indices:admin/mapping/auto_put). I believe that this request does a BulkShardRequest (indices:data/write/bulk[s]) which spawns a BulkShardRequest[Primary] (indices:data/write/bulk[s][p]) and BulkShardRequest[Replica]
(indices:data/write/bulk[s][r])

The error is coming up during the derivative BulkShardRequest[Primary] and BulkShardRequest[Replica] requests.

After tracing through the code it looks like the assumption that the transient headers will still be available on the receiving end of a transport request on a local node may not be accurate. From doing some testing, it looks like direct requests still go through the same OutboundHandler and InboundHandler as transport requests.

Flow in the OutboundHandler

In an OutboundHandler the threadContext is sent as headers in the OutboundMessage
The threadcontext is serialized here
and in the ThreadContext it looks like only the request headers (not the transient headers) are written to the OutputStream: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/common/util/concurrent/ThreadContext.java#L812-L831

Flow in the InboundHandler

It gets the headers from the InboundMessage
It stashes the local threadcontext and puts the headers from the InboundMessage

I'm having some trouble tracing all of the calls, but it looks like more work may need to be done to skip the serialization-deserialization of the transient headers on local node calls.

I'm still doing testing on the RC to see if the issue is reproducible there.

DarshitChanpura · 2023-07-24T14:35:30Z

CI passed on #2765's merge to main (without any retries) so I don't believe that is the root cause. However, the failures seem oddly specific to requests involving action: indices:admin/mapping/auto_put (See this run for example)

All failures point to this source:

[{"type":"security_exception","reason":"No user found for indices:admin/mapping/auto_put"}],"type":"security_exception","reason":"No user found for indices:admin/mapping/auto_put"},"status":500

stephen-crawford · 2023-07-24T19:30:26Z

[Triage] This is being actively looked at by @DarshitChanpura.

DarshitChanpura · 2023-07-24T19:54:11Z

There are 3 mains areas of concern that were identified:

this failure was hidden behind test retries, and could not be uncovered until recently
we don't have 100% confidence in identifying whether Adds a check to skip serialization-deserialization if request is for same node #2765 introduces this
We may see 500s with "No user found for .." message in 2.9 clusters

and in the ThreadContext it looks like only the request headers (not the transient headers) are written to the OutputStream:

@cwperks From what I see in OutboundMessage class, the transient headers are being written to output stream: https://github.com/opensearch-project/OpenSearch/blob/3ee42920fa886bdf566f1bb7b1666ffa72d36f54/server/src/main/java/org/opensearch/common/util/concurrent/ThreadContext.java#L321-L325

reta · 2023-07-25T12:35:51Z

@DarshitChanpura do you need a hand with figuring out the cause?

DarshitChanpura · 2023-07-25T14:22:58Z

@DarshitChanpura do you need a hand with figuring out the cause?

Yes, please.

cwperks · 2023-07-25T17:55:03Z

In my local testing, the sender (SecurityInterceptor) and the receiver (SecurityRequestHandler) are using different threadpools even when handling a local node request. I don't think persistent threadcontext headers would help here if there are different threadpools.

A concept of a ThreadContextStatePropagator was added recently. This may help transmit the necessary transient headers on a local node.

reta · 2023-07-25T18:23:55Z

I don't think persistent threadcontext headers would help here if there are different threadpools.

We've been looking with @DarshitChanpura and the hypothesis here (not confirmed yet) is that the thread context may not work here: the SecurityInterceptor thread may finish before the SecurityRequestHandler (basically the SecurityRequestHandler won't see the stashed thread context but the previous one)

cwperks · 2023-07-25T18:29:56Z

One thing that's interesting, is that I see that sometimes it bypasses the sender during local requests and the logic to bypass is actually in core.

In TransportService, sendRequest first calls on getConnection
getConnection returns a localConnection if its the local node
The localNodeConnection calls sendLocalRequest which does not call on the AsyncSender at all. Requests to other nodes use the async sender

In the failing tests the AsyncSender is called somehow (on a local node request) which always precedes the failures seen in CI.

cwperks · 2023-07-25T19:17:50Z

In doing more local testing, I think I see an identity crisis happening where core thinks the local node is node1 and the security plugin thinks its node2. I'm looking into how this can happen now. Both the security plugin and core should be aware of node they are running on without any conflicts.

I added logging statements in TransportService.getConnection and then again inside SecurityInterceptor.sendRequestDecorate which is called directly after getConnection on remote node requests to determine that there was a discrepancy between what core thought the localNode was and what the security plugin thought the localNode was.

reta · 2023-07-25T20:25:48Z

I think I nailed it folks:

    public static DiscoveryNode getLocalNode() {
        return localNode;
    }

We run 3 nodes in single JVM, with 3 security plugins, only one of them wins. @DarshitChanpura I think at this moment - this is 100% test related problem that should not leak into production.

DarshitChanpura · 2023-07-25T20:34:28Z

This localNode is set by using clusterService.localNode() method upon node bootstrap:
https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L1379

And security plugin uses this localNode reference to then compare if request is for sameNode:

security/src/main/java/org/opensearch/security/transport/SecurityInterceptor.java

Lines 149 to 150 in 37aacdc

    
           final DiscoveryNode localNode = OpenSearchSecurityPlugin.getLocalNode(); 
        
           boolean isSameNodeRequest = localNode != null && localNode.equals(connection.getNode());

i'm not super-convinced on what would happen in a multi-node cluster when the request ends up in the same node. Will the thread always be shared between SecurityInterceptor and SecurityRequestHandler in that case?

cwperks · 2023-07-25T20:39:47Z

We run 3 nodes in single JVM, with 3 security plugins, only one of them wins. @DarshitChanpura I think at this moment - this is 100% test related problem that should not leak into production.

I think so too, I was not able to replicate the issue on the RC.

That being said I think the code path here in the SecurityInterceptor and the corresponding code in the receiver is dead code. In reality, the user is not serialized on local node requests because it takes the bypass route described here

DarshitChanpura · 2023-07-25T21:03:44Z

I will update here with an RCA and next steps soon.

davidlago · 2023-08-01T17:08:10Z

Closing this one as the major blocker was resolved. Future flaky tests can have their own issues.

DarshitChanpura · 2023-08-08T13:40:18Z

I will update here with an RCA and next steps soon.

Posted results here: #2724 (comment)

DarshitChanpura added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Jul 17, 2023

davidlago added the flaky-test Flaky Test issue label Jul 17, 2023

peternied removed the untriaged Require the attention of the repository maintainers and may need to be prioritized label Jul 17, 2023

peternied mentioned this issue Jul 19, 2023

Prevent message collection from being updated after message count has been received #2180

Merged

3 tasks

DarshitChanpura mentioned this issue Jul 24, 2023

Revert "Adds a check to skip serialization-deserialization if request is for same node (#2765) (#2973) #3046

Closed

3 tasks

stephen-crawford added the triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. label Jul 24, 2023

davidlago assigned DarshitChanpura Jul 25, 2023

cwperks mentioned this issue Jul 28, 2023

Investigate CI Failures #3064

Closed

davidlago assigned RyanL1997 Jul 31, 2023

davidlago unassigned DarshitChanpura Jul 31, 2023

davidlago closed this as completed Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021

[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021

DarshitChanpura commented Jul 17, 2023 •

edited

Loading

peternied commented Jul 17, 2023

cwperks commented Jul 19, 2023

peternied commented Jul 19, 2023

DarshitChanpura commented Jul 21, 2023

DarshitChanpura commented Jul 21, 2023

cwperks commented Jul 21, 2023 •

edited

Loading

cwperks commented Jul 24, 2023 •

edited

Loading

DarshitChanpura commented Jul 24, 2023

stephen-crawford commented Jul 24, 2023

DarshitChanpura commented Jul 24, 2023 •

edited

Loading

reta commented Jul 25, 2023

DarshitChanpura commented Jul 25, 2023

cwperks commented Jul 25, 2023 •

edited

Loading

reta commented Jul 25, 2023

cwperks commented Jul 25, 2023 •

edited

Loading

cwperks commented Jul 25, 2023 •

edited

Loading

reta commented Jul 25, 2023 •

edited

Loading

DarshitChanpura commented Jul 25, 2023 •

edited

Loading

cwperks commented Jul 25, 2023

DarshitChanpura commented Jul 25, 2023

davidlago commented Aug 1, 2023

DarshitChanpura commented Aug 8, 2023

[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021

[BUG] [CI] Investigate Flaky test failure for windows CI tasks #3021

Comments

DarshitChanpura commented Jul 17, 2023 • edited Loading

peternied commented Jul 17, 2023

cwperks commented Jul 19, 2023

peternied commented Jul 19, 2023

CI / test (citest, windows-latest, 11) (pull_request) Failing after 29m

DarshitChanpura commented Jul 21, 2023

DarshitChanpura commented Jul 21, 2023

cwperks commented Jul 21, 2023 • edited Loading

cwperks commented Jul 24, 2023 • edited Loading

DarshitChanpura commented Jul 24, 2023

stephen-crawford commented Jul 24, 2023

DarshitChanpura commented Jul 24, 2023 • edited Loading

reta commented Jul 25, 2023

DarshitChanpura commented Jul 25, 2023

cwperks commented Jul 25, 2023 • edited Loading

reta commented Jul 25, 2023

cwperks commented Jul 25, 2023 • edited Loading

cwperks commented Jul 25, 2023 • edited Loading

reta commented Jul 25, 2023 • edited Loading

DarshitChanpura commented Jul 25, 2023 • edited Loading

cwperks commented Jul 25, 2023

DarshitChanpura commented Jul 25, 2023

davidlago commented Aug 1, 2023

DarshitChanpura commented Aug 8, 2023

DarshitChanpura commented Jul 17, 2023 •

edited

Loading

cwperks commented Jul 21, 2023 •

edited

Loading

cwperks commented Jul 24, 2023 •

edited

Loading

DarshitChanpura commented Jul 24, 2023 •

edited

Loading

cwperks commented Jul 25, 2023 •

edited

Loading

cwperks commented Jul 25, 2023 •

edited

Loading

cwperks commented Jul 25, 2023 •

edited

Loading

reta commented Jul 25, 2023 •

edited

Loading

DarshitChanpura commented Jul 25, 2023 •

edited

Loading