Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace uninterruptible wait #26

Merged
merged 2 commits into from
Oct 21, 2024
Merged

Replace uninterruptible wait #26

merged 2 commits into from
Oct 21, 2024

Conversation

Bouncheck
Copy link
Contributor

Current cassandra-stress when set up to work for specific duration will wait that duration even if all consumer threads fail.
This PR adds 2 modifications:

  • removes the uninterruptible wait and in default case replaces it with awaiting on CountDownLatch used for monitoring ongoing consumer jobs. Maximum wait time is set to configured duration.
  • Adds another error settings option, that changes the behaviour to await for set duration or until first consumer thread fails instead of waiting for all of them to end.

Fixes scylladb/scylla-tools-java/issues/168

@Bouncheck
Interrupt StressAction wait if all threads end before duration.
d271e92 
Instead of using `Uninterruptibles.sleepUninterruptibly` for set duration
await on latch for completion of all threads or until duration elapses.
@Bouncheck
Add failFast error option
cf3140d 
Adds an option that will stop other consumer threads in case one of them fails
@Bouncheck
Copy link
Contributor Author

I was unsure if this should be created in scylla-tools-java repo or here. LMK if I should recreate it elsewhere.

@fruch
Copy link

fruch commented Oct 21, 2024

I was unsure if this should be created in scylla-tools-java repo or here. LMK if I should recreate it elsewhere.

this is a good start, and later if we need that backported, we'll need to apply to scylla-tools-java and older releases

@CodeLieutenant CodeLieutenant added the enhancement New feature or request label Oct 21, 2024
@CodeLieutenant CodeLieutenant merged commit 6d3313a into scylladb:master Oct 21, 2024
2 checks passed
@dkropachev dkropachev mentioned this pull request Oct 28, 2024
Open
CodeLieutenant added a commit to CodeLieutenant/scylla-cluster-tests that referenced this pull request Nov 21, 2024
@CodeLieutenant
fix(cassandra-stress): bump c-s version to 3.17.0
0c659c7 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
@CodeLieutenant CodeLieutenant mentioned this pull request Nov 21, 2024
Merged
2 tasks
CodeLieutenant added a commit to CodeLieutenant/scylla-cluster-tests that referenced this pull request Nov 21, 2024
@CodeLieutenant
fix(cassandra-stress): bump c-s version to 3.17.0
d96f633 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
CodeLieutenant added a commit to CodeLieutenant/scylla-cluster-tests that referenced this pull request Nov 23, 2024
@CodeLieutenant
fix(cassandra-stress): bump c-s version to 3.17.0
060696c 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
CodeLieutenant added a commit to CodeLieutenant/scylla-cluster-tests that referenced this pull request Nov 23, 2024
@CodeLieutenant
fix(cassandra-stress): bump c-s version to 3.17.0
03c7a2b 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
02997a6 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
3a51e66 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)

# Conflicts:
#	defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
f34103a 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)

# Conflicts:
#	defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml
@mergify mergify bot mentioned this pull request Nov 26, 2024
Merged
2 tasks
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
51b8390 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
This was referenced Nov 26, 2024
Merged
Merged
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
324a187 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)

# Conflicts:
#	defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
b9a9d49 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
7e39ba6 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
This was referenced Nov 26, 2024
Merged
Merged
Merged
mergify bot pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 26, 2024
@CodeLieutenant @mergify
fix(cassandra-stress): bump c-s version to 3.17.0
40ba2eb 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
@mergify mergify bot mentioned this pull request Nov 26, 2024
Merged
2 tasks
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 27, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
ad7b1c9 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 27, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
c5fa8f2 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 27, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
6216a42 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 27, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
03545a6 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 28, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
ecb6535 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 28, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
01a9efc 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 28, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
56b7e8b 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 28, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
4f27721 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Nov 28, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
e38e7e4 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)

# Conflicts:
#	defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml
fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this pull request Dec 1, 2024
@CodeLieutenant @fruch
fix(cassandra-stress): bump c-s version to 3.17.0
358eb0a 
Main reason for version change:

Using cassandra-stress 3.17 to mittigate
- Switch to ReplicaOrdering.RANDOM for select LBPs
  [32](scylladb/cassandra-stress#32)

Other Noticable Changes since the last version used in SCT:

- Add support for hostname verification
  [31](scylladb/cassandra-stress#31)
- Print thread dump on specific signals
  [27](scylladb/cassandra-stress#27)
- Replace uninterruptible wait
  [26](scylladb/cassandra-stress#26)
- Make it use DCAwareRoundRobinPolicy unless rack is provided
  [21](scylladb/cassandra-stress#21)
- feature(docker): adding support for dependabot
  [19](scylladb/cassandra-stress#19)

Signed-off-by: Dusan Malusev <[email protected]>
(cherry picked from commit 02997a6)

# Conflicts:
#	defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CodeLieutenant CodeLieutenant CodeLieutenant approved these changes

Assignees

@Bouncheck Bouncheck

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

cassandra-stress can keep running even if thread had failed
3 participants
@Bouncheck @fruch @CodeLieutenant