Absence of existing elements with large suite with many scenarios

[johnsickels]

I have a large suite with 65 scenarios, multiple sites, identical assertions. When I run only 5 of the scenarios, they pass, but when I attempt all 65 the same assertions fail. I've identified which items are absent in the tests:

flo-trending ul li
flo-most-watched .most-watched-container div

Steps to repro:

• Run the suite as is and see mostly success.
• Uncomment the nested forEach to unleash 12*5 more scenarios. (lines 86-91)
• Observe failures:

   ✕  Has more than 4 trending items 
   …  Actual value: 0 
   ✕  Has more than 3 most watched videos 
   …  Actual value: 0 

The lis in these assertions exist in a smaller suite, but are null in a suite with many scenarios. I see null when I comment the length while attempting all 65 scenarios:

context.comment(floTrending.length.toString());

Here is the code:

/**
 * Smoke - Platform Front-End
 *
 * Tests a sample of all public pages.
 * Tests the homepage of several sites.
 */

import { Flagpole } from "flagpole";
import { iAssertionContext } from "flagpole/dist/interfaces";

const THRESHOLD_LOAD_TIME = 5000;
const basicPageChecks = (context: iAssertionContext) => {
  context
    .assert("HTTP status is 200", context.response.statusCode)
    .equals(200)
    .assert(
      `Should load within the threshold, ${
        Number(context.response.loadTime) / 1000
      } seconds`,
      context.response.loadTime
    )
    .optional.lessThan(THRESHOLD_LOAD_TIME);
};
const sites = [
  {
    base: "https://www.flotrack.org",
    name: "FloTrack",
  },
  {
    base: "https://www.flowrestling.org",
    name: "FloWrestling",
  },
  {
    base: "https://www.flograppling.com",
    name: "FloGrappling",
  },
  {
    base: "https://www.flobowling.com",
    name: "FloBowling",
  },
  {
    base: "https://www.flofc.com",
    name: "FloFC",
  },
];
const pages = [
  "/articles",
  "/events",
  "/events/6180217-2018-di-ncaa-xc-championships/videos?playing=6275915&limit=20",
  "/rankings",
  "/results",
  "/training",
  "/films",
  "/live/8160-2019-bu-john-thomas-terrier-classic/",
  "/video/6329008-tasty-race-bullis-school-4x400m-hs-national-record",
  "/search?q=video&page=1&limit=10",
  "/login",
  "/signup",
];
const suite = Flagpole.Suite("Smoke - Platform Front-End");

sites.forEach((site) => {
  suite
    .html(`${site.name} Homepage Loads`)
    .open(site.base)
    .next(basicPageChecks)
    .next(async (context) => {
      context
        .assert("Header Exists", await context.find("flo-header i svg"))
        .length.greaterThan(0);

      context
        .assert(
          "Has more than 4 trending items",
          await context.findAll("flo-trending ul li")
        )
        .length.greaterThan(4);

      context
        .assert(
          "Has more than 3 most watched videos",
          await context.findAll("flo-most-watched .most-watched-container div")
        )
        .length.greaterThan(2);
    });
  // pages.forEach((page) => {
  //   suite
  //     .html(`Page Test - ${site.name} ${page}`)
  //     .open(site.base + page)
  //     .next(basicPageChecks);
  // });
});

Flagpole: 2.2.12
Node: 12.13.1

[jasonbyrne]

Still investigating this. I did get it to work by doing this

 setTimeout(() => {
    pages.forEach((page) => {
      suite
        .html(`Page Test - ${site.name} ${page}`)
        .open(site.base + page)
        .next(basicPageChecks);
    });
  }, 100);

This makes the execution of those pages out of band with the homepage checks. So it does seem to be some sort of concurrency thing or them stepping on each other's toes... which i think we've seen happen before. Will need some more digging

[jasonbyrne]

Okay... still don't know the root cause and it does need to be addressed... BUT... with the new reactors in 2.4.0 the suite.setConcurrencyLimit works now. I set it to 5, for example, which limits it to five active scenario http requests at once. This resolved it. So it's obviously something with the responses stepping on each other.