[RFC]: Add the .test method to CSF factories

[kasperpeulen]

RFC: Add the .test method to CSF factories

In modern frontend development, Storybook has become the go-to tool for building and documenting UI components. It allows developers to create "stories" that highlight various component states, aiding both development and documentation. Yet, integrating comprehensive testing within Storybook can be challenging. This post examines the current testing limitations in Storybook and proposes a new syntax for seamlessly incorporating tests with a jasmine-style syntax on top of static stories.

Are stories tests?

Storybook provides testing features through hooks like play and beforeEach and leverages Vitest for a fast, browser-based test runner. However, using Storybook for testing means learning a new testing syntax and best practices built on top of the Component Story Format (CSF).

Every story, even those without a play hook, functions as a render test. Rendering a component in a known configuration and verifying that it doesn’t break is a basic yet valuable form of validation.

Even though those stories are run automatically in CI with our Vitest integration, traditional tests usually have clear assertions and specifications.

The Limitations of the play Function

The play function in Storybook allows users to add those assertions on top of a story. However, its limitation of supporting only one play function per story complicates the process of writing multiple tests against the same component state.

A common workaround is to create separate "test stories" with their own play functions. This approach often involves duplicating base stories and adding different play functions to test various behaviors:

export const meta  = {
  component: AddLabelForm,
} satisfies Meta<typeof AddLabelForm>

type Story = StoryObj<typeof meta>;

export default meta;

export const ManyLabels: Story = {
  args: { labels: [{...}, {...}] }}
};

export const SubmittingInvalidLabelShowsInlineError: Story = {
  ...ManyLabels,
  async play() { 
    await userEvent.type(canvas.getByText('Label:'), 'invalid label');
    await userEvent.click(canvas.getByRole('button'));
  
    await expect(...).toHave(...);
  }
})

export const AutocompletionSuggestionsAreShownWhenTyping: Story = {
  ...ManyLabels,
  async play() {
    await userEvent.type(canvas.getByText('Search label'), 'a label');
    await userEvent.click(canvas.getByRole('button'));
  
    await expect(...).toHave(...);
  }
})

export const NoLabels: Story = {
  args: { labels: [] }}
};

export const NoAutocompletionSuggestionsAreShownWhenTyping: Story = {
  ...NoLabels,
  async play() {
    await userEvent.type(canvas.getByText('Search label'), 'a label');
    await userEvent.click(canvas.getByRole('button'));
  
    await expect(...).toHave(...);
  }
})

This best practice is not obvious from an API perspective, but even if you discover this, it's not ideal. These test stories clutter the sidebar, which is problematic if Storybook is used primarily for documentation. And when you click on it, you might not expect the story to take 15 seconds to finish. Additionally, using PascalCase for lengthy spec names reduces readability.

We're essentially treating AddLabelForm and SubmittingInvalidLabelShowsInlineError as both a story and a test at the same time. But are they truly the same? Arguably not—they serve different purposes.

Embrace “Jasmine” test syntax on top of static stories

In the CSF factories proposal, we introduced this new factory syntax for writing stories:#30112

import { config } from '#.storybook/preview';
import { AddLabelForm } from './AddLabelForm';

const meta = config.meta({ component: AddLabelForm });

export const ManyLabels = meta.story({ 
  args: { labels: [{...}, {...}] } 
});

export const NoLabels = meta.story({ 
  args: { labels: [] } 
});

This proposal is about adding a .test method on the story object.

import { config } from '#.storybook/preview';
import { AddLabelForm } from './AddLabelForm';

const meta = config.meta({ component: AddLabelForm });

// the story can be seen as the arrange step in testing
export const ManyLabels = meta.story({ 
  args: { labels: [{...}, {...}] } 
});

// 👇 easy to write (and read!) full sentence specs!
ManyLabels.test('submitting invalid label shows inline error', async ({ canvas }) => {
  // act/assert
  await userEvent.type(canvas.getByText('Label:'), 'invalid label');
  await userEvent.click(canvas.getByRole('button'));
  
  await expect(...).toHave(...);
});

ManyLabels.test('autocompletion suggestions are shown when typing', async ({ canvas }) => {
  await userEvent.type(canvas.getByText('Search label'), 'a label');
  await userEvent.click(canvas.getByRole('button'));
  
  await expect(...).toHave(...);
});

export const NoLabels = meta.story({ 
  args: { labels: [] } 
});

NoLabels.test('no autocompletion suggestions are shown when typing', async ({ canvas }) => {
  await userEvent.type(canvas.getByText('Search label'), 'a label');
  await userEvent.click(canvas.getByRole('button'));
  
  await expect(...).toHave(...);
});

This model aligns with the natural thinking of stories as documented UI states and tests as actions performed on those states. Here, stories represent the "arrange" step of a test, while the test method handles the "act" and "assert" phases, mirroring familiar testing patterns.

The UI

To prevent tests from cluttering the sidebar, they only appear when you open a specific story.

You can also search for specific tests in the search bar.

Play gives you extra control over the “arrange” step

We created the args abstraction because it enables you to quickly define your key UI states, while also allowing you to adjust props in the controls panel interactively.

const meta = config.meta({ component: AddLabelForm });

export const ManyLabels = meta.story({ 
  args: { labels: [{...}, {...}] } 
});

However, sometimes you need more flexibility. What if you must set up mocks before rendering the component? What if you need to open a dropdown after the story has been rendered?

From a testing perspective, you can think of the play hook as providing full control over the arrange step in your tests:

export const LabelFormSettings = meta.story({ 
  async play({ mount, canvas }) {
    // create mock data
    const label = await db.labels.create({
      data: {
        title: 'Some label',
        createdBy: 'storybookjs',
      },
    });
    // pass it to the component
    await mount(<AddLabelForm id={label.id}/>);
    // open settings dropdown
    await userEvent.click(canvas.getByText('Label settings'));
  },
});

Sharing data

We can share data between the story and the test in a typesafe way:

export const LabelForm = meta.story({ 
  async play({ mount, canvas }) {
    const label = await db.labels.create({
      data: {
        title: 'Some label',
        createdBy: 'storybookjs',
      },
    });
    await mount(<AddLabelForm id={label.id}/>);
    return { label }
  },
});

AddLabelForm.test('submitting invalid label shows inline error', async ({ label }) => {
  await userEvent.click(canvas.getByText(label.title));
});

Overwriting parameters for a test

We can support an API that overrides story props such as parameters and tags:

const meta = config.meta({ component: AddLabelForm });

// the story can be seen as the arrange step in testing
export const ManyLabels = meta.story({ 
  args: { labels: [{...}, {...}] } 
});

ManyLabels.test('submitting invalid label shows inline error', 
  {
    parameters: {
      // Disables Chromatic's snapshotting on a test level
      chromatic: { disableSnapshot: true },
    },
    tags: ['some-tag']
  }, 
  async ({ canvas }) => {
    await userEvent.type(canvas.getByText('Label:'), 'invalid label');
    await userEvent.click(canvas.getByRole('button'));
  
    await expect(...).toHave(...);
});

[DEfusion]

This looks great and is better than what I was imagining when commenting on #22361

I love that you've put tons of thought into this especially things like them not cluttering the sidebar until you open the parent story, just come up with an awesome icon for them 😄

[Sidnioulz]

A derivative of the AAA pattern consists of writing data suites for tests, and then running tests on each item of a data suite.

How could we do that with stories acting as test data? Could I bundle together a bunch of stories (e.g. different variants of a button) and check they all satisfy a behaviour (e.g. all become aria-disabled once clicked and until the click handler returns, or all contain an aria-label and role button)?

I don't know how this should be written so here are a bunch of proposals in no particular order and to which I have no particular attachment, just for the sake of triggering conversation.

// meta providing an each method?
meta
  .each([Primary, Secondary, Tertiary, Frogginary])
  .test('prevent double-submit by setting aria-disabled while the handler is running for variant %s', () => {
  // act
  // assert
})

// With a test global?
test
  .each([Primary, Secondary, Tertiary, Frogginary])
  ('prevent double-submit by setting aria-disabled while the handler is running for variant %s', () => {
  // implicit arrange?
  // act
  // assert
})

// If there is a test global, does it enable other syntax formats, e.g.
test('prevent double-submit by setting aria-disabled while the handler is running', () => {
  arrange(Primary); // or render(Primary), etc. naming here indicates purpose and is not an actual suggestion
  // act
  // assert
})

// Leading to an explicit handling of arrange in test.each?
test
  .each([Primary, Secondary, Tertiary, Frogginary])
  ('prevent double-submit by setting aria-disabled while the handler is running for variant %s', (story) => {
  arrange(story);
  // act
  // assert
})

[shilman]

On a related note we're considering bringing modes to Storybook

https://www.chromatic.com/docs/modes/

[Sidnioulz]

So you did tell me about modes! I missed that message!

On the naming: I feel modes is a fantastic name for that. It aligns with the vocabulary that Figma uses for contextual information that informs the values of Figma variables, and I think that's great news for DS folks.

I feel like modes and stories-as-data-suites are two distinct concepts that can be combined. If I went back to the OG RFC syntax and wanted to loop over my data, because my data is the object that provides the test method, I'd have an unusual syntax:

[Primary, Secondary, Tertiary, Frogginary].forEach(
  (story) => story.test(
    // Note the need to name / uniquely identify stories, I don't know if this has been discussed yet?
    `prevent double-submit by setting aria-disabled while the handler is running for variant ${story.name}`,
    () => {
      // act and assert, assuming arrange is done by the story
    }
  )
)

But this opens up a possibility for stories themselves to have an each method that takes modes as parameters.

const modes = ["light mobile", "dark mobile", "light desktop", "dark desktop"]

[Primary, Secondary, Tertiary, Frogginary].forEach(
  (story) => story.each(modes).test(
    `prevent double-submit by setting aria-disabled while the handler is running for variant ${story.name} and modes %s`,
    (modes) => {
      // now we probably need an explicit arrange where we pass the modes
      // act and assert
    }
  )
)

Of course you could create an eachMode method that hides away all the mode arranging and test duplication/naming, or any other such mechanism.

That begs the question whether end users need to be able to disable modes for a story's tests. If so, can that be done by defining modes on the story and having the test routine pick that up automatically ? If so, then all the conversation about providing API for explicit mode targeting is moot, isn't it?

[IntusFacultas]

This would be a dream come true, I wholeheartedly approve of this proposal.

The tiniest bit of feedback would be perhaps we can try to visually distinguish the tests in the sidebar when expanded so that they're more obviously "this is a test rather than a traditional story", that would be great since we also use Storybook for non-technical folks to be able to explore the current state of the application development.

Perhaps something like replacing the bookmark with a small "T" or "Test" (space depending) in some kind of badge styling?

[Sidnioulz]

@shilman do you provide tags for these test stories? I could augment https://github.com/Sidnioulz/storybook-addon-tag-badges's default config to display the badges already, if so.

[shilman]

Since .test will be a core feature we'll re-design the sidebar around it. It will probably show up with a different icon at the very least and we might even handle its interactions differently such as making it easy to hide/show all tests as a first class interaction

[domyen]

Do Storybook features like tags and parameters apply to these tests?

[shilman]

Yes. Under the hood a test would be very similar to a story and I believe we'll make it possible to configure tags/parameters/etc as an optional second argument to the test method.

[samsamit]

This looks awesome! This system could also benefit the ability to provide afterEach, beforeEach, ie. for tests in global or local scope. When Vitest is already being used under the hood implementing more vitest functionalities in this .test method would provide more usecases for this and might make writing Vitest tests unessessary in the future.

[Sidnioulz]

There is another RFC for afterEach: #29583

[MarceloPrado]

I love this! We've having similar organizational challenges here at Brex with the current approach. Having a clear separation between what's interaction testing and what's meant as documentation/snapshot testing would be awesome

[scottrippey]

Excellent proposal. I love the syntax, and love the idea of using play to arrange.

I do have a suggestion for the UI. IMO, the sidebar is not the right place to show these tests. Each storybook "tells a story" and the sidebar is how we navigate that story.
The tests are not a part of that story. They're just an implementation detail. Almost like documentation about the behavior.

So I would suggest these tests belong in the bottom "Component Tests" panel. It currently shows the steps of the current test. Imagine if this panel was split, and the left side was the list of tests. You select one from the list and watch it run.

[mpiroc]

Hi, I mentioned this feedback in yesterday's Storybook 9 webinar and was asked to share it here:

I use BDD-style (i.e. nested describes and its) organization for all of my tests. If Storybook only supports flat test suites (i.e. just a list of tests), I won't migrate my existing component tests to Storybook after this RFC is implemented.

With this RFC, I would like to be able to use all of vitest's organization features, including nested describes and its.

[shilman]

Thanks for sharing @mpiroc. I have questions!

Storybook already has an organization system that produces the hierarchical structure in its sidebar. If we were to generate Vitst tests that with the same structure would that be sufficient? If not, what's the biggest thing missing?

If we were to add a describe-style construct to organize your tests underneath a component, would that satisfy your needs?

[kettanaito]

Given the test-related hierarchy with describe and its potential visual clash with the rest of the items in the story tree, I would strongly suggest to consider displaying test suites and test cases in a separate view. Something like a "Tests" tab, perhaps?

In the end, tests aren't stories and it would be great to be able to view all the tests reflecting the user's describe/it/test nesting in the UI (and get inspired from Playwright with the "Go to test" button that opens respective test cases in your IDE!).

This UI is simply more future-proof, allowing you to build better experienced on it with time, like re-running a subset of tests, for example, from Storybook directly.

[kettanaito]

We've also had a great discussion with @MichaelArestad about the best way to present the testing information to the user. I still believe that a separate tab would be the best way forward. Storybook has quite a lot of horizontal real estate, which can be used to a great effect to display test steps, reference the source code, have rich actions next to the test cases (re-run, go to source, etc).

I also suggest outlining the user stories first. When would a Storybook user reach out to a test? When would they want to find one? Are they exploring a new code base, did they come from a failed CI—what do they wish to do at this moment? The answers to all of those will help shape the UI to serve the users better.

[Stadly]

This is grand! What is the status? Is there a branch or (draft) PR to look at?

[shilman]

Prototype here #31551

[lifeiscontent]

@shilman might be worth doing something like this via test.extend, you could probably model the same APIs where extend here would be able to allow you to have typed contexts for the test, and potentially use it to setup the metadata for storybook.

https://vitest.dev/guide/test-context.html#test-extend

[shilman]

This something that @kasperpeulen has explored. I don't know where we landed with that though.

[yannbf]

Hey everyone! Thank you so much for your comments and thoughts. While we can still add comments in this RFC, I'd like to invite you all to a new channel in our discord, #sb-test-early-access.

We will be implementing this RFC and would like to soon provide you canary versions with some prototypes for early feedback. We'd also like to hear more about your motivations, needs, etc, so we can note them down to consider. Please join and share your thoughts!

Pinging you all of you who have shown interest ❤️
@alextompkins @Blitz2145 @Catchpowle @DEfusion @gurobokum @honey32 @IntusFacultas @k1tikurisu @lifeiscontent @lishaduck @lukasjelonek @MarceloPrado @mikegreiling @mishaberezin @mmirus @mpiroc @noah-potter @nus3 @Parkreiner @pdemarino @philwolstenholme @rajeshshou @samsamit @scottrippey @Sidnioulz @skinread @Stadly @tommy-mitchell

[Stadly]

Nice! I just get this when visiting the link:

[yannbf]

If you're not part of our discord yet, you can join by clicking here! Then the channel should be available.

[yannbf]

Hey there! We just posted instructions on discord on how to try out our first canary release that includes the test syntax work. Please check it out!

[yannbf]

Hey everyone! Sorry again for the ping (last time, I promise!), but it's quite important that we gather feedback while we are developing the feature!
We already have a prerelease (based on Storybook 9) that contains the work, and we'd really appreciate if you tried it out and filled a small form to share your experience.

Check out the proposal here, and the feedback form at the bottom.

Thank you so so much 🙏

@cabbiepete @alextompkins @Blitz2145 @Catchpowle @DEfusion @gurobokum @honey32 @IntusFacultas @k1tikurisu @lifeiscontent @lishaduck @lukasjelonek @MarceloPrado @mikegreiling @mishaberezin @mmirus @mpiroc @noah-potter @nus3 @Parkreiner @pdemarino @philwolstenholme @rajeshshou @samsamit @scottrippey @Sidnioulz @skinread @Stadly @tommy-mitchell

[lishaduck]

Are there any plans to support Svelte CSF?

[shilman]

I believe there's a proposal somewhere. @JReinhold can you please share?

[JReinhold]

@shilman its not public yet, as we haven't aligned on it within the team yet.