[Discussion] Unit testing of SST

[mkilp]

Hi Everyone,

In light of some people being added to the repo as volunteer maintainers and me wanting to contribute more in the future as well, I'd like to open a discussion about how we could tackle testing the SST repo.

One approach that comes to mind is using sst state export or sst diff to generate a static export of the pulumi up command.
The only problem I see with this is that it requires a prior deploy (at least to my knowledge).

I'd love to look into this myself and try constructing something but want to get some input from folks here what they thing might be a good approach. Happy to put the work in with some guidance.

cheers

marv

[vimtor]

some ideas i had while reviewing lots of prs:

1. tidy up every example in the examples/ folder and run snapshot testing across all examples
2. create a github app or actions that let maintainers deploy specific examples on demand (to avoid the aws account abuse)
3. create a github action that builds the platform, cli and docs

i haven't had time to tackle any of these. number 3 should be the easiest win. if anyone wants to open a pr with it i’ll gladly review

[mkilp]

@vimtor I can take care of 3 today and test it in my fork.

Regarding the others, especially 2) - If SST supplied an "empty" AWS account to diff against, I feel like we could make it work in an action.

My idea:

1. Actions gets short lived token via OIDC provider to SST "test" AWS account with a READ ONLY role
2. Then runs a sst diff from whatever unit test / example folder is currently testing
3. sst diff should produce a reproducible output that we can compare against what it should look like. This should include every single change the component under test makes because the AWS account is empty.
4. Nothing gets ever actually deployed (maybe later like you said for maintainers?) and we just make sure the cli works as intended and all generated values are correct e.g. Names, VPCs, Routes, IPs, IAM roles etc. which are needed for high level components. For example: a sst.Function requires an IAM Execution role, so I would assume sst diff creates that?

Still a couple of things to really figure out here, especially how we hook into the CLI commands and what language we run the unit tests in (TS vs. Go - I vote TS) but those are just my 2 cents.

[jamesgibbons92]

Yeh I would avoid anything like that that requires connecting to a real cloud provider. Most of what you mentioned there to be validated can be done in unit / integration tests I imagine.

Pulumi have some guides for unit and integration testing, we can look through this for best approach (and also see what the providers themselves do in their repos).
https://www.pulumi.com/docs/iac/guides/testing/
I had a quick skim but it seems we might be able to mock various parts of the pulumi engine, but needs a proper look to see what makes sense for sst.

On the typescript / component side we can start adding unit tests to platform/test (there are 2 in there now, look like someone experimenting not sure if they work). There are some components where we have complex logic which we can easily start writing tests for now - for example the step functions component has a few testable methods (e.g. serializing the definition) - LLMs should be good at helping write tests for these so might be good to get started with quickly.

The go packages should also have their own unit tests written in go as well ideally, I haven't had a look at how testable they are just yet