-
Notifications
You must be signed in to change notification settings - Fork 6
TestingTerminology
I've been involved in loads of discussions about this recently, and it's really hard to even discuss as nobody uses consistent terms for anything. From the bottom up:
These test a single class in isolation. Any even remotely non-trivial dependencies, including internal classes to the application, will be mocked. No dependency injection framework will be booted. These will hence generally be fast, but (conjecture!) will be relatively brittle due to the large amount of mocking.
- Some people call anything that can be run in a testing framework a "unit" test, try and use the term "JUnit test" for these?
- Some people would prefer "automated test" over "JUnit test", but the distinction between tests run as part of your build, and embedded into the same runtime as the application, are distinctly different from external tests (say, driven from a shell script or Cucumber).
These test an area of the system. This covers anything from a single class of "user" code, and whatever libraries you trust not to do anything stupid, to whole subsystems of the system. Critically, this'll be calling internal functions, or using subsystems / class hierarchies that wouldn't normally be exposed.
"Libraries" may include systems like in-memory databases; so long as it doesn't need to exist before hand, and you can run many of them in parallel, it's in. Starting a whole DI system for this is fine.
- Frequently confused with unit tests by lazy people.
- Occasionally known as "component tests".
- "But what if the test shows the brokenness in the wrong place?" What if your mock is wrong? What if your mock is right but doesn't break like the real library?
- "Where do I start?" With something so simple it's going to be a unit test.
- Short name: "Agg tests".
Start as little of the system as possible, and pound it's external API. Again, in-memory databases and other resources are the order of the day. Shouldn't be relying on any external system being up at all.
These tests have disproportionately higher value to outsiders as they're understandable, as they're phrased in APIs that people will have seen. In Choob's case, the primary "API" is IRC. This is fine.
- "Acceptance tests", per QA definition, cannot be effectively written by developers, or anyone who's seen the code; they must be very black box. This can be approximated by the developer.
- Sometimes called "integration test", but it seems much more logical to use that for the following.
- Short name: "A/Ts" ("aye-tees").
Like an acceptance test, but running against production resources; a real database, a real IRC server, really hitting the internet. Only run by people who know what they're doing; not part of the normal process.
- "Shouldn't we be running everything against a production-like environment?" Yep, but it's a pain in the ass so meh. Maybe we can make the acceptance tests run against production resources, if available?
- Unit tests have significantly limited value, hence mocking has limited value
- Integration tests are generally something done by low-level people (say, Hibernate and PircBOT), not us
i.e.
- Aggregate tests for internals, like plugin loader development
- Acceptance tests for plugins and anything external facing