Template Ranking

[dbanty]

Not all templates are created equal, and we want to indicate something about what to expect from a template at a glance when choosing a template in Rover (or looking at templates in the docs).

Open Questions

1. What are the criteria by which we judge templates?
2. How do we indicate, at a glance, how the templates stack up?
3. What should a more detailed view of each template (in the docs) show?
4. What is the process for promoting/demoting a template when either the criteria change or the template changes?
5. What are the criteria for removing a template that degrades too far (e.g., no longer maintained)?

Possible Criteria

Each of these probably warrants its own thread to discuss:

1. Is the template created/sponsored/maintained by Apollo?
2. Is there an active effort to keep the template's dependencies up to date (e.g., Renovate PRs are being merged after being tested)?
3. How much of the Federation spec does the template support (e.g., using subgraph compatibility tests)?
4. How many of the template requirements are met (especially when adding new requirements, like the new subgraph security config)?

Possible Display Options (Rover)

1. A set of emoji for each major requirement/criteria that are met (e.g., adding 🧩✅ could mean that federation and CI requirements are met).
2. Have tiers of templates and separate them in the list by tier (e.g., Gold means all requirements are met, Silver means some minimal set of requirements are met, and Bronze means no guarantee about quality).
3. Have a score for each requirement and display a template's score next to it (e.g., 100 is perfect, 50 meets half of our expectations).
4. Only display templates that meet some quality threshold. Hide the rest unless the user opts in with a --include-outdated-templates option or similar.

[dbanty]

Is the template created/sponsored/maintained by Apollo?

I don't think this is a good criterion to have. While we can guarantee the quality of our templates, there's no reason they are inherently better than a well-maintained community template. It also raises the question of "what counts as an Apollo template?" Only templates for Apollo frameworks? Templates where Apollo has commit access? Templates initially created by Apollo?

[patrick91]

I think we can have official templates for our own libraries. I imagine we can have templates for apollo client for the various frameworks (next, remix, react native) and also for apollo servers.

I don't think we should make official templates for community products (like Strawberry for example).
Maybe we can have an additional filter for "verified" templates, ie templates made by owner of that specific library, but it might be over kill 😊

[dbanty]

That makes sense to me; I guess the question still exists of whether or not we mark official templates as official in some way? Does that add any value to the user?

[dbanty]

Is there an active effort to keep the template's dependencies up to date (e.g., Renovate PRs are being merged after being tested)?

This is very important. I have used several templates where the effort to update them made me wish I'd just started from scratch instead. The hard part here is going to be validating this. Perhaps an initial manual validation step to say that Renovate or Dependabot is enabled, then an automated check on the number of open/stale PRs?

[patrick91]

I think we can definitely do this for our templates. For community ones, maybe it is not that necessary?

[dbanty]

I don't think it should be a requirement to be added, but templates that are kept up to date should be preferred in whatever ranking system we select. I, for one, when choosing a template, would like to know whether or not it's up to date. So, for example, if we were going with the tier system, you'd have to be using Renovate or similar to be considered 🥇gold tier, and without it the best you could do is 🥈silver tier.

[patrick91]

How much of the Federation spec does the template support (e.g., using subgraph compatibility tests)?

I think we add filters for this as well, I'm assuming we'll also support non federation templates in future (once GraphOS is more capable?)

[dbanty]

Right, that would be a different type of template, though. These are subgraph-specific templates, but we would have a similar system for other types of templates in the future.

[EverlastingBugstopper]

This all sounds great to me, and I really love the idea of gold/silver/bronze tier templates and displaying that info in Rover. Also agree on the --include-outdated-templates flag - though I might name it something like --include-bronze-templates but that's a small thing 🤷🏼

Is the template created/sponsored/maintained by Apollo?

agree, probably not necessary

Is there an active effort to keep the template's dependencies up to date (e.g., Renovate PRs are being merged after being tested)?

agree, seems important, need renovate to be 🥇

How much of the Federation spec does the template support (e.g., using subgraph compatibility tests)?

running compat tests seems good. i think we should only require federation 2 compatibility to be gold so if a new library spins up that doesn't support fed 1 they aren't penalized

How many of the template requirements are met (especially when adding new requirements, like the new subgraph security config)?

gold = renovate + all fed compat tests. could see an argument for needing a 90% threshold here or some lag time between when a new directive is released and when the template supports it.
silver = 90% of fed compat tests
bronze = some fed compat tests (could do base percentage or could say you just need to support some of the base federation directives to get here)

[dbanty]

some of the base federation directives to get here

I guess the minimum is that you can do @link and @key—basically resolve and contribute to entities. Unless there's something else that we'd consider MVP which we should add 🤔

[EverlastingBugstopper]

@link and @key make sense to me