-
Notifications
You must be signed in to change notification settings - Fork 11
Reusable Recipes
Roboconf Specification Request: 237
Status: Closed (2015-03-25)
Editors:
- Vincent Zurczak, Linagora
Roboconf has few users, and most of them have written their own recipes, in Puppet, Bash, etc. Sometimes, they even wrote recipes for a same application, and they did not know it.
Making recipes reusable is about 3 things:
- Describe good practices to share recipes.
- Define a default tool to deal with recipes reusability.
- Setup a central place to share and/or contribute recipes.
This is what this specificiation is about: defining and explaining the choice we are going to make about this problematic.
Before talking recipes, we should talk about build tools. With Roboconf, the default build tool is Maven. This is because the Roboconf model has quite a lot of validation rules that were implemented in Java. Maven is a robust tool with a wide community of users and contributors. It is also well-integrated with CI servers. Another advantage was that Maven repositories can host packaged artifacts (and people are used to these repositories).
Besides, there are bridges between Maven and other build tools. SBT (Scala), Apache Ivy and Gradle can retrieve artifacts from Maven repositories.
So, it made sense to use it. For the moment, it can be used to validate and package an application. And it makes almost no doubt if will be used also to retrieve imports/reusable recipes.
A reusable recipe is a graph portion with resources for a Roboconf extension.
Example: a Roboconf graph that declares a Tomcat component, and Bash scripts to handle its life cycle with the Roboconf Bash plugin.
A reusable recipe can be a complete Roboconf application, or it can be a subset of a complete Roboconf application (the subset being a Roboconf application itself, except it is incomplete).
Thus, a reusable recipe is a Roboconf application.
If it contains instance definitions, they will be ignored.
Let's now talk about what we want to do with reusable recipes.
Two kinds of users are considered: a contributor (or a user) is somebody that uses Roboconf but that is not mandatory a developer on the project. The second kind of user is a Roboconf team member. Their needs to all should be satisfied by the solution that will come out.
- As a user, I would like to share one or several recipes on my own (no validation required by the Roboconf team).
- As a user, I want to be able to share recipes on Git, SVN, Maven, whatever.
- As a user, I want to easily find recipes other shared.
- As a user, I want to easily improve shared recipes.
- As a user, I do not want to be forced to use Maven to benefit from reusable recipes.
- As a Roboconf team member, I want to distinguish recipes maintained by the team from external recipes.
- As a Roboconf team member, I want to spend the minimal amount of time to release recipes.
- As a Roboconf team member, I want the community management about reusable recipes to be easy.
- As a Roboconf team member, I want the documentation management to be easy.
- As a Roboconf team member, I want reusable recipes to work with Maven.
Yes, there are some contradictory requirements.
But we can fulfill them all.
Roboconf's DSL has an import keyword.
This keyword allows to read a graph or instances portion from other files than the one referenced in the application.properties. The constraint is that the import is relative to the file that declares the import. No absolute file path, no URL. This was made this way for two reasons.
First, we needed this keyword if we wanted one day to create and share recipes. And then, we do not want to reference any file because we want the Roboconf archives to be complete. Roboconf should be able to read its model without an internet access. And it should be able to read its model even in 15 years. The fact that recipes may require an internat access is out of the scope of the Roboconf model. But the graph and the instances set must be complete and available in the deployment archives.
This import keyword allows to solve two requirements.
Basically, to reuse a recipe, we only need it to be copied within my model. And this can be done with Maven and with other tools (even by hand). To prevent conflicts in file naming, applications have a namespace attribute. So, to use an existing recipe in my own graph, I only need to import...
import reusable-recipe-namespace/graph-name.graph;
And that's it. Namespaces are used for that. They do not prevent conflicts between component names.
Summary
| Satisfied Requirement | Solution |
|---|---|
| 5 | Use the import keyword. |
| 10 | Use the import keyword. |
Reusable recipes are Roboconf applications.
They can be shared under any form: ZIP archive, Maven artifact, source repository (SVN, Git, CVS...). A user can share them under the shape he (or she) needs. Obviously, for Maven users, and more generally, it is not a good practice to rely on snapshots repositories for a real project.
But it should be possible to import recipes from a source server.
With Maven, this could be achieved with the Maven ANT plugin. In the generate-sources phase, Maven can invoke an ANT script to checkout sources and compile them. For many reasons, it is dirty as hell, but that will work. We will see later if this kind of requirement needs to be addressed with a better solution (maybe nobody will reference recipes from Git, we never know...).
And of course, since recipes are Roboconf applications, we can package them and upload them to Maven repositories thanks to the Roboconf Maven plugin.
Let's take a look at our requirements table.
| Satisfied Requirement | Solution |
|---|---|
| 1 | Share it wherever you want. |
| 2 | Share it wherever you want. |
| 5 | Use the import keyword. |
| 10 | Use the import keyword. |
The interest of having reusable recipes is also about building a community where people can help each other and share things or tips. However, there need to be some organization so that a community can work. Besides, the Roboconf team has to setup something, let's call it a core, to see a community appear and grow.
This is why the Roboconf team should share, document and organize the recipes it already has. Since we are entirely based on Git, they will be hosted on GitHub. There are two options. We can either have one Git repository per recipe. Or we can have a big one where we aggregate all the ones with the Roboconf team stamp.
To keep the documentation simple, we need a central repository to gather all the recipes. So, we need a roboconf-recipes repository that will list all the recipes we have with a good (and probably generated automatically) readme file. On the other hand, we would need every recipe to be managed separately. Every "official" recipe will result in a Maven artifact. And a Maven artifact means a tag. So, it is better to handle every recipe in its own repository. If we had all our recipes in a single Git repository, we would release and tag all the recipes every time we want to release one. This is very inefficient (for Maven repositories that will host similar recipes but with different versions, and also for our own management).
So, the conclusion is that for Roboconf official recipes (those managed by the Roboconf team or its relative), we need one Git repository by recipe. And we also need a central repository to aggregate all the recipes repositories (Git modules). Said this way, it would be more simple to have the recipes on a SVN (but anyway...).
Let's take a look at our requirements table.
| Satisfied Requirement | Solution |
|---|---|
| 1 | Share it wherever you want. |
| 2 | Share it wherever you want. |
| 4 | Every recipe maintained by the Roboconf team is in its own Git repository. So, forks and pull requests will work. We cannot force anything with respect to external recipes. |
| 5 | Use the import keyword. |
| 6 | Official recipes are managed in Roboconf's Git repositories. |
| 7 | Recipe repositories are very small and managed independently. |
| 10 | Use the import keyword. |
We also need to address documentation and search issues.
This is the role of the centralized repository (or aggregator). Every recipe should have a readme with standardized information. A part of this information should be reused in the main readme of the aggregator repository.
This central repository should also classify recipes in categories.
Here is a proposition.
+ databases
++ mysql
+++ 5.0
+++ 5.1
++ mongodb
+++ ...
Basically, we have categories at the root. We then have Software below. And inside, we have versions. And under the version, we have a link toward another Git repository. Since we only have references, it is easy to put a same Software in different categories. Categories will include initially databases, web-servers, applications-servers and messaging-servers.
The root of this repository should also contain a pom.xml file, so that we can plug continuous integration on our recipes and make sure they are valid (at least for the Roboconf model).
Still at the root, there should be a generated readme.md file.
It would lists all the Software we can deploy with the referenced recipes. This readme file should distinguish between recipes managed by the Roboconf team (or contributors) and external ones.
Let's get back to our requirements table.
| Satisfied Requirement | Solution |
|---|---|
| 1 | Share it wherever you want. |
| 2 | Share it wherever you want. |
| 3 | Use categories to classify recipes and lists them all on the main readme. |
| 4 | Every recipe maintained by the Roboconf team is in its own Git repository. So, forks and pull requests will work. We cannot force anything with respect to external recipes. |
| 5 | Use the import keyword. |
| 6 | Official recipes are managed in Roboconf's Git repositories. |
| 7 | Recipe repositories are very small and managed independently. |
| 9 | Make recipes categorization straight-forward and generate the main readme automatically. |
| 10 | Use the import keyword. |
There is one last requirement to cover.
As a Roboconf team member, I want the community management about reusable recipes to be easy.
And we have not yet specified how we would distinguish an official recipe from an external one.
In fact, that's easy.
- Official recipes are Maven-based and follow conventions.
- They have net.roboconf.recipes as their group ID.
- The artfiact ID looks like mysql-5.0 optionally followed by a suffix. Example: mysql-5.0 is valid. mysql-5.0-my-client is also valid (at least, for the naming convention). So, we must find the Software name followed by its version and an optional suffix.
- They are hosted in git repositories that belong to the Roboconf organization.
- When released, they will be uploaded on a Maven repository.
Other recipes are considered as external.
External recipes can be referenced in the readme of the central repository. They will not be referenced as git modules.
This completes our requirements table.
| Satisfied Requirement | Solution |
|---|---|
| 1 | Share it wherever you want. |
| 2 | Share it wherever you want. |
| 3 | Use categories to classify recipes and lists them all on the main readme. |
| 4 | Every recipe maintained by the Roboconf team is in its own Git repository. So, forks and pull requests will work. We cannot force anything with respect to external recipes. |
| 5 | Use the import keyword. |
| 6 | Official recipes are managed in Roboconf's Git repositories. |
| 7 | Recipe repositories are very small and managed independently. |
| 8 | What is in the community is managed by the community according to rules and conventions. |
| 9 | Make recipes categorization straight-forward and generate the main readme automatically. |
| 10 | Use the import keyword. |
Why would people contribute recipes to our central repository?
That's a very good question. Why would they give up their group ID to net.roboconf.recipes? One solution could be to define a roboconf-contributors group in the organization. This group would have access to the recipe repositories in read and write modes. They would be elected to this group after several contributions (just like Apache or Eclipse do in their projects). That would make contributing more appealing (and motivating).
We could also consider the use of a very open license for recipes, like MIT.
This would make people less reluctant to contribute.
During the implementation, a little change was brought.
Rather than having a central Git repository to aggregate recipes repositories, it was decided it would be more simple to have a list on the web site. We also dropped automatic listing of recipes. Again, it is more simple to contribute such an update on the web site.