-
Notifications
You must be signed in to change notification settings - Fork 220
Support for Macros #178
Comments
Unless I see some really compelling examples I lean against this proposal. -1 My initial thoughts:
My main worry may be unwarranted but I'm afraid this will open Cucumber up to creating a mess of documentation. Though given that is still possible now I feel this would tip the boat. |
TL;DR-1, Gherkin is not a programming/scripting language and The whole thingI've been thinking about this for quite a while and I'm not in favor of this feature. I second Joseph's thoughts on the potential maintainability issues and the lurking problems that disappeared with the removal of GivenScenario. I believe Gherkin's (the syntax, not the parser) strength lies in its simplicity and the fact it is not a programming language. Every keyword is a simple documentation label telling what is being described in the system under test. They have very little programmatic meaning, if not none at all. In other words, you can (and should, says Liz Keogh ;)) write scenarios which are not to be automated and yet be immensely useful as a way to communicate and document features and drive development. The That shift of focus could foster writers to express implementation details too soon and not where it belongs. Instead of pushing them down to step definitions or - even better - to some domain layer/DSL, they could easily fit them into macros. This is quite a big concern in the Cucumber world today: many people are still unsure what level of abstraction is right or wrong; many are even not aware of such concepts as an "implementation detail". We are slowly recovering from the web_steps era in Cucumber and I think the In Gherkin, a step is the atom: it is the smallest piece of information. Steps represent a context, an action or an outcome but their meaning is defined somewhere else: in a step definition (it looks like that name was chosen for a reason ;)). In other words, the definition of a step is not expressed at the Gherkin level and this is a good thing: it leaves room for discussion as it decouples the behaviour from the implementation. I think the Gherkin syntax is good at what it was made for because it is not a programming language. It is an efficient tool for filling the gap between "the business", "the devs" and the code, when it comes to communicating specifications and their validations. I feel that adding the @josephwilk: for my part, yes I'd like to see #steps go away. But that's another story. :) |
Was +1 a couple weeks ago. Now -1. I'm for the idea of letting step definition writers (QA folks in our company) use english to write stepdefs. As for the proposed solution, I realized that it's actually going to add confusion for Cucumber users. I like to keep separation of concerns wherein feature files contain business requirements whereas the stepdefs contain the test scripts. The solution described above seems to blur the line of separation. |
I'm not for or against this at this point, but I'd like to balance out the comments from @josephwilk and @jbpros. The request seems to be to find a way for non-technical people to combine re-usable sets of steps in a way that doesn't require developers to have to be involved. Before implementation is considered, I would want to know if there is value in this? And not from a developer perspective, but from the perspective of the primary audience of this feature. There appears to be some perceived value since it has been requested. Are there only two alternatives right now: copy and pasting steps or getting with a programmer? Copy and pasting has its own annoyances. Getting with a programmer also can be problematic and costly. While Gherkin is not a programming language, I don't think finding a way to make it more useful for one of its intended audiences is a bad thing. This is assuming one of Cucumber's core audiences is non-technical business analysts and QA folks. Macro may not be the right way to solve this problem, but first we should decide if non-technical people would benefit enough from a feature that allowed them to reduce unnecessary duplication or incur unnecessary cost (or dependency on the programmer). If that is real and valuable then we should explore different approaches to solve that problem. For empathy... If you couldn't write language-specific step definitions and you maintained the feature suite for my project and you wanted to write new scenarios based on similar usages of steps from other scenarios, how would you want to go about that? What are your options and how would you prioritize them? If anyone here is working with non-technical analysts it would be awesome to get some real feedback. |
I'm currently working with non-technical testers and analysts. The analysts work on the feature files and they work great as they are now because they focus on writing business requirements, not test scripts. Given I have a role of <Role> in the project
Then I <Can or Cannot Create> an instance in the project
Examples: Authorized Roles
| Role | Can or Cannot Create |
| Project Owner | Can Create |
| Developer | Can Create |
Examples: Unauthorized Roles
| Role | Can or Cannot Create |
| Network Admin | Cannot Create |
| Non-Member | Cannot Create | The testers, on the other hand, work on the step definitions and that's where the test scripts are written. Currently, we've implemented the Page Object pattern to help make the syntax easier to write: When /^I log in with the following credentials: (.*), (.*)$/ do |username, password|
page = LoginPage.new
page.visit
page.should_be_valid
page.fill_in_username_field username
page.fill_in_password_field password
page.submit
end On the other hand, it would be easier for or non-technical testers to write in a language that's close to english. When I think of it now, the solution should probably be a DSL that can work alongside (not part of) Gherkin. Something that is meant to be used by stepdef writers who don't have a strong programming background. Something that allows them to write the above as:
|
To clarify for @josephwilk and @jbpros yes, my primary reason for suggesting this so that we can remove Rather than just yanking out #steps, I'd like to look for a better way to give this kind of user what they want. If we're going to support this feature at all, I think we need something declarative that will allow IDEs and other tools to let users click through to macro definitions (or whatever we call them) without having to execute anything to understand the mapping. At the moment, you don't know whether you're looking at a nested step or a regular step, but if these were defined in the Gherkin it would be easier for a tool to analyse the Gherkin statically and see what was happening. Some suggestions:
How about
Let's keep brainstorming on the idea and listening to users like @relaxdiego who obviously have a need for something like this, and see if we can figure out a solution that works better than [1] like this? |
To clarify @relaxdiego are in favour or against this feature? I thought your where against it from your previous post. |
@josephwilk I'm for the idea of giving stepdef writers the ability to write stepdefs in English. I was against the proposed solution but @mattwynne's clarification makes it sound like it's headed in the right direction after all. Can I be conditionally in favor of it? My concerns:
|
Thanks @relaxdiego thats helpful. @zdennis I think one problem we have is that this is not the right medium to get feedback and examples from non-technical analysts. If we want to learn more from those people I feel we need to move this discussion somewhere more suitable. Having said that I'm not sure what that medium is 😔 |
@josephwilk I'll answer your question tonight when I get some more time. In the meantime, I've started a little poll about the removal of |
I am torn. On the one hand, I greatly appreciate the desire for non technical people writing feature files to group and reuse sets of step definitions. On the other hand, every time I have seen this played out it has caused difficulties with maintainability of feature files. In the past, I have extracted each steps implementation and mixed it into world; then I have written a step that explains the permutation (i.e "Given a user who may access the flim flam machine") and in that step definition called the extracted methods. Remember, Gherkin is not something that is handed from a business owner to the developer. It's the start of a conversation between the people who know what should be built, and the people who are building it. |
@zspencer maybe we should stop thinking of this feature as a way to group and reuse sets of step definitions. Rather, we should think of it as something that allows step definition writers (not feature file writers) to use an English-like language for creating step definitions. In this case, it's really an alternative to Ruby when writing stepdefs. It's a language that is not part of Gherkin, but something totally different and only used for stepdefs. |
@relaxdiego right. This is a way for non-technical people to create good declarative scenarios from a small number of basic building block steps, without having to work in Ruby files. Crucially, it also gives those people visibility about what goes on inside that step, thereby building more trust in what the scenario is doing. Ideally, they should have a good trusting relationship with the programmer who implements the step definition code (that calls nicely extracted World methods) but this isn't the context that everyone operates in. |
The word "macro" sounds a bit techy to me. It may be clear enough to a broad range of casual computer users from experience with spreadsheets, though. The basic problem I see is this: I think it's beneficial for testers or testers paired with business people to be able to create new scenario variants without having to go get a programmer to help them. When they do so, however, I see lots of duplication in the scenarios. They become more imperative and less declarative. I would like to help them learn the rudiments of programming (avoiding duplication, symbolic representation of larger concepts) while remaining in gherkin. I would like to give them more of a sense of being masters of their own fate, and less beholden to the programmers, without having to retreat to record-and-playback tools. |
So @gdinwiddie do I take it that you like the idea, just not the name? Can you suggest any alternative names? I was thinking |
Naming is hard. Maybe For those who think this feature is a bad idea, it's not intended to replace coded steps. It's an overlap. There's not a distinct point where you should make a switch from gherkin to a programming language. |
Or |
I don't understand why we have to invent a keyword for this. I'm also getting mixed messages. At one point I thought this was turning into an alternative to using Ruby when creating stepdefs (which i think is the right solution) but then it's beginning to sound like it's an extension to Gherkin (which I'm not so sure is a good idea) If this feature is be meant for testers who don't have a programming background, then the solution should really just be a DSL that they can use for writing step definition files. The file that they write should be saved in the step_definitions directory so that there will be no confusion about where the tester will save what. In this solution, there's no need to invent new keywords such as Macro or Step Fragment because Cucmber already knows to look i the step_definitions directory for executing each step. |
@relaxdiego actually, having a Ruby DSL is precisely how I personally use, and recommend other teams try to use Cucumber. You can already do that. However, I'm raising this ticket because I meet lots of people who would like to be able to refactor their features, raising the level of abstraction in their scenarios by effectively extracting a high-level step, without going into a Ruby file. |
No matter how you slice or dice it, this feature request is about adding functions or subroutines [1] to the gherkin language. The reason for doing so is to make it easier for non-programmers to create more declarative scenarios on their own. By making it easier for non-programmers to work on their own we are also undermining one of Cucumber's essential values - Collaboration. The people who want this feature don't know how to program. If they did they would create their subroutines in their Cucumber implementation's programming language (Ruby, Java, JavaScript etc). Guess what a person without programming skill and without a programmer buddy will do with subroutines. They will create a mess. Down the road people will blame this mess on Cucumber, which I'm not really interested in. But more importantly, the poor people using this feature will create a mess for themselves. And they won't realise this until they have a big stinking pile of poo. I understand why people want this feature, but I don't see anything good coming out of it. There is no shortcut to creating high level, declarative scenarios. You either have to learn to program or work closely with someone who does. [1] http://en.wikipedia.org/wiki/Function_(computer_science) |
This hasn't been the case in our experience. Right now, we're using nested 'web' and 'low level' steps. The process thus far has been:
Here's a sample stepdef from our project:
So far, this has actually enhanced collaboration in our team because now the BAs and the testers can join in the discussion regarding how a step should be executed. Compare that to our previous process where only the programmers know how to read the stepdef, while BAs and testers are limited to the feature files because they can't understand the stepdefs. Note the last phrase in the above paragraph. That's been one of the reasons why Cucumber has been misused (i.e. tendency for a team to write imperative steps). Because stepdefs are written in a language that is alien to many team members, the tendency of the team is to write imperative steps so that the BAs and the testers know what the heck is going on. With nesting/macros/whatever, this doesn't have to be the case. Granted the proposal as it stands right now may need some more improvement, that doesn't invalidate the whole thing. This is useful and, I believe, will improve collaboration rather than undermine it. |
-1 The idea of 'non-technical' people combining a re-usable set of steps to create automated tests has been around for quite some time. It is and has always been fundamentally flawed. Cucumber by its design makes it feasible for non-technical people to specify what a test should do. But as soon as this slips into how a test should do something then some degree of technicality is required. At the very least every feature should fail before it passes. This requires technical knowledge - you have to know its failing for the correct reason, and you have to know that additional functionality that is added is now responsible for making it pass. One of the biggest counter arguments to using Cucumber is that it makes everything so difficult. This feature will exacerbate this problem by vastly increasing the complexity of feature files, and encouraging the creation of features that are highly coupled and prone to breaking. The problem of combining a number of operations in a sensible and repeatable way has been well addressed by programming languages. They have the tools (and we all know these extend far beyond just having subroutines) for dealing with this problem with grace and elegance. Cucumber currently encourages people to avoid using the appropriate tools for this composition problem by allowing nested steps. Allowing nested features will make things even worse. The place for nesting is in the programming language that implements the step definition , because only a programming language has sufficient power to implement this composition effectively. Finally the re-use of step definitions is highly counterproductive and overrated. Generally all it does is restrict the language business users can use to express themselves and encourage a kind of programming by proxy. The essential beauty of cucumber is the clear separation between features and step definitions. This was horribly muddied by web steps and early implementations of feature sets, and unfortunately cucumber still hasn't recovered from this. |
I've been thinking about an alternative implementation. Perhaps plain-text steps should just be in a .step file that could be handled similarly to steps written in any other language. Something like:
Some of the comments on this issue, here and elsewhere, seem quite militant that implementation of steps should be left to real programmers using a real programming language. I find this attitude anti-collaborative. It smacks of "don't you worry your pretty little head about this, we'll take care of the complicated stuff." Sometimes we should remind ourselves that testers and analysts are capable adults who can also do things that we cannot. Technical knowledge and programming ability isn't a boolean value. We should not strive for a sharp distinction between the realm of a tester and that of a programmer. I agree that web-steps was a mistake, but not because it gave low-level imperative step definitions. It was a mistake because it took those definitions out of the control of the team using them. It's beneficial to have an overlap between what can be done in "plain language" and what can be done in a programming language. It's important, though, that it be easy for teams to make their own decisions about what gets done where, based on the needs of the project and the capabilities of the participants. The ability to refactor imperative scenarios into declarative ones by using "extract step definition" is a huge win, in my opinion. Having to simultaneously reimplement these steps in a different language is a bigger, more error prone, step than I would prefer, even as a programmer. |
+1 to @gdinwiddie's comments. Also, I created a sample app for a fictitious framework in the hope of providing a more concrete example of what I'd like to see happen. See https://github.com/relaxdiego/norm_app |
Just to be clear I believe that that testing and programming are heavily overlapping activities and that both features and steps should be written collaboratively. However I'm also absolutely clear that writing features is a distinctly different activity to implementing steps. If you make writing features a programming activity then you undermine its core function which is to capture business knowledge. It should be possible for anyone with domain knowledge to write features. If you don't apply programming disciplines to the activity of writing step definitions you end up with a set of step definitions that hampers the writing of new features. This can take many forms including
Generally I believe that the level of programming discipline required to implement a really useful set of step definitions is quite minimal, and can be learnt by all sorts of people, programmers, business analysts, testers even business owners with the right guidance. Cucumber has all the tools (and more) to do this. However there is a general trend to make this far more difficult than it needs to be, and then claim that we need additional functionality like 'support for macros', because writing step definitions is too difficult. Cucumber already has a reputation for being difficult and unnecessary. Encouraging the effective use of what the tool already gives us, might be more productive than adding additional features that the tool doesn't need. |
Encouraging the effective use of Cucumber might not help if how the suggested use is in conflict with how the tool is currently designed. Cucumber is touted as a tool that allows for collaboration between POs, BAs, testers, and developers. However, when you look at its current design, and if you take away Gherkin, you are really still left with a tool for developers. Gherkin right now isn't enough. While it gives POs, BAs, and testers a glimpse of what the team is trying to design, going beyond that is not possible. How will they know that the devs are testing the right thing? We can't expect POs, BAs, and testers to learn Ruby considering that they already have their own concerns re the team's work. This then puts a lot of burden on the devs to maintain Cucumber and probably even explain to the non-programmers the ins and outs every now and then. However, if we allow the understanding of the ins and outs of Cucumber to spread out to the rest of the team (i.e. allow stepdefs to read like natural language bulletpoints) then there will be better collaboration within it. |
I'm not actually interested in arguing about the rights and wrongs of this feature. Cucumber already supports From what I can tell, this feature is most used by teams where there is a lot of ownership and interest in the Cucumber features by people whose strength isn't writing and maintaining Ruby code. My suggestion is that we recognise that, and move this feature into a medium that doesn't involve them having to open up Ruby files. Having them expressed declaratively in Gherkin would have the additional benefit of being able to use tools to reflect on the relationships between these steps without having to execute them. Right now, the only way we can find out about nested steps is by running the scenarios, and I think this is a disadvantage. |
Matt, If you are going to move Cucumber along this path could I ask that you consider some sort of extension architecture. Is the idea perhaps that there is a simple core gherkin and then extensions for this sort of functionality e.g. macros In addition could there be refactorings that move things currently in gherkin into extensions? e.g. example groups, tables. I think this approach might allow you to meet the needs of both audiences. All best Andrew |
I've been thinking a lot more about this feature and my attempt at a compromise above. I've come to the conclusion that any change to the gherkin language is absolutely fundamental. My extension proposal would result in the fragmentation of gherkin - minimal gherkin, gherkin with loops etc.. Gherkin is now pretty much a language specification. So I've discarded any thoughts of a minimal cucumber. The disruption of the gherkin language caused by removing things is a very high a price to pay for any improvements in the purity of the language. We are in a situation where removing functionality from Gherkin is very disruptive and we have to be extra careful about adding functionality to gherkin - it will be very hard to go back if we get things wrong. My opposition to this change is stronger than ever. The cure is far worse than the disease. However it also means that I remove all of my objections to any additional syntax in step definitions or even some intermediate .steps file that help people write steps in the way that George or Mark are proposing. Better to leave in nested steps or even enhance them than to move the functionality into gherkin. All best Andrew |
NOTE: I originally posted this to the Cucumber mailing list but realized later that I should've posted it here Here's a proof of concept framework that I wrote the past couple of days to illustrate the benefits of natural language in test scripts. I call this framework 'Norm'. Note that this is an executable application, not just a bunch of text files. It also has unit tests to keep out the bugs that I know of. We start with listing the requirements which looks something like the feature files but not exactly: directives/requirements/create_a_project.requirements
The difference with this and feature files apart from the filename extension is that it doesn't require Given/When/Then keywords (although you're welcome to start your sentences with those words if you really want to). The requirement declaration can be written in free form, using multiple lines if desired. Here's a multi-line example: directives/requirements/log_in.requirements
Note that you can optionally add borders and header separators for the example tables. Also note that you only need to use the 'Requirement:' keyword and have no need for a 'Requirement Outline:' keyword. That is, you can use the 'Requirement' keyword for formats that are similar to 'Scenario' and 'Scenario Outline' in the latest version of Cucumber. That's for the requirement level. Now for the test script level: Each requirement can be matched by any of the defined test scripts. For create_a_project.requirements, its corresponding test scripts may be found at: directives/test_cases/create_a_project.test_cases
Test Case files have a different syntax from Requirements files. For instance:
Finally, the steps level Each line in the Setup, Script, and Cleanup section of a test script can be matched by a step defined in the directives/utilities/steps/user_steps.rb
This part is pretty much the same as how steps are defined in Cucumber. The difference is that steps are accessible from within test cases only. Requirements cannot use steps directly. Please note that I have no plans for maintaining this proof of concept. It's just meant to illustrate how it might 'feel' if Cucumber implemented this feature. For more information on how to play with Norm, please see the readme at https://github.com/relaxdiego/norm |
Revived discussion: https://groups.google.com/group/cukes/browse_thread/thread/f313f906671f782 |
To put some oil on the fire: I started a little project in Github called Macros4Cuke. It results in a gem macros4cuke that adds a macro-facility to Cucumber. With it, it is possible to rewrite the original example from Matt like this: Scenario: The macro is defined here...
Given I define the step "When I [sign in as <username>]" to mean:
"""
Given I have visited the homepage
When I click "sign in"
And I fill in "Username" with "<username>"
And I fill in "Password" with "password"
And I click "Submit"
"""
Scenario: Greet user
Given a user named matt
When I [sign in as "matt"]
Then I should see a greeting "hello matt" The nice thing about all this, is that no change in Cucumber is required. |
3 different people have asked me about this in the last 24 hours, so I'm going to write it up and we can discuss how it might work
The goal is to allow non-technical people to compose high-level, declarative steps from low-level steps without leaving the familiar, safe, Gherkin layer.
Here's my suggestion. We add a new
Macro:
keyword to Gherkin which allows you to define a step that maps to other steps.Example
The idea is that by doing this declaratively at the Gherkin level, we can do away with the gnarly code in Cucumber for supporting #steps. In fact, for a first iteration of this feature, Cucumber would not even need to know about the macros - Gherkin can unpack them and send the low-level steps to Cucumber for invocation.
Some details:
.feature
file will be applicable to any scenario.<angle brackets>
. No regexps.I'm agnostic about whether this is a good idea. I know it will make Cucumber more usable to semi-technical people, but that will come at the cost of building and maintaining this extra complexity in Gherkin. I certainly am not motivated enough to build it myself but I am happy to discuss how it would be done and help someone else built it.
The text was updated successfully, but these errors were encountered: