Cucumber reports

Christoph Kappel · 16 October 2021 · specification-by-example, cucumber, showcase · Reading time 7 minutes

In this article

Cucumber
Conclusion

In my previous post about [specification-by-example]({% post_url 2021-09-09-specification-by-example %}) I touched the general idea about the method and also about some supporting frameworks to execute the tests and create some kind of feedback.

I must admit the result was pretty much geared towards web-based frameworks like FitNesse and Concordion and I haven’t touched the reporting facilities of Cucumber. In this post I want to correct that and show three different ways to generate reports.

Cucumber &

If you check the reporting-page of Cucumber, there are lots different options to generate reports. Some are fancier, some have more details and numbers, it is up to you pick yours. Before having a look at some of the options, let us start with the built-in reporting first:

Builtin / online &

All you have to do to get some online reports is to set the option publish = true:

@RunWith(Cucumber.class)
@CucumberOptions(
        plugin = "pretty",
        publish = true,
        features = "src/test/resources/features"
)
public class TodoCucumberFixture {
}

Once this is set, the next run of Cucumber ends with this information box:

Since the reports really disappear after the given time (go ahead and give it a try) here is a screenshot with some hints how it looks like and what is included:

For a default this is pretty nice and well readable, although online-only (as far as I know) might be a problem for some cases.

Back to our list, next!

Cluecumber &

I must admit I kind of liked and hated the name, so this is my pick, obviously. The setup is pretty easy, just add the plugin:

<build>
    <plugins>
        <plugin>
            <groupId>com.trivago.rta</groupId>
            <artifactId>cluecumber-report-plugin</artifactId>
            <version>${cluecumber-report-plugin.version}</version>
            <executions>
                <execution>
                    <id>report</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>reporting</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <sourceJsonReportDirectory>${project.build.directory}/cucumber-report</sourceJsonReportDirectory>
                <generatedHtmlReportDirectory>${project.build.directory}/generated-report</generatedHtmlReportDirectory>
            </configuration>
        </plugin>
    </plugins>
</build>

And configure Cucumber to save the output as JSON and you are ready to go:

@CucumberOptions(
        plugin = {"pretty", "json:target/cucumber-report/cucumber.json"},
        publish = true,
        features = "src/test/resources/features"
)

Since we’ve added another plugin, we have to call it right after our test run:

$ mvn -f todo-service-cucumber/pom.xml test cluecumber-report:reporting

Here is another screenshot of the report generated by Cluecumber:

I probably have to add some delays to see something in the results diagram, but still this report is easy to read and informative.

Serenity BDD (aka Thucydides) &

Just to get this out upfront: I am quite a fan of the Serenity movie, but I have no idea why so many projects must adopt this name. (Maybe these projects have some bored SEO experts who need a challenge?)

I really preferred the previous name, especially because of this quote which really describes why we need more reporting:

More generally, Thucydides developed an understanding of human nature to explain behavior in such crises as plagues, massacres, and civil war.

— https://en.wikipedia.org/wiki/Thucydides

Coming back to the tool itself: This isn’t another plain reporting plugin for Cucumber, but a drop-in replacement for the test runner. It comes with some additional test steps, but all that is needed for a first run is to set up the fixture like this:

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
        plugin = "pretty",
        features = "src/test/resources/features",
        objectFactory = SerenityObjectFactory.class
)
public class TodoCucumberFixture {
}

And after a run of following command:

$ mvn -f todo-service-cucumber-with-serenity/pom.xml test serenity:reports -Dserenity.reports=single-page-html,navigator serenity:aggregate

We are greeted with the normal Gherkin output along with some ASCII-art: (Don’t ask; just more line noise)

  _____   ___   ___   _____     ___   _____     _     ___   _____   ___   ___
 |_   _| | __| / __| |_   _|   / __| |_   _|   /_\   | _ \ |_   _| | __| |   \
   | |   | _|  \__ \   | |     \__ \   | |    / _ \  |   /   | |   | _|  | |) |
   |_|   |___| |___/   |_|     |___/   |_|   /_/ \_\ |_|_\   |_|   |___| |___/

Create a todo with title and description and check the id.(create-a-todo;create-a-todo-with-title-and-description-and-check-the-id.)
--------------------------------------------------------------------------------
2021-10-16 17:20:25,440 INFO  [net.thu.cor.mod.TestOutcome] (main) SetUserStory Create a todo
2021-10-16 17:20:25,445 INFO  [net.thu.cor.mod.TestOutcome] (main) SetUserStory Create a todo
  Given I create a todo with the title "title1"                              # dev.unexist.showcase.todo.domain.todo.TodoSteps.given_set_title(java.lang.String)
  And the description "description1"                                         # dev.unexist.showcase.todo.domain.todo.TodoSteps.and_set_description(java.lang.String)
  Then its id should be 1                                                    # dev.unexist.showcase.todo.domain.todo.TodoSteps.then_get_id(int)
2021-10-16 17:20:25,841 INFO  [null] (main)
  _____   ___   ___   _____     ___     _     ___   ___   ___   ___
 |_   _| | __| / __| |_   _|   | _ \   /_\   / __| / __| | __| |   \
   | |   | _|  \__ \   | |     |  _/  / _ \  \__ \ \__ \ | _|  | |) |
   |_|   |___| |___/   |_|     |_|   /_/ \_\ |___/ |___/ |___| |___/

Once the test runs are completed Serenity-BDD generates some beautiful and clickable reports. I won’t cover all the pages of it, but here is a screenshot of the index page:

Conclusion &

We have seen some options to generate reports, from simple ones to multi-paged reports there is probably everything you and/or your department desires. Just collect your requirements and pick the one with the least trade-offs.

My personal pick is probably Serenity-BDD, because I like the additions to the test steps and I really dig there output format.

As always, my showcase can be found here:

https://github.com/unexist/showcase-acceptance-testing-quarkus

unexist.dev