Cucumber reports

Christoph Kappel · 16 October 2021 · specification-by-example, cucumber, showcase · Reading time 7 minutes

In this article

Cucumber
Conclusion

In my previous post about [specification-by-example]({% post_url 2021-09-09-specification-by-example %}) I touched the general idea about the method and also about some supporting frameworks to execute the tests and create some kind of feedback.

I must admit the result was pretty much geared towards web-based frameworks like FitNesse and Concordion and I haven’t touched the reporting facilities of Cucumber. In this post I want to correct that and show three different ways to generate reports.

Cucumber

If you check the reporting-page of Cucumber, there are lots different options to generate reports. Some are fancier, some have more details and numbers, it is up to you pick yours. Before having a look at some of the options, let us start with the built-in reporting first:

Builtin / online

All you have to do to get some online reports is to set the option publish = true:

@RunWith(Cucumber.class)
@CucumberOptions(
        plugin = "pretty",
        publish = true,
        features = "src/test/resources/features"
)
public class TodoCucumberFixture {
}

Once this is set, the next run of Cucumber ends with this information box:

Since the reports really disappear after the given time (go ahead and give it a try) here is a screenshot with some hints how it looks like and what is included:

For a default this is pretty nice and well readable, although online-only (as far as I know) might be a problem for some cases.

Back to our list, next!

Cluecumber

I must admit I kind of liked and hated the name, so this is my pick, obviously. The setup is pretty easy, just add the plugin:

<build>
    <plugins>
        <plugin>
            <groupId>com.trivago.rta</groupId>
            <artifactId>cluecumber-report-plugin</artifactId>
            <version>${cluecumber-report-plugin.version}</version>
            <executions>
                <execution>
                    <id>report</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>reporting</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <sourceJsonReportDirectory>${project.build.directory}/cucumber-report</sourceJsonReportDirectory>
                <generatedHtmlReportDirectory>${project.build.directory}/generated-report</generatedHtmlReportDirectory>
            </configuration>
        </plugin>
    </plugins>
</build>

And configure Cucumber to save the output as JSON and you are ready to go:

@CucumberOptions(
        plugin = {"pretty", "json:target/cucumber-report/cucumber.json"},
        publish = true,
        features = "src/test/resources/features"
)

Since we’ve added another plugin, we have to call it right after our test run:

$ mvn -f todo-service-cucumber/pom.xml test cluecumber-report:reporting

Here is another screenshot of the report generated by Cluecumber:

I probably have to add some delays to see something in the results diagram, but still this report is easy to read and informative.

Serenity BDD (aka Thucydides)

Just to get this out upfront: I am quite a fan of the Serenity movie, but I have no idea why so many projects must adopt this name. (Maybe these projects have some bored SEO experts who need a challenge?)

I really preferred the previous name, especially because of this quote which really describes why we need more reporting:

More generally, Thucydides developed an understanding of human nature to explain behavior in such crises as plagues, massacres, and civil war.

— https://en.wikipedia.org/wiki/Thucydides

Coming back to the tool itself: This isn’t another plain reporting plugin for Cucumber, but a drop-in replacement for the test runner. It comes with some additional test steps, but all that is needed for a first run is to set up the fixture like this:

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
        plugin = "pretty",
        features = "src/test/resources/features",
        objectFactory = SerenityObjectFactory.class
)
public class TodoCucumberFixture {
}

And after a run of following command:

$ mvn -f todo-service-cucumber-with-serenity/pom.xml test serenity:reports -Dserenity.reports=single-page-html,navigator serenity:aggregate

We are greeted with the normal Gherkin output along with some ASCII-art: (Don’t ask; just more line noise)

  _____   ___   ___   _____     ___   _____     _     ___   _____   ___   ___
 |_   _| | __| / __| |_   _|   / __| |_   _|   /_\   | _ \ |_   _| | __| |   \
   | |   | _|  \__ \   | |     \__ \   | |    / _ \  |   /   | |   | _|  | |) |
   |_|   |___| |___/   |_|     |___/   |_|   /_/ \_\ |_|_\   |_|   |___| |___/

Create a todo with title and description and check the id.(create-a-todo;create-a-todo-with-title-and-description-and-check-the-id.)
--------------------------------------------------------------------------------
2021-10-16 17:20:25,440 INFO  [net.thu.cor.mod.TestOutcome] (main) SetUserStory Create a todo
2021-10-16 17:20:25,445 INFO  [net.thu.cor.mod.TestOutcome] (main) SetUserStory Create a todo
  Given I create a todo with the title "title1"                              # dev.unexist.showcase.todo.domain.todo.TodoSteps.given_set_title(java.lang.String)
  And the description "description1"                                         # dev.unexist.showcase.todo.domain.todo.TodoSteps.and_set_description(java.lang.String)
  Then its id should be 1                                                    # dev.unexist.showcase.todo.domain.todo.TodoSteps.then_get_id(int)
2021-10-16 17:20:25,841 INFO  [null] (main)
  _____   ___   ___   _____     ___     _     ___   ___   ___   ___
 |_   _| | __| / __| |_   _|   | _ \   /_\   / __| / __| | __| |   \
   | |   | _|  \__ \   | |     |  _/  / _ \  \__ \ \__ \ | _|  | |) |
   |_|   |___| |___/   |_|     |_|   /_/ \_\ |___/ |___/ |___| |___/

Once the test runs are completed Serenity-BDD generates some beautiful and clickable reports. I won’t cover all the pages of it, but here is a screenshot of the index page:

Conclusion

We have seen some options to generate reports, from simple ones to multi-paged reports there is probably everything you and/or your department desires. Just collect your requirements and pick the one with the least trade-offs.

My personal pick is probably Serenity-BDD, because I like the additions to the test steps and I really dig there output format.

As always, my showcase can be found here:

https://github.com/unexist/showcase-acceptance-testing-quarkus

unexist.dev