One of the best aspects of BDD is that they are human friendly, plain english, narrative, easy for a human (developer) to translate his\her thought process into actions, represents end-user behavior and a living documentation. By reading the output of a BDD test execution, anyone can understand the quality at that moment and the coverage provided by those tests. Above and beyond it invites better collaboration between PO and developers to verify the acceptance criteria in an automated fashion. It helps the functional verification part of the lifecycle.
When we move beyond functional checks, Is there anything similar to convert load and performance acceptance criteria? Right now most of the tools are engineer\tester focused tools, record and playback based and GUI driven.
If not BDD, Taurus seems to solve some critical problems that exist in this space. Except Locust, python based load testing tool, most of the popular tools such as JMeter, Gatling, Tsung, Visual Studio load test are all GUI driven and very hard to get them working for distributed Agile teams where more than one person would want to contribute, bug fix, maintain. Not very easy to code review when another colleague changes the test and accept the pull request and much more troubles.
Taurus is pretty cool,
- write load tests in plain english (yaml), although its not narrative compared to BDD, Taurus is definitely more readable.
- source control and easy for code reviews
- unified DSL to author the tests and more than once tool of choice to run the test (JMeter, Tsung, Gatling etc)
- readable real time reporting
- Integration with Blazemeter reporting services, user will see cool charts @ no cost..
- platform independent.. has Python and Java dependency and can run on any platform where Python and Java can run
Here is a quick sneak peek
This is all you need to run a get request on bing.com
--- scenarios: demo: requests: - url: "http://www.bing.com" method: GET
Here is how you can add load profile to it
execution: concurrency: 100 iterations: 100 hold-for: 10m scenario: demo
Put this in one .yml file and you are all set to run a load test for 10 mins with 100 concurrent users 100 iterations..couldn’t get easier than this..
So, how to get started?
Visit this page for detailed instructions – http://gettaurus.org/docs/Installation/
Once you install BZT..lets write a simple first load test.
My intention is to create a load test to find 90% response time when 100 concurrent users hit this page http://finance.yahoo.com/q?s=aapl continuously for 5 mins
First, we’ll create the scenario. It needs scenarios top level node and requests underneath. Demo is the name of the scenario.
scenarios: demo: store-cache: false requests: - url: "http://finance.yahoo.com/q?s=aapl" method: GET
Second, we’ll setup the load profile. “execution” is the root node that holds the load profile properties and scenario to run (demo).
execution: concurrency: 100 iterations: 100 hold-for: 5m scenario: demo
Remaining steps below are all optional.
Taurus uses JMeter by default, however, you may instruct to choose something else as well. If that tool of choice doesn’t exist, BZT will download, install it for you and run the test.
settings: default-executor: jmeter
Putting it together
--- settings: default-executor: jmeter execution: concurrency: 2 iterations: 2 hold-for: 30s scenario: demo scenarios: demo: store-cache: false requests: - url: "http://finance.yahoo.com/q?s=aapl" method: GET
When you run the test, you should see something similar to below result (note: I reduced the load profile while running from my laptop)
Depending on the platform you run the test, you might also see this cool report console as well.. its out of box
Now the icing on the cake is its super easy to integrate this with Blazemeter reporting and distribute some pretty reports around..
modules: blazemeter: token: e504c64....... reporting: - module: blazemeter report-name: Test run test: 25 user load 25 iterations for 5 mins project: demo
In the above code, you will have to key in your blazemeter API key.
Amend this to your existing test and run the test again..you will likely see some cool reports like below
You can download the logs from blazemeter report as well
In addition to all this, your console output should leave you with a link to this .html report for distribution..
This whole process can get better but Taurus is awesome and made traditional JMeter load testing much easier.. I love it.