relaxdiego (Mark Maglana's Technical Blog)

How We Use Cucumber, The Sequel

May 24, 2012
Est. read: 6 minutes

In a previous post, I explained how we used Cucumber. Now, after about a month of using the recommended style, we’ve noticed some ‘usability problems’ and introduced our own styles.

Readability of stepdefs

The following is how our steps used to be defined:

Then /^I can disable notifications$/i do
  Notifier.fill_in 'username', 'jdoe'
  Notifier.fill_in 'password', '123qwe'
  Notifier.click   'login'
  Notifier.click   'disable'

  Notifier.should_be_disabled
end

It’s nice, clean, and readable…for a programmer. However, stepdefs are not exclusive to programmers: testers also need to be able to review them to know if the test is thorough or accurate. We have smart testers who know how to find edge cases or break an application, but they don’t know how to code. Things like dot notation and remembering to enclose a string with quotations are alien to them. They already understand English though and since we found that it’s easy to modify the above code to read like English, we thought it would be better to write stepdefs that the Testers can use within stepdefs. One example of such a stepdef is:

Then /^Fill in the (.+) field with (.+)$/i do |field_name, value|
  field_name = field_name.split.join('_').downcase
  Notifier.fill_in field_name, value
end

Now, the step listed at the top is defined like this:

Then /^I can disable notifications$/i do
steps %{

  * Fill in the Password field with 123qwe
  * Click the Login button
  * Click the Disable button

  * The Notifier should be disabled

}
end

Our testers still had to deal with regex and the steps %{ } delimiter, but that’s a lot easier compared to the previous style. Now, their tests read a lot like the the test specs that they’re used to.

Fragmented test scripts and cognitive dissonance

One of the bigger pains that we’ve encountered is the fragmentation of our test scripts for a given scenario. For example, in the following:

Scenario Outline: Withdraw an amount given a balance
    Given my account has a balance of <Balance>
     When I withdraw <Requested Amount>
     Then the amount will be <Dispensed or Not>

    Scenarios:
      | Balance | Requested Amount | Dispensed or Not |
      |    $100 |              $50 | Dispensed        |
      |    $100 |             $150 | Not Dispensed    |

We’d end up with our test script for the above outline distributed across the three steps (Given/When/Then). This results in a few problems:

Given these problems, we had to change some things drastically. Here’s what we’re currently experimenting with:

Scenario Outline:
  * Given the account has a balance of <Balance>, the system will <Dispense or Not> <Requested Amount>

    Examples:
      | Balance | Requested Amount | Dispense or Not |
      |    $100 |              $50 | Dispense        |
      |    $200 |             $199 | Dispense        |
      |    $100 |             $150 | Not Dispense    |
      |    $200 |             $201 | Not Dispense    |

What we changed:

Second, we rewrote the underlying stepdef as follows:

TestCase /^If the account has a balance of (\d+), the system will dispense (\d+)$/i do |balance, requested_amount|

  Setup %{
    * Ensure my account has a balance of #{ balance }
  }

  Teardown %{
    * Reset my account to 0 when done
  }

  Script %{
    * Insert card into the slot
    * Press the 1, 2, 3, and 4 keys
    * Press the OK key
    * Press the Withdraw key
    * Enter the amount #{ requested_amount }
    * Press the OK key
    * The amount #{ requested_amount } should be dispensed
  }

end

(We also created a separate stepdef for the ‘Not Dispense’ scenario)

What we changed:

Finally, we created a separate directory for all the steps used in the above test case. So now our features directory looks like this:

features/
 |
 |- requirements/
 |   |
 |   |- withdrawal/
 |   |   |
 |   |   `- withdraw_amount.feature
 |   |
 |   `- deposit/
 |
 |- test_cases/
 |   |
 |   |- withdrawal_test_cases.rb
 |   |
 |   `- desposit_test_cases.rb
 |
 |- utilities/
 |   |
 |   |- functions/
 |   |
 |   `- steps/
 |       |
 |       |- accounts_steps.rb
 |       |
 |       |- atm_steps.rb
 |       |
 |       `- web_steps.rb
 |
 `- support/

The above structure is complemented with the following conventions:

Results so far

I understand that this approach has diverged from the Cucumber community’s accepted practice, however, the results so far have been very promising:

Conclusion

As I mentioned above, I’m aware that this is a significant divergence from generally accepted Cucumber practice (In fact this is more influenced by the Robot Framework than Cucumber) so we’re taking it slowly to see if there’s a catch. Overall though, we feel good about this new direction.

UPDATE I started working on a prototype framework that better supports what I’m trying to achieve. It’s tentatively named Norm ManaMana.