Rspec and Cucumber : Re run intermitent specs or cucumber features failures

26 Dec Rspec and Cucumber : Re run intermitent specs or cucumber features failures

Sometimes when you have a large test suite chances are that some of them fail randomly and when this happen you would not want to re run everything again so here it is the configuration I used to have for Rspec and cucumber in order to re run only those failing specs and features, in my case in this project we are using circleci.

This is how it looks my circle.yml setup:

machine:
  timezone:
    America/Mexico_City
  node:
    version: 5.11.0
  environment:
    VARIABLE_SETUP_TIMEOUT: 50
dependencies:
  pre:
    - bundle exec rake prepare_assets

test:
  override:

    - RAILS_ENV=test bundle exec rspec_runner:all_rspec:
      parallel: true
      files:
        - spec/controllers/**/*_spec.rb
        - spec/helpers/**/*_spec.rb

    - mkdir -p $CIRCLE_TEST_REPORTS/cucumber:
      parallel: true
    - bundle exec rake cucumber_runner:start:
      parallel: true
      files:
        - features/section_one/**/*.feature
        - features/section_two/**/*.feature

    - bundle exec rake cucumber_runner:rerun_failing_scenarios:
      parallel: true

For cucumber:

Basically you need to indicate the `rerun` and a file to save failed features:

  bundle exec cucumber rerun --out cucumber_failures.txt

This is how it looks my rake task:

namespace :cucumber_runner do

 FAILING_CUCUMBER_SCENARIOS_FILENAME = 'failing_scenarios.txt'

 desc 'Run cucumber tests in using the circle ci configuration'
 task :start do

   # delete previous failing cucumber test scenarios filename
   if File.exists?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
     File.delete("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
   end

   # exit 0, we don't want to fail here
   exec("bundle exec cucumber --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber -f rerun --out #{FAILING_CUCUMBER_SCENARIOS_FILENAME}; exit 0")
 end

 desc 'Re run cucumber scenarios that failed in the first intent'
 task :rerun_failing_scenarios do

   # we don't need to run cucumber again if all scenarios passed
   unless File.zero?("#{FAILING_CUCUMBER_SCENARIOS_FILENAME}")
     # run cucumber with failing scenarios only
     exec("bundle exec cucumber @#{FAILING_CUCUMBER_SCENARIOS_FILENAME} --format junit --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber")
   end
 end
 
 task all_rspec: :environment do
   Rake::Task["spec:start"].execute
   Rake::Task["spec:retry"].execute
 end
end

For Rspec

Basically in order to re run failed specs in the first run, you need to specify a file name in your Rspec setup, which will save the status of each spec in the first try and after that you are able to run only the ones that failed using the tag –only-failures:

bundle exec rspec # first try
bundle exec rspec --only-failures # run only failures

Rspec setup:

RSpec.configure do |config|

  config.example_status_persistence_file_path = "rspec_failures.txt"

end

Rake task for continuos integration software:
As you know when you run several commands in your CI if the first one fails that won’t let run the spec retry, in order to avoid that we need to catch the exception as you see in the circle.yml configuration I’m running a custom rake tasks and that looks like this:

content of Rakefile file:

require File.expand_path('../config/application', __FILE__)

Rails.application.load_tasks

begin
  require 'rspec/core/rake_task'
  RSpec::Core::RakeTask.new("spec:start") do |t|
    t.rspec_opts = "--format documentation"
    t.verbose = false
    t.fail_on_error = false # don't stop the whole suite
  end

  RSpec::Core::RakeTask.new("spec:retry") do |t|
    t.rspec_opts = "--only-failures"
    t.verbose = false
  end
rescue LoadError
end

Afterwards you can easily run the whole rspec test suite and re-run only the ones that failed in the first try:

bundle exec rake rspec_runner:all_rspec

That’s it hope you find useful this post, keep in touch.

H