Small test suites that run frequently are a great way of catching a lot of regression bugs fast. Most of the time, all tests in this type of test suite pass but occassionally tests fail due to either a new regression bug or a random seed generating a new test scenario that catches a bug that has always existed. When tests fail it is in most cases often due to only one underlying bug. Consequently PinDown should be setup to debug one bug at a time as fast as possible.

Note This setup is optimized to debug one bug at a time as fast as possible. The assumption is that in most cases there is only one bug.

Test Setup

This type of test setup works as follows:

  1. PinDown launches a small test suite (a script written by the user). All test jobs run on the computer farm. A schedule (cronjob) set in the PinDown TestHub determines when PinDown launches the test suite.

  2. PinDown supervises the test results while the test suite is running

  3. If PinDown detects a failure then PinDown stops the test suite immediately by killing all its jobs on the farm

  4. PinDown debugs the failures and sends out bug reports

  5. PinDown goes back to step 1

Optimal Settings

The optimal PinDown settings for this test setup is as follows:

Frequent Launches

A small test suite should be setup to run frequently as the main purpose with a small test suite is to catch many (but not all) bugs fast.

PinDown has a unique feature where it can basically run non-stop without the risk of launching multiple jobs of the same test suite in parallel (which would just be a waste). You do this by setting the cronjob in PinDown TestHub to launch every say 15 minutes whereby PinDown will wake up, check if this project is already running and only if not, kick off a new run. This means you can launch the test suite more frequently as you don’t have to add extra margin to the test schedule in case the computer farm is busy.

The cronjob for each project is specified in PinDown TestHub (see How to schedule a cron job)

Stop on New Failure

If there is a failure in this test setup, it is most often only due to one underlying bug. Consequently it makes sense to stop the test suite as soon as a failure is detected and start debugging immediately. This is done with the following completion criteria:

group -name "test" -commands {
  set_completion -source "newfailure" -path "" -keywords "" -filter "";
};

This command makes PinDown stop the test phase when a new failure is detected, i.e. a failure that is not already attached to an open bug in the database. For more info on available completion criterias check the set_completion man page.

Debug One Bug At A Time

PinDown debugs using several computer slots on the farm in parallel (as defined by the set_debug_bandwidth command. The default value is 10 slots.

In this test setup the assumption is that there is most often only one underlying bug so consequently we should use all 10 slots to debug each bug (as opposed to debugging several bugs in parallel, each using fewer slots). This is controlled by set_diagnosis_optimization -limit option.

set_diagnosis_optimization -target "time" -limit "10" -minpass "1" -minopenbug "1";

Example

Here is an example of PinDown setup for a short test suite. Search for "Optimized for small test suites" to see the relevant settings.

pindown_config.txt
/*--------------------------------------------------------

 PinDown Setup - Small Test Suite

---------------------------------------------------------*/
// Server paths
license -flexlm "8055@flexlm.srv.local";
set_webinterface -path "http://vm.verifyter.com:8080/PinDown/";
set_results_database -location "vm:5432" -database "pindown" -username "postgres" -password "ENC:v1aVbzyLZ-2x53HA";

// Use all computer slots to debug one bug at a time
set_debug_bandwidth 10;
set_diagnosis_optimization -target "time" -limit "10" -minpass "1" -minopenbug "1"; // Optimized for small test suites

// Clear bugs for fresh analysis each time and do not check the setup
clear_results_database; // During setup the database should be cleared until the flow has been validated
check_setup "false";

// Email Setup
set_mail -host "localhost"
         -port "25"
         -from "PinDown <daniel@verifyter.com>"
         -encryption "none"
         -append "@verifyter.com"
         -user ""
         -password ""
         -to "maillist_project2@verifyter.com"
         -owner "daniel.hansson@verifyter.com"
         -sendtocommitters "true"
         -sendonnew "true";

// Kill all test jobs
set_kill_tests "kill_farm_jobs.sh";

// Kill PinDown itself (it runs on the farm too)
set_kill_pindown "kill %PINDOWN_PID%";

// Repositories
set_repository -option "P4=/depot/perforce/p4";
set_repository -option "FORCE_SYNC=-f";
set_repository -option "USE_TICKET";
set_repository -option "P4PORT=p4p-verifyter:1900";
set_repository -type "p4" -location "//depot/Hardware/verifyter/main/pindown/src/..." -username "daniel" -password "ENC:YsTtrzDASQNrw" -checkoutarea "verifyter/main/pindown/src";
set_repository -type "p4" -location "//depot/Hardware/verifyter/main/pindown/bin/..." -username "daniel" -password "ENC:YsTtrzDASQNrw" -checkoutarea "verifyter/main/pindown/bin";
set_earliest_revision -days "15";

// Test Results
group -name "test" -commands {
  extract -type "configlabel" -label "configLabel" -source "filename" -path "project_cat/.*/reg/mrl_.*" -keywords "";
  extract -type "buildpass" -path "%configLabel%/vcs/compile.log" -keywords "simv up to date";
  extract -type "buildfail" -path "%configLabel%/vcs/compile.log" -keywords "Error|syntax error|Compilation aborted";

  extract -type "testname" -path "%configLabel%/vcs/stat/log.sanity.*" -keywords "PASS|FAIL" -filter "";
  extract -type "testpass" -path "%configLabel%/vcs/stat/log.sanity.*" -keywords "PASS" -filter "PROJECT_HOME is not currently set";
  extract -type "testfail" -path "%configLabel%/vcs/stat/log.sanity.*" -keywords "FAIL";
  extract -type "replace" -label "testname" -text ".*:" -with "";
  extract -type "replace" -label "testname" -text " .*" -with "";
  extract -type "replace" -label "configLabel" -text ".*\/reg\/" -with "";

  set_completion -source "newfailure" -path "" -keywords "" ; // Optimized for small test suites
  set_completion -source "file" -path "done.txt" -keywords "";
};

// Checkout
checkout_repositories -latest -pollforchange;

// Launch tests and wait for test results
set_test_runarea "runarea/test";
set_test_checkoutarea "/fileserver2/pindown/%PROJECT%/checkoutarea/test";
set_test_log_file -path "runtests_log.txt";
shell "qsub -cwd -V -P bnormal -l os_bit=64,qsc='h|i|j|l|k',os_version='WS7.0' -N runtests_dev -o runtests_log.txt -e runtests_log.txt /fileserver1/pindown/dev/runtests.csh" -test;
wait -completion "test" -sleep "1m" -timeout "110m";

// Extract Test Results
extract -group "test" -file "test_result.xml";
read_results "test_result.xml";

// Debug failures
set_diagnosis_checkoutarea "/fileserver2/pindown/%PROJECT%/checkoutarea/debug";
set_diagnosis_runarea "runarea/debug";
diagnose -runtests {
  checkout_repositories -requested;
  shell "qsub -cwd -V -P bnormal -l os_bit=64,qsc='h|i|j|l|k',os_version='WS7.0' -N runtests_dev -o rundebug_log.txt -e rundebug_log.txt /fileserver1/pindown/dev/rundebug.csh" -diagnosis;
  wait -completion "diagnosis" -sleep "1m" -timeout "90m" -onfail "complete";
  extract -group "diagnosis" -file "debug_result.xml";
  read_results "debug_result.xml";
};

// Send Report
send_mail;