1. Integrating PinDown into regression testing

The labs show how to integrate PinDown with the regression test environment at hand. There are many different regression environments but the principals are the same. We use a simple regression environment built around an open source core to demonstrate these principals. The core used here is the Y80e processor from OpenCores http://opencores.org/project,y80e .

Integration is best understood by starting with the PinDown flow. PinDown follows a straightforward flow: Run tests (optional step), extract test results, group by error signature, then run debug if any failures were found. In the debug cycle PinDown will rerun select failing tests on past revisions, trying to find revisions that see those test pass. The debug cycle is run on several past revisions in parallel.

All mentioned steps require some setup and configuration by the user, except for grouping error signatures, which is an internal PinDown function. Here is the flow chart:

PinDown flow
Figure 1. PinDown Flow

2. LAB1 Running Regression Tests

The labs pick up where the installation guide left off. After a complete and successful installation you should have access to TestHub. TestHub is the control and monitoring dashboard for PinDown. For the labs TestHub can be found at http://localhost:8080/PinDown/ .

2.1. Creating New Projects

For this exercise the lab1 project has already been created for you.

Note If you were to create a new project from scratch you’d have to go to the Settings tab in TestHub and click on New Project. Enter a name for the project and click on Create Blank Project.
TASK

Verify that the lab1 project exists:

2.2. Configuration via pindown_config.txt

When a new project is created PinDown creates an entry in the DB and creates a local project directory under the /opt/pindown directory. The /opt/pindown was selected during installation and is any directory of your choice with sufficient disk space for the project. In the /opt/pindown/lab1 project directory there will be an empty pindown_config.txt configuration file:

TASK

Locate the empty pindown_config.txt in the PinDown folder:

$ cd /opt/pindown/lab1
$ ls
pindown_config.txt

In the next sections we will add all necessary configurations and commands to this file.

2.3. License Server Setup

PinDown has two license mechanism, flexlm and proprietary. When deployed PinDown will use flexlm. When in trial mode, or in this lab, PinDown can use the proprietary mechanism instead. Here is the syntax for the two options. For more details check the license command in man pages.

Flexlm configuration:

license -flexlm "2304@flexlm-lic-server";

Proprietary configuration:

license -server "http://localhost:8080/PinDown/";

2.4. Web Interface Setup

PinDown was configured to run on localhost port 8080. The port is configured in Tomcat’s server.xml file during installation. This URL is communicated to the PinDown debugger via the set_webinterface setting in the config file.

It is possible to stop runs by clicking on the Stop-button on the PinDown TestHub web page related to this project. What actually happens when you click on the Stop-button is defined by the set_kill_pindown-command. In this project we run PinDown locally. The following command kills the local PinDown process.

set_kill_pindown "kill %PINDOWN_PID%";

2.5. Results Database Setup

The results database server is a PostgreSQL server running on localhost port 5432. The database is called pindown by default, accessible for user postgres, password postgres. Configure database settings with set_results_database command.

By default the database will collect all regression results. The results can be flushed from the DB with the clear_results_database command, if required.

2.6. Check Setup

The check_setup-command controls whether PinDown should automatically verify that PinDown itself is correctly setup. However, we have not set it up yet, so let’s keep it off.

check_setup "off";
TASK

Add below commands to pindown_config.txt:

### General settings ###
license -flexlm "45601@heimdal";
set_webinterface -path "http://localhost:8080/PinDown/";
set_results_database -location "localhost:5432" -database "pindown" -username "postgres" -password "postgres";
clear_results_database;
set_kill_pindown "kill %PINDOWN_PID%";
check_setup "off";
Note Notice that the database password will be encrypted by PinDown on first invocation.

2.7. Email

PinDown supports the SMTP protocol. SMTP by default uses TCP port 25. SMTP connections secured by SSL, known as SMTPS, default to port 465 (see set_mail).

SMTP: set_mail -host "localhost" -port "25" -from "PinDown: %PROJECT_NAME% <christian.graber@verifyter.com>" -encryption "none" -append "@verifyter.com" -user "chrisg" -password "banana" ...
SMTPS: set_mail -host "smtp.gmail.com" -port "465" -from "PinDown: %PROJECT_NAME% <christian.graber@verifyter.com>" -encryption "ssl" -append "@verifyter.com" -user "christian.graber@verifyter.com" -password "banana" ...
Note PinDown will automatically encrypt plain text passwords on first invocation.

PinDown will always send a bug report to the committer of that bug, unless explicitly disabled. Typically the only time this option is disabled is during bringup or maintenace of the project.

Enable bug report to committer:

-sendtocommitters "true"

Disable bug report to committer:

-sendtocommitters "false"

PinDown allows fine-grained tuning of email recipients lists (CC list):

-topass "joe@example.com;jane@example.com" ('No failures' recipients)
-tofail "joe@example.com;jane@example.com" ('Always failing tests and/or bugs' recipients)
-tobadcommit "joe@example.com;jane@example.com" ('Bugs only' recepients)

The owner of the project will receive all emails including administrative emails:

-owner "john@example.com"
TASK

Add email setup to pindown_config.txt:

### Configure email ###
set_mail -host "localhost" -port "25" -from "PinDown: %PROJECT_NAME% <%USER_NAME%@localhost>" -encryption "none" -append "@training.verifyter.com" -user "" -password "" -topass "" -tofail "" -tobadcommit "" -owner "%USER_NAME%@localhost" -sendtocommitters "true" -sendonnew "true";
Note All CC lists for this lab are empty. Only the user will receive emails via -owner option.

2.8. Repository access

PinDown can support multiple revision control systems. By default support for Subversion, Perforce, GIT, Mercurial and Clearcase is available. Contact Verifyter to check on support for your unlisted RCS. In this lab we use Subversion. Using the PinDown command set_repository you specify which folders of the revision control system that are applicable to this project.

One of the key settings you typically want to adjust to your specific project needs is set_earliest_revision. This option lets you specify how far back in revision history PinDown should search.

TASK

Add repository setup to pindown_config.txt:

### Configure repository ###
set_repository -type "svn" -location "svn://localhost/training/y80e/branches/labs" -username "%USER_NAME%" -password "svnpassword" -checkoutarea "pilot";
set_earliest_revision -weeks "4";

2.9. Checking out repositories

There are multiple options for checking out repositories. You can check out at the latest revision, or at the latests revision only if there was a new checkin since last run, or at a specific past revision (see checkout_repositories).

checkout_repositories -latest;
checkout_repositories -latest -pollforchange;
checkout_repositories -revision "3265";
TASK

Add below lines to checkout the repositories:

### Checkout repository ###
shell "$PINDOWN_WORKFOLDER/cleantest.sh > runtest.log 2>&1" -test;
checkout_repositories -latest;
shell "ln -s $TEST_CHECKOUTAREA/pilot" -test;

2.10. Running tests

PinDown needs test results to work with. In the first step of the flow PinDown will run tests. Alternatively PinDown can skip this step and go directly to extraction when test results are already available.

PinDown run test flow
Figure 2. PinDown flow run tests

In this example project tests are run simply by calling the script runtest.sh. This script first builds the DUT, then calls several tests and logs their results.

runtest.sh:

#!/bin/bash
script=`basename "$0"`
echo ----------------------------------
echo PinDown $script: Initiate
echo `date`
echo ----------------------------------
echo "test_runarea: " $TEST_RUNAREA
echo "test_checkoutarea: " $TEST_CHECKOUTAREA
cd $TEST_CHECKOUTAREA/pilot/run
echo ----------------------------------
echo PinDown $script: Build
echo `date`
echo ----------------------------------
./build_y80e.sh
echo ----------------------------------
echo PinDown $script: Run Tests
echo `date`
echo ----------------------------------
for test in bit_ops dat_mov io_ops jmp_ops alu_ops
do
    echo "running " $test
    ./run_y80e.sh $test > z80.$test.log
done
echo ----------------------------------
echo PinDown $script: Create summary
echo `date`
echo ----------------------------------
cd $TEST_RUNAREA
touch done.nfo

The build script is a simple call to iverilog. Results are logged in build_y80e.log

build_y80e.sh:

iverilog -o y80e.vvp -v -I../rtl pilot.v &> build_y80e.log

The run script will call the compiled executable from the build step. Results are logged in test specific log files.

run_y80e.sh

vvp v80e.vvp -lxt2 +TEST_NAME=$test.vm +TEST_DEFAULT=$test"d.vm" +TEST_FORCEBUG=$forcebug &> z80.$test.log

The command shell -test is used to launch the runtest.sh script in test run-area (which is why the -test option is set). All scripts and the pindown config file are located in the $PINDOWN_WORKFOLDER (in this lab set to /opt/pindown/lab2).

The command set_test_log_file is set to the output log (runtest.log) in order for it to be accessible from the PinDown TestHub by clicking on the Test log-button.

TASK

Add below lines to pindown_config.txt to run the test suite:

### Run regression ###
set_test_log_file -path "runtest.log";
shell "$PINDOWN_WORKFOLDER/runtest.sh >> runtest.log 2>&1&" -test;
TASK

Go to TestHub, project lab1 and click the Start-button.

After successful completion you should see this log in TestHub:

Starting up pindown job
Sat Apr 30 14:54:00 CEST 2016
Started by Anonymous
Starting project: lab1
workfolder: /opt/pindown/lab1
Running as verifyterdh
In folder /opt/pindown/lab1 owned by verifyterdh
PinDown starting...
Starting shellcommand: $PINDOWN_WORKFOLDER/cleantest.sh > runtest.log 2>&1
PinDown creating run area for the test phase
Using folder: /opt/pindown/lab1/runarea/test/run1
Preparing to check out these repositories at the latest revisions:
svn://localhost/training/y80e/branches/labs at 133 (latest) Earliest is 131
Using folder: /opt/pindown/lab1/checkoutarea/test
Progress 0% - Apr 30 2016 14:54:05
Progress 100% - Apr 30 2016 14:54:08
SVN checkOutRevision: Checking out 133 from svn://localhost/training/y80e/branches/labs into /opt/pindown/lab1/checkoutarea/test/pilot
Starting shellcommand: ln -s $TEST_CHECKOUTAREA/pilot
Starting shellcommand: $PINDOWN_WORKFOLDER/runtest.sh >> runtest.log 2>&1&
PinDown diagnosis has finished.

3. LAB2 Setting up Extraction

At the end of lab1 we called runtest.sh. This script builds the DUT and runs tests. To build the DUT it calls build_y80e.sh. Similarly it calls run_y80e.sh for each test in the suite. Both scripts run in the checkout area and produce log files in the checkout area. PinDown extracts pass/fail information from these log files.

PinDown extract flow
Figure 3. PinDown flow extract results

3.1. Extracting test results

PinDown distinguishes between build and test failures. The build log will indicate whether a build was successful or not. There can be more than one build if the DUT supports multiple configurations. In this example only one configuration is available.

TASK

Go to TestHub and start lab2

TASK

Have a look at the build log in lab2/runarea/test/runX/pilot/run/build_y80e.log.

pilot/run/build_y80e.log:

...
PARSING INPUT
LOCATING TOP-LEVEL MODULES
   top_levl
 ... done, 0.01 seconds.
ELABORATING DESIGN
 ... done, 0 seconds.
RUNNING FUNCTORS
 -F cprop ...
 -F nodangle ...
 ... 1 iterations deleted 136 dangling signals and 0 events.
 ... 2 iterations deleted 136 dangling signals and 78 events.
CALCULATING ISLANDS
 ... done, 0.01 seconds.
CODE GENERATION
 ... invoking target_design
 ... done, 0.01 seconds.
STATISTICS
lex_string: add_count=2466 hit_count=4396
Icarus Verilog version 0.9.7  (v0_9_7)

As is often the case build logs don’t indicate explict pass phrases. Instead you can make sure that the log reached the end by picking up a phrase that shows up on completion. Here we can pick up the word STATISTICS, assuming that if the script reached STATISTICS the build was successful. If that is not an option, you can assign a default pass status to a build already in the config file.

There can be many different failure signatures. The best way to find out about these failure signatures is to ask someone who knows the test environment. Typically these signatures are already tracked somewhere. If that is not an option you can let the build fail artifically and record the signature. That will be your starting point. Most likely there’s more than one signature and you will have to add them one-by-one as they show up. In the pilot project we know that the signatures are syntax error or I give up or No such file or directory.

TASK

Now have a look at the test log files:

pilot/run/z80.alu_ops.log:

  ...
$monitor at time=3719180: MEM_ADDR='h00c2 MEM_DATA_OUT='h30
$monitor at time=3719380: MEM_ADDR='h00c3 MEM_DATA_OUT='h30
pilot-test-result  ---------- FAILED: ALU operation failed ----------
pilot-test-time time=3719440

pilot/run/z80.io_ops.log:

...
$monitor at time=654180: MEM_ADDR='h00c2 MEM_DATA_OUT='h13
$monitor at time=654380: MEM_ADDR='h00c3 MEM_DATA_OUT='h13
pilot-test-result  ---------- PASSED ----------
pilot-test-time time=654440

Here we have clear pass/fail phrases, as is the case for most regression environments. Some regression environments do not have a distinct fail phrase. Instead they will have several failure signatures. These signatures are typically already extracted in some regression scripts. Again, it’s best to ask someone who knows the test environment.

So now we know the pass/fail criteria for build and test runs. The next step is to teach PinDown to recognize them. This feature is called extraction in PinDown lingo. The central PinDown command to achieve this is the extract command. You can find details on this command in PinDown’s man pages. The extract commands are put into what is called an extraction group in the config file. Each group has a name and is denoted by brackets:

group -name "test" -commands {
  ...
}

3.2. Extracting build log

Inside this group we will place the extraction commands. Start with build extraction. Head over to the /opt/pindown/lab2 project directory:

TASK

Add extraction for build results to pindown_config.txt in lab2:

### Extract test results ###
set_extract_filenames;
group -name "test" -commands {
# Extract build name
extract -type "configlabel" -source "filename" -path "pilot/run/build_y80e.log" -keywords "";
extract -type "replace" -label "configlabel" -text ".*run\/" -with "";
extract -type "replace" -label "configlabel" -text "\.log" -with "";
# Extract build result
extract -type "buildpass" -path "pilot/run/build_y80e.log" -keywords "STATISTICS";
extract -type "buildfail" -path "pilot/run/build_y80e.log" -keywords "syntax error|I give up|No such file or directory";
};
extract -group "test" -file "test_result.xml";

Here’s what this code does. It will first place the entire path to the build log file into variable configlabel. It will then use extract -type "replace" commands to truncate the path to the string build_y80e. This will be the name for the build, or configuration.

Note PinDown uses the term build and configuration interchangeably.

Then the code will extract the actual pass/fail condition for the build, utilizing the key words we found above. Finally, the result of this extraction will be written into an XML file, test_result.xml.

The command set_extract_filenames adds the file name of the log data.

This process is typically iterative. That means you’d start with the first command, check the results, add another command, check the results again etc., until you converge to the desired result. Ideally you’d want to comment out the scripts that remove the checkoutarea, the checkout command and the call to runtest.sh during these iterations. This will save a lot of time because the results are already available and do not need to be regenerated. For the labs this step is optional. In your own regression environment it is recommended, unless you also have a fast test suite.

### Checkout repository ###
//shell "$PINDOWN_WORKFOLDER/cleantest.sh > runtest.log 2>&1" -test;
//checkout_repositories -latest;
//shell "ln -s $TEST_CHECKOUTAREA/pilot" -test;
### Run regression ###
set_test_log_file -path "runtest.log";
//shell "$PINDOWN_WORKFOLDER/runtest.sh >> runtest.log 2>&1&" -test;

PinDown has some runtime options to enable this iteration. Go to the PinDown runarea for the latest run, e.g. runarea/test/run10, and call below command:

TASK

Run PinDown from command line in test area to extract build results:

$ cd runarea/test/run10 (pick latest run directory)
pindown -config /opt/pindown/lab2/pindown_config.txt -debug -runhere

You can examine the output of running this command in the terminal, to check for any syntax errors. It will look something like this when successful (you will find it 1-2 pages up amongst many other commands; PinDown is very verbose when -debug is enabled):

...
09:49:47 [DEBUG] - added %configlabel%pilot/run/build_y80e.log
...

Once the call to PinDown has terminated successfully you’ll find the file test_result.xml in the run area. Look for the buildResult string in this file. This line will reflect what PinDown has extracted from the build log file. In our example you want to find a pass status and a result line that includes the STATISTICS key word. The configuration label should be called build_y80e.

runarea/test/run10/test_result.xml:

...
<com.verifyter.pindown.BuildResult buildResult="%buildpass%pass" buildResultLine="pilot/run/build_y80e.log:STATISTICS">
...
<com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%build_y80e"/>
...

3.3. Extracting test log

The procedure to extract test results is basically the same. The only difference is that you want to extract a few more statistics from your tests. Aside from test name and pass/fail status, PinDown also wants to know the test runtime and the specific random seed used with this test, if any. Add below test extraction commands inside the extraction group and under the build extract commands in your config file.

TASK

Add extraction for test results to pindown_config.txt:

{
  ...
# Extract test name
extract -type "testname" -path "pilot/run/z80.*.log" -keywords "TEST_NAME=";
# Clean up test name
extract -type "replace" -label "testname" -text ".*TEST_NAME=" -with "";
extract -type "replace" -label "testname" -text "^[ \t]+" -with "";
extract -type "replace" -label "testname" -text "\.vm$" -with "";
# Extract test result
extract -type "testpass" -path "pilot/run/z80.*.log" -keywords "---------- PASSED";
extract -type "testfail" -path "pilot/run/z80.*.log" -keywords "---------- FAILED";
# Extract test runtime
extract -type "testend" -path "pilot/run/z80.*.log" -keywords "pilot-test-time;column_delimiter= ;$2";
extract -type "replace" -label "testend" -text ".*time=" -with "";
};

This code will first extract the entire path to the test log file, including the matched line, into a variable called testname. Then it will discard everything but the actual test name using extract -type "replace" commands. The resulting test name will be e.g. alu_ops. Note that we’re matching the string TEST_NAME= in the log to find the test name.

Next is the extraction of the pass/fail status and testend. The latter provides the test runtime. The example project does not use randomization. Call PinDown again with the completed extraction group to produce a new test_result.xml:

TASK

Run PinDown from command line in test area to extract test results:

$ cd runarea/test/run11 (pick latest run directory)
pindown -config /opt/pindown/lab2/pindown_config.txt -debug -runhere

After a successful call to PinDown, the result in the XML should look like below. Shown for one passed and one failed test only. You should have more results.

test_result.xml:

...
<com.verifyter.pindown.Test testEnd="%testend%1936840" testName="%testname%bit_ops" testResult="%testfail%fail" testResultLine="...---------- FAILED ----------..."/>
...
<com.verifyter.pindown.Test testEnd="%testend%654440" testName="%testname%io_ops" testResult="%testpass%pass" testResultLine="pilot/run/z80.io_ops.log:pilot-test-result  ---------- PASSED ----------"/>
...

3.4. Adding completion criteria

The last step is to add a completion criteria (see set_completion. PinDown understands different criteria for completion. Here we pick a simple one, which is synchronizing on a specific file called done.nfo. The idea here is that PinDown will launch the test suite on the farm. The PinDown script needs to continue after the launch. So make sure the call to runtest.sh is non-blocking (i.e. call ending with &). PinDown will then extract results in regular intervals, waiting for completion of the run. In this case PinDown will wait for the file done.nfo to show up in the run area. The wait-command waits until one of the completion criterias in the test group has been satisfied, by specifying wait -completion "test". After completion the results are extracted into an XML file, which in turn is read into PinDown using the read_results-command. Add below commands to your config to complete this step.

TASK

Add completion criteria inside the extraction group at the very end:

{
  ...
  # Completion criterium
  set_completion -source "file" -path "done.nfo" -keywords "";
};
TASK

Add wait and read commands after the extraction group:

### Wait for regression to finish and extract results ###
wait -completion "test" -sleep "4s" -timeout "15m";
extract -group "test" -file "test_result.xml";
read_results "test_result.xml";
Note You already have the extract command and you need to insert the other commands above and below.

Finally, go to TestHub and kick off the lab2 project.

TASK

Go to TestHub and start lab2

When everything ran successfully you should see the real time status advancing, resulting in several tests found.

PinDown real time status

Note that if the computer is quick you may only see the final result directly. You will only get partial results as in the sceen-shot above if not all tests have completed when PinDown checks the results the first time.

4. LAB3 Setting up Automatic Debug

4.1. Automatic debug flow

This lab deals with the automatic debug flow. After running tests and extracting failures we are ready to debug them. The debug phase is similar to the test phase. In debug PinDown runs a single test, called pilot, on past revisions. This is done on several past revisions in parallel. Typically 10 parallel runs are used. PinDown will select the pilot and past revisions automatically. The pilot is the shortest test in each error signature group. Grouping is an internal PinDown function that does not require any setup from the user.

PinDown debug flow
Figure 4. PinDown flow debug cycle

4.2. Running the pilot

PinDown will check out a past revision and hand over control to a user defined script, rundebug.sh. In this script the user has to rerun the test provided by PinDown. Test name, and optional test parameters, are communicated by the environment variables that PinDown defines. These variables are available in any script PinDown launches. Please check the man page for the shell-command for a full list of available variables that you can use in your shell scripts. All environmental variables that PinDown sets are listed in a local file in the runarea called pindown_variables.sh (there is also one file for .csh).

The only variable we need here is the one containing the pilot test name. This variable is called $DEBUG_TESTS and in this example the name of the pilot test is dat_mov. From within the rundebug.sh-script you can access this variable:

$ echo $DEBUG_TESTS
dat_mov

The tests in this example project don’t have any options. If they had some, they could be either included in the test name or extracted separately. Often there are only a few options and the easiest way to access them is to include them in the test name. So re-running a test in this project is very simple. You only have to call the run script, passing in the name of the test you want to run:

$ ./run_y80e.sh $DEBUG_TESTS

Here is a copy of the script that builds and re-runs the pilot test. This script, rundebug.sh, is called from the config file:

rundebug.sh:

#!/bin/bash
script=`basename "$0"`
echo ----------------------------------
echo PinDown $script: Initiate
echo `date`
echo ----------------------------------
echo "diagnosis_testarea: " $DEBUG_RUNAREA
echo "diagnosis_checkoutarea: " $DEBUG_CHECKOUTAREA
ln -s $DEBUG_CHECKOUTAREA/pilot pilot
echo ----------------------------------
echo PinDown $script: Build
echo `date`
echo ----------------------------------
cd $DEBUG_CHECKOUTAREA/pilot/run
./build_y80e.sh
echo ----------------------------------
echo PinDown $script: Run Tests
echo `date`
echo ----------------------------------
cd $DEBUG_CHECKOUTAREA/pilot/run
echo "debugging $DEBUG_TESTS ..."
./run_y80e.sh $DEBUG_TESTS > z80.$DEBUG_TESTS.log
echo ----------------------------------
echo PinDown $script: Create summary
echo `date`
echo ----------------------------------
cd $DEBUG_RUNAREA
touch done.nfo

As is the case for the test phase, the pilot is run in the checkout area. After creating a symbolic link to ease extraction of results, the script changes into the checkout area. Then it builds the DUT and calls the run script, passing in the pilot test name.

4.3. Extracting results

The next step after running the pilot is to extract the results. The debug phase has an extraction group just like the test phase. This group will contain all the extraction commands for extracting debug results. As you can imagine, these commands will be quite similar to the ones used in the test phase, as long as the log files are the same. And in this example project they are actually identical. If that is the case, you don’t have to reproduce the extraction group and can skip it altogether. Only if there are differences in the way results are extracted, even a single difference, the extraction group needs to be there.

Here is the syntax for debug extraction group. It would contain commands similar to the test extraction group, but can be skipped altogether, since it is idential to the test extraction group in this case. Skip this group:

group -name "diagnosis" -commands {
  ...
}
Note PinDown uses the names diagnosis and debug interchangeably.

4.4. Configuring the debug loop

The final step in configuring the debug phase is setting up the actual debug loop. In this loop PinDown is instructed to check out past revisions, run the pilot on each and extract the XML. Before we do that, we need to add a few more debug specific settings. Most importantly the debug bandwidth (see set_debug_bandwidth. This parameter determines how many runs in parallel are launched in debug. Typically this parameter is set to 10, meaning PinDown will check out 10 different workspaces and launch the pilot in each.

The set_diagnosis_optimization option specifies the behavior of the binary search algorithm. Typically you are fine with the default setting. Adjustements are only necessary in very particular situations.

Head over to the /opt/pindown/lab3 project directory:

TASK

Add debug specific settings:

### Debug general settings ###
set_debug_bandwidth 10;
set_diagnosis_optimization -target "time" -limit "16" -minpass "10" -minopenbug "10";

Now add the actual debug loop and debug commands. The flow is basically the same as the test phase flow:

TASK

Add the debug loop and debug commands:

### Debug loop ###
diagnose -runtests {
  shell "$PINDOWN_WORKFOLDER/cleandebug.sh > debug.log 2>&1" -diagnosis;
  checkout_repositories -requested;
  shell "$PINDOWN_WORKFOLDER/rundebug.sh >> debug.log 2>&1&" -diagnosis;
  wait -completion "diagnosis" -sleep "10s" -timeout "15m";
  extract -group "diagnosis" -file "debug_result.xml";
  read_results "debug_result.xml";
};
### Send full report ###
send_mail;
Note The command send_mail after the debug loop will send out the summary report by email.

That’s it. The pindown_config.txt file is now complete.

4.5. Running automatic debug

TASK

Head over to testhub and launch lab3.

Here we’re not showing a copy of the realtime status output, since it would simply be too long. Instead, after successful debug, you will receive the summary bug report by email which should look very similar to the one below:

lab3 regression: PinDown has found 1 bug (committed by Christian) - Full Report

PinDown bug report email
Figure 5. PinDown bug report

4.6. Interpreting results

The email report contains the most important information in the box on top. There you will find the name of the failing test, the error signature, the name of the committer, the revision number of the bad commit, the commited files and for small committs also the lines that have changed. There’s a link to the log file too. In most cases this information is enough to debug the problem on the spot. Some problems are tricker and require e.g. to re-run the failing test with waveforms turned on. PinDown allows to add specific information to the bug report explaining how to reproduce the problem.

For a more complete explanation see the PinDown Getting Started Guide.

5. LAB4 Debugging setup problems

This last lab is a little different. So far the focus was on reproducing the PinDown flow for the given regression environment. This is fairly straightforward until you encounter a problem. This lab reproduces the exact same flow, but has two problems. You are expected to find these problems with some guidance.

5.1. Integration Points

A good place to start debugging is at the integration points. That is where PinDown is interfacing with the regression environment. This will allow you to determine whether the problem is in PinDown’s setup or in the regression environment. The test phase and the debug phase each have their own integration points.

Test phase integration points:

*PinDown*                       *Regression Environment*

call runtest.sh
                                execute runtest.sh
                                -> produces runtest.log
                                -> produces test logs z80.*.log

extract logs
-> write test_result.xml

Test phase log files:

Generated by the runtest.sh-script
runtest.log:     runarea/test/run*/runtest.log
z80.*.log:       runarea/test/run*/pilot/run/z80.*.log
Generated by PinDown when extracting the results:
test_result.xml: runarea/test/run*/test_result.xml

Debug phase integration points:

*PinDown*                       *Regression Environment*

specify pilot test
-> in diagnosis_tests.xml
-> in pindown_variables.sh/.csh

call rundebug.sh
                                retrieve pilot test name
                                execute rundebug.sh
                                -> produces rundebug.log
                                -> produces test log z80.*.log
extract logs
-> write debug_result.xml

Debug phase files:

Generated by PinDown:
debug_tests.xml:       runarea/debug/run*/iteration*/try*/pindownID*/debug_tests.xml
pindown_variables.sh:  runarea/debug/run*/iteration*/try*/pindownID*/pindown_variables.sh
pindown_variables.csh: runarea/debug/run*/iteration*/try*/pindownID*/pindown_variables.csh
Generated by the rundebug.sh-script
rundebug.log:          runarea/debug/run*/iteration*/try*/pindownID*/rundebug.log
z80.*.log:             runarea/debug/run*/iteration*/try*/pindownID*/pilot/run/z80.*.log
Generated by PinDown when extracting the results:
debug_result.xml:      runarea/debug/run*/iteration*/try*/pindownID*/debug_result.xml
Note The debug runareas have deeper paths than the test runarea. This is because during debug several runareas are used in parallel, typically 10, whereas in the test phase there is only one runarea. The paths to the debug runareas start with an iterationX-folder which contains all debug runareas on which tests have been kicked off in parallel (typically 10 tests in parallel on 10 different debug runareas). The tryY-folder contains each attempt, normally there is just one try, but if there are problems with setup, farm load etc there will be subsequent tries. The pindownIDi-folders are the actual debug run areas, where i is 1-10 if we run 10 jobs in parallel.

The easiest way to debug with integration points is to work your way backwards through the log files. For example, if the problem shows up in the debug phase,

  • Check what the pilot test’s name is

  • Check if expected results for this pilot are in debug_results.xml.

  • If they are not, then check if test log file z80.*.log exists.

  • If it does, and no errors are reported, the problem is in the extraction step in PinDown.

  • If it doesn’t, check rundebug.log if something went wrong running the pilot.

  • etc.

5.2. Debugging Issue One

Let’s get started:

TASK

Head over to testhub and launch lab4.

At first glance everything seems fine. The build is reported as passed and the tool seems to be running some tests. But when the report arrives it complains that no test were run. And indeed, when looking at the real time status zero tests are reported as run.

lab4 regression: Builds completed, but no tests were run (farm full or flow broken)

PinDown setup problem 1
Figure 6. Issue One

Head over to the /opt/pindown/lab4 project directory:

What could have gone wrong?

Following the flow backwards, here is what you can check:

  • Are there any test results in test_result.xml?

  • Do the test log files z80.*.log exist?

  • Were any errors reported in z80.*.log?

  • Were any errors reported in runtest.log?

Note If you see a broken symbolic link to a test log or any other test-specific issue, first ask yourself: is this an issue with a single test or does this affect all tests? We are looking for a problem here which prevents any test results from being found, not a problem with a single test.

5.2.1. Which one is it?

Basically, either the tests were not run properly (issue in /opt/pindown/lab4/runtests.sh) or the extraction of the test results doesn’t work (issue in /opt/pindown/lab4/pindown_config.txt). Which one is it?

If the problem is in runtests.sh, on which line is there a problem according to the logs (runtests.log, pilot/run/z80.*.log)? If the problem on the other hand is in pindown_config.txt the problem has to do with the extraction of test names. Use a unix grep-command with the same path and keywords as extract -type "testname" is set to in the pindown config file and run this grep-command from the latest runarea and see if it works.

Can you figure it out?

TASK
  • Try to find the problem and fix it.

  • Kick off lab4 again in TestHub to check.

  • Success criteria: PinDown finds test results

When you have fixed the problem PinDown will find test results, including some failures. It will try to debug the failures but encounters a second problem. Please proceed to the next section.

5.3. Debugging Issue Two

So you have found and fixed the first problem. But when running lab4 again you received a report complaining that debug was aborted.

lab4 regression: Debug was aborted.

PinDown setup problem 2
Figure 7. Issue Two

First notice that the test phase now has a record of tests run and looks fine. As the subject line of the report suggests, the new problem is now in the debug phase. The PinDown email report is actually very specific about what is missing:

Configuration: "config build_y80e" - Extracted OK!
Buildresult: Got "pass" - Extracted OK!
Testname: Error, got no value, expected "dat_mov"
Testresult: Error, got no value
Run area: /opt/pindown/lab4/runarea/debug/run4/iteration1/try4/pindownID1
Log files: pilot/run/build_y80e.log

So the build was fine. But there are no results for either the test name or the test result (pass/fail status).

First of all, go into the debug run area reported in the email. Details about the test PinDown wants to run is always available in these files:

  • diagnosis_tests.xml

  • pindown_variables.sh/.csh

The result should have been generated in the following file:

  • debug_result.xml

Trying to work your way backwards, in the /opt/pindown/lab4 project directory, here’s what you can check:

  • What is the requested pilot test’s name?

  • Is there a result for the pilot in debug_result.xml?

  • Does the test log for the pilot z80.*.log exist?

  • Were any errors reported in z80.*.log?

  • Were any errors reported in rundebug.log?

Note If you see a broken symbolic link to a test log or any other test-specific issue, first ask yourself: is this the pilot test that PinDown is trying to run? If not, ignore it. The debug flow is only aborted if the requested pilot test cannot be re-run on the same revision where it failed in the test phase.

5.3.1. Which one is it?

Basically, either the tests were not run properly (issue in /opt/pindown/lab4/rundebug.sh) or the extraction of the test results doesn’t work (issue in /opt/pindown/lab4/pindown_config.txt). Which one is it?

If the problem is in rundebug.sh, on which line is there a problem according to the logs (rundebug.log, pilot/run/z80.*.log)? If the problem on the other hand is in pindown_config.txt the problem has to do with the extraction of test names. Use a unix grep-command with the same path and keywords as extract -type "testname" is set to in the pindown config file and run this grep-command from the runarea reported in the email.

Can you figure it out?

TASK
  • Try to find the problem and fix it.

  • Kick off lab4 again in TestHub to check.

After you fixed the problem the run should come back clean and report a bug. That’s it.

Congratulations on finishing the PinDown setup labs!!

lab4 regression: PinDown has found 1 bug (committed by Christian) - Full Report

PinDown clean run
PinDown clean run part 2