1. Launching PinDown from Testhub

PinDown can be controlled and monitored from an application in your browser, referred to as Testhub. In a default installation on your local machine, Testhub is accessed at http://localhost:8080/PinDown.

PinDown testhub
Figure 1. PinDown Testhub

Testhub has multiple tabs. The main tab to control PinDown regression runs is the Test tab. Regressions can be launched either manually, simply by clicking on the Start button, or by a cron job. The cron scheduler is active when the clock symbol next to the Start button is not grayed out.

Note If a regression is in progress it will not be disrupted by an attempt of the cron scheduler to launch another job. The same is true when an attempt is made to manually launch a regression with the Start button, while the previous regression is still in progress.

2. How to schedule a CRON job

Next to the Start button on Testhub’s Test tab, click on the link labelled with Schedule to bring up the cron menu.

Finding the CRON menu
Figure 2. Finding the CRON menu
CRON menu
Figure 3. CRON menu

The cron menu allows to add, edit or delete cron jobs. Add or select a project to edit. This will bring up the edit dialog. If unsure about cron syntax, get help from the wikipedia on Cron.

For instance, 8 20 * * * launches the project every day at 8:08 PM.

CRON editor
Figure 4. CRON editor

3. Launching PinDown from command line

Start PinDown by calling curlProjectStart.sh pilot_s from the PinDown folder. In the demo installation the PinDown folder is located at, /home/chrisg/projects/Demos/pilot_pd. This directory was selected as the main PinDown directory (folder) during installation. All project directories are sub-directories of this main directory. The project directories contain all project specific configuration files, log files and regression results.

$PINDOWN_FOLDER:$ curlProjectStart.sh pilot_s

4. Interpreting Results

PinDown communicates results through two channels: By email and by displaying a summary in Testhub.

As soon as PinDown diagnosis is conclusive, a bug report email will be sent out. This happens while diagnosis is still going on. After a bug email has been sent out, PinDown will either chase down another bug, or wrap up diagnosis and send a full report on all currently open bugs. Information on currently open bugs comes directly from the database.

There are two kinds of bugs: Build bugs and test case bugs. A build bug occurs when something went wrong during testbench compilation and a test bug occurs when a test run completed successfully, but returned a fail status as result.

Build bugs are recognized by the keyword Build Failure on the Test: line of the report, and test bugs are recognized by an actual test case name on the Test: line, see example below. The associated testbench is found on the Build: line, for both types of bugs.

Example 1. Test bug report
Test: bit_ops (in total 1 failures)
Build: build_y80e
Example 2. Build bug report
Test: Build Failure (in total 1 failures)
Build: build_y80e

5. Complete example of PinDown bug report email

PinDown will send a bug report email to the committer as soon as a bug is found during debug.

Bug report email
Bug report email
Figure 5. PinDown bug report email

5.1. Report top box

The box in the top section of the report has the most essential information.

Example 3. Report top box
Test: dat_mov
Build: build_y80e
Error:
pilot/run/z80.dat_mov.log:
$monitor at time=1428380: MEM_ADDR='h00c1 MEM_DATA_OUT='h22
$monitor at time=1428580: MEM_ADDR='h0000 MEM_DATA_OUT='h00
$monitor at time=1429180: MEM_ADDR='h00c1 MEM_DATA_OUT='h22
$monitor at time=1429380: MEM_ADDR='h00c2 MEM_DATA_OUT='h22
$monitor at time=1429580: MEM_ADDR='h00c3 MEM_DATA_OUT='h22
pilot-test-result ---------- FAILED ----------
Error at failing revision: pilot/run/z80.dat_mov.log:pilot-test-result ---------- FAILED ----------
Validated: true (PinDown made the failing test pass again by only removing the bad commit. This was done locally, nothing was committed)
Committer: Christian
Commit Message:
6368. Enter bug
Committed Files:
svn://svn.verifyter.com/Demos/pilot/branches/story2/rtl/datapath.v
Committed Changes (shown for small commits):
/* */
/* address alu */
/* */
/*****************************************************************************************/
assign bsign_ext = {addb_in[7], addb_in[7], addb_in[7], addb_in[7],
addb_in[70], addb_in[7], addb_in[7], addb_in[7]};
always @ (aluop_reg or adda_in or addb_in or bsign_ext) begin
case (aluop_reg)
`ALUOP_ADS: adda_out = adda_in + {bsign_ext[15:8], addb_in[7:0]};
`ALUOP_BADD,
Log:
/home/chrisg/work/checkoutarea/pilot_s/pilot/run/z80.dat_mov.log
Other Failures In Same Bug:
bit_ops on build build_y80e
alu_ops on build build_y80e

This section shows at a glance:

  • Bug No: Bugs are numbered in increasing order. The bug number is required if a bug is to be reproduced by PinDown.

  • Test: Shows which type of failure was encountered, build or test case failure. Shows the testcase name in the latter case. The number of failing pilot testcases associated with this bug is shown in parathesis.

  • Build: Associated build configuration.

  • Error: The encountered error signature at head

  • Error at failing revision: The error signature at the failing revision

  • Validated: If true, then the failing test was fixed by removing the bad commit. If false, the failing test could not be fixed.

  • Committer: Committer’s user name

  • Commit Message: Perforce commit message at the tipping point showing the all important revision number, here 6368

  • Committed Files: List of files committed at that revision

  • Log: Path to the extracted failing log file in the checkoutarea (temporary) and Path to the extracted failing log file in the testarea (permanent)

Note The checkoutarea is a sub-directory in the project directory. This area is temporary and will be regenerated with every regression. To access a copy of the log file when the checkoutarea is no longer available, go to the runarea directory listed in the second path, marked “(copied)”.

Of all the information in the top section, the revision number is probably the most important. This revision is the revision of the commit that broke the regression, or simply the bug. This number can be used to get detailed information about the commit, e.g. svn log -r3232 or svn diff -r3232. All other information in the report email provides additional information for debugging, but is not all that essential as the information in the top box. Below is a brief description of each section, as labeled above:

5.2. Failure Signatures

Failure Signatures
1 distinct error message is occurring in 1 test failure:
Error 1 (from failures 1):
pilot-test-result ---------- FAILED ----------
Build: build_y80e
Failure 1: dat_mov
Error: pilot-test-result ---------- FAILED ----------
Log: /home/chrisg/projects/Demos/pilot_pd/pilot_s/checkoutarea/test/pilot/run/z80.dat_mov.log

Failure signatures show the actual line that was extracted by PinDown from the log file of a failing testcase or build. In above example one bug was found and one failing testcase is associated with this bug. Each testcase is listed with its name, including the seed (if applicable), the one line error signature and the location of the log file.

5.3. Reproduce Bugs

Reproduce Bugs
/home/chrisg/projects/Demos/pilot_pd/pindown
-config /home/chrisg/projects/Demos/pilot_pd/pilot_s/pindown_config.txt
-reproduce X (where X is the bug no). For more information go to PinDown TestHub

Bugs can be reproduced either from the testhub, or on the command line. The reproduce bug section in the report email shows how to do it from the command line. Two options are available from the command line:

  1. Local checkout driven by user.

  2. Checkout in PinDown project directory under PinDown control.

For option one, the report email provides the label at which regressions failed. This label is ToT at the time when the regression was originally launched. Next step is a checkout by the user into a local user directory at this label, or alternatively at ToT. Then the usual build and run steps are executed, just as in any other regular test run. As testlist the pilot testcase with seed and runtime options need to be provided. The runtime options can be found in the original testlist for the regression run.

For option two, simply execute the specified command as specified. Contrary to the first approach, this option is completely under PinDown control and the results can be found in the PinDown project directory’s runarea, e.g. /home/chrisg/projects/Demos/pilot_pd/pilot_s/runarea/.

Note When PinDown is launched from the command line, it will not be stopped if a regression is already in progress. You need to make sure that no regression is in progress when launching from the command line.

5.4. Test Results

Test Results
Diagnosis completed.
5 tests run (4 pass, 1 fail -> 1 new, 0 closed). 1 builds created (1 pass, 0 fail -> 0 new, 0 closed). Run 1/15 min
Checkout area: /home/chrisg/projects/Demos/pilot_pd/pilot_s/checkoutarea/test
Test area: /home/chrisg/projects/Demos/pilot_pd/pilot_s/runarea/test/run139

This section is a snapshot of the real-time regression status for builds and testcases. In this case, we have only one build, which was successfully compiled, and 5 test cases, with one test case that failed. This example test case ran very fast and only required 1 min out the 15 min allowed before timeout.

5.5. Recent Commits

Recent Commits
3261. set dat_mov CMP_ERR_L (chrisg, Oct 28 2:41PM)
3260. backout build defines (chrisg, Oct 28 2:37PM)
3258. reconfigured build defines (chrisg, Oct 28 7:20AM)
3257. disable bit_ops CMP_ERR_L (chrisg, Oct 28 7:07AM)
3246. new test feature (chrisg, Oct 25 7:58AM)
3245. pilot reformat (chrisg, Oct 25 7:56AM)
3244. update to run script (chrisg, Oct 25 7:56AM)
3232. introduced bypass feature (chrisg, Oct 25 7:14AM)
3191. forcebug needs to come last (chrisg, Oct 16 5:25PM)
3189. fix top levl (chrisg, Oct 16 4:42PM)
(none of these commits have been tested before)

The recent commits section is meant as debugging help. It shows the most recent commits, including commit messages, starting from ToT.

5.6. Tested Revisions

Tested Revisions
The following repository was tested at revision 3261 (last update 3261 was made Oct 28 2013 14:41):
svn://svn.verifyter.com/Demos/pilot/trunk (last update 3261)

This section is a snapshot of the subversion repository as seen by PinDown. It shows the repositories with the additional information on what repository is kept at a specific label (if applicable) and what repositories are ToT. In the example only 1 repositories is ToT. In this repository PinDown will step back in revision history during automatic debug.

6. Common Tasks

6.1. Selecting recipients for the bug report emails

By default PinDown sends bug report emails to the committer of the bug, and to the owner of the project. Often additional recipients are desired, e.g. the project manager. To serve as example, here are the settings for project pilot_s:

set_mail -host "localhost" -port "25" -from "PinDown: <chrisg@verifyter.com>"
-encryption "none" -append "@verifyter.com" -user "chrisg" -password "ENC:Fwwo9EuEvxU"
-topass "chrisg@verifyter.com" -tofail "chrisg@verifyter.com" -tobadcommit "chrisg@verifyter.com"
-owner "chrisg@verifyter.com" -sendtocommitters "true" -sendonnew "true";

The options allow to send emails to the listed recipients on certain events:

  • -topass: all testcases have passed

  • -tofail: any bug, with revision # or ‘always failing’ type

  • -tobadcommit: only bugs with revision #

  • -owner: every email, including status emails, for monitoring purposes

6.2. Adjusting PinDown’s RCS history window

PinDown goes back in the revision history to find tipping points. How far it goes back is adjustable by the user. The adjustment is also done in pindown_config.txt, located in the project directory. The parameter to look for is set_earliest_revision. Below are a few examples for alternative settings, you would of course only use one at a time:

set_earliest_revision –weeks “2”;
set_earliest_revision –days “4”;
set_earliest_revision –revision “3232";
set_earliest_revision –revision “PILOT_S 5.0.16”;

6.3. Adjusting PinDown’s parallel jobs

During diagnosis PinDown runs multiple regressions in parallel. By default the number of parallel regressions is set to 10. This setting also determines the number of required PinDown runtime licenses. Adjustments can be made in pindown_config.txt:

set_test_bandwidth 10;

6.4. Assigning a different user

If you want to run a project under a different user name, you can bind the project to a different user at any time. It is not necessary to create a new project for that. Any existing project can be re-assigned.

To assign a different user, go to the Settings tab in testhub, under Assign Unix credentials to this project, click to open the credentials pop-up box, enter the UNIX user name and password of the assignee. Make sure that the proper project is selected, visible in the upper right corner of testhub. In the screenshot below, the project pilot_q is assigned to the new user pilot.

Assigning Unix credentials
Figure 6. Assigning Unix credentials

Then click on Bind to Project. A green checkmark will show up inside the Bind to Project button, if successful. At this point you are ready to switch back to the Test tab. Simply start your project as usual with the Start button and it will now run under the newly assigned user name.

Successful bind to project
Figure 7. Successful bind to project

7. Cloning a Project

A clone is an exact copy of an existing project, just with a different name. The clone is a great starting point for a new project, if the flow of the new project is similar to an existing one. To clone a project, from the Settings tab in testhub select New Project.

Create new project
Figure 8. Create new project

The New Project dialog box will come up. Fill in the name for your new project, in the example pilot_q, and select which existing project will be used as a template, here pilot_s (Creating a blank project would require to fill in configuration files from scratch). Then click on Copy Project.

New project dialog box
Figure 9. New project dialog box

Then select your new project, i.e. pilot_q, from the drop down menu in the upper right corner. If you can’t see your new project, reload the page in the browser. And select the Test tab.

Select new project
Figure 10. Select new project

At this point, you might want to make some adjustments to your configuration. So far the new project is an exact copy of the original and will run e.g. the exact same testlist as the original. It will also run under the same user as the original. If you want to change the user for this new project, see section Assigning a different user under Common Tasks. Once you’re done with any modifications, launch your new project by clicking on the Start button.

Launch new project
Figure 11. Launch new project

You can easily see that a regression is in progress by the rotating circle inside the Start button, the spinner, and the advancing progress report in the Test tab. You can now periodically check the progress in testhub, or simply wait for PinDown to send out a report email when the regression found a bug. Then see chapter on Interpreting results on how to read the report email.

8. Glossary

Tipping Point (TP)

The tipping point describes a pass to fail transistion of a build or testcase. The bug found is the revision number of the failing build or testcase.

Top of Tree (ToT)

Top of tree refers to the most recent changelist of a repository.