1. Test Environment
We use a simple regression environment built around an open source core to demonstrate PinDown debug capabilities. The core used here is the Y80e processor from OpenCores http://opencores.org/project,y80e .
PinDown follows a straightforward flow: Run tests (optional step), extract test results, group by error signature, then run debug if any failures were found. In the debug cycle PinDown will rerun select failing tests on combinations of past revisions until all tests pass again. The debug cycle is run on several past revisions in parallel.
Here is the flow chart:
The following scenarios each demonstrate a specific debug scenario. Before you explore debug, make sure the environment is up and running and shows all test passing.
Once the run has completed you should see an all pass email report from PinDown in your inbox.
2. Scenario 1. Debugging a single failure
The first scenario covers the simplest scenario. A single bug that makes a single test fail. At this point all tests are passing. That means you actually have to make the test fail by inserting the bug and committing the bug to the repository.
Your repository history might look similar to below. In this example test t1 will fail because of a bad commit in revision r8. PinDown is guaranteed to find the right commit. It won’t return revision r2 as a simple binary search would.
|SVN password in this lab is svnpassword.|
Once the run has completed you will find a bug report from PinDown in your inbox. You will find the most important information about this bug in the red box. A bug is a bad commit. You will find the revision number of this commit and the name of the committer. In this case it’s your user name, because you have committed this bug to the repository in the prior step. The commit message should show your commit message: "Bug No 1".
Another important line in this report is the Validated line. It says true if PinDown was able to make the test in the report pass again at top-of-tree. This is almost always the case. And if it says true you have proof that this bad commit really is the culprit.
For more details on how to read a bug report email see the Pindown Getting Started Guide.
3. Scenario 2. Debugging dependent bugs
PinDown can find bugs in any complex scenario. You might have a situation where two bugs depend on each other. This could be because they are part of an interface, but that’s not the only scenario bugs can depend on each other. What PinDown will do in a scenario like this is to make the test pass again by unrolling all dependent bugs or combinations thereof.
Your repository history might look similar to below. Here test t1 will fail because of a bad commit in revision r2 and another bad commit in revision r5. And both of these bad commits depend on each other here, by design. PinDown is not limited to two dependent bugs. This is only an example.
|The challenge here is that there are two bad commits, both of which have to be fixed in order to make the test pass again.|
Once PinDown has finished you will find a bug report in your inbox again. This time two bugs were found. Each bug has its own red box. The title of the first bug will say:
Bug No: 1 (new bug) - Shares test with bug no 2. Both bugs had to be unrolled to make the test pass again.
And the title of the second bug will say:
Bug No: 2 (new bug) - Shares test with bug no 1. Both bugs had to be unrolled to make the test pass again.
So the titles will explicitly tell you about the dependencies found.
As expected, you will find that the two bugs that depend on each other are "Bug No 1" and "Bug No 2".
|Whether these two commits are inseparable and together contains only one bug, or whether there are two bugs, one in each commit, that both affect the same test, does not matter for PinDown. PinDown shows the minimum number of commits that have to be unrolled to make the test passing again. Consequently it can handle any amount of dependencies or bugs.|
4. Scenario 3. Debugging overlaying failures
In the third scenario we have two bad commits that make the same test fail. However, the first bad commit has been fixed already by the time you run regression. That means there are overlapping error signatures, but only one bad commit is left at top-of-tree.
Your repository history might look similar to below. Here test t1 will fail because of a bad commit in revision r5. The commit in revision r2 has already been fixed in revision r7 before you run your regression. PinDown is guaranteed to find the right bad commit. A standard binary search would point at revision r2 as the bad commit, which is not correct.
|The challenge here is that there is no pass-fail transition when the bad commit was introduced as the test was already failing.|
Once PinDown has finished you will find a bug report again in your inbox. This time PinDown shows a single bug again. Notice that the bug found is "Bug No 2". Despite overlapping error signatures PinDown found the correct remaining bug.
This concludes the automatic debug lab. As you have seen, PinDown does not rely on pass-fail transitions. Scenario 3 is not possible to solve by looking at pass-fail transitions because the test was already failing when the bad bug was committed, and this bad commit is the only bug that has not been fixed.
PinDown can also handle dependencies between commits. In scenario 2 PinDown showed that there were two commits that both had issues and that both commits had to be unrolled before the test was passing (i.e validated). Whether these two commits are inseparable and together just contains one bug, or whether there are two bugs, one in each commit, that both affect the same test, does not matter for PinDown. PinDown shows the minimum number of commits that have to be unrolled to make the test passing again. Consequently it can handle any amount of dependencies or bugs.
The scenarios in this lab are kept at a minimum for practical reasons. To see how PinDown behaves in larger systems please check out the PinDown demo videos on Verifyter’s web page.