This user guide explains the PinDown Bug Reports. What does the different types of bug reports mean? What is the confidence level in each type of bug report?
There is a single bad commit. When unrolling it the test passes again (validated).
Multiple commits had to be unrolled in order to make the test pass again (validated). This means there is at least one bad commit amongst the list of commits that had to be unrolled, but PinDown cannot say which one.
If a test is not failing due to a regression bug (i.e. a recent mistake) then it is marked as an always-failing-bug. This means that even if we try older revisions this test has never passed, at least not within the debug window specified by the set_earliest_revision command. When using constrained random tests this occurs for those seeds that creates a new test scenario that discover a bug that has always existed. It may also occur if there are problems with the IT environment or the PinDown setup.
PinDown was not able to make the test pass again by unrolling any commit. The bug report points to the first commit that causes this test to fail, which is often the bad commit, but that is not certain. The issue introduced in this bad commit may have been fixed in a later revision.
Normally PinDown manages to identify a tipping point where a test that passed on one revision starts to fail on the next revision. However sometimes there are a lot of build failures or no results at all between the last passing revision and the earliest failing revision. This is called an extended tipping point. In this case the bug report shows all commmits from the last pass until the earliest failing revision, including the results for all revisins in between (build failure or no results). Most likely the bad commit is one of the list revisions, but that is not certain. Any problems may have been fixed in a later revision.
The user may setup PinDown to send progress reports before it has reached a conclusion. Progress reports can be setup to send a report just once in order to get the grouping of failure signatures ("buckets") or continuously during debug, once per iteration, in order to supervise the progress during debug all the way until the conclusion.
The test failed during the test phase but when PinDown re-run the same test and seed on the same revision it passed.
An open bug was not checked whether it is still open, due to IT or setup issues.