This user guide explains the PinDown Bug Reports. What does the different types of bug reports mean? What is the confidence level in each type of bug report?
There is a single bad commit. When unrolling it the test passes again (validated).
Multiple commits had to be unrolled in order to make the test pass again (validated). This means there is at least one bad commit amongst the list of commits that had to be unrolled, but PinDown cannot say which one.
If a test is not failing due to a regression bug (i.e. a recent mistake) then it is marked as an always-failing-bug. This means that even if we try older revisions this test has never passed, at least not within the debug window specified by the set_earliest_revision command. When using constrained random tests this occurs for those seeds that creates a new test scenario that discover a bug that has always existed. It may also occur if there are problems with the IT environment or the PinDown setup.
Same as above, but with an extra helpful section on historical causes. This section lists which file updates that are normally linked to this type of failure message. Sometimes it is useful to understand what type of sub-module that is behind certain types of failure messages in order to quickly know where to fix the issue.
PinDown was not able to make the test pass again by unrolling any commit. The bug report points to the first commit that causes this test to fail, which is often the bad commit, but that is not certain. The issue introduced in this bad commit may have been fixed in a later revision.
Normally PinDown manages to identify a tipping point where a test that passed on one revision starts to fail on the next revision. However sometimes there are a lot of build failures or no results at all between the last passing revision and the earliest failing revision. This is called an extended tipping point. In this case the bug report shows all commmits from the last pass until the earliest failing revision, including the results for all revisins in between (build failure or no results). Most likely the bad commit is one of the list revisions, but that is not certain. Any problems may have been fixed in a later revision.
The user may setup PinDown to send progress reports before it has reached a conclusion. Progress reports can be setup to send a report just once in order to get the grouping of failure signatures ("buckets") or continuously during debug, once per iteration, in order to supervise the progress during debug all the way until the conclusion.
Same as above, but with an extra helpful section on historical causes. This section lists which file updates that are normally linked to this type of failure message. Sometimes it is useful to understand what type of sub-module that is behind certain types of failure messages in order to quickly know where to fix the issue. As this is a progress report and not a final report we don’t know yet if the final report will point to any updates to the listed files or if in this case it is something new and different that is the cause of the problem.
The test failed during the test phase but when PinDown re-run the same test and seed on the same revision it passed.
The test failed during the test phase but when PinDown re-run the same test and seed on the same revision it did not get any results at all. Farm is full or flow is broken.
An open bug was not checked whether it is still open, due to IT or setup issues.
Two good commits fail after being merged together