The results consist of both configurations and tests which are available in a summary file generated by the user. Each row contains the result from one test and its associated compilation results.

Note A summary file may contain just one row with one result or consist of a long list of test results. The only difference is whether you extract from a single summary file (logs/allresults.log) or multiple files (e.g. logs/result.*log). The definition of a summary file is that it is generated by the user and contains one result per row.

Example

All results are available in one summary file (allresults.log) generated by the user, including the compilation results.

logs/allresults.log
Test Results

config  build   time test  seed     result testtime
---------------------------------------------------
configA success 12.3 test1 22442244 PASSED 122.7
configA success 12.3 test2 12341234 FAILED 234.3
configA success 12.3 test3 45674567 PASSED 217.1
configB success 14.7 test4 84666431 PASSED 371.1
configB success 14.7 test5 27333434 FAILED 42.3
Note Extracting from a summary file has one downside compared to extracting directly from the test logs: the failure messages are not present. PinDown extracts the failure plus 5 lines above and 5 line below the message, which often contain interesting information. However not in the case of a summary file.

PinDown Extraction

Step 1. Generate Summary Report

Step 1 is to run the script that generates the test and build results summary report (see file pindown_config.txt below).

pindown_config.txt
group -name "test" -commands {

  // Step 1. Generate test report
  shell "/full/path/generateReports.sh";

  // Step 2. Extracting both build and test results from columns
  extract -type "configlabel" -path "logs/allresults.log" -keywords "success|failure;column_delimiter= ;$1";
  extract -type "buildpass" -path "logs/allresults.log" -keywords "success";
  extract -type "buildfail" -path "logs/allresults.log" -keywords "failure";
  extract -type "buildend" -path "logs/allresults.log" -keywords "success|failure;column_delimiter= ;$3";
  extract -type "testname" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$4";
  extract -type "testseed" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$5";
  extract -type "testpass" -path "logs/allresults.log" -keywords "PASSED";
  extract -type "testfail" -path "logs/allresults.log" -keywords "FAILED";
  extract -type "testend" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$7";

  // Step 3. Clean the data. Remove the filename and decimal values on the build and test times
  extract -type "replace" -label "configlabel" -text "[^:]*:" -with "";
  extract -type "replace" -label "buildend" -text "[^:]*:" -with "";
  extract -type "replace" -label "buildend" -text "\..*" -with "";
  extract -type "replace" -label "testname" -text "[^:]*:" -with "";
  extract -type "replace" -label "testseed" -text "[^:]*:" -with "";
  extract -type "replace" -label "testend" -text "[^:]*:" -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract Test and Build Results

In step 2 we extract the test results from the lines in the summary report that contain either "PASSED" or "FAILED" and correspondingly for the build result we extract the lines that contain either "success" or "failure". Depending on the result type we select the relevant column. The column delimiter is defined as one space. The column number 1 contains the configuration name, column no 3 contains the build end time, column no 4 contains the test name and column no 7 contains the test end time. The pass/fail criteria for test and build results is whether the keywords are anywhere on the line. We don’t check just a column for the pass/fail criteria.

Step 3. Clean Results

In step 3 the results are cleaned to remove the file names (each match returns filepath:content, but here we are only interested in the content). Also the build and test end times need to be cleaned to only contain whole seconds as decimals are not allowed.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Note PinDown reads the results continously while the tests are running. That is why it is necessary to generate the report in Step 1 so that each time PinDown wants to check the status there is a summary report to read from. If this is not done then PinDown will only be able to read the results after the entire test suite has completed and such summary file is available. If PinDown does not have knowledge about the status of running tests then it is not possible to use its mechanism to stop an ongoing run if enough test failures have been detected - a feature that saves time and computer resources.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_end" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%12" buildResult="%buildpass%pass" buildResultLine="logs/allresults.log:configA success 12.3 test3 45674567 PASSED 217.1">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%configA"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="logs/allresults.log:configA success 12.3 test1 22442244 PASSED 122.7" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/allresults.log:Test Results&lt;PinDownNL&gt;logs/allresults.log:&lt;PinDownNL&gt;logs/allresults.log:config  build   time test  seed     result testtime&lt;PinDownNL&gt;logs/allresults.log:---------------------------------------------------&lt;PinDownNL&gt;logs/allresults.log:configA success 12.3 test1 22442244 PASSED 122.7&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/allresults.log:configA success 12.3 test2 12341234 FAILED 234.3&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/allresults.log:configA success 12.3 test3 45674567 PASSED 217.1&lt;PinDownNL&gt;logs/allresults.log:configB success 14.7 test4 84666431 PASSED 371.1&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="logs/allresults.log:configA success 12.3 test3 45674567 PASSED 217.1" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="logs/allresults.log"/>
        </com.verifyter.pindown.BuildResult>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%14" buildResult="%buildpass%pass" buildResultLine="logs/allresults.log:configB success 14.7 test5 27333434 FAILED 42.3">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%configB"/>
            <com.verifyter.pindown.Test testEnd="%testend%371" testName="%testname%test4" testResult="%testpass%pass" testResultLine="logs/allresults.log:configB success 14.7 test4 84666431 PASSED 371.1" testSeed="%testseed%84666431"/>
            <com.verifyter.pindown.Test testEnd="%testend%42" testName="%testname%test5" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/allresults.log:---------------------------------------------------&lt;PinDownNL&gt;logs/allresults.log:configA success 12.3 test1 22442244 PASSED 122.7&lt;PinDownNL&gt;logs/allresults.log:configA success 12.3 test2 12341234 FAILED 234.3&lt;PinDownNL&gt;logs/allresults.log:configA success 12.3 test3 45674567 PASSED 217.1&lt;PinDownNL&gt;logs/allresults.log:configB success 14.7 test4 84666431 PASSED 371.1&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/allresults.log:configB success 14.7 test5 27333434 FAILED 42.3&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;" testSeed="%testseed%27333434"/>
            <com.verifyter.pindown.LogFile path="logs/allresults.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>