The test results are available in a summary file generated by the user. Each row contains the result from one test.

There are no configurations in this scenario just one build used by all tests.

Note A summary file can contain just one row with one test result or consist of a long list of test results. The only difference is whether you extract from a single summary file (logs/allresults.log) or multiple files (logs/result.*log). The definition of a summary file is that it is generated by the user and contains one result per row.

Example

All test results are available in one summary file (allresults.log) generated by the user. The compilation result is present in a different file (logs/compile.log).

logs/allresults.log
Test Results
------------
test1 22442244 PASSED 122.7
test2 12341234 FAILED 234.3
test3 45674567 PASSED 217.1
logs/compile.log
Compiling project
...
Done!
CPU time 50.7 seconds
Note Extracting from a summary file has one downside compared to extracting directly from the test logs: the failure message is not present. PinDown extracts the failure message plus 5 lines above and 5 lines below, which often contain interesting information, but in the case of a summary file.

PinDown Extraction

Step 1. Generate Summary Report

Step 1 is to run the script that generates the test results summary report (see the pindown_config.txt below).

pindown_config.txt
group -name "test" -commands {

  // Step 1. Generate test report
  shell "/full/path/generateReport.sh";

  // Step 2. Extracting build result
  extract -type "buildpass" -path "logs/compile.log" -keywords "CPU time";
  extract -type "buildfail" -path "logs/compile.log" -keywords "Error|error|failed";

  // Step 3. Extract the test results from columns
  extract -type "testname" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$1";
  extract -type "testseed" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$2";
  extract -type "testpass" -path "logs/allresults.log" -keywords "PASSED";
  extract -type "testfail" -path "logs/allresults.log" -keywords "FAILED";
  extract -type "testend" -path "logs/allresults.log" -keywords "PASSED|FAILED;column_delimiter= ;$4";

  // Step 4. Clean the data. Remove the filename and decimal values on the build and test times
  extract -type "replace" -label "testname" -text "[^:]*:" -with "";
  extract -type "replace" -label "testseed" -text "[^:]*:" -with "";
  extract -type "replace" -label "testend" -text "[^:]*:" -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract the Compilation Result

Step 2 is to extract the compilation result in logs/compile.log.

Step 3. Extract Test Results

In step 3 we extract each line from the summary report that contains one of the keywords "PASSED" and "FAILED". Depending on the result type we select the relevant column where the column delimiter is defined as one space. The column number 1 contains the test name, column number 2 contains the seed and column number 4 contains the test end time. The fail criteria is if the entire line contains the keyword "FAILED" anywhere (we don’t look at just column no 3). Correspondingly the pass criteria is if the line contains the keyword "PASSED".

Step 4. Clean Results

In step 4 the results are cleaned to remove the file names (each match returns filepath:content, but here we are only interested in the content part). Also the test end times need to be cleaned to only contain whole seconds as decimals are not allowed.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Note PinDown reads the results continously while the tests are running. That is why it is necessary to generate the report in Step 1 so that each time PinDown wants to check the status there is a summary report to read from. If this is not done then PinDown will only be able to read the results after the entire test suite has completed and such summary file is available. If PinDown does not have knowledge about the status of running tests then it is not possible to use its mechanism to stop an ongoing but incomplete run if enough test failures have already been detected. This is a feature that saves time and computer resources.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildResult="%buildpass%pass" buildResultLine="logs/compile.log:CPU time 50.7 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%build"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="logs/allresults.log:test1 22442244 PASSED 122.7" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/allresults.log:Test Results&lt;PinDownNL&gt;logs/allresults.log:------------&lt;PinDownNL&gt;logs/allresults.log:test1 22442244 PASSED 122.7&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/allresults.log:test2 12341234 FAILED 234.3&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/allresults.log:test3 45674567 PASSED 217.1&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="logs/allresults.log:test3 45674567 PASSED 217.1" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="logs/allresults.log"/>
            <com.verifyter.pindown.LogFile path="logs/compile.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>