The results consist of both configurations and tests. Both the configuration names and the test names are found inside log files together with the build and test results. It is not possible to get the configuration or test names from the file paths or file names.

Example

The test results are available in files whose names do not give any clue about the test name or configuration name. There is one log per test and this log contains both the test names and the configuration names as well as the test results.

The build results associated with each configuration are available in the compilation logs. The compilation of the different configurations are done in sequence and the output is stored to the same compilation log for all configurations (compilation/compile.log).

testlogs/results_a.log
Command: run -test test1
Build: config_A
seed 22442244
Test Result: PASSED
CPU time 122.7 seconds
testlogs/results_b.log
Command: run -test test2
Build: config_A
seed 12341234
Compare mismatch 34( got) !=33 (expected)
Test Result: FAILED
CPU time 234.3 seconds
testlogs/results_c.log
Command: run -test test3
Build: config_A
seed 45674567
Test Result: PASSED
CPU time 217.1 seconds
testlogs/results_d.log
Command: run -test test4
Build: config_B
seed 31576784
Test not run
Test Result: FAILED
CPU time 10.9 seconds
testlogs/results_e.log
Command: run -test test5
Build: config_B
seed 89516342
Test not run
Test Result: FAILED
CPU time 9.9 seconds
compilation/compile.log
Build config_A
Compiling project
...
Done!
CPU time 12.3 seconds
Build config_B
Compiling project
Syntax Error: iff <- unknown command
...
CPU time 14.7 seconds

PinDown Extraction

Step 1. Extract Configuration Names and Test Results

Step 1 consists of extracting the configuration name (configlabel) from inside each test log (see the file pindown_config.txt below). The result contains filepath:content, i.e. both the file name and the content that matches the keyword in this search. In order to associate the test names and test results together with the configuration names we use the file, as we know that the content of each file contains information for one test. We do this by cleaning away the content so we only have the file path left to the test log. The label "configlabel" is now set to a list of all test logs. Then we extract the test names and test results by referring to this label ("%configlabel%"). By doing so PinDown understands how to group the extracted data.

pindown_config.txt
group -name "test" -commands {

  // Step 1. Extract the config name and then use the file it was extracted from when
  // extracting the test results
  extract -type "configlabel" -path "testlogs/results.*log" -keywords "Build";
  extract -type "replace" -label "configlabel" -text ":.*" -with "";
  extract -type "testname" -path "%configlabel%" -keywords "run \-test";
  extract -type "testseed" -path "%configlabel%" -keywords "seed";
  extract -type "testpass" -path "%configlabel%" -keywords "PASSED";
  extract -type "testfail" -path "%configlabel%" -keywords "FAILED";
  extract -type "testend" -path "%configlabel%" -keywords "CPU time";

  // Step 2. Restore the config name to its original value and then select actual config name
  extract -type "restore" -label "configlabel" -containing "";
  extract -type "replace" -label "configlabel" -text ".*Build: " -with "";

  // Step 3. Extract the compilation results
  extract -type "configlabel" -path "compilation/compile.log" -keywords "Build";
  extract -type "buildpass" -path "compilation/compile.log" -keywords "CPU time";
  extract -type "buildfail" -path "compilation/compile.log" -keywords "Error|error|failed";
  extract -type "buildend" -path "compilation/compile.log" -keywords "CPU time";
  extract -type "replace" -label "configlabel" -text ".*Build " -with "";

  // Step 4. Merge the compilation and test results belonging to the same build
  extract -type "merge" -label "configlabel" -containing "";

  // Step 5. Clean the test results
  extract -type "replace" -label "testname" -text ".*run -test " -with "";
  extract -type "replace" -label "testseed" -text ".*seed " -with "";
  extract -type "replace" -label "testend" -text ".*CPU time " -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
  extract -type "replace" -label "buildend" -text ".*CPU time " -with "";
  extract -type "replace" -label "buildend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Set the Configuration Names to the Correct Values

In step 2 we set the label "configlabel" to the correct names. So far we used it to group the results in each test log together, but now that this has been done we need to set the label "configlabel" to the actual names of the configurations. We do that first by restoring "configlabel" back to the original value, i.e filepath:content. This time we remove the file path instead, and also the "Build: "-part of the content in order to set the label "configlabel" to the actual configuration names (i.e. config_A, config_B). OK, so now we have good configuration names associated with the complete test results. The only thing lacking is the build results.

Step 3. Extract Build Results

In step 3 we extract the build results, including the configuration names, from the file "compilation/compile.log". Then we clean up the names of the configurations to be identical to the ones we got in step 2.

Step 4. Merge Build and Test Results

In step 4 we merge the compilation results with the test results. The test results (step 1-2) contain all test results plus the name of the configurations, but no build results. The compilation results (step 3) contain the name of the configurations and the associated build results but no test results. So in this step we merge all that so we get all results grouped nicely together by merging results for the same configuration together.

Step 5. Clean Results

In step 5 the results are cleaned up. The test names are extracted from the content of a file and consequently it contains the result in the format filepath:content where the file path shows which file the content was extracted from. In this case the file paths and the "run -test "-parts of the content are removed in order to only keep the actual test names. The same is done for the seed numbers. The test end times and build end times must be numbers without decimals. In this case we take the whole number of seconds that the test or compilation lasted.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message as they may contain interesting information. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_restore" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_end" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_merge" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%12" buildResult="%buildpass%pass" buildResultLine="compilation/compile.log:CPU time 12.3 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%config_A"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="testlogs/results_a.log:Test Result: PASSED" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;testlogs/results_b.log:Command: run -test test2&lt;PinDownNL&gt;testlogs/results_b.log:Build: config_A&lt;PinDownNL&gt;testlogs/results_b.log:seed 12341234&lt;PinDownNL&gt;testlogs/results_b.log:Compare mismatch 34( got) !=33 (expected)&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;testlogs/results_b.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;testlogs/results_b.log:CPU time 234.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="testlogs/results_c.log:Test Result: PASSED" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="compilation/compile.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/results_a.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/results_b.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/results_c.log"/>
        </com.verifyter.pindown.BuildResult>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%14" buildResult="%buildfail%fail" buildResultLine="&lt;PinDownContextStart&gt;compilation/compile.log:...&lt;PinDownNL&gt;compilation/compile.log:Done!&lt;PinDownNL&gt;compilation/compile.log:CPU time 12.3 seconds&lt;PinDownNL&gt;compilation/compile.log:Build config_B&lt;PinDownNL&gt;compilation/compile.log:Compiling project&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;compilation/compile.log:Syntax Error: iff &lt;- unknown command&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;compilation/compile.log:...&lt;PinDownNL&gt;compilation/compile.log:CPU time 14.7 seconds&lt;PinDownNL&gt;">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%config_B"/>
            <com.verifyter.pindown.Test testEnd="%testend%10" testName="%testname%test4" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;testlogs/results_d.log:Command: run -test test4&lt;PinDownNL&gt;testlogs/results_d.log:Build: config_B&lt;PinDownNL&gt;testlogs/results_d.log:seed 31576784&lt;PinDownNL&gt;testlogs/results_d.log:Test not run&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;testlogs/results_d.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;testlogs/results_d.log:CPU time 10.9 seconds&lt;PinDownNL&gt;" testSeed="%testseed%31576784"/>
            <com.verifyter.pindown.Test testEnd="%testend%9" testName="%testname%test5" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;testlogs/results_e.log:Command: run -test test5&lt;PinDownNL&gt;testlogs/results_e.log:Build: config_B&lt;PinDownNL&gt;testlogs/results_e.log:seed 89516342&lt;PinDownNL&gt;testlogs/results_e.log:Test not run&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;testlogs/results_e.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;testlogs/results_e.log:CPU time 9.9 seconds&lt;PinDownNL&gt;" testSeed="%testseed%89516342"/>
            <com.verifyter.pindown.LogFile path="compilation/compile.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/results_d.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/results_e.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>