The results consist of both configurations and tests, both of which form part of the file names or file paths.

Each configuration has its own build results and a list of tests. Here we first extract the configuration names from the file paths, followed by the associated build results. Then we extract the test names from the file paths, mapped to each configuration, followed by the associated test results.

Example

Each test produces a log file whose name is the name of the test. Each compilation produces a log in a folder whose name is the name of the configuration.

testlogs/configA/test1.log
seed 22442244
Test Result: PASSED
CPU time 122.7 seconds
testlogs/configA/test2.log
seed 12341234
Compare mismatch 34( got) !=33 (expected)
Test Result: FAILED
CPU time 234.3 seconds
testlogs/configA/test3.log
seed 45674567
Test Result: PASSED
CPU time 217.1 seconds
testlogs/configB/test4.log
seed 84666431
Test Result: PASSED
CPU time 371.1 seconds
testlogs/configB/test5.log
seed 27333434
Unexpected value 22. Expected 23
Test Result: FAILED
CPU time 42.3 seconds
compilation/configA/compile.log
Compiling project
...
Done!
CPU time 12.3 seconds
compilation/configB/compile.log
Compiling project
...
Done!
CPU time 14.7 seconds

PinDown Extraction

Step 1. Extract Build Results

Step 1 is to extract the configuration name from each sub-folder of the compilation folder (see the file pindown_config.txt below). Then by referring to the configuration names ("%configlabel%") PinDown is able to associate each build result with the respective configuration.

pindown_config.txt
group -name "test" -commands {

  // Step 1. Extracting build results
  extract -type "configlabel" -source "filename" -path "compilation/.*" -keywords "";
  extract -type "buildpass" -path "%configlabel%/compile.log" -keywords "CPU time";
  extract -type "buildfail" -path "%configlabel%/compile.log" -keywords "Error|error|failed";
  extract -type "buildend" -path "%configlabel%/compile.log" -keywords "CPU time";

  // Step 2. Extract the test results - first the complete lines
  extract -type "replace" -label "configlabel" -text "compilation\/" -with "";
  extract -type "testname" -source "filename" -path "testlogs/%configlabel%/.*log" -keywords "";
  extract -type "testseed" -path "%testname%" -keywords "seed";
  extract -type "testpass" -path "%testname%" -keywords "PASSED";
  extract -type "testfail" -path "%testname%" -keywords "FAILED";
  extract -type "testend" -path "%testname%" -keywords "CPU time";

  // Step 3. Clean the test results
  extract -type "replace" -label "testname" -text "testlogs/config./" -with "";
  extract -type "replace" -label "testname" -text ".log" -with "";
  extract -type "replace" -label "testseed" -text ".*seed " -with "";
  extract -type "replace" -label "testend" -text ".*CPU time " -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
  extract -type "replace" -label "buildend" -text ".*CPU time " -with "";
  extract -type "replace" -label "buildend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract Test Results

In step 2 we extract the test names from test log names, but instead of writing the full path we refer to the configuration folder ("%configlabel%") as part of the path, which allows PinDown to understand which configuration each test is associated with. In a similar fashion we then subsequentially associate the test results with each test log file by referring to the testname label ("%testname%").

Step 3. Clean Results

In step 3 the results are cleaned up so that the test names are only the actual test names and not the complete file paths. The seeds are extracted from the content of the test log files and consequently contain the result in the format filepath:content where the file path shows which file the content was extracted from. In this case the file path and the "seed "-part of the content is removed in order to only keep the seed number. The test end times and build end times must be numbers without decimals. In this case we use the whole number of seconds that the test or compilation lasted.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message as they may contain interesting information. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_end" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%12" buildResult="%buildpass%pass" buildResultLine="compilation/configA/compile.log:CPU time 12.3 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%configA"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="testlogs/configA/test1.log:Test Result: PASSED" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;testlogs/configA/test2.log:seed 12341234&lt;PinDownNL&gt;testlogs/configA/test2.log:Compare mismatch 34( got) !=33 (expected)&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;testlogs/configA/test2.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;testlogs/configA/test2.log:CPU time 234.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="testlogs/configA/test3.log:Test Result: PASSED" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="compilation/configA/compile.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/configA/test1.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/configA/test2.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/configA/test3.log"/>
        </com.verifyter.pindown.BuildResult>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%14" buildResult="%buildpass%pass" buildResultLine="compilation/configB/compile.log:CPU time 14.7 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%configB"/>
            <com.verifyter.pindown.Test testEnd="%testend%371" testName="%testname%test4" testResult="%testpass%pass" testResultLine="testlogs/configB/test4.log:Test Result: PASSED" testSeed="%testseed%84666431"/>
            <com.verifyter.pindown.Test testEnd="%testend%42" testName="%testname%test5" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;testlogs/configB/test5.log:seed 27333434&lt;PinDownNL&gt;testlogs/configB/test5.log:Unexpected value 22. Expected 23&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;testlogs/configB/test5.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;testlogs/configB/test5.log:CPU time 42.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%27333434"/>
            <com.verifyter.pindown.LogFile path="compilation/configB/compile.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/configB/test4.log"/>
            <com.verifyter.pindown.LogFile path="testlogs/configB/test5.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>