An index file contains a list of result logs from which the results should be extracted. The index file contains both configurations names and test names and paths to their respective results log files. It is not possible to find these names inside the log files.

Example

All logs are listed in one index file (report_files.txt). The names of the tests and configurations are only available in the index file, not in the logs themselves. In this example each log is added to the index file as soon as the compilation or test has completed. This means the order is chronological and does not say anything about which configuration each test belongs to.

logs/report_files.txt
Build    Test    File
------------------------------------------
config_A compile file_170504_1223.log
config_B compile file_170504_1225.log
config_A test1   file_170504_1227.log
config_B test4   file_170504_1229.log
config_A test2   file_170504_1245.log
config_B test5   file_170504_1247.log
config_A test3   file_170504_1248.log

Log folder: logs
logs/file_170504_1223.log
Compiling project
...
Done!
CPU time 12.3 seconds
logs/file_170504_1225.log
Compiling project
...
Done!
CPU time 14.7 seconds
logs/file_170504_1227.log
seed 22442244
Test Result: PASSED
CPU time 122.7 seconds
logs/file_170504_1227.log
seed 84666431
Test Result: PASSED
CPU time 371.1 seconds
logs/file_170504_1245.log
seed 12341234
Compare mismatch 34( got) !=33 (expected)
Test Result: FAILED
CPU time 234.3 seconds
logs/file_170504_1247.log
seed 27333434
Unexpected value 22. Expected 23
Test Result: FAILED
CPU time 42.3 seconds
logs/file_170504_1248.log
seed 45674567
Test Result: PASSED
CPU time 217.1 seconds

PinDown Extraction

Step 1. Extract the Log Folder

Step 1 is to extract the folder where all logs are stored (see the file pindown_config.txt below). We look for the keyword "Log folder" and store the matching line in a list, which is the only extraction type that is not directly associated with a build result or a test result. The type is a list of variables, but in this case we only set it to one value.

pindown_config.txt
group -name "test" -commands {

  // Step 1. Extract the logs folder
  extract -type "list" -path "logs/report_files.txt" -keywords "Log folder";
  extract -type "replace" -label "list" -text ".*Log folder: " -with "";

  // Step 2. Extract the configuration names and compilation results
  extract -type "configlabel" -path "logs/report_files.txt" -keywords " compile";
  extract -type "replace" -label "configlabel" -text ".*:" -with "";
  extract -type "replace" -label "configlabel" -text ".*file_" -with "%list%/file_";
  extract -type "buildpass" -path "%configlabel%" -keywords "CPU time";
  extract -type "buildfail" -path "%configlabel%" -keywords "Error|error|failed";
  extract -type "buildend" -path "%configlabel%" -keywords "CPU time";
  extract -type "restore" -label "configlabel" -containing "";
  extract -type "replace" -label "configlabel" -text " .*" -with "";
  extract -type "replace" -label "configlabel" -text ".*:" -with "";

  // Step 3. Extract the test results per configuration
  extract -type "testname" -path "logs/report_files.txt" -keywords "%configlabel%.*test";
  extract -type "replace" -label "testname" -text ".*file_" -with "%list%/file_";
  extract -type "testseed" -path "%testname%" -keywords "seed";
  extract -type "testpass" -path "%testname%" -keywords "PASSED";
  extract -type "testfail" -path "%testname%" -keywords "FAILED";
  extract -type "testend" -path "%testname%" -keywords "CPU time";
  extract -type "restore" -label "testname" -containing "";
  extract -type "replace" -label "testname" -text ".* test" -with "test";
  extract -type "replace" -label "testname" -text " .*" -with "";

  // Step 4. Clean the test results
  extract -type "replace" -label "testseed" -text ".*seed " -with "";
  extract -type "replace" -label "testend" -text ".*CPU time " -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
  extract -type "replace" -label "buildend" -text ".*CPU time " -with "";
  extract -type "replace" -label "buildend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract Build Results

In step 2 we first extract the configuration names from the index file called "report_files.txt" by looking for rows containing the keyword "compile". Then we remove all text on each row except the compilation file name. We add the log folder which we extracted in step 1 in order to have a complete path to each log file. Now the label "configlabel" is set the full paths to the compilation logs. While referring to this label ("%configlabel%") we extract the build pass/fail and the build end time from these compilation logs. This label referral is what makes PinDown understand how to which configuration each build result belongs.

After this has been done we restore the label "configlabel" to its original values and then remove all text on each row except the clean configuration names. We now have good clean configuration names grouped correctly with the associated build results.

Step 3. Extract Test Results

In step 3 we extract the test names from the "report_files.txt" file, by setting the keyword to look for both the label "%configlabel%" and the keyword "test". This will pick up the tests and the label referral will let PinDown understand which configuration each test belongs to. Then we add the log folder to the test log paths in order to get the full paths to the test logs. Referring to the test name label ("%testname%), which contains all test logs, we extract the test pass/fail results plus the test end times. Again, it is the label referral which makes PinDown understand which test name each set of test results belong to.

The last thing we do is to restore the label "testname" to its original values and then instead of the files clean out everything except the test names. We now have good clean test names grouped correctly with its test results.

Step 4. Clean Results

In step 4 we clean the results so that the test seeds only consist of a number and the build and test end times only contain whole seconds as decimals are not allowed.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_list" label="list"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="list"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_end" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_restore" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_restore" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="buildend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%12" buildResult="%buildpass%pass" buildResultLine="logs/file_170504_1223.log:CPU time 12.3 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%config_A"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="logs/file_170504_1227.log:Test Result: PASSED" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/file_170504_1245.log:seed 12341234&lt;PinDownNL&gt;logs/file_170504_1245.log:Compare mismatch 34( got) !=33 (expected)&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/file_170504_1245.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/file_170504_1245.log:CPU time 234.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="logs/file_170504_1248.log:Test Result: PASSED" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1223.log"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1227.log"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1245.log"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1248.log"/>
        </com.verifyter.pindown.BuildResult>
        <com.verifyter.pindown.BuildResult buildEnd="%buildend%14" buildResult="%buildpass%pass" buildResultLine="logs/file_170504_1225.log:CPU time 14.7 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%config_B"/>
            <com.verifyter.pindown.Test testEnd="%testend%371" testName="%testname%test4" testResult="%testpass%pass" testResultLine="logs/file_170504_1229.log:Test Result: PASSED" testSeed="%testseed%84666431"/>
            <com.verifyter.pindown.Test testEnd="%testend%42" testName="%testname%test5" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/file_170504_1247.log:seed 27333434&lt;PinDownNL&gt;logs/file_170504_1247.log:Unexpected value 22. Expected 23&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/file_170504_1247.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/file_170504_1247.log:CPU time 42.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%27333434"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1225.log"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1229.log"/>
            <com.verifyter.pindown.LogFile path="logs/file_170504_1247.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>