The test names form part of the file names or file paths. Extract the test name first and then associate all other test results (e.g. seed, time, pass/fail) found in the log file to this test name.

There are no configurations in this example, just one build used by all tests.

Example

Each test produces a log file whose name is the name of the test. The compilation result is present in a different file (logs/compile.log).

logs/test1.log
seed 22442244
Test Result: PASSED
CPU time 122.7 seconds
logs/test2.log
seed 12341234
Compare mismatch 34( got) !=33 (expected)
Test Result: FAILED
CPU time 234.3 seconds
logs/test3.log
seed 45674567
Test Result: PASSED
CPU time 217.1 seconds
logs/compile.log
Compiling project
...
Done!
CPU time 50.7 seconds

PinDown Extraction

Step 1. Extract the Compilation Result

Step 1 is to extract the compilation result in logs/compile.log (see the file pindown_config.txt below).

pindown_config.txt
group -name "test" -commands {

  // Step 1. Extracting build result
  extract -type "buildpass" -path "logs/compile.log" -keywords "CPU time";
  extract -type "buildfail" -path "logs/compile.log" -keywords "Error|error|failed";

  // Step 2. Extract the test results - first the complete lines
  extract -type "testname" -source "filename" -path "logs/test.*log" -keywords "";
  extract -type "testseed" -path "%testname%" -keywords "seed";
  extract -type "testpass" -path "%testname%" -keywords "PASSED";
  extract -type "testfail" -path "%testname%" -keywords "FAILED";
  extract -type "testend" -path "%testname%" -keywords "CPU time";

  // Step 3. Then clean the test results
  extract -type "replace" -label "testname" -text "logs/" -with "";
  extract -type "replace" -label "testname" -text ".log" -with "";
  extract -type "replace" -label "testseed" -text ".*seed " -with "";
  extract -type "replace" -label "testend" -text ".*CPU time " -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract Test Results

In step 2 we extract the test names from the file name of each test log. Initially the entire file path is set to the label "testname" which is subsequentially referred to ("%testname%") when extracting the other test results (seed, pass/fail and test end time). It is this association that makes PinDown understand which test results that belong together.

Step 3. Clean Results

In step 3 the results are cleaned up so that the test name is only the actual test name and not the complete file path. The seed is extracted from the content of a file and consequently it contains the result in the format filepath:content where the file path shows which file the content was extracted from. In this case the file path and the "seed " part of the content is removed in order to only keep the seed number. The test end time must be a number without decimals. In this case we take the whole number of seconds that the test lasted.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message as they may contain interesting information. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildResult="%buildpass%pass" buildResultLine="logs/compile.log:CPU time 50.7 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%build"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="logs/test1.log:Test Result: PASSED" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/test2.log:seed 12341234&lt;PinDownNL&gt;logs/test2.log:Compare mismatch 34( got) !=33 (expected)&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/test2.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/test2.log:CPU time 234.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="logs/test3.log:Test Result: PASSED" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="logs/compile.log"/>
            <com.verifyter.pindown.LogFile path="logs/test1.log"/>
            <com.verifyter.pindown.LogFile path="logs/test2.log"/>
            <com.verifyter.pindown.LogFile path="logs/test3.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>