The test names are found inside log files together with the test results. It is not possible to get the test names from the file paths or file names. There is one log file per test or one log file containing the logs of all test results run in sequence.

There are no configurations in this example, just one build used by all tests.

Example

The test results are available in files whose names do not give any clue about the test name. The test name can only be extracted from inside each log file. The compilation result is present in a different file (logs/compile.log).

logs/results_a.log
Command: run -test test1
seed 22442244
Test Result: PASSED
CPU time 122.7 seconds
logs/results_b.log
Command: run -test test2
seed 12341234
Compare mismatch 34( got) !=33 (expected)
Test Result: FAILED
CPU time 234.3 seconds
logs/results_c.log
Command: run -test test3
seed 45674567
Test Result: PASSED
CPU time 217.1 seconds
logs/compile.log
Compiling project
...
Done!
CPU time 50.7 seconds

PinDown Extraction

Step 1. Extract the Compilation Result

Step 1 is to extract the compilation result in logs/compile.log (see the file pindown_config.txt below).

pindown_config.txt
group -name "test" -commands {

  // Step 1. Extracting build result
  extract -type "buildpass" -path "logs/compile.log" -keywords "CPU time";
  extract -type "buildfail" -path "logs/compile.log" -keywords "Error|error|failed";

  // Step 2. Extract the test results - first the complete lines
  extract -type "testname" -path "logs/results.*log" -keywords "run \-test";
  extract -type "testseed" -path "logs/results.*log" -keywords "seed";
  extract -type "testpass" -path "logs/results.*log" -keywords "PASSED";
  extract -type "testfail" -path "logs/results.*log" -keywords "FAILED";
  extract -type "testend" -path "logs/results.*log" -keywords "CPU time";

  // Step 3. Clean the test results
  extract -type "replace" -label "testname" -text ".*run \-test " -with "";
  extract -type "replace" -label "testseed" -text ".*seed " -with "";
  extract -type "replace" -label "testend" -text ".*CPU time " -with "";
  extract -type "replace" -label "testend" -text "\..*" -with "";
};
extract -group "test" -file "test_results.xml";

Step 2. Extract Test Results

In step 2 we extract the test names from the content each test log in the same way we are extracting the other test results (seed, pass/fail and test end time). Each result type is looking for a different keyword. PinDown associates the seed, results and test end time with the test name that is present in the same test log file, provided there is one test result per log file. If all test results are listed in just one log file then this extraction works too, provided the tests were run sequentially.

Step 3. Clean Results

In step 3 the results are cleaned up so that the test name is only the actual test name. Note that each result is in the format filepath:content where the file path shows which file the content was extracted from. In this case the file path and the "run -test " part of the content are removed in order to only keep the actual test name. The seed is cleaned in the same way. The test end time must be a number without decimals. In this case we take the whole number of seconds that the test lasted.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Note In this example each file contains the result from one test, but the same extraction also works if all results are available in one log file. However, the benefit of having one result file per test is that it is more stable to extract the test result in case one of the test produces incomplete results.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message as they may contain interesting information. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_fail" label="buildfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_seed" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_end" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testseed"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testend"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildResult="%buildpass%pass" buildResultLine="logs/compile.log:CPU time 50.7 seconds">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%build"/>
            <com.verifyter.pindown.Test testEnd="%testend%122" testName="%testname%test1" testResult="%testpass%pass" testResultLine="logs/results_a.log:Test Result: PASSED" testSeed="%testseed%22442244"/>
            <com.verifyter.pindown.Test testEnd="%testend%234" testName="%testname%test2" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/results_b.log:Command: run -test test2&lt;PinDownNL&gt;logs/results_b.log:seed 12341234&lt;PinDownNL&gt;logs/results_b.log:Compare mismatch 34( got) !=33 (expected)&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/results_b.log:Test Result: FAILED&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;logs/results_b.log:CPU time 234.3 seconds&lt;PinDownNL&gt;" testSeed="%testseed%12341234"/>
            <com.verifyter.pindown.Test testEnd="%testend%217" testName="%testname%test3" testResult="%testpass%pass" testResultLine="logs/results_c.log:Test Result: PASSED" testSeed="%testseed%45674567"/>
            <com.verifyter.pindown.LogFile path="logs/compile.log"/>
            <com.verifyter.pindown.LogFile path="logs/results_a.log"/>
            <com.verifyter.pindown.LogFile path="logs/results_b.log"/>
            <com.verifyter.pindown.LogFile path="logs/results_c.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>