Extraction of linting results is special in two ways: there are no pass messages for linting (no news is good news) and there are no test names. Instead the test name is set to a combination of the file name and the error ID in the lint error message in order to debug multiple issues per file. All linting error messages are defined as failures and a lack of an error message is defined as a pass.

There are no configurations in this scenario,just one lint log with error messages

Example

A lint report produces lists of line numbers that contain code that is breaking some rule. Each error message has an error level, an error identification number, a file name and a line number.

logs/lint.log
Lint Report:
------------
processor/bpu.v:1451 ERROR E3454 Instance name required for module
processor/alu.v:3451 ERROR E3231 Range index out of bound
processor/wbc.v:1733 ERROR E2123 Drive strength cannot be given to a net

PinDown Extraction

Step 1. Extract Lint Errors

In step 1 we extract the lint errors (see the file pindown_config.txt below). We set the test name to "file_errorlevel_errorcode" to allow multiple different issues to be debugged in each file. However we don’t include the line numbers in the test name because that varies when the files are updated.

pindown_config.txt
group -name "test" -commands {

   // Step 1. Test name is set to file_ERRORLEVEL_ERRORCODE. In linting there are no
  // pass messages, so each test is a failure
  extract -type "testname" -path "logs/lint.log" -keywords "ERROR|FATAL";
  extract -type "testfail" -path "logs/lint.log" -keywords "ERROR|FATAL";
  extract -type "replace" -label "testname" -text "logs/lint.log:" -with "";
  extract -type "replace" -label "testname" -text ":[0-9 ]*" -with "_"; // remove line number
  extract -type "replace" -label "testname" -text "ERROR " -with "ERROR_";
  extract -type "replace" -label "testname" -text "FATAL " -with "FATAL_";
  extract -type "replace" -label "testname" -text " .*" -with ""; // remove message

  // Step 2. Add default passes as passes are not present in linting reports
  extract -type "testname" -source "value" -path "%DEBUG_TESTS%" -keywords "";
  extract -type "testpass" -source "value" -path "%testname%" -keywords "";
  extract -type "remove" -label "testname" -containing "^all$";
};
extract -group "test" -file "test_results.xml";

Step 2. Set Default Passes

In step 2 we add default passes as passes are not present in linting reports. During debug Pindown runs linting on older revisions until it gets a pass, i.e the lint error message is no longer reported. No news is good news. However PinDown needs to know the name of the passing test. Consequently we set the requested test to pass by default, by using the PinDown variable %DEBUG_TESTS%, which is set to the requested test during debug. Note that PinDown will not overwrite the same name with a pass if it is already registered as a failure, so it will be only added if the test was not already extracted as a failure from the lint log in step 1. In the test phase PinDown has not requested any tests so consequently %DEBUG_TESTS% is set to the keyword "all" and this we want to remove because it is not a real failure.

The last line is the one that actually runs the extraction as defined by the group "test" and writes the result to the file test_results.xml.

Test Results

The result of running pindown_config.txt is written to the test_results.xml file.

Note that for failures multiple lines are extracted from the log files in order to see the lines above and below the failure message as they may contain interesting information. The actual error message is shown between PinDownMatchStart and PinDownMatchEnd.

test_results.xml
<?xml version="1.0" encoding="UTF-8"?>
<com.verifyter.pindown.TestResultList formatOwner="Verifyter" formatVersion="1.1" tool="PinDown">
    <com.verifyter.pindown.TestResults>
        <com.verifyter.pindown.ExtractionCommands>
            <com.verifyter.pindown.ExtractionCommand command="extract_config_label" label="configlabel"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_build_pass" label="buildpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_fail" label="testfail"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_replace" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_name" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_test_pass" label="testpass"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_remove" label="testname"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_results" label=""/>
            <com.verifyter.pindown.ExtractionCommand command="extract_repository" label="repository"/>
            <com.verifyter.pindown.ExtractionCommand command="extract_revision" label="revision"/>
        </com.verifyter.pindown.ExtractionCommands>
        <com.verifyter.pindown.BuildResult buildResult="%buildpass%pass" buildResultLine="Default set to pass">
            <com.verifyter.pindown.TestedConfiguration configurationLabel="%configlabel%build"/>
            <com.verifyter.pindown.Test testName="%testname%processor/bpu.v_ERROR_E3454" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/lint.log:Lint Report:&lt;PinDownNL&gt;logs/lint.log:------------&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/lint.log:processor/bpu.v:1451 ERROR E3454 Instance name required for module&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;"/>
            <com.verifyter.pindown.Test testName="%testname%processor/alu.v_ERROR_E3231" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/lint.log:Lint Report:&lt;PinDownNL&gt;logs/lint.log:------------&lt;PinDownNL&gt;logs/lint.log:processor/bpu.v:1451 ERROR E3454 Instance name required for module&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/lint.log:processor/alu.v:3451 ERROR E3231 Range index out of bound&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;"/>
            <com.verifyter.pindown.Test testName="%testname%processor/wbc.v_ERROR_E2123" testResult="%testfail%fail" testResultLine="&lt;PinDownContextStart&gt;logs/lint.log:Lint Report:&lt;PinDownNL&gt;logs/lint.log:------------&lt;PinDownNL&gt;logs/lint.log:processor/bpu.v:1451 ERROR E3454 Instance name required for module&lt;PinDownNL&gt;logs/lint.log:processor/alu.v:3451 ERROR E3231 Range index out of bound&lt;PinDownNL&gt;&lt;PinDownMatchStart&gt;logs/lint.log:processor/wbc.v:1733 ERROR E2123 Drive strength cannot be given to a net&lt;PinDownMatchEnd&gt;&lt;PinDownNL&gt;"/>
            <com.verifyter.pindown.LogFile path="logs/lint.log"/>
        </com.verifyter.pindown.BuildResult>
    </com.verifyter.pindown.TestResults>
</com.verifyter.pindown.TestResultList>