Extract test results into a file structured according to the pindown XML format.

The XML file is then read using the command read_results. Only test results appearing in such a XML file is considered during PinDown debug. Test results from previous runs are stored in the database and are also considered during debug.

The test results achieved during extraction can also be used to determine whether the ongoing test run has completed according to the complettion criterias as specified by the PinDown command set_completion. There are two different completion criterias which make use of extracted test results: counting test or builds and comparing them to a certain value and checking for a new failure, i.e. a failure that is not present in the database.

Different types of test results, e.g. test name and test pass, test fail, are linked to each other by using either labels or pointing to the same files or web pages. The first alternative is to define the relationship explicitly by using labels. You can only refer to one label, but you can refer to the same label multiple times across the same or different options (-path, -keyword, -filter) without any limit.

The second alternative exist if the test results are extracted from the same files or web pages. In this case PinDown assumes the result types appear in the same order across columns or lines. If that is the case you just need to specify the files/web pages, no labels needed, and PinDown will automatically match the results together.

When setting up the extraction commands it is recommended to run PinDown with the following command line options:

pindown -extract
-debug

sets the verbosity level to debug, which makes PinDown print out step by step how it is constructing the XML file

-runhere

makes PinDown run in the local folder instead of creating a new test run folder and run in that folder, which is what happens during normal operation.

Syntax

extract -type resulttype [-label string] [-source sourcetype] -path string -keywords "keyword1|keyword2…" [-filter "keyword1|keyword2"];

extract -type resulttype [-label string] [-source sourcetype] -path string -keywords "keyword1|keyword2…" [-filter "keyword1|keyword2"] -assign string -default string [-prio int];

extract -type "replace" -label string -text string -with string;

extract -type "keep|remove|restore|move|merge" -label string -containing string;

extract -group string -file file;

type

Type of result to extract. Result types are either associated with build results (e.g. a compilation error) or test results (e.g. a failing test). An extraction of test results can only end up as a test result or a build result. However there is a third category of extraction types, data editing, which are used to manipulate test and build results data.

Build Result Types Test Result Types Comments

name

configlabel, testlist

testname

name of configuration, testlist or test

pass

buildpass

testpass

definition of pass

fail

buildfail

testfail

definition of fail

options

configoption, configvalue

runtimeoption, runtimevalue

options and its values associated with the name

metric

buildmetric

testmetric

any metric associated with the name, e.g. performance, size

count

buildcount

testcount

count and compare to completion critiera, e.g. if 14 tests extracted then stop if the completion criteria was 10.

count

-

works as buildcount and testcount types, but is not related to build or test results.

repository

repository

-

checkout information - repository names. Not necessary if PinDown performs the checkout.

revision

revision

-

checkout information - repository revisions. Not necessary if PinDown performs the checkout.

systemerror

systemerror

-

Extract system errors, which are associated with the set_error_response for this system error. Allows actions to be taken on specific error messages, e.g retesting the test

seed

-

testseed

seed used in random test environment

start time

buildstart

teststart

Time when build or test started. Useful as PinDown can identify short tests for retesting during debug.

end time

buildend

testend

Time when build or test ended. Useful as PinDown can identify short tests for retesting during debug.

owner

-

testowner

Default owner of the test. Also see set_testmanager for how to set a default owner.

Data Editing Types
list

An intermediate list not associated with neither build or test results

replace

Replace text for a specified label

keep

Only keep lines that matches the associated keyword.

remove

Remove lines that matches the associated keyword

restore

Restore a label to the original values

move

Move data from one label to another label of the same type (approved types are configlabel, testlist, testname and list). Define two comma-separated labels: -label "fromlabel,tolabel".

merge

Merge test or build results. If one label is specified then the first failing result is selected and any sub-sequent results with the same test name, configuration label or test list only add result types (e.g. run times) if they were not already populated by the first result (approved types are configlabel, testlist and testname). The exception are test results which are always added if build results were merged so that no test results are ever lost. If all results with the same name pass then the last passing result is selected. If all results are unknown then the last unknown result is selected. If a comma-separated list of labels is provided then the results associated with the last label is selected first. If the other result is a failure and the first result was a pass, or if the other result is a pass and the first result was unknown, then the other result, including all its result types (e.g. run times) takes higher priority during the merger. The name will always be picked up from the last declared label in that list. Note that the earlier labels in this list are only allowed to provide one result each, whereas the last declared label may have many results. This in order to prevent a large number of combinations. Build results are only merged if they share the same revision. Test results are only merged if they are part of the same configuration/testlist.

label

Label the extracted values in order to be able to refer to them with another command. If no label is entered then the default label name is the same as the result type, e.g. the default label name for the type "testname" is "testname". Labels can be used to extract the same result type to different labels in order to process the data in the different labels differently.

Normally the -label option only takes one label value. However for extract -type "move" you must specify two labels (-label "fromlabel, tolabel"). Also for extract -type "merge" you need to specify two or more labels (-label "mergefrom1, mergefrom2, … , mergeto).

source

Source defines what type of path that the -path option is referring to. Source can be set to either "log","filename","value" or "http". If source is not specified then the default value is "log".

Source Path Example Example output

log

files

-source "log" -path "example.txt" -keywords "line 1";

This is line 1

filename

files, folders

-source "filename" -path "example.txt" -keywords "file";

example.txt

value

text

-source "value" -path "baseconfig" -keywords "";

baseconfig

http

URL

-source "http" -path "http://www.verifyter.com" -keywords "Verifyter.*title";

Verifyter </title>

File "example.txt" (used in above examples):

This is line 1
This is line 2
Last line

path

The path defines where the source to extract is located. Regular expressions can be used to define many files or folders. See source above for more information on different source types.

During debug it is also possible to set this path to the requested values: %DEBUG_TESTS%, %DEBUG_CONFIGURATION% or %DEBUG_TESTLIST%. The source should be set to value when using these. During the test phase these values return "all".

keywords

The keyword defines what lines of the source to extract. Regular expressions is supported. You can use several different keywords, where each may contain regular expressions, separated with |, e.g.

"regex1|regex2|regex3"

You can also define the column number you wish to extract by using the following keyword format:

"keyword;column_delimiter=X;$n"

The first part keyword works as a normal keyword: the lines containing the keyword will be extracted. The two other sections, separated by semicolon (;) defines which column in these lines that should be extracted. The first of these two section, column_delimiter specifies what character that delimits the columns, e.g. a space or comma. The last section specifies the column number, which can be 1 or larger.

For example, using the "example.txt"-file above we can extract the numbers (1 and 2) using the following keyword:

"This;column_delimiter= ;$4"

This command selects the two first line that contains "This" and then using space as the column delimiter selects the 4th column which contains the number.

filter

Filter defines what lines of the source to not extract. Regular expressions is supported. You can entire several different keywords, where each can use regular expressions, separated with |.

assign

Used by extraction types buildfail and testfail (for which it is default set to "fail") and buildpass and testpass (for which it is default set to "pass"). It is also possible to set it to an expression in the format "if(operator value)". E.g. to define a test cycle count as a failure if it exceeeds 9500 cycles, it is possible to write:

extract -type "testfail" -path "results.txt" -keywords "Happy after .* cycles" -assign "if(>9500)" -default "unknown";

Allowed operators are ==, !=, <, , >= and >. Allowed values are integers for all operators. For == and != it is also possible to use a string or a regular expression, like this:

extract -type "buildpass" -path "compile.log" -keywords "Compilation Result;column_delimiter=,;$2" -assign "if(==ok)" -default "unknown";
extract -type "buildfail" -path "compile.log" -keywords "Compilation Result;column_delimiter=,;$2" -assign "if(!=ok)" -default "unknown";
compile.log (referred to above):
34..
76..
99..
100..done!
Compilation Result: 435.3 s, result ok, 2018-08-23 06:55
Finished

In the example above the build result is set to pass if the second column contains the word "ok" (it doesn’t have to be exactly equal), otherwise fail . The column delimiter is defined as a comma.

default

When no failure or pass messages can be found then result takes its value from this default option. If neither the pass or fail result type finds a corresponding message then the command with default value of the result type with the highest priority is selected, which is normally the fail type, unless the user has explicitly said it to be otherwise (see prio).

Any time assign is explicitly defined then the default option has to also be defined.

Normally is should always be defined to the reserved keyword "unknown", but it is also possible to defined it to either "pass" or "fail" (also reserved keywords).

prio

By default failures have prio 1 and passes prio 2 but if you want to swap that around you can use this option. Notice that this is the only option that is an integer (1 or 2) and not a string, so its values should not be surrounded by quotation marks

text

Used with extraction type replace to identify the text that this command should operate on.

with

Used with extraction type replace to replace the text identified with the -text option with this.

The extraction type replace supports using & to mean replace with the match defined in the text. If you actually want to replace with the character & then you have to type \\&

It is also possible to replace a text in one label with the content of another label by referring to %otherlabel% in this field. If you refer to %otherlabel:orig% you get the original values of the label (:orig is a reserved keyword - it is illegal to have : in a label name), which is useful in case the values have already been modified by another replace command.

During debug it is also possible to set this path to the requested values: %DEBUG_TESTS%, %DEBUG_CONFIGURATION% or %DEBUG_TESTLIST%. The source should be set to value when using these. During the test phase these values return "all".

containing

Used with extraction types keep, remove, restore, move and merge. Lines matching this string are either kept, removed, restored, moved or merged depending on the extraction type.

group

Identify the extraction group, either "test" or "diagnosis", that should be used to extract the test results into an XML file.

file

The name of the XML file to which the extracted test results should be written to.

Examples

The best place to start is the Extraction User Guide where complete extraction setups are shown for different types of verification environments and log structures. Here we show examples of specific commands, not complete setups.

Example 1 - Replace Text

Different examples of replacing text in the specified label.

Example 1.1 Replacing text with another text

Replacing text in the test names, from the start of the line until "test_" with just "test_" instead.

extract -type "replace" -label "testname" -text ".*run -test .*_test_" -with "test_";

Example 1.2 Removing text

extract -type "replace" -label "testname" -text ".*run -test " -with "";

Example 1.3 Removing filename of match

Each match is in the format "filename:content". Here we want to remove the file name and the separating ":" in order to just save the content.

extract -type "replace" -label "testname" -text "^[^:]*:" -with "";

Example 1.4 Removing content of match

This is the opposite of Example 1.3. Now we want to remove the content part of the match.

extract -type "replace" -label "testname" -text ":.*" -with "";

Example 1.5 Replacing text with the content of another label

Here we want to change each configuration name to its associated config option value (%configvalue%) plus the text "config" added as a suffix.

extract -type "replace" -label "configlabel" -text "config_" -with "%configvalue%_config_";

Example 1.6 Replacing text with the its own match (&)

Replacing the match with itself (&) surrounded by some text

extract -type "replace" -label "configlabel" -text "test_.* " -with "test: & seed:";

Example 1.7 Using regex groups to swap matches

label "testname" contains:
c2t10
c2t2
c2t4
c2t5

Then we apply an regex with groups to swap the matches

extract -type "replace" -label "testname" -text "([^0-9]*[0-9]*){1}(t.*)" -with "$2$1";
label "testname" now contains:
t10c2
t2c2
t4c2
t5c2

Example 2 - Remove Tests or Builds

Removing tests or builds

Example 2.1 Remove tests containing certain text

Remove tests containing the text "test_usb5" or "test_usb6"

extract -type "remove" -label "testname" -containing "test_usb[5-6]";

Example 2.2 Remove all tests

Remove all tests associated with label "testname"

extract -type "remove" -label "testname" -containing "";

Example 2.3 Keep certain builds, remove the rest

Keep builds containing the text "usb1", "usb2" or "usb3". Remove the other builds.

extract -type "keep" -label "configlabel" -containing "test_usb[1-3]";

Example 3 - Restore Label Values

Restore values to their original values in order to select different parts of the matching text for different purposes.

In this example we extract the configuration names by looking for the keyword "Build" in logs. This returns matches in the format filepath:content. First we remove the content parts in order to only keep the filepaths. We subsequentially refer to these filepaths to get the test names and associate them with the correct configuration names.

After this exercise we want to restore the configuration names to their actual values. These are found in the content parts of the match.

// Get the configuration names and associate them with the test names located in the same files
extract -type "configlabel" -path "testlogs/results.*log" -keywords "Build";
extract -type "replace" -label "configlabel" -text ":.*" -with "";
extract -type "testname" -path "%configlabel%" -keywords "run \-test";

// Restore the configuration names and set them to their actual names.
extract -type "restore" -label "configlabel" -containing "";
extract -type "replace" -label "configlabel" -text ".*Build: " -with "";

Instead of restoring the labels it is also possible to refer to the original value of the label. Using this method instead the example above would look like this:

// Get the configuration names and associate them with the test names located in the same files
extract -type "configlabel" -path "testlogs/results.*log" -keywords "Build";
extract -type "replace" -label "configlabel" -text ":.*" -with "";
extract -type "testname" -path "%configlabel%" -keywords "run \-test";

// Refer to the original confignames and set them to their actual names.
extract -type "replace" -label "configlabel" -text ".*" -with "%configlabel:orig%";
extract -type "replace" -label "configlabel" -text ".*Build: " -with "";

Example 4 - Move Data to Another Label

Moving data to another label is useful in advanced cases where you need to extract multiple sets of builds or tests which come in very different formats. You can modify each set according to different rules and only once they are all complete you move them back together under a common label.

You can only move data between labels of the same type.

Example 4.1 Move builds into another label

Move the build results in label "configlabel2" whose configuration names contain the text "build_usb" into the list of build results associated with the label "configlabel"

extract -type "move" -label "configlabel2,configlabel" -containing "build_usb";

Example 4.2 Move a list into another list

Move the content of label "logfiles_disc34" into the label called "all_logfiles". This label type is a "list" which is the only label type that is not associated with a build or test result.

extract -type "move" -label "logfiles_disc34,all_logfiles" -containing "";

Example 4.3 Move data from a list into a different type

Move the content of label "all_logfiles" (type list) into the label "configlabel" (type configuration label). These labels are of different types so now you can’t use extract -type "move". Instead you have to extract the content of one label into the other.

extract -type "configlabel" -label "configlabel" -source "value" -path "%all_logfiles%" -keywords "";

Example 5 - Merge Data

Merging data allows different results to be extract in different ways and the in the end they are merged together to form well structured build results. It can be used to extract compilation results and test results in two different structures and then merged together. Another example is to extract multiple build steps separately and then merge the results together

There are two types of merging: merging within a label and merging between two different labels.

Note that the difference between move (see Example 4) and merge is that after a move the label will contain more configurations or tests, but after a merger that number does not increase. It can be a good idea to do a merger after a move in order to remove any redundancies.

Example 5.1 Merging multiple configurations

Merging all configurations in label "configlabel". All results with the same configuration name will be merged together.

extract -type "merge" -label "configlabel" -containing "";

Example 5.2 Merging one configuration into multiple configurations

Merging the one and only configuration in label "buildstep1" (only one allowed) with each of the many configurations in the label "configlabel". After the merger only the label "configlabel" survives and all its configuration names are preserved. The label "buildstep1" completely disappears, but its results, e.g. failures are merged into the label "configlabel"

extract -type "merge" -label "buildstep1,configlabel" -containing "";

Example 6 - Referring to two labels

PinDown does not allow the user to refer to two labels in one extraction command, but it is possible to solve the same problem by referring to two labels in two consecutive commands as shown in this example.

Example: A result file contains the following data:

results.txt
configuration c1
test t1
test t2
test t3
configuration c2
test t1
test t2
test t3

and another file contains the seeds

seeds.txt
c1 t1 34565434
c1 t2 78554344
c1 t3 12224534
c2 t1 44345434
c2 t2 99943345
c2 t3 83858841

In this case it would make sense to do the following extraction, but it is forbidden!

pindown_config.txt
extract -type "configlabel" -path "results.txt" -keywords "configuration";
extract -type "testname" -path "results.txt" -keywords "test";
extract -type "replace" -label "configlabel" -text ".*configuration " -with "";
extract -type "replace" -label "test" -text ".*test " -with "";

// Illegal referral to two labels in one command
extract -type "testseed" -path "seeds.txt" -keywords "%configlabel% %testname%";

However it is legal to refer to two labels in two consecutive commands:

pindown_config.txt
extract -type "configlabel" -path "results.txt" -keywords "configuration";
extract -type "testname" -path "results.txt" -keywords "test";
extract -type "replace" -label "configlabel" -text ".*configuration " -with "";
extract -type "replace" -label "test" -text ".*test " -with "";

// Legal referral to two variables in two commands
extract -type "replace" -label "testname" -text ".*" -with "%configlabel% &";
extract -type "testseed" -path "seeds.txt" -keywords "%testname%";

// Remove the config label from the test name after the seed has been extracted
extract -type "replace" -label "testname" -text ".* " -with "";

The magic is the replace , which replaces the entire test name with the name of its associated configuration, a space and the test name itself (& means the match in the -text field). Each test name will consequently be "c1 t1", "c1 t2" etc. In the last line the test name is restored to just be the actual test name again ("t1", "t2" etc).

Example 7 Extracting runtime options

If you need to extract runtime options, i.e. settings that relate to the test that you are running, then here is a way to do it. This method is stable even if the runtime options are not always exactly the same. If the argument order changes or there sometimes appears an extra optional value this method of extraction changes them to be in the same order each time.

Indata

something PLUSARGS=+NOR_A=bool\ +NOR_B=grunge\ NOR_C=ignore something

Extraction commands used

extract -type "runtimeoption" -path "%testname%" -keywords "PLUSARGS=";
extract -type "runtimevalue" -path "%testname%" -keywords "PLUSARGS=";

extract -type "replace" -label "runtimeoption" -text ".*PLUSARGS.*" -with "PLUSARGS";

// First remove all text except the PLUSARGS values. Adding the text PINDOWN as first text as a delimiter.
extract -type "replace" -label "runtimevalue" -text ".*PLUSARGS=" -with "PINDOWN";
extract -type "replace" -label "runtimevalue" -text "([^\\]) .*" -with "$1\\ ";

// Extract each option in the desired order.
extract -type "replace" -label "runtimevalue" -text "^(.*)PINDOWN(.*?)(\+NOR_B=[^\\]*)(\\.*)$" -with "$1$3\\ PINDOWN$2$4"
extract -type "replace" -label "runtimevalue" -text "^(.*)PINDOWN(.*?)(\+NOR_A=[^\\]*)(\\.*)$" -with "$1$3\\ PINDOWN$2$4"

// Finally remove the delimiter and all options not picked.
extract -type "replace" -label "runtimevalue" -text "\\* *PINDOWN.*" -with "";

Resulting option PLUSARGS has value

NOR_B=grunge\ NOR_A=bool

And NOR_C is not used.

Example 8 Passes higher priority than failures

If passes should have higher priority (normally failures have higher priority) then the -prio option should be used. If you only want to capture the RTL compilation results and filter any error that is due to test compilation then this is one way to do it.

extract -type "configlabel" -label "configlabel" -source "value" -path "core_.*" -keywords "";
extract -type "buildpass" -path "%configlabel%_hybrid.log" -keywords "simv up to date" -assign "pass" -default "unknown" -prio 1;
extract -type "buildfail" -path "%configlabel%_hybrid.log" -keywords "Error" -assign "fail" -default "unknown" -prio 2;