Operation

The goal of the Validation Tool (VTool) is to programmatically ascertain if a given data product is PDS compliant (or "valid"). Typically, this means the data product is well formed, complete, syntactically and semantically correct, and that it conforms to all applicable PDS standards. The standards themselves are defined in the PDS Standards Reference.

This section describes how to use VTool in order to perform automated validation of data products. The following topics can be found in this section:

Note: The command-line examples in this section have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution.

Tool Execution

VTool can be executed in various ways. This section describes how to run VTool and how to generate reports, as well as the behaviors and caveats of the tool.

Command-Line Options

The following table contains command-line options available to VTool:

Command-Line OptionDescription
-t, --target <labels,URLs,dirs>Explicitly specify the targets (label files, directories, and URLs) to validate. Targets can be specified implicitly as well (example: VTool label.lbl). For more details on target specification, see the Specifying Targets section.
-d, --dict <.full files>Specify the Planetary Science Data Dictionary full file name and any local dictionaries.
-l, --log-file <file (optional)>(Not available in this release) Specify the file name for the machine-readable log. A file specification is optional. If no file name is given, then the log will be written to standard out. For more details on writing the log to a file, see the Report and Log Generation section.
-r, --report-file <file>Specify the file name for the human-readable report. Default is to write to the standard out if this option is not specified. This report, however, will not print to standard out if this option is missing AND the log file option is specified with no file name. For more details on how to create reports, see the Report and Log Generation section
-s, --report-styleSpecify the standard human-readable report format. Valid values are "full" for a full view, "min" for a minimal view, "sum" for a summary view, "json" for a JSON view, or "xml" for an XML view. Default is to generate a full report if this option is not specified. for more details on these report styles, see the Report Formats section
-v, --verboseSpecify the message severity level and above to include in the human-readable report (1=Info, 2=Warning, 3=Error). Default is warning and above (level 2).
-I, --include <paths>Specify paths to search for files referenced by pointers in a label. Separate each path with a comma. Default is to always look in the directory of the label, then search the specified directories.
-F, --no-followDo not follow or check for the existence of files referenced by pointer statements in a label.
-f, --forceEnable standalone label fragment validation. Default is to not validate them.
-c, --config <file>Specify a configuration file to set the default values for VTool.
-L, --localValidate files only in the input directory rather than recursively traversing down the sub-directories.
-e, --regexp <expressions>Specify file patterns to look for when validating a directory. Separate each pattern with a comma. Each pattern should be surrounded in quotes ("*.LBL", "*.FMT") to avoid having the system shell mistakingly interpreting them (This has been seen on non-Windows systems). Pattern matching is case-insensitive in Windows, but case-sensitive for other systems.
-X, --ignore-file <expressions>Specify file patterns to ignore when validating a directory. Separate each pattern with a comma. Patterns should be surrounded in quotes ("*.TXT", "*.TAB") to avoid having the system shell mistakingly interpreting them (This has been seen on non-Windows systems). Pattern matching is case-insensitive in Windows, but case-sensitive for other systems.
-D, --ignore-dir <expressions>Specify the directory patterns to ignore. Separate each pattern with a space. Patterns should be surrounded by quotes ("LABEL", "DOC*") to avoid having the system shell mistakingly interpreting them (This has been seen on non-Windows systems). Pattern matching is case-insensitive in Windows, but case-sensitive for other systems.
-p, --progressEnable validation progress reporting. Default is no. When this option is enabled, the current directory being validated will be written to standard error, followed by a series of asterisk "*" symbols, which represents a file being validated.
-h, --helpDisplay VTool usage.
-V, --versionDisplay VTool version.

Execute VTool

This section demonstrates execution of the tool using the command-line options. The examples below execute the tool via the batch/shell script. Alternate methods for executing the tool can be found in the Tool Setup section.

  • Validating Against a Single Dictionary
  • Validating Against Multiple Dictionaries
  • Validating Files with a Specific File Pattern
  • Ignoring Sub-Directories During Validation
  • Ignoring Files with a Specific File Pattern
  • Checking For Referenced Files in Different Locations
  • Not Following a Pointer
  • Standalone Label Fragment Validation
  • Progress Reporting
  • Changing Tool Behaviors With The Configuration File

Validating Against a Single Dictionary

The following command demonstrates the validation of a single data product label against the PSDD:

% VTool LABEL.LBL -d pdsdd.full
        

If there is a dictionary error, the tool will not perform label validation.

Validating Against Multiple Dictionaries

The following command demonstrates the validation of a single data product label against the PSDD and a local dictionary:

% VTool LABEL.LBL -d pdsdd.full, localdd.full
        

For more information on how the tool behaves when multiple dictionaries are passed in, see the Multiple Dictionary Support section.

Validating Files with a Specific File Pattern

The following command demonstrates the validation of multiple data product labels in a specified target directory, where validation only occurs on file names that end in ".LBL" or begins with the letters "MER":

% VTool $HOME/DIR -d pdsdd.full -e "*.LBL", "MER*"
        

Ignoring Sub-Directories During Validation

By default, the tool will recursively traverse down a directory tree when a target directory is specified, validating files within the sub-directories. Sub-directories can be ignored with the ignore directories option.

The following command demonstrates the validation of multiple data product labels in a specified target directory, where directories named "EXTRAS" or "LABEL" are ignored.

% VTool $HOME/DIR -d pdsdd.full -D "EXTRAS", "LABEL"
        

The local option can also be used to stop the tool from recursing down a directory tree.

The following command demonstrates the same validation as above, but without recursion:

% VTool $HOME/DIR -d pdsdd.full -L
        

Ignoring Files with a Specific File Pattern

The ignore files option can be used to tell the tool which files to ignore during validation.

The following command demonstrates the validation of multiple data product labels contained in a directory except for files ending in a ".IMG" or ".TAB":

% VTool $HOME/DIR -d pdsdd.full -X "*IMG", "*TAB"
        

Checking For Referenced Files in Different Locations

If a data product label contains a pointer statement that references a file, it will always assume it is co-located with the label. If it cannot be found there, then VTool will look for that referenced file in the paths specified by the include directories option.

The following command demonstrates the validation of a data product label that contains a pointer statement to a file located in a directory called DIR.

% VTool LABELPTR.LBL -d pdsdd.full -I $HOME/path
        

If the data product label contains pointer statements that reference files located in two different locations, then multiple paths can be specified.

The following command demonstrates the validation of a data product label that contains pointer statements that reference files located in two different locations, path1 and path2.

% VTool LABELPTR2.LBL -d pdsdd.full -I $HOME/path1, $HOME/path2
        

Not Following a Pointer

If a data product label points to a label fragment, VTool has the capability of validating this label without having to validate the associated label fragment with the no follow pointers option. This would be used in cases where the label fragment is not complete or in cases where the user wishes to simply check that the parent label structure is PDS compliant.

The following command demonstrates the validation of a data product label that contains a label fragment, but specifying to the tool to not follow the pointer:

% VTool LABEL.LBL -d pdsdd.full -F
        

Standalone Label Fragment Validation

The force option is used to force standalone label fragment validation. When this option is specified, standalone label fragment validation will occur on files whose extension ends in a .FMT

The following command demonstrates the validation of multiple data products, including performing standalone label fragment validation on any files it finds with a .FMT extension:

% VTool $HOME/DIR -d pdsdd.full -f
        

Progress Reporting

Validation progress reporting is enabled with the progress option. With progress reporting enabled, VTool will print, to the standard error, the current location that is being validated followed by a series of asterisk "*" symbols that represents a file it has encountered. The default behavior is to have this feature disabled.

The following command demonstrates the validation of multiple data products found in a directory, DIR, and its sub-directories with progress reporting enabled.

% VTool /home/user/DIR -d pdsdd.full -p
        

The progress reporting would look something like the following:

Validating file(s) in: file:/home/user/DIR
*
Validating file(s) in: file:/home/user/DIR/subDir1
*****
Validating file(s) in: file:/home/user/DIR/subDir2
***

...
        

In the above example, 1 file was validated in the DIR directory, 5 files were validated in DIR/subDir1, and 3 files were validated in DIR/subDir2.

Changing Tool Behaviors With The Configuration File

A configuration file can be passed into the command-line to change the default behaviors of the tool and to also provide users a way to perform validation with a single option. For more details on how to setup the configuration file, see the Using a Configuration File section.

The following command demonstrates performing validation using a configuration file:

% VTool -c config.txt
        

Specifying Targets

Targets are validated in the order in which they are specified on the command-line. They can be specified implicitly and explicitly.

To specify targets implicitly, it is best to specify them first on the command-line before any other options.

The following command demonstrates the validation of a single data product label, specified implicitly, against the PSDD:

% VTool LABEL.LBL -d pdsdd.full
        

The following command demonstrates the validation of multiple data product labels, both specified implicitly, against the PSDD:

% VTool LABEL.LBL, $HOME/DIR -d pdsdd.full
        

Implicit targets should not be specified after options that allow multiple arguments (see example below). Unexpected results will occur.

% VTool -d pdsdd.full LABEL.LBL
        

In this example, VTool will inadvertently treat the implicit target, LABEL.LBL, as a dictionary file.

Targets can be specified both implicitly and explicitly at the same time. Targets specified implicitly are validated first, followed by those that are specified explicitly with the target option.

The following command demonstrates the validation of multiple data product labels, specified both implicitly and explicitly, against the PSDD:

% VTool LABEL1.LBL LABEL2.LBL -d pdsdd.full -t LABEL3.LBL, $HOME/DIR
        

In this example, LABEL1.LBL and LABEL2.LBL will get validated first, then LABEL3.LBL and the labels in $HOME/DIR will get validated next.

Report and Log Generation

This section describes how to generate a human-readable report and machine-readable log of a validation run (machine-readable log generation not available in this release). This is done in the following ways:

  • Writing a Human-Readable Report to File
  • Writing a Human-Readable Report to Standard Out
  • Writing the Machine-Readable Log to Standard Out with No Human-Readable Report
  • Writing the Machine-Readable Log to File, Writing the Human-Readable Report to Standard Out
  • Writing Both the Machine-Readable Log and Human-Readable Report to File

Writing a Human-Readable Report to File

Specify the report file option to write the human-readable report to file. The report style option is used to specify the report format. If the style option is not specified, the default is to show a full report.

The following command demonstrates writing the human-readable, full report to a file named report.txt:

% VTool LABEL.LBL -d pdsdd.full -r report.txt
        

Writing a Human-Readable Report to Standard Out

Do not specify the report file option to write the human-readable report to standard out.

The following command demonstrates the validation of a single data product label against the PSDD, where the human-readable, full report is written to standard out:

% VTool LABEL.LBL -d pdsdd.full
        

The examples below will demonstrate how to generate a machine-readable log of a validation run.

Writing the Machine-Readable Log to Standard Out with No Human-Readable Report

Use the log file option with no file specification and do not specify the report file option.

The following command demonstrates writing the machine-readable log to standard out with no human-readable report generated:

% VTool LABEL.LBL -d pdsdd.full -l
        

Writing the Machine-Readable Log to File, Writing the Human-Readable Report to Standard Out

Specify the report file option and the log file option with a file name specification.

Note: When a directory is being validated and the log will be written to a file, the log should be written to a location different from the target directory. Otherwise, the tool will attempt to validate the log and it will be seen in the report.

The following command demonstrates writing the machine-readable log to a file named log.xml and writing a human-readable, full report to standard out.

% VTool LABEL.LBL -d pdsdd.full -l log.xml
        

Writing Both the Machine-Readable Log and Human-Readable Report to File

Specify the report file option and the log file option with a file specification. The file names should be different. The log and report cannot be written to the same file.

The following command demonstrates writing the machine-readable log to a file named log.xml and writing the human-readable report to a file named report.txt

% VTool LABEL.LBL -d pdsdd.full -l log.xml -r report.txt
        

Writing the Machine-Readable Log to Standard Out, Writing the Human-Readable Report to File

Specify the report file option and the log file option with no file specification.

The following command demonstrates writing the machine-readable log to standard out and writing the human-readable report to a file named report.txt

% VTool LABEL.LBL -d pdsdd.full -l -r report.txt
        

Multiple Dictionary Support

VTool allows multiple dictionary files to be passed in through the command-line via the "-d" option. When passing in multiple dictionaries, element and object definitions found in the dictionaries are merged together internally. In the case where definitions are found to be identical in each of the dictionary files being passed in, the valid value list from both dictionaries are combined.

The following example demonstrates the VTool behavior when passing in two dictionary files:

Suppose a PSDD, pdsdd.full, and a local dictionary, local-dd1.full, is being passed into VTool, where the PSDD contains the following DATA_SET_ID definition:

OBJECT = ELEMENT_DEFINITION
  NAME = DATA_SET_ID
  STATUS_TYPE = APPROVED
  GENERAL_DATA_TYPE = IDENTIFIER
  UNIT_ID = NONE
  STANDARD_VALUE_TYPE = FORMATION
  MAXIMUM_LENGTH = 40
  DESCRIPTION = "
     The data_set_id element is a unique
     alphanumeric identifier for a data set or a data product.
     The data_set_id value for a given data set or
     product is constructed according to flight project
     naming conventions.  In most cases the data_set_id is an
     abbreviation of the data_set_name.
     Example value: MR9/VO1/VO2-M-ISS/VIS-5-CLOUD-V1.0.
     Note:  In the PDS, the values for both data_set_id and
     data_set_name are constructed according to standards
     outlined in the Standards Reference."
  STANDARD_VALUE_SET = {
     "A-5-DDR-ASTERMAG-V1.0",
     "A-5-DDR-ASTEROID-SPIN-VECTORS-V3.0",
     "A-5-DDR-ASTNAMES-V1.0",
     "A-5-DDR-POLE-POSITION-REF-V1.0",
     "A-5-DDR-POLE-POSITION-V1.0",
     "A-5-DDR-TAXONOMY-V1.0",
     "ARCB-L-RTLS-3-70CM-V1.0",
     "ARCB-L-RTLS-4-70CM-V1.0",
     ....
     }
END_OBJECT = ELEMENT_DEFINITION
END
        

and the local dictionary also contains DATA_SET_ID, but with a different set of standard values:

OBJECT = ELEMENT_DEFINITION
  NAME = DATA_SET_ID
  STATUS_TYPE = APPROVED
  GENERAL_DATA_TYPE = IDENTIFIER
  UNIT_ID = NONE
  STANDARD_VALUE_TYPE = FORMATION
  MAXIMUM_LENGTH = 40
  DESCRIPTION = "
     The data_set_id element is a unique
     alphanumeric identifier for a data set or a data product.
     The data_set_id value for a given data set or
     product is constructed according to flight project
     naming conventions.  In most cases the data_set_id is an
     abbreviation of the data_set_name.
     Example value: MR9/VO1/VO2-M-ISS/VIS-5-CLOUD-V1.0.
     Note:  In the PDS, the values for both data_set_id and
     data_set_name are constructed according to standards
     outlined in the Standards Reference."
  STANDARD_VALUE_SET = {
     "VENUS"
     "EARTH"
  }
END_OBJECT = ELEMENT_DEFINITION
END
        

the STANDARD_VALUE_SET attribute from both definitions will be combined. As a result, the STANDARD_VALUE_SET will have the following values:

...
  STANDARD_VALUE_SET = {
     "A-5-DDR-ASTERMAG-V1.0",
     "A-5-DDR-ASTEROID-SPIN-VECTORS-V3.0",
     "A-5-DDR-ASTNAMES-V1.0",
     "A-5-DDR-POLE-POSITION-REF-V1.0",
     "A-5-DDR-POLE-POSITION-V1.0",
     "A-5-DDR-TAXONOMY-V1.0",
     "ARCB-L-RTLS-3-70CM-V1.0",
     "ARCB-L-RTLS-4-70CM-V1.0",
     ....
     "VENUS"
     "EARTH"
  }
...
        

Using a Configuration File

A configuration file is used to set the default behaviors of the tool. It consists of a text file made up of keyword/value pairs. The configuration file follows the syntax of the stream parsed by the Java Properties.load(java.io.InputStream) method. The following rules apply to the content of configuration files:

  • Blank lines and lines which begin with the hash character "#" are ignored.
  • Values may be separated on different lines if a backslash is placed at the end of the line that continues below.
  • Escape sequences for special characters like a line feed, a tabulation or a unicode character, are allowed in the values and are specified in the same notation as those used in Java strings (e.g. \n, \t, \r).
  • Since backslashes (\) have special meanings in a configuration file, keyword values that contain this character will not be interpreted properly by the tool even if it is surrounded by quotes. A common example would be a Windows path name (e.g. c:\VTT_EN_1-1\target). Use the forward slash character instead (c:/VTT_EN_1-1/target) or escape the backslash character (c:\\VTT_EN_1-1\\target).

Note: Any option specified on the command-line takes precedence over any equivalent settings placed in the configuration file.

The following table contains valid keywords that can be specified in the configuration file:

Property KeywordAssociated OptionValid Value(s)
vtool.target-tSpecify labels,directories, and URLs
vtool.dict-dSpecify dictionary files
vtool.log-l(Not available in this release) To write the log to standard out, set to 'true'. To write the log to a file, set to a file name. Otherwise, do not specify this keyword at all (or simply set to 'false').
vtool.report-rSpecify the human-readable report file name (do not specify keyword if wanting to write to standard out)
vtool.style-sSpecify the standard human-readable report format ("full", "min", or "sum"). If not specified, then a full report will be generated
vtool.verbose-vSpecify the message severity level and above to include in the human-readable report(1=Info, 2=Warning, 3=Error). Default is warning and above (level 2).
vtool.includepaths-ISpecify paths to search for files refererenced by pointers in a label
vtool.follow-FSet to 'false' to not look for files referenced by pointer statements in a label, set to 'true' otherwise
vtool.force-fSet to 'true' to enable standalone label fragment validation, 'false' otherwise
vtool.recursive-LSet to 'false' to not traverse a directory, set to 'true' otherwise
vtool.regexp-eSpecify file patterns to search for when validating a directory.
vtool.ignorefile-XSpecify file patterns to ignore when validating a directory.
vtool.ignoredir-DSpecify directory patterns to ignore.
vtool.progress-pSet to 'true' to enable progress reporting, 'false' otherwise.

The following example demonstrates how to set a configuration file:

# This is a VTool configuration file

vtool.target = ./TEST_DIR
vtool.dict = pdsdd.full
vtool.log = log.xml
vtool.report = rpt.txt
vtool.recursive = false
vtool.includepaths = /home/path
vtool.regexp = "*.LBL"
        

This is equivalent to running the tool with the following options:

-t ./TEST_DIR -d pdsdd.full -l log.xml -L -e "*.LBL" -I /home/path -r rpt.txt
        

The following example demonstrates how to set a configuration file with multiple values for a keyword:

# This is a VTool configuration file with multiple values

vtool.target = TEST.LBL, ./TEST_DIR
vtool.dict = pdsdd.full, localdd.full
vtool.recursive = true
vtool.includepaths = /home/path1, /home/path2
vtool.regexp = "*.LBL", "*.FMT"
        

This is equivalent to running the tool with the following options:

-t TEST.LBL, ./TEST_DIR -d pdsdd.full, localdd.full -e "*.LBL", "*.FMT" \
-I /home/path1, /home/path2
        

The following example demonstrates how to set a configuration file with multiple values that span across multiple lines:

# This is a VTool configuration file with multiple values
# that span across multiple lines

vtool.target = TEST.LBL, \
               ./TEST_DIR
vtool.dict = pdsdd.full, \
             localdd.full
vtool.recursive = true
vtool.regexp = "*.LBL", \
                 "*.FMT"
        

The following example demonstrates how to override a setting in the configuration file.

Suppose the configuration file named config.txt is defined as follows:

# This is another VTool configuration file

vtool.target = ./TEST_DIR
vtool.dict = pdsdd.full
vtool.log = log.xml
vtool.recursive = false
vtool.regexp = "*.LBL"
        

If validating everything in TEST_DIR is desired rather than just files that end in "LBL" as is defined in the configuration file, then the following command demonstrates how to perform this behavior:

% VTool -c config.txt -e "*"
        

Report Formats

This section describes the contents of the validation report formats. The links below detail the validation results of the same run for each format.

The tool can represent a validation report in three different formats: a full, a summary , and a minimal format. The report style option is used to change the formatting. When this option is not specified on the command-line, the default is to generate a full report.

Full Report

In a full report, the location, severity, and textual description of each detected anomaly is reported. A 'PASS', 'FAIL', or 'SKIP' keyword is displayed next to each file to indicate when a file has passed, failed, or skipped PDS validation, respectively.

Summary Report

In a summary report, the set of detected anomalies are summarized by reporting the number of occurrences for each type of anomaly and providing the location of one example for that anomaly.

Minimal Report

In a minimal report, only the number of anomalies detected are reported for each file. The anomalies are grouped by severity level.

JSON Report

In a JSON report, the location, severity, message type, and offending statement (when possible) of each detected anomaly is reported. Note that currently, the tool only supports validation runs of only a single data product label when generating this type of report.

XML Report

In an XML report, the location, severity, message type, and offending statement (when possible) of each detected anomaly is reported.

Exit Status Values

The Validation Tool returns an exit status based on the PDS validation results or if an application failure has occurred. This section details the values that can be returned as a result of a single validation run.

The following table shows what exit values can be returned when the tool validates one or more data product labels in a single run:

Integer ValueBinary ValueMeaning
00000 0000The data product label(s) passed the PDS validation step with no errors or warnings.
10000 0001An Application error has occurred.
20000 0010A System error has occurred.
320010 0000One or more files skipped during the validation run.
640100 0000One or more validation warnings were encountered during the validation run.
960110 0000One or more files skipped and validation warnings were encountered during the validation run.
1281000 0000One or more validation errors were encountered during the validation run.
1601010 0000One or more files skipped and validation errors were encountered during the validation run.
1921100 0000One or more validation warnings and errors were encountered during the validation run.
2241110 0000One or more files skipped. In addition, validation warnings and errors were encountered during the validation run.