Planetary Data System
Key Word Value Tool (KWVTOOL) User’s Guide
(UG)
Draft Release Version 1
Prepared by:
Michael Cayanan/Dale Schultz
Document Custodian: Valerie
Henderson
Approved by:
________________________________________________
Dan Crichton
PDS Project Manager
Jet Propulsion Laboratory
CHANGE LOG
Revision |
Date |
Description |
Author |
Start Draft |
May, 2004 |
Initial draft |
D. Schultz |
Release 1 |
|
Draft Release 1 – Edited all |
V. Henderson |
|
August 2005 |
Update Document |
M. Cayanan |
|
|
|
|
|
|
|
|
|
|
|
|
Table of Contents
2.5.2 Process a Group of Files
2.5.3 Using a New Data Dictionary
2.5.4 An example of a KWVTOOL report
4.2 –d Enable Data Dictionary index file
4.3 –df
Enable Data Dictionary full file
4.4 –di Do not search subdirectories for label files
to be processed.
4.6 –fin
<file name> Specifies name of file containing list of file names to
process
4.7 –ivd
<file name> Specifies a file containing directories to skip.
4.8 -ivf
<file name> Specifies name of file containing a list of file extensions
to skip
4.9 -lef <file name> Create a Log File of all Skipped Files
4.11 -r Specify Report File Name
4.12 -sf Specify Keywords to Skip
4.13 –nol3d
Do Not Check For Level 3 Labels in Input Files
Project name: Planetary Data System (PDS)
Program set name: PDS Key Word Value Extraction Tool
Abbreviation: kwvtool
Version number: 2.2
Platforms supported: Windows, Solaris, Linux
This document provides information on the use of the PDS KWVTOOL software set. This software creates a list of all keywords and their values found in a PDS label file or in a group of PDS label files.
This document provides:
This document is intended for those who are responsible for ensuring that labels submitted to the PDS conform to PDS standards. This document assumes familiarity with the PDS data preparation process and label design requirements. The data preparation process is described in the PDS Data Preparation Workbook (JPL Document D-7669, Part 1); label file requirements are described in full in Chapter 5 of the PDS Standards Reference (http://pds.jpl.nasa.gov/documents/sr/Chapter05.pdf). See Section 1.6.
PDS labels are written in the Object Description Language (ODL). ODL consists of a series of lines of the form "keyword = value", with certain keywords (for example, OBJECT) being used to delimit named groups of keywords within the label. For a description of the PDS implementation of ODL, see Chapter 12 of the PDS Standards Reference (http://pds.jpl.nasa.gov/documents/sr/Chapter12.pdf).
The keywords and objects are themselves defined in the Planetary Science Data Dictionary (PDSDD). The Dictionary exists as a Word file for human readers, as a text file for use by the software, and is available via an online look up tool at (http://pds.jpl.nasa.gov/tools/data_dictionary_lookup.cfm).
The PDS KWVTOOL reads PDS labels, extracts all keyword/keyword value pairs, deletes any duplicates, and creates a two-part report of the unique pairs. The list is divided into one section for keywords not found in the data dictionary and one part for keywords that are found in the data dictionary.
·
Planetary Science Data Dictionary Document,
· Planetary Data System Data Preparation Workbook, JPL D-7669, Part 1. (http://pds.jpl.nasa.gov/documents/dpw/index.html)
The PDS KWVTOOL software set consists of the following files:
Windows
kwvtool.exe - executable
pdsdd.full - Data Dictionary full file
make_index.exe – utility that creates the pdsdd.idx
Solaris and Linux
lvtool – executable
pdsdd.full - Data Dictionary full file
make_index - – utility that creates the pdsdd.idx
PDSDD.FULL and MAKE_INDEX are included in the ddict distribution package. PDSDD.FULL will change often and should be downloaded frequently from: http://pds.jpl.nasa.gov/tools/software_download.cfm. All of the central node validation software tools are also available at this site. Each time PDSDD.FULL is downloaded it is important to run MAKE_INDEX again to create a new index. Once at that url select “for data producers”. When the new url opens scroll down the Software Id/Version ID window until the desired software tool is highlighted. Click on the selected tool once and select the submit button. When the new information is displayed click once on the appropriate download.
PDSDD.FULL can be obtained by scrolling through the Software Id/Version ID window to pdsdd(CAT1R40):PDS Toolbox Data Dictionary Version *****(the version number will be changed from time to time and is not shown here). Select MULTIPLE from the download list.
MAKE_INDEX can be obtained by scrolling through the Software Id/Version ID window to make_index, selecting it, and then selecting the submit button.
Click once on the appropriate download.
Download the zip or tar file for kwvtool, make_index, and data dictionary file to the hard drive.
Use make_index to make pdsdd.idx by typing:
make_index pdsdd.full
The PDS KWVTOOL is a command line tool and will need to be run from a command prompt.
To invoke kwvtool from the same directory in which both kwvtool and the default Data Dictionary files are located, enter the command:
kwvtool –d pdsdd.idx -f <input-file> -r <output-file>
where <input-file> is the name of the label file to process, and <output-file> is the name of the report file to be created. If the kwvtool cannot locate the Data Dictionary files it needs, it will generate an error message.
Kwvtool may also be started from a different directory that the one in which kwvtool and the Data Dictionary files are located. For example, on a Unix system, if the kwvtool and dictionary files are located in directory /usr/local/pds, enter the command:
/usr/local/pds/kwvtool
-f <input-file> -r <report-file> -d
/usr/local/pds/pdsdd.idx
The -d option tells kwvtool the path and name of the PDS Data Dictionary index file.
If using the Data Dictionary full file, then use the –df option instead of –d:
/usr/local/pds/kwvtool
-f <input-file> -r <report-file> -df
/usr/local/pds/pdsdd.full
The –df option tells kwvtool the path name of the PDS Data Dictionary full file. You will need to also make sure the make_index program (version 1.7 or higher) is located in the same path as kwvtool since it uses that to create the index file internally when given the full file as an input.
To get a brief description of the command line parameters, at the command prompt type:
kwvtool
The parameters are also explained in detail in the COMMAND REFERENCE section of this document.
For further help or information, contact the PDS Operator at pds_operator@jpl.nasa.gov
The following examples will help in using kwvtool immediately. The chapter, “What the KWVTOOL Does,” explains the key word value extraction process in more detail.
While reviewing the following examples, refer to the “Command Reference” chapter, which explains the command line options available for kwvtool, and the “Kwvtool Messages” chapter, which explains the error and warning messages generated by kwvtool.
The following examples assume that both kwvtool and the Data Dictionary reside in the current directory.
Note: In all cases
below, you can substitute the full file instead of the index file by using the
–df argument instead of –d:
kwvtool
–df pdsdd.full….
Remember to have
make_index version 1.7 or higher residing in the same directory as kwvtool.
Assume there is a label file called “example1.lbl” as shown below: To process that one file enter the command:
kwvtool –d
pdsdd.idx -f example1.lbl -r myreport.rpt
This tells kwvtool to compare the keywords in example1.lbl against the keywords in pdsdd.idx and to write the report to myreport.rpt.
If there are several label files in the directory, enter the command:
kwvtool –d pdsdd.idx -f *.lbl -r myreport.rpt
This tells the kwvtool to extract the keywords from all the files that have the extension of lbl in the current directory, compare them against the keywords in pdsdd.idx and to write the report to myreport.rpt. Kwvtool was used in this fashion on a directory containing a large number of label files to generate the following report:
Assume that the existing Data Dictionary contains too general a set of rules for the object class TABLE. The generic, PDS-supplied TABLE object class includes an optional keyword, DESCRIPTION, that should be required in all of the labels. The Data Dictionary may be modified according to the instructions in the “Toolbox Data Dictionary” chapter of the PDS Toolbox Overview and define a new object class, called FEATURE_TABLE, which requires the presence of the keyword DESCRIPTION in every FEATURE_TABLE object. Enter the command:
kwvtool -f
example1.lbl -r newreport.rpt -d mydd.idx
to tell kwvtool: “Process file ‘example1.lbl’ again and write the results to file ‘newreport.rpt’, only this time use the Data Dictionary index file ‘mydd.idx’”. (The file “mydd.idx” must be a Data Dictionary index file generated by the make_index program.)
____________________________________________________________
************************************************************************
* *
* Planetary Data System KWVTOOL *
* Selected Keywords and Values *
* Extractor Version 1.9 *
* *
*
Mon May 03
* *
* Data Dictionary Name: pdsdd.idx *
* Data Dictionary Version not found *
* Data Dictionary
Generated:
* *
* Data Dictionary Validation is ON *
* *
* WARNING! *
*----------------------------------------------------------------------*
* Keywords will be truncated at 30 characters *
* values will be truncated at 95 characters *
* No indication will be given that truncation has occurred *
************************************************************************
WARNING! These selected keywords are not in the Data Dictionary
SOURCE_IMAGE_ID 383B23
SOURCE_IMAGE_ID 383B42
SOURCE_IMAGE_ID 421B21
SOURCE_IMAGE_ID 421B23
SOURCE_IMAGE_ID 421B73
SOURCE_IMAGE_ID 421B74
SOURCE_IMAGE_ID 421B75
SOURCE_IMAGE_ID 421B76
SOURCE_IMAGE_ID 421B77
TYPE BDV
TYPE VICAR2
----------------------------------------------
These selected keywords are in the Data Dictionary
BYTES 1024
BYTES 12
BYTES 13
BYTES 2000
BYTES 4
BYTES 5
BYTES 5000
BYTES 9
FILE_RECORDS 1001
FILE_RECORDS 1026
FILE_RECORDS 175
FILE_RECORDS 180
FILE_RECORDS 2048
FILE_RECORDS 2128
FILE_RECORDS 2129
FILE_RECORDS 23
FILE_RECORDS 2396
FILE_RECORDS 2917
FILE_RECORDS 310
FILE_RECORDS 512
FILE_RECORDS 56
FILE_RECORDS 602
FILE_RECORDS 720
FILE_RECORDS 806
FILE_RECORDS 809
IMAGE_ID E2W3599
IMAGE_ID EJZ1928
IMAGE_ID F-MIDR.20S145;1
IMAGE_ID MG88S045
LINE_FIRST_PIXEL 1
************************************************************************
* *
* This is the end of the Keyword / Value Tool Report *
* *
*
Mon May 03
* *
************************************************************************
Report File “myreport.rpt “
This report was several hundred lines long and has been edited to make it smaller. In the first section there were 9 occurrences of SOURCE_IMAGE_ID. Actually there may have been more but any duplicates were deleted by kwvtool. There were only 2 unique occurrences of TYPE. It is possible in this case that 9 different values for the same keyword is legitimate but that may not always be true. This provides a list of all the values for each keyword in the processed label files. This first section of the report also identifies SOURCE_IMAGE_ID and TYPE as keywords that are not in the data dictionary.
The second section contains a list of keywords that were in the data dictionary along with the unique keyword values of each.
KWVTOOL makes a list of all keywords and the keyword value. This list is then alphabetized. The keywords in the list are checked against the data dictionary and those that are not found are flagged. The keywords that are flagged will appear in the section of the report that is for keywords not found in the data dictionary. The remaining keywords will be placed in the report section for keywords that were found in the data dictionary. If a keyword/keyword value pair is a duplicate it will be deleted. The resulting report will contain a complete list of all keywords found in the label(s) and all of the values that were found for each keyword.
Default: -na (no aliasing)
The -a command line option enables the aliasing feature of kwvtool. This option is useful when processing a label which was developed according to another Data Dictionary, or to identify out-of-date keywords and objects in the label.
When the -a option is turned on, kwvtool will compare the object classes in the label with the list of object class aliases in the Data Dictionary, and replace them with their official names. Object classes that were changed are not identified in the report file. An example of such a change is:
OBJECT =
TABLE_STRUCTURE
which will be replaced by
OBJECT = TABLE
if TABLE_STRUCTURE is defined to be an alias for object class TABLE in the Data Dictionary.
Kwvtool will then compare the keywords in the label with the list of keyword aliases in the Data Dictionary and replace them with their official names.
After aliasing is finished, kwvtool will continue identifying all keywords and values, using the official names instead of the aliases when it searches the Data Dictionary.
Note that only full class names will be replaced. For example, EDR_TABLE_STRUCTURE will not become EDR_TABLE just because TABLE_STRUCTURE is defined as an alias for TABLE. The same is true for the keyword replacement. Remember that kwvtool does not change the actual label file at all. The changes to the label remain in effect only for the duration of the kwvtool run.
Default: -d pdsdd.idx
Kwvtool compares the keywords found in the label file(s) with a data dictionary. Kwvtool will then write a report identifying those keywords that were not in the data dictionary as well as those that were.
The default is -d pdsdd.idx. Kwvtool will look for pdsdd.idx if no argument is given. If it can not fine the file the program will issue an error message and stop. If another data dictionary is to be used it can be specified with this parameter. If another data dictionary is specified be sure run make_index with the alternate dictionary. The following command would tell kwvtool to use mydd.idx as an index into a data dictionary. Make_index will put the name of the actual data dictionary at the head of the index file.
kwvtool -d mydd.idx -f
*.lbl -r myreport.rpt
Default: -df pdsdd.full
The –df command line option enables the user to give the Data Dictionary full file instead of the index file as an input. When this is given, kwvtool will make the index file internally by making a system call to the make_index program. It will then use this internally created index file to continue verifying the labels. For this reason, the make_index program (version 1.7 or higher) must be located in the same directory as kwvtool or the path to the program must be set in the PATH variable of the user system.
If there was an error in creating the Data Dictionary index file, then it will display something like the following:
**************Creating
Data Dictionary index file**************
PDS Make Dictionary
Index - Version 1.7
The data dictionary
file you specified does not exist or cannot be read.
The data dictionary
file you specified does not contain any object or element de
finitions.
Data Dictionary index
file could not be created, Verification will cease
Check that the Data
Dictionary full file exists in the directory path you have specified
If this happens, make sure that the correct path to the Data
Dictionary Full file is specified and make sure that the make_index.
Either –d or –df is a mandatory parameter. Kwvtool will not run if
either parameter is not specified.
The default is for kwvtool to search all subdirectories for labels to process. This option will prevent kwvtool from searching all subdirectories for labels to be processed. The search will begin in the directory specified with the file name(s) in the -f option or in the current directory if no path is specified. The -di option only works for DOS at this time.
Default: None.
Off Switch: None. Specify a label file to be processed.
The -f command line option specifies one or more files are to be processed. Kwvtool will search all subdirectories of the pathname of the file specified in this option. To disable this searching feature (in DOS), use the -di option.
Specify just one filename:
kwvtool -f
mylabel.lbl . . .
Or a wildcard pattern:
kwvtool
–f c:\home\joeuser\*.lbl . . . (in unix -f
/home/joeuser/”*.lbl”)
Or a series of filenames and / or wildcard patterns:
kwvtool -f
/home/joeuser/*.lbl *.fmt catalog.lbl
In the last case, all files with extension “.lbl” in the directory /home/joeuser, all files in the current directory with the extension “.fmt”, and the file “catalog.lbl” in the current directory will be processed by kwvtool. The directory specifications and wildcards should be those expected by the particular operating system.
Default: No file list to check.
This option allows the user to specify a file containing a list of files to be processed. The format of the file is one file name per line, path is assumed to be included if the file to be processed is not in the current working directory.
Default: no file.
The file contains a list of directories the user does not want to be checked for files to process. Each directory is on a separate line, names are case sensitive. There can be up to 30 entries. Each directory file specification must be less than 100 characters in length. A sample contents of a file is:
D:\SOFTWARE
D:\CATALOG
D:\INDEX
-ivf <file name>
Any file with an extension in this list will not be processed. Some types of file extensions that might be included in this list are: c, h, txt, img. The list is a string of file extensions that are comma separated. The program is case sensitive. The list may not contain blanks and can only be 200 characters in length. Lower case characters can be included in the string. A sample string is:
C,H,TXT,IMG,c,h
This string of characters tells kwvtool not to validate any file having an extension of C (or c), H (or h), TXT, or IMG. The string is saved in an ASCII file with a name of choice.
Default: Do not create a log file.
Directs KWVTOOL to keep a log file of all files that are skipped because of the
-ivf, -ivd, or -l3d options and specifies a file name for the log.
Default: -na
This option disables alias substitution. To turn on alias substitution, use the -a option.
Default: none, a report file must be specified.
The -r command line specifies the name of the report file to which kwvtool writes the report.
kwvtool -r
/home/joeuser/label.rpt ...
or
kwvtool –r
c:\pds\testarea\label.rpt…
The –sf option specifies a file containing keywords to be skipped. Keywords are comma separated in a single string not to exceed 200 characters, do not include any blanks. The program is case sensitive.
The default is to make sure that the file being validated is
in fact a label. Specifying this option will make kwvtool not check what file
is being validated.
The data dictionary index is incorrect or missing
KWVTOOL will not proceed without a data dictionary.
There must either be a pdsdd.idx and the associated pdsdd.full or the equivalent must be specified in the command line.
KWVTOOL now gives the user the option to input either the pdsdd.idx file or the pdsdd.full file. If the full file is desired, you will need to use the -df option instead of the -d option like so,
kwvtool
-df c:\data\pdsdd.full
KWVTOOL uses the MAKE INDEX program (version 1.7 or higher) to create the data dictionary index file in this case. Because of this dependency, you must make sure that the MAKE INDEX program is in the same directory as the KWVTOOL program or that the PATH environment is set properly to where the MAKE INDEX program can be located.
Additional Changes:
are/are not in the data dictionary. Originally, it was in this format, but was
changed in version 2.0, which broke the PDS PERL Tools.
Creating KWVTOOL
requires include files that are supplied as part of lvtool, odlc,
and lablib. It will also require object modules created
from files that are
supplied with
lvtool. It will need the odlc library
and the lablib(not lablib3)
library.
Changes for this
version:
New feature that displays a unit ID
value in the report.
Properly handles local data dictionary
keywords that are greater than
30 characters long
Fixed program to correctly exclude
directories to skip if that option
is set.
Program now removes
"JunkZZZxx" files after execution.
Updated usage description of
"-ivd" in the help of this program.
Made code more robust in its checking
of directories to exclude.
The use of -t may cause the program to go into an endless loop. –nt is unnecessary as it is the default.
The program will not accept the -np option and -p is the default.
There is no difference in the program behavior if the -v or -nv options are used.
The report incorrectly states that keywords will be truncated at 30 characters.
PDSDD.FULL and MAKE_INDEX are not included in the release of KWVTOOL.
TBD
TBD