Planetary Data System

Key Word Value Tool (KWVTOOL) User’s Guide (UG)

 

 

 

 

 

 

Draft Release Version 1

August 05, 2005

 

 

 

Prepared by:  Michael Cayanan/Dale Schultz

 

Document Custodian:  Valerie Henderson                                          

 

 

 

 

 

Approved by:

 

 

________________________________________________

Dan Crichton

PDS Project Manager

 

 

 

 

 

 

 

 

 

 

 

 


Jet Propulsion Laboratory

Pasadena, California

 


CHANGE LOG

                       

 

 

Revision

Date

Description

Author

Start Draft

May, 2004

Initial draft

D. Schultz

Release 1

 

Draft Release 1 – Edited all

V. Henderson

 

August 2005

Update Document

M. Cayanan

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Table of Contents

1.0       Introduction.. 1

1.1       Software Identification.. 1

1.2       Purpose.. 1

1.3       Document Scope.. 1

1.4       Audience.. 1

1.5       Overview... 1

1.6       Applicable Documents.. 2

2.0       Getting Started.. 3

2.1       PDS KWVTOOL Setup Process.. 3

2.2       Installing the KWVTOOL. 3

2.3       Starting kwvtool. 4

2.4       Getting Help.. 4

2.5       Using KWVTOOL – Examples.. 5

2.5.1                 Processing One File. 5

2.5.2                 Process a Group of Files. 5

2.5.3                 Using a New Data Dictionary. 6

2.5.4                 An example of a KWVTOOL report 6

3.0       What KWVTOOL does.. 9

4.0       Command Reference.. 9

4.1       –a   Enable Aliasing.. 9

4.2       –d   Enable Data Dictionary index file.. 10

4.3       –df Enable Data Dictionary full file.. 10

4.4       –di   Do not search subdirectories for label files to be processed. 11

4.5       -f  Specify Label Files(s) 11

4.6       –fin <file name> Specifies name of file containing list of file names to process.. 12

4.7       –ivd <file name> Specifies a file containing directories to skip. 12

4.8       -ivf <file name> Specifies name of file containing a list of file extensions to skip   12

4.9       -lef  <file name>  Create a Log File of all Skipped Files.. 12

4.10    -na   Disable Aliasing.. 12

4.11    -r  Specify Report File Name.. 13

4.12    -sf  Specify Keywords to Skip.. 13

4.13    –nol3d Do Not Check For Level 3 Labels in Input Files.. 13

5.0       KWVTOOL Messages.. 13

Appendices.. 13

Appendix A - Release Notes.. 13

Version 2.2 Release Notes.. 14

Version 1.9 Release notes.. 14

Version 1.9 Known Bugs.. 15

Appendix B - Acronyms.. 15

Appendix C- Glossary.. 15

 

 

 

 

 

 


1.0           Introduction

 

1.1             Software Identification

 

Project name:                        Planetary Data System (PDS)

Program set name:              PDS Key Word Value Extraction Tool

Abbreviation:                         kwvtool

Version number:                    2.2

Platforms supported:            Windows, Solaris, Linux

 

 

1.2             Purpose

This document provides information on the use of the PDS KWVTOOL software set. This software creates a list of all keywords and their values found in a PDS label file or in a group of PDS label files.

 

1.3             Document Scope  

This document provides:

 

  • A description of the components of the PDS KWVTOOL software set
  • Information on how to install and run the software
  • A description of the command line parameters
  • A description of the messages that that result from the extraction process.

 

1.4             Audience

This document is intended for those who are responsible for ensuring that labels submitted to the PDS conform to PDS standards. This document assumes familiarity with the PDS data preparation process and label design requirements. The data preparation process is described in the PDS Data Preparation Workbook (JPL Document D-7669, Part 1); label file requirements are described in full in Chapter 5 of the PDS Standards Reference (http://pds.jpl.nasa.gov/documents/sr/Chapter05.pdf). See Section 1.6.

 

 

 

1.5             Overview

 

PDS labels are written in the Object Description Language (ODL). ODL consists of a series of lines of the form "keyword = value", with certain keywords (for example, OBJECT) being used to delimit named groups of keywords within the label. For a description of the PDS implementation of ODL, see Chapter 12 of the PDS Standards Reference (http://pds.jpl.nasa.gov/documents/sr/Chapter12.pdf).

 

The keywords and objects are themselves defined in the Planetary Science Data Dictionary (PDSDD). The Dictionary exists as a Word file for human readers, as a text file for use by the software, and is available via an online look up tool at (http://pds.jpl.nasa.gov/tools/data_dictionary_lookup.cfm).

 

The PDS KWVTOOL reads PDS labels, extracts all keyword/keyword value pairs, deletes any duplicates, and creates a two-part report of the unique pairs.  The list is divided into one section for keywords not found in the data dictionary and one part for keywords that are found in the data dictionary.

 

1.6             Applicable Documents

 

 

·        Planetary Science Data Dictionary Document, August 28, 2002, Planetary Data System, JPL D-7116, Rev. E. (http://pds.jpl.nasa.gov/documents/psdd/psdd.pdf)

 

·        Planetary Data System Data Preparation Workbook,  JPL D-7669, Part 1. (http://pds.jpl.nasa.gov/documents/dpw/index.html)


 

2.0           Getting Started

2.1             PDS KWVTOOL Setup Process

The PDS KWVTOOL software set consists of the following files:

 

Windows

kwvtool.exe - executable

pdsdd.full  - Data Dictionary full file

make_index.exe – utility that creates the pdsdd.idx

 

Solaris and Linux

lvtool – executable

pdsdd.full  - Data Dictionary full file

make_index - – utility that creates the pdsdd.idx

 

PDSDD.FULL and MAKE_INDEX are included in the ddict distribution package. PDSDD.FULL will change often and should be downloaded frequently from: http://pds.jpl.nasa.gov/tools/software_download.cfm.  All of the central node validation software tools are also available at this site.  Each time PDSDD.FULL is downloaded it is important to run MAKE_INDEX again to create a new index.  Once at that url select “for data producers”.  When the new url opens scroll down the Software Id/Version ID window until the desired software tool is highlighted.  Click on the selected tool once and select the submit button.  When the new information is displayed click once on the appropriate download.

 

PDSDD.FULL can be obtained by scrolling through the Software Id/Version ID window to pdsdd(CAT1R40):PDS Toolbox Data Dictionary Version *****(the version number will be changed from time to time and is not shown here).  Select MULTIPLE from the download list.

 

MAKE_INDEX can be obtained by scrolling through the Software Id/Version ID window to make_index, selecting it, and then selecting the submit button.

Click once on the appropriate download.

 

2.2             Installing the KWVTOOL

Download the zip or tar file for kwvtool, make_index, and data dictionary file to the hard drive.

 

Use make_index to make pdsdd.idx by typing:

 

make_index pdsdd.full

 

 

2.3             Starting kwvtool

The PDS KWVTOOL is a command line tool and will need to be run from a command prompt.

 

To invoke kwvtool from the same directory in which both kwvtool and the default Data Dictionary files are located, enter the command:

 

kwvtool –d pdsdd.idx -f <input-file> -r <output-file>

 

where <input-file> is the name of the label file to process, and <output-file> is the name of the report file to be created.  If the kwvtool cannot locate the Data Dictionary files it needs, it will generate an error message.

 

Kwvtool may also be started from a different directory that the one in which kwvtool and the Data Dictionary files are located.  For example, on a Unix system, if the kwvtool and dictionary files are located in directory /usr/local/pds, enter the command:

 

/usr/local/pds/kwvtool -f <input-file> -r <report-file> -d

                                                                              /usr/local/pds/pdsdd.idx

 

The -d option tells kwvtool the path and name of the PDS Data Dictionary index file. 

 

If using the Data Dictionary full file, then use the –df option instead of –d:

 

/usr/local/pds/kwvtool -f <input-file> -r <report-file> -df

                                                                              /usr/local/pds/pdsdd.full

 

The –df option tells kwvtool the path name of the PDS Data Dictionary full file. You will need to also make sure the make_index program (version 1.7 or higher) is located in the same path as kwvtool since it uses that to create the index file internally when given the full file as an input.

 

2.4             Getting Help

To get a brief description of the command line parameters, at the command prompt type:

 

            kwvtool

 

The parameters are also explained in detail in the COMMAND REFERENCE section of this document.

 

For further help or information, contact the PDS Operator at pds_operator@jpl.nasa.gov

 

 

2.5             Using KWVTOOL – Examples

The following examples will help in using kwvtool immediately. The chapter, “What the KWVTOOL Does,” explains the key word value extraction process in more detail.

 

While reviewing the following examples, refer to the “Command Reference” chapter, which explains the command line options available for kwvtool, and the “Kwvtool Messages” chapter, which explains the error and warning messages generated by kwvtool.

 

The following examples assume that both kwvtool and the Data Dictionary reside in the current directory.

 

Note: In all cases below, you can substitute the full file instead of the index file by using the –df argument instead of –d:

 

            kwvtool –df pdsdd.full….

 

Remember to have make_index version 1.7 or higher residing in the same directory as kwvtool.

 

2.5.1       Processing One File

 

Assume there is a label file called “example1.lbl” as shown below:  To process that one file enter the command:

 

kwvtool –d pdsdd.idx -f example1.lbl -r myreport.rpt

 

This tells kwvtool to compare the keywords in example1.lbl against the keywords in pdsdd.idx and to write the report to myreport.rpt.

 

2.5.2       Process a Group of Files

 

If there are several label files in the directory, enter the command:

 

            kwvtool –d pdsdd.idx -f *.lbl -r myreport.rpt

 

This tells the kwvtool to extract the keywords from all the files that have the extension of lbl in the current directory, compare them against the keywords in pdsdd.idx and to write the report to myreport.rpt.  Kwvtool was used in this fashion on a directory containing a large number of label files to generate the following report:

 

 

2.5.3       Using a New Data Dictionary

 

Assume that the existing Data Dictionary contains too general a set of rules for the object class TABLE.  The generic, PDS-supplied TABLE object class includes an optional keyword, DESCRIPTION, that should be required in all of the labels.  The Data Dictionary may be modified according to the instructions in the “Toolbox Data Dictionary” chapter of the PDS Toolbox Overview and define a new object class, called FEATURE_TABLE, which requires the presence of the keyword DESCRIPTION in every FEATURE_TABLE object.  Enter the command:

 

kwvtool -f example1.lbl -r newreport.rpt -d mydd.idx

 

to tell kwvtool: “Process file ‘example1.lbl’ again and write the results to file ‘newreport.rpt’, only this time use the Data Dictionary index file ‘mydd.idx’”.  (The file “mydd.idx” must be a Data Dictionary index file generated by the make_index program.) 

 

 

2.5.4       An example of a KWVTOOL report

 

 

____________________________________________________________

************************************************************************

*                                                                      *

*               Planetary Data System KWVTOOL                          *

*               Selected Keywords and Values                           *

*                     Extractor Version 1.9                            *

*                                                                      *

*                      Mon May 03 18:18:59 2004                        *

*                                                                      *

*       Data Dictionary Name: pdsdd.idx                                *

*    Data Dictionary Version not found                                 *

*  Data Dictionary Generated: October 20, 2003                         *

*                                                                      *

*                  Data Dictionary Validation is ON                    *

*                                                                      *

* WARNING!                                                             *

*----------------------------------------------------------------------*

* Keywords will be truncated at 30 characters                          *

* values will be truncated at 95 characters                            *

* No indication will be given that truncation has occurred             *

************************************************************************

 

WARNING!  These selected keywords are not in the Data Dictionary

 SOURCE_IMAGE_ID                    383B23

 SOURCE_IMAGE_ID                    383B42

 SOURCE_IMAGE_ID                    421B21

 SOURCE_IMAGE_ID                    421B23

 SOURCE_IMAGE_ID                    421B73

 SOURCE_IMAGE_ID                    421B74

 SOURCE_IMAGE_ID                    421B75

 SOURCE_IMAGE_ID                    421B76

 SOURCE_IMAGE_ID                    421B77

 TYPE                               BDV

 TYPE                               VICAR2

 

----------------------------------------------

 

 

These selected keywords are in the Data Dictionary

 

 BYTES                              1024

 BYTES                              12

 BYTES                              13

 BYTES                              2000

 BYTES                              4

 BYTES                              5

 BYTES                              5000

 BYTES                              9

 FILE_RECORDS                       1001

 FILE_RECORDS                       1026

 FILE_RECORDS                       175

 FILE_RECORDS                       180

 FILE_RECORDS                       2048

 FILE_RECORDS                       2128

 FILE_RECORDS                       2129

 FILE_RECORDS                       23

 FILE_RECORDS                       2396

 FILE_RECORDS                       2917

 FILE_RECORDS                       310

 FILE_RECORDS                       512

 FILE_RECORDS                       56

 FILE_RECORDS                       602

 FILE_RECORDS                       720

 FILE_RECORDS                       806

 FILE_RECORDS                       809

 IMAGE_ID                           E2W3599

 IMAGE_ID                           EJZ1928

 IMAGE_ID                           F-MIDR.20S145;1

 IMAGE_ID                           MG88S045

 LINE_FIRST_PIXEL                   1

 

************************************************************************

*                                                                      *

*        This is the end of the Keyword / Value Tool Report            *

*                                                                      *

*                      Mon May 03 18:19:02 2004                        *

*                                                                      *

************************************************************************

 

                        Report File “myreport.rpt “

 

This report was several hundred lines long and has been edited to make it smaller.  In the first section there were 9 occurrences of SOURCE_IMAGE_ID.  Actually there may have been more but any duplicates were deleted by kwvtool.  There were only 2 unique occurrences of TYPE.  It is possible in this case that 9 different values for the same keyword is legitimate but that may not always be true.  This provides a list of all the values for each keyword in the processed label files. This first section of the report also identifies SOURCE_IMAGE_ID and TYPE as keywords that are not in the data dictionary.

 

The second section contains a list of keywords that were in the data dictionary along with the unique keyword values of each.

 

 

 


 

3.0           What KWVTOOL does

KWVTOOL makes a list of all keywords and the keyword value.  This list is then alphabetized. The keywords in the list are checked against the data dictionary and those that are not found are flagged.  The keywords that are flagged will appear in the section of the report that is for keywords not found in the data dictionary.  The remaining keywords will be placed in the report section for keywords that were found in the data dictionary.  If a keyword/keyword value pair is a duplicate it will be deleted.  The resulting report will contain a complete list of all keywords found in the label(s) and all of the values that were found for each keyword.

 

4.0           Command Reference

4.1             –a   Enable Aliasing

Default: -na (no aliasing)

 

The -a command line option enables the aliasing feature of kwvtool.  This option is useful when processing a label which was developed according to another Data Dictionary, or to identify out-of-date keywords and objects in the label.

 

When the -a option is turned on, kwvtool will compare the object classes in the label with the list of object class aliases in the Data Dictionary, and replace them with their official names.  Object classes that were changed are not identified in the report file.  An example of such a change is:

 

OBJECT = TABLE_STRUCTURE

 

which will be replaced by

 

OBJECT = TABLE

 

if TABLE_STRUCTURE is defined to be an alias for object class TABLE in the Data Dictionary.

 

Kwvtool will then compare the keywords in the label with the list of keyword aliases in the Data Dictionary and replace them with their official names.

 

After aliasing is finished, kwvtool will continue identifying all keywords and values, using the official names instead of the aliases when it searches the Data Dictionary.

 

Note that only full class names will be replaced.  For example, EDR_TABLE_STRUCTURE will not become EDR_TABLE just because TABLE_STRUCTURE is defined as an alias for TABLE.  The same is true for the keyword replacement.  Remember that kwvtool does not change the actual label file at all.  The changes to the label remain in effect only for the duration of the kwvtool run.

 

4.2             –d   Enable Data Dictionary index file

 

Default: -d pdsdd.idx

 

Kwvtool compares the keywords found in the label file(s) with a data dictionary.  Kwvtool will then write a report identifying those keywords that were not in the data dictionary as well as those that were.

 

The default is -d pdsdd.idx.  Kwvtool will look for pdsdd.idx if no argument is given.  If it can not fine the file the program will issue an error message and stop.  If another data dictionary is to be used it can be specified with this parameter.  If another data dictionary is specified be sure run make_index with the alternate dictionary.  The following command would tell kwvtool to use mydd.idx as an index into a data dictionary.  Make_index will put the name of the actual data dictionary at the head of the index file.

kwvtool -d mydd.idx  -f  *.lbl  -r myreport.rpt

 

4.3             –df Enable Data Dictionary full file

 

Default: -df pdsdd.full

 

The –df command line option enables the user to give the Data Dictionary full file instead of the index file as an input. When this is given, kwvtool will make the index file internally by making a system call to the make_index program. It will then use this internally created index file to continue verifying the labels. For this reason, the make_index program (version 1.7 or higher) must be located in the same directory as kwvtool or the path to the program must be set in the PATH variable of the user system.

 

If there was an error in creating the Data Dictionary index file, then it will display something like the following:

 

**************Creating Data Dictionary index file**************

PDS Make Dictionary Index - Version 1.7

 

The data dictionary file you specified does not exist or cannot be read.

The data dictionary file you specified does not contain any object or element de

finitions.

 

Data Dictionary index file could not be created, Verification will cease

Check that the Data Dictionary full file exists in the directory path you have specified

 

If this happens, make sure that the correct path to the Data Dictionary Full file is specified and make sure that the make_index.

 

Either –d or –df is a mandatory parameter. Kwvtool will not run if either parameter is not specified.

 

4.4             –di   Do not search subdirectories for label files to be processed.

The default is for kwvtool to search all subdirectories for labels to process. This option will prevent kwvtool from searching all subdirectories for labels to be processed.  The search will begin in the directory specified with the file name(s) in the -f option or in the current directory if no path is specified. The -di option only works for DOS at this time.

 

4.5             -f  Specify Label Files(s)

 

Default:  None.

Off Switch: None. Specify a label file to be processed.

 

The -f command line option specifies one or more files are to be processed. Kwvtool will search all subdirectories of the pathname of the file specified in this option.  To disable this searching feature (in DOS), use the -di option.

 

Specify just one filename:

 

kwvtool -f mylabel.lbl . . .

 

Or a wildcard pattern:

 

kwvtool –f  c:\home\joeuser\*.lbl . . .  (in unix -f  /home/joeuser/”*.lbl”)

 

Or a series of filenames and / or wildcard patterns:

 

kwvtool -f /home/joeuser/*.lbl *.fmt catalog.lbl

 

In the last case, all files with extension “.lbl” in the directory /home/joeuser, all files in the current directory with the extension “.fmt”, and the file “catalog.lbl” in the current directory will be processed by kwvtool.  The directory specifications and wildcards should be those expected by the particular operating system.

 

4.6             –fin <file name> Specifies name of file containing list of file names to process

Default: No file list to check.

 

This option allows the user to specify a file containing a list of files to be processed.  The format of the file is one file name per line, path is assumed to be included if the file to be processed is not in the current working directory.

 

4.7             –ivd <file name> Specifies a file containing directories to skip.

Default: no file.

 

The file contains a list of directories the user does not want to be checked for files to process.  Each directory is on a separate line, names are case sensitive.  There can be up to 30 entries.  Each directory file specification must be less than 100 characters in length. A sample contents of a file is:

 

D:\SOFTWARE

D:\CATALOG

D:\INDEX

 

4.8             -ivf <file name> Specifies name of file containing a list of file extensions to skip

-ivf <file name>

 

Any file with an extension in this list will not be processed.  Some types of file extensions that might be included in this list are: c, h, txt, img. The list is a string of file extensions that are comma separated. The program is case sensitive.  The list may not contain blanks and can only be 200 characters in length.  Lower case characters can be included in the string.  A sample string is:

            C,H,TXT,IMG,c,h

This string of characters tells kwvtool not to validate any file having an extension of C (or c), H (or h), TXT, or IMG.  The string is saved in an ASCII file with a name of choice.

 

4.9             -lef  <file name>  Create a Log File of all Skipped Files

Default: Do not create a log file.

 

Directs KWVTOOL to keep a log file of all files that are skipped because of the

-ivf, -ivd, or -l3d options and specifies a file name for the log.

 

4.10         -na   Disable Aliasing

Default: -na

 

This option disables alias substitution.  To turn on alias substitution, use the -a option.

 

4.11         -r  Specify Report File Name

 

Default: none, a report file must be specified.

 

The -r command line specifies the name of the report file to which kwvtool writes the report.

 

kwvtool -r /home/joeuser/label.rpt ...

or

kwvtool –r c:\pds\testarea\label.rpt…

 

4.12         -sf  Specify Keywords to Skip

 

The –sf option specifies a file containing keywords to be skipped.  Keywords are comma separated in a single string not to exceed 200 characters, do not include any blanks.  The program is case sensitive.

 

4.13         –nol3d Do Not Check For Level 3 Labels in Input Files

 

The default is to make sure that the file being validated is in fact a label. Specifying this option will make kwvtool not check what file is being validated.

 

 

5.0           KWVTOOL Messages

 

 

The data dictionary index is incorrect or missing

KWVTOOL will not proceed without a data dictionary.

 

There must either be a pdsdd.idx and the associated pdsdd.full or the equivalent must be specified in the command line.

 

Appendices

Appendix A - Release Notes

 

Version 2.2 Release Notes

 

 KWVTOOL now gives the user the option to input either the pdsdd.idx file or the pdsdd.full file. If the full file is desired, you will need to use the -df option instead of the -d option like so,

 

               kwvtool -df c:\data\pdsdd.full

 

KWVTOOL uses the MAKE INDEX program (version 1.7 or higher) to create the data dictionary index file in this case. Because of this dependency, you must make sure that the MAKE INDEX program is in the same directory as the KWVTOOL program or that the PATH environment is set properly to where the MAKE INDEX program can be located.

 

Additional Changes:

 

  •  Modified output report to indent once it starts reporting keywords that

            are/are not in the data dictionary. Originally, it was in this format, but was

            changed in version 2.0, which broke the PDS PERL Tools.

 

  • Fixed a bug where KWVTOOL would crash when checking a file against a directory the user wants to exclude from processing

 

  • Fixed a bug to prevent program crash when a keyword value is longer than 250 characters

 

Version 1.9 Release notes

Creating KWVTOOL requires include files that are supplied as part of lvtool, odlc,

and lablib.  It will also require object modules created from files that are

supplied with lvtool.  It will need the odlc library and the lablib(not lablib3)

library.

 

Changes for this version:

 

         New feature that displays a unit ID value in the report.

        

         Properly handles local data dictionary keywords that are greater than

         30 characters long

         

         Fixed program to correctly exclude directories to skip if that option

         is set.

        

         Program now removes "JunkZZZxx" files after execution.

        

         Updated usage description of "-ivd" in the help of this program.

        

         Made code more robust in its checking of directories to exclude.

        

Version 1.9 Known Bugs

The use of -t may cause the program to go into an endless loop.  –nt is  unnecessary as it is the default.

The program will not accept the -np option and -p is the default.

There is no difference in the program behavior if the -v or -nv options are used.

The report incorrectly states that keywords will be truncated at 30 characters.

PDSDD.FULL and MAKE_INDEX are not included in the release of KWVTOOL.

Appendix B - Acronyms

TBD

 

Appendix C- Glossary

TBD