0000711: Add utility command for a logical compare of 2 files

Notes
(0000874) Falk Reichbott (administrator) 2015-08-07 00:26	The new DIFF command use the all the read capabilities of the CONV command to compare the clear net content of two data sources independent of the data format and representation. This is a comparison of logical identical data, based on the neutral FLAM5 elements. The DIFF function does no cross correlation to limit a difference to a minimum. If one byte or element more or less in the data, then the compare will fail for the whole rest. This command don't support wild cards. As an example, the DIFF command can be used to compare the logical content of tow XML files, where one file is in UTF-16LE and compressed conform to RFC1950/51/52 (GZIP) and the other one is a host data set compressed and encrypted in EBCDIC as FLAMFILE member. The DIFF command builds a neutral element list (UTF-8) and compares the parsed XML elements. The following features are supported: * Different kinds of remote access protocols (IP-Native, MQ-Series, SSH) * Different kinds of data sources (Files, Streams, Tables) * Different kinds of archive formats (ZIP, FLAMFILE, TAR) * Different kinds of I/O methods (Binary, Character, Text, Record, FLAM) * Different data encodings (HEX (Base16), Base32, Base64) * Different encryption methods (FLAM, PGP) * Different compression methods (GZIP, BZIP2, XZ) * Automatic decoding of data encodings during read operations * Auto-detection and decoding of length fields in data blocks * Auto-detection of character sets and language based conversion * Powerful character conversion based on CCSIDs * Different logical data formatting (BIN, TEXT, XML, ...) Depending on the data formats, a dedicated method is used, to compare the logical data records or streams. If a compare of the data formats makes no sense (XML with records or blocks) then an error is issued. For example: If you use read.file='filename' and compare.file='filename' and the first file is binary and the second file is the compressed version of this binary data, then a list of binary blocks with different length are build. In this case it makes no sense to compare the elements (this are several block of different size). In this case we compare the logical data stream and not the element list. If the first file are records from a host data set and the second file contains text records with rest elements (suppressed trailing white spaces and the delimiter) then we compare only the net record elements. If the first file block oriented then we include the remaining rest element from the text parser in a data stream compare. With the switch 'FULLY' you can activate additional checks depending on the data formats, for example: * TEXT/TEXT: Compare the remaining rest (trailing whitespace and delimiter) * ASA-RECORD/ASA-RECORD: Compare additional the control character * REL-RECORD/REL-RECORD: Compare additional the slot numbers * XML/XML: Compare additional comments, formatting (whitespaces) and DTD All these additional checks are on top of the compare of the logical net data. If you want to verify if the data still the same independent of the platform and representation, then don't use this switch. If you have defined a report file and you activate the switch 'OFFSET' then byte offsets and byte values for each difference are written to the report file. ATTENTION: This can cause in a very large report file. To get syntax information, please use: ----------------------------------------------------------------------- &{pgm} SYNTAX DIFF ----------------------------------------------------------------------- To get help for a parameter, please use: ----------------------------------------------------------------------- &{pgm} HELP DIFF.parameter[.parameter[...]] ----------------------------------------------------------------------- To read the manual page for a parameter, please use: ----------------------------------------------------------------------- &{pgm} MANPAGE DIFF.parameter[.parameter[...]] or &{pgm} HELP DIFF.parameter[.parameter[...]] MAN ----------------------------------------------------------------------- To generate the user manual for the command, please use: ----------------------------------------------------------------------- &{pgm} GENDOCU DIFF=filename ----------------------------------------------------------------------- Parameters can be defined via command line (directly or by parameter file) or via properties taken from the corresponding property file .EXAMPLE ----------------------------------------------------------------------- &{pgm} DIFF read.file='./swift.txt.gz' compare.file='~.swift.fba' report='out.txt' ----------------------------------------------------------------------- This test reads text records and remaining rest elements from a text file which was compressed and for example encoded in ASCII-CP1252 and contains 0x0D0A as record delimiter. On the other side a FBA host data set in EBCDIC is read and the ASA control character are detached as attributes and trailing spaces are removed. Both files are converted to UTF-8 and only the net record element are compared. ----------------------------------------------------------------------- &{pgm} DIFF read.xml(file='test.xml.bz') compare.xml(file='test.xml') report='out.txt' fully offset ----------------------------------------------------------------------- This example requires and compares to XML files including comments, DTD white spaces element by element and the report file contains the byte offsets for each difference. If you don't use the 'FULLY' switch, then a data stream compare is done, only start tags, stop tages, attributes and the corresponding values and tag data elements are compared and all element type changes must be in sync. ----------------------------------------------------------------------- &{pgm} DIFF read.text(file='./test.txt.xz' suptws) compare.record(file='~.test.fba' suppad) report='~.test.out' offset ----------------------------------------------------------------------- The example reads a UNIX text from a LZMA compressed file and compares the text records with records from a FBA data set. That this compare will work the trailing white spaces on the UNIX file and the padding characters on the FIX blocked host file must be removed. The ASA control character is detached by default, both text data are converted to UTF-8 and only the net record data are compared. To simplify the use of this compare feature we provide a sample utility program SCFCUDIF. This program takes options and a list of file names from command line for a logical compare of the resulting net data after decoding, decryption, decompression, character conversion and formatting in neutral FLAM elements. It use the FLUCDIFF subprogram and write the report to STDOUT. The first file is compared with all remaining files in the list and the program stops at the first error. ----------------------------------------------------------------------- Required parameter: :> SCFCUDIF "[-f] [-o] [-m] [-r] [reportfile] filename1 filename2 [filename3 ... filenameN]" Options: -f Make a fully compare of data or elements -o Print byte offsets and values to report file -m Minimize to protocol output on STDERR -r The first given filename is used for the report file ----------------------------------------------------------------------- You can use this utility program to compare each kind of supported file format. It supports comma separated list, so that it can simply used on mainframe system. The example below is taken from the JOBLIB(SCFCUDIF) member of your FLAM installation. ----------------------------------------------------------------------- //SCFCUDIF JOB 'SCFCUDIF',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,REGION=0M //* //-------------------------------------------------------------------- // //* SAMPLE JOB TO CALL THE SCFCUDIF SAMPLE PROGRAM //* //* THE SCFCUDIF SAMPLE PROGRAM USES THE FLUC SUB PROGRAM DIFF //* FOR A LOGICAL DATA COMPARE. FOR THIS, YOU CAN PROVIDE OPTIONS AND //* YOU MUST SUPPLY TWO OR MORE FILE NAMES. TO SEE THE SYNTAX AND ALL //* OPTIONS RUN THE PROGRAM WITHOUT PARAMETER. THE COMPLETE HELP IS //* WRITTEN TO SYSPRINT. //* //-------------------------------------------------------------------- //SCFCUDIF EXEC PGM=SCFCUDIF,REGION=0M, // PARM=('-o,DD:INPUT,~.odat.psfba.gz') //STEPLIB DD DSN=&SYSUID..FLAM.LOAD,DISP=SHR //SYSOUT DD SYSOUT= //SYSPRINT DD SYSOUT=* //INPUT DD DSN=TST.IDAT.PSFBA,DISP=SHR //*-------------------------------------------------------------------- ----------------------------------------------------------------------- The sample program SCFCUDIF is available on all platforms and the source is part of each FLAM installation (sample or SRCLIBC). You can simply adjust this program and use the provided build instructions (makefile or SBUILD job) to make your own utility.

Issue History
Date Modified	Username	Field	Change
2015-07-31 15:42	Falk Reichbott	New Issue
2015-07-31 15:42	Falk Reichbott	Status	new => assigned
2015-07-31 15:42	Falk Reichbott	Assigned To	=> Falk Reichbott
2015-08-05 19:32	Falk Reichbott	Product Version	5.2 => 5.1.07
2015-08-05 19:32	Falk Reichbott	Target Version	5.1.07 => 5.1.08
2015-08-05 19:32	Falk Reichbott	Description Updated	View Revisions
2015-08-07 00:26	Falk Reichbott	Note Added: 0000874
2015-08-07 00:26	Falk Reichbott	Status	assigned => resolved
2015-08-07 00:26	Falk Reichbott	Fixed in Version	=> 5.1.08
2015-08-07 00:26	Falk Reichbott	Resolution	open => fixed

Relationships