FLCREC-API
FLUC Record Interface
FLUC Record Interface

This interface provides record- or element-oriented sequential read and write access to original files. Original files can be binary, text or XML data sets as well as GZIP-, BZIP2-, XZ-, PGP-, ZIP, FLAM4 files or any other kind of original local or remote data set format supported by FLAM. It includes all conversion and formatting capabilities of the Frankenstein Limes Universal Converter (FLUC). These conversion capabilities are also available as an in-memory variant (FCRCONV/RUNV/FINV) without file I/O. The COBOL sample 'SOFCRIPS' shows how it can be used to convert an XML transaction to a copy book (second FCRCONP call) for SEPA instant payments.

The interface provides functions similar to the record-oriented file I/O functions used with COBOL or PL/1 on mainframes. Only the function for opening files differs in that it takes a file, a format and a state string. You can specify them using the syntax of our command line (FLCL). The FLAM return code (rtc) can be mapped to a message with FCRMSG() and the complete FLAM error trace can be fetched with FCRTRC(). Additionally, this interface provides functions for getting help and syntax information for the file and format string parameters.

There is also a logging facility which causes log messages to be collected in memory. These log messages can be queried when desired.

Additionally, a set of conversion functions is available, which can be used when accessing elements (format.element()), for example to parse XML documents in COBOL or PL/I. An example can be found in a section below.

Micro Focus EDZ support

The interface does support Micro Focus Enterprise Server for SystemZ (MF-EDZ). It is usable on Linux and Windows, 32 and 64 bit systems with Cobol and PL1.

There are some special functions implemented, mainly to provide a possibility for a workaround concerning the native pointer handling bug if the AMODE directive is required. In this case (e.g AMODE=31), the native pointer provided by the locate functions (FCRLOC(), FCRCONP(), ...) is not usable with the ADDRESS OF statement (invalid pointer value). The pointer is mapped with the corresponding MF-EDZ routine, but the pointer is still in native format. For this case, the function FCRCPY() is provided to implement a transparent workaround.

In the opposite direction, i.e. when writing, FCRPTR() converts a local AMODE=31 pointer to a native pointer to allow, for example, structures with length and pointer for variable length formats in table support.

Often to AMODE=31 is only required to determine job or system names from control blocks. In this case FCRSYM() can be used for it.

For use with Micro Focus Enterprise Server the following environment variables might be used.

FLAM4MF=unset:
The filename needs to be specified as either DD:name or //'name' if the Micro Focus support is needed. Otherwise normal FLAM4 use is done.
If DD:name or //'name' cannot be accessed an error is returned.
FLAM4MF=encoding:
Micro Focus libraries will be used and an error is returned if they are not found.
The encoding is used to automatically convert all input character strings from this to the local system encoding. The output character strings are converted from the local system to this encoding.
The command flcl info get.enc prints a list of all supported encodings.
For convenience the strings :EBCDIC or :ASCII might be used. If the encoding strings begins with 'IBM' big endian is assumed and binary values will be byte swapped.
FLAM4MF=YES:
Micro Focus libraries will be used and an error is returned if they are not found.
No character conversion is done.
FLAM4MF=NO:
Micro Focus support is switched off, standard FLAM4 use.
This must be done in order to work with filenames starting with DD:
FLAM4MF_TRACEFILE=unset
Without trace file name tracing of library calls is omitted.
FLAM4MF_TRACEFILE=filename
trace output of library function calls is written to filename
FLAM4MF_STATIC_SYSVAR=unset
No static system variables are read into the environment.
FLAM4MF_STATIC_SYSVAR=filename
Static system variables are read from filename into the environment (for FCRSYM()).
FLAM4MF_DYNAMIC_SYSVAR=unset
No dynamic system variables are read into the environment.
FLAM4MF_DYNAMIC_SYSVAR=filename
Dynamic system variables are read from filename into the environment (for FCRSYM()).
The second file name can be used in conjunction with the JCL User Exit to provide the dynamic system variables for FCRSYM().
The implementation can be done with this interface using FCROPN(fomat.record() write.text(file=filename ccsid=local))

Interface standards

Each function is provided as a separate load module. All parameters are call-by-reference and function do not have a return value. There are 4 types of parameters:

POINTER:   pointer to an address (usually 32 (PIC S9(9) COMP) or 64 bit)
INTEGER:   pointer to a 32 bit number in two's complement (PIC S9(9) COMP)
STRING[x]: pointer to a byte (8 bit) array of length x (PIC X(x))
STRING:    pointer to a variable length byte (8 bit) array (PIC X(n))

The type INTEGER has local endianness. On mainframes, this is usually big endian. This means that the most significant byte is stored first (i.e. at the lowest memory address) and the least significant byte is stored last. On x86 platforms, the byte order is little endian. This means that the byte order is reversed.

Input strings may be null-terminated in which case the corresponding length pointer may be NULL. If a length value is provided, the string may not be null-terminated. All output strings are null-terminated. The output length is set only if the pointer is not NULL. This is only valid for printable string output. Binary byte arrays are not null-terminated. The return code parameter can be a NULL pointer in which case no return value is set.

File definition and format strings

When reading: Through the file definition string, you specify how the read operation interprets and transforms your original data to produce an internal neutral FLAM5 element list from it. The format string describes how this element list will then be formatted into a sequence of records. You also have the option of getting the raw elements by specifying element formatting in the format string (format.element()). An optional state string is returned when opening a file for reading which contains some metadata about the contents.

When writing: The format string defines how the provided records or elements will be formatted into a neutral FLAM5 element list. The file definition string describes how this element list will be written to a file. The state string can be used to set additional metadata that provides FLUC with additional information about the data. The converted data can be written to multiple targets in parallel by specifying multiple I/Os through the OUTPUT object. It is possible to specify multiple WRITE overlays or OUTPUT objects to write the same data in different formats to different files, reading the input only once.

Formatting

When reading or writing, you can define how to format the data (binary, character, text, xml, ...), use different conversions (Base64, OpenPGP, GZIP, CHRSET, ...) executed sequentially and different kinds of I/O methods (block, record, text, FLAM4, ...). This means that you can read and write compressed, encrypted and encoded files, where you can change the character set and other things as part of the write and read operation.

Through the format string, you specify how the data is transformed into a sequential list of records or elements. For example, if the data contains text, you can specify which character set the output should be in and so on. Please use the interactive help function FCRHLP() for more information.

Beside record (default) and element formatting, you can also choose binary, character, text and XML formatting. An FCRGET() call returns the formatted content of one element. For example, a text record with delimiter or a binary block. Please be aware, that such a block could potentially contain up to 1 GB of data.

For read operations, you can use the auto detection capabilities of FLUC.

FCROPN("read.file='filename'","format.record()")

This results in any kind of file (encoded, encrypted and or compressed) being converted to records. If the content is XML, then the XML data is pretty printed into records. If it is text, then the data is parsed based on the containing delimiters. If it is binary and record lengths are detected, then the records are provided one by one. If it is binary and no record lengths are known or detected, then the data is wrapped into records.

Record formatting

Record formatting is the default format. If the format string does not start with format, the format string is expected to be the inner portion of format.record(). So, the format string reclen=80 is equivalent to format.record(reclen=80). Record formatting will result in the expected results for a record-oriented I/O interface.

Element formatting

FLAM5 elements are parsed data elements with a type, a length, a value and more. If the element data contains printable characters, then these characters are encoded in UTF-8 by default, but can also be converted as needed.

With element formatting, you can read and write a serialized form of FLAM5 elements. For example, it can be used to tokenize an XML document and read these tokens (elements) for further processing. The serialized element format (version 0) is described by the structure FlmElmRec0. It basically consists of a set of 32 bit integers followed by element data and metadata.

Other formatting modes

The formatting modes binary, character, text or XML can also be used. The converted element data field is returned as a record, which could be a line of a text with a delimiter or the whole chunk of binary data as a block. These formatting modes are designed to be used via the byte interface, but available on the record interface as well. Be careful when using these modes as results could be not as expected.

Element conversion

When using element formatting, you can use several individual element data conversion modules for per-element data conversion. If no converter is used, the element data is simply copied into application memory to build the element structure (FlmElmRec0). A set of functions with 'v' at the end of the function name can be used to set a custom element data converter. A converter must be opened before using it, which can done by calling FCROPNV with a corresponding conversion string. The conversion string describes how the data is converted from the neutral format of a FLAM5 element to the representation in the application memory (when reading) or how the application memory must be interpreted to form the corresponding neutral FLAM5 element data type (when writing). The functions FCRGETV, FCRLOCV and FCRPUTV accept an additional conversion handle as parameter which is obtained from FCROPNV() to replace the default/standard conversion. The FCRGETV, FCRLOCV and FCRPUTV functions require to know the data format that will be read in advance, which is not always the case. Instead, you can also use the regular FCRGET, FCRLOC and FCRPUT functions and use the function FCRCONV to convert the data, if needed. In fact, this function works on arbitrary data and may also be used independently.

There is no limit on the amount of element converters that can be opened. The output length of most converts can be controlled in two ways: By passing appropriate length values in the conversion string or or by passing a buffer of appropriate length at call time.

The converter handle must be closed with FCRCLSV to release all associated resources.

Some usage scenarios for element converters:

  • Selective character conversion
  • Number conversion from/to BCD/binary integer
  • Removal of whitespace

Table support

With version 5.1.16 of FLAM the table support was introduced. For the record interface you can activate an end of table support if you read from a file (file string). If you activate ENDOFT in the format string at write and use FCRSTN() in conjunction with an output file name containing [table], then you can split the data in different files. But at read you must now handle the FLMRTC_EOT error like an FLMRTC_EOF. If you don't get data the reason could be EOF or EOT. With EOT the next read gives the records conform to the new table. At EOF you get still no data and still EOF. If you activate ENDOFT at read you can use the new function FCRGTN() to get the name of the current table after FCROPN() or after an EOT was signalled. This can be used to interpret the data correctly if more than one table is in a file. If you want to write more than one table to a file, you can work with the table format detection, but normally it is better to define the table format with the new function FCRSTN(), before you provide the data. If you set the table name the data must match this table format (row specification). The table format detection is disabled if FCRSTN() is used. You can reactivate the automatic table detection at write by calling FCRGTN().

Environment variables

For all default character conversions, it is useful to set the environment variable LANG. Other used environment variables of FLAM can be found in the FLCL manual. With version 5.1.19 a new function was introduced (FCRENV()) to load the FLAM environment . This function can be used before the first API call to establish the same environment used by FLAM utilities, subsystems and so on. This give the application developer the possibility to adjust the environment before the first real call is done. Until version 5.1.18, each opening function has read the system variables on z/OS. This is now part of the FCRENV() function to give complete control about the environment to users of the API. To fetch a symbol from the environment, the function FCRSYM() can be used.

Special EBCDIC code page support

On system using EBCDIC a special support for critical punctuation characters was implemented (see FLCL manual). This support converts the several punctuation character from a certain EBCDIC code page to the local character set defined over the LANG variable (if LANG not defined the default is 1047). Below you can find the list of character with different code points in the different supported EBCDIC code pages, which are part of the first 128 Unicode code points.

   CRITICAL PUNCTUATION CHARACTERS:
      ! $ # @ [ \ ] ^ ` { | } ~
   SUPPORTED EBCDIC CODE PAGES FOR COMMAND ENTRY:
      "IBM-1140","IBM-1141","IBM-1142","IBM-1143",
      "IBM-1144","IBM-1145","IBM-1146","IBM-1147",
      "IBM-1148","IBM-1149","IBM-1153","IBM-1154",
      "IBM-1156","IBM-1122","IBM-1047","IBM-924",
      "IBM-500","IBM-273","IBM-037","IBM-875","IBM-424",
      "IBM-277","IBM-278","IBM-280","IBM-284","IBM-285",
      "IBM-297","IBM-871","IBM-870","IBM-1025","IBM-1112",
      "IBM-1157"

This conversion is required to interpret the command syntax correctly. These CLP strings are the major part used by this interface. To work with this API the user must build such CLP string. For this often literals are used. On EBCDIC systems you can define in which CCSID (code page) the literals are provided by the compiler. For example, in COBOL the default, if the CODEPAGE parameter not defined, is 1140.

Your application could get variables from outside (e.g. file names) in the local character set (e.g. 1141). Your literals are in 1140 and you must build a CLP string with literals and variables. To support this kind of inconsistent code pages (since version 5.1.19 of FLAM) escape sequences (&xxx;) the CCSID areas (&nnnn;...&nnnn;) are supported (see FLCL manual). Below you can find an examples for a CLP command string how to use it.

    WORKING-STORAGE SECTION.
    01  C2-DATA                    PIC X(200)
           VALUE 'CONV.MEMORY(READ.CHAR(TABLE='&TLD;.PAIN008.XMLTAB')
   -            'WRITE.RECORD(TABLE='&TLD;.PAIN008.FIXTAB'))'.
or
    WORKING-STORAGE SECTION.
    01  C2-DATA                    PIC X(200)
           VALUE '&1140;CONV.MEMORY(READ.CHAR(TABLE='~.PAIN008.XMLTAB')
   -            'WRITE.RECORD(TABLE='~.PAIN008.FIXTAB'))'.

An unsupported CCSID (e.g. 0) can be used to define the local character set as code page (default case). To get an area for the literal code page you must add a CCSID escape sequence (&1140;). If you have a variable part, you must switch to the local character set (&0000;) or the correct CCSID for this variable and if the literal continued switch back to the literal CCSID (&1140;). If you need such moving characters in your literals you can use the corresponding escape sequence for it. Below you can find all escape sequences for the critical punctuation characters.

   ! = &EXC;   - Exclamation mark
   $ = &DLR;   - Dollar sign
   # = &HSH;   - Hashtag (number sign)
   @ = &ATS;   - At sign
   [ = &SBO;   - Square bracket open
   \ = &BSL;   - Backslash
   ] = &SBC;   - Square bracket close
   ^ = &CRT;   - Caret (circumflex)
   ` = &GRV;   - Grave accent
   { = &CBO;   - Curly bracket open
   | = &VBR;   - Vertical bar
   } = &CBC;   - Curly bracket close
   ~ = &TLD;   - Tilde

This two feature gives you the possibility to build CLP strings Independent of the EBCDIC code page used for literals and as local or system character set.

Compile and Link

On mainframes (z/OS, ...)

The interface is provided as DLL and each function of the DLL is also available as separate load module.

For dynamic linking of the load module (fetch) the hlq.FLAM.LOAD library must be part of the STEPLIB concatenation. A static link in COBOL or PL1 works as well with the separate load modules.

If you link one of the load modules statically to your application (each load module contains all required entries of the interface), you must include the DLL import files below to resolve the missing external references and set the binder option DYNAM(DLL):

hlq.FLAM.IMPORT(FLCBYT)
hlq.FLAM.IMPORT(FL5CORE)

To link dynamically against the DLL you must include the import DLL file form the IMPORT library below:

hlq.FLAM.IMPORT(FLCRECLB)

Assembler programs need to activate the LE runtime system (PIPI) before any call. The calling convention is standard OS (save area and R1 points to the flagged parameter list (each parameter call be reference, R15 is not used (no return code))).

On other platforms (Unix, Windows, ...)

The interface is only available as dynamic link library (DLL) or shared object (SO). You can link it dynamically, statically or load the library at runtime (dlopen(), LoadLibrary()) with the common DLL/SO mechanism of your operating system.

Sample programs

A sample program in C with name SCFCRCPY can be found as part of the installation package for mainframe systems in the library SRCLIBC(SCFCRCPY), with the corresponding compile and link step in JOBLIB(SBUILD). For other platforms (Windows, UNIX) the sample program source of SCFCRCPY is located in the sample directory and the compile and link procedures can be found in the Makefile of the same directory.

Additionally, COBOL samples with name SOFCRGET (uses FCRGET to read any kind of supported file format and writes the EBCDIC records to a host dataset), SOFCRXML (reads a clear or encoded XML file and writes the content as dump), SOFCREXV (parses an XML file based on several element converters and does operations with it) and SOFCRINT/FLT (reads records from a dataset, converts the strings to several different integer/floats values and calculates the sum) can be found in SRCLIB(SOFCRGET/SOFCRXML/SOFCREXV/SOFCRINT). The corresponding compile and link procedures are also located as separate steps in JOBLIB(SBUILD). SOFCRGET/INT uses record formatting and SOFCRXML/EXV uses element formatting.

Example for reading XML files with COBOL

This simple XML document contains some strings and integer numbers within some nested XML tags:

<foo>
   <address>
      <name>  Max Herre        </name>
      <number>    13           </number>
   </address>
</foo>

The file (FCROPN) can be opened with the file and format strings below. In this example, we use a static allocation.

FILE-STRING:        "read.xml(file='DD:INPUT' nocmnt)"
FORMAT-STRING:      "format.element()"

If the file is compressed/encrypted/encoded in a supported format, it is decoded first. Character set conversion takes place automatically, if necessary. The internal neutral representation of XML elements is always a UTF-8 string. The nocmnt flag suppresses XML comments from the document while reading.

We need a couple of converters which are created with FCROPNV and an appropriate conversions string:

CONVERSION-STRING1: "write.string(chrset(ccsid=DEFAULT))"
CONVERSION-STRING2: "write.string(chrset(ccsid=DEFAULT whitespace=collapse) padding=right)"
CONVERSION-STRING3: "conv.integer(from(format.str()) to(format.bin(signed)))"

The first converter converts an element to a string with the system's default character set. This is used to convert the XML tag names in order to be able to compare them in our code.

The second converter also converts an element to a string with default CCSID, but additionally removes leading, trailing and duplicate successive whitespace and pads the string on the right to fill our output buffer that we pass to the interface functions. This is used to convert the "name" field of the document and store it in a fixed-length data structure.

The last converter is used to convert the string representation of a number to a signed number in two's complement format in system endianness. The output buffer passed must be 1, 2, 4 or 8 bytes long as these are the supported binary integer lengths.

Now that everything is setup, we can start reading XML elements with FCRLOC. If we encounter the start of an XML tag, we use the first converter to retrieve the tag name in our local charset and do a string compare. For the string data in the "name" tag, we use the second converter to store it in a COBOL data structure (COPYBOOK). If the element contains a number, we use the number converter to store the number as 32 bit signed bianry integer as "PIC 9(8) COMP".

Here is the example for reading the above XML file in a kind of simplified pseudo code:

pvFil=FCROPN("read.xml(file='DD:INPUT' nocmnt)","format.element()")
pvKyw=FCROPNV("write.string(chrset(ccsid=DEFAULT))")
pvStr=FCROPNV("write.string(chrset(ccsid=DEFAULT whitespace=collapse) padding=right)")
pvInt=FCROPNV("conv.integer(from(format.str(marker=period)) to(format.bin(signed)))")
uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
if (uiTyp=STARTELM && acKyw=="foo") then begin
   uiTyp=FCRLOC(pvFil)
   if (uiTyp==ENDSTARTELM) then begin
      uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
      if (uiTyp=STARTELM && acKyw=="address") then begin
         uiTyp=FCRLOC(pvFil)
         if (uiTyp==ENDSTARTELM) then begin
            if (uiTyp=STARTELM && acKyw=="name") then begin
               uiTyp=FCRLOC(pvFil)
               if (uiTyp==ENDSTARTELM) then begin
                  uiTyp=FCRLOC(pvFil,acDat)
                  if (uiTyp=DATA) then begin
                     FCRCONV(pvStr,acDat,32,CB.NAME)
                     uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
                     if (uiTyp!="ENDELM" && acKyw!="name")
                        ERROR
                     end
                  end
               end
            end
            uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
            if (uiTyp=STARTELM && acKyw=="number") then begin
               uiTyp=FCRLOC(pvFil)
               if (uiTyp==ENDSTARTELM) then begin
                  uiTyp=FCRLOC(pvFil,acDat)
                  if (uiTyp=DATA) then begin
                     FCRCONV(pvInt,acDat,4,CB.NUMBER)
                     uiTyp=FCRLOCV(pvFil,acKyw,pvKyw)
                     if (uiTyp!="ENDELM" && acKyw!="name")
                        ERROR
                     end
                  end
               end
            end
         end
      end
   end
end
FCRCLSV(pvInt)
FCRCLSV(pvStr)
FCRCLSV(pvKyw)
FCRCLS(pvFil)

FCRLOC returns a pointer to the neutral FL5 element. Alternatively, FCRGET can be used to retrieve a copy of the element data written to an application-provided buffer. The corresponding functions with a 'V' at the end (FCRLOCV/GETV) provide FCRLOC/FCRGET functionality, but with implicit data conversion by passing an open converter handle (created with FCROPENV). If the type of data is unknown in advance, the data can also be read with FCRLOC/FCRGET first, and then converted explicitly with FCRCONV. This function performs explicit conversions on arbitrary buffers. Therefore, it is not limited to data processed by the record interface.