* More usage of regular expressions
** Support new command GREP in FLCL
*** Use this new command in CLIST FLGREP
** Make FMTERR usable for TVD format
** Correct match and replace statistics
** Add more context to error messages
* Use zACS in QA process of FLAMSTC (no findings, only 0C4 possible)
* Use character conversion without pre-calculation (RawRun) only if no factory to reuse the handle available
* Add NoRun switch to FLAM and ICNV commands
* Support new SAF policies LOG.INPUT and LOW.ENTROPY
** LOG.INPUT to prevent the logging of input parameter strings (could contain credentials (passwords, keys, PINs))
** LOW.ENTROPY to prevent the use (environment variable FL_ALLOW_LOW_ENTROPY) of low entropy for random values
* Add new chapter for security considerations to manuals
* Bugfixes
** Fix use of free'ed memory in table support concerning default ccs_id default delimiter if directory walk used
*** if no CCSID or delimiter for the columns/elements defined, then the determined default value was lost till the second file
*** The default value was stored in the FMTTAB handle which is closed and the pointer then points to already freed memory
*** The result are too long or wrong CCSIDs or damaged delimiters
*** The problem could be solved by defining the CCSIDs and element delimiters
** Correct compression mode in member list for FLAMFILEs on UNIX/Windows (for MODE=ADC and old FLAM encryption)
5.1.27-32770
ATTENTION: For character set conversion, IBM Unicode Services are used on z/OS
by default if possible and if it improves conversion speed. In a few cases, the
behavior for incomplete or non-convertible characters will differ from the old
FLAM behavior. To prevent this change in behavior, you can set the system symbol
&FLIBMUS to OFF, but this will result in more CPU and memory utilization for simple
character conversions (see chapter 'Use of IBM Unicode Services' for more
information).
ATTENTION: Incorrect handling of surrogates in the character conversion module was
fixed for UTF-16LE. This fix changes behavior and breaks backward compatibility.
* Regular expressions (Perl-compatible) can now be used for:
** Generating output file names from input filenames (Input to Output Name Mapping)
** Validation or replacement of string elements
** Validation or replacement of string records during text formatting
** Validation or replacement of string records during binary formatting for records
** Validation or replacement for columns in table support
* A tool to test regular expressions is available as object run.regexp() of the UTIL command
* A new CLIST named FLGREP is available to search for a regex pattern using the directory walk on z/OS
* About and version information was reworked
* Rework handling of environment variables and add new built-in functions HLPENV and LSTENV
* Complete rework of character conversion module
** Character conversion now differentiates between illegal characters (EILSEQ) and malformed encoding (ENOMSG)
** Uses the same conversion handle if user table and report file are the same
** Uses fast conversion algorithms in more cases
** Reduced memory utilization and increased overall performance
* TR instruction is used on SystemZ for single byte to single byte character translations in character conversion module
** Significantly increases performance and reduces CPU utilization compared to IBM Unicode Services
** Substitution or identity mapping must be activated as error handling (CHRMODE=STOP will be 10x slower)
** A new environment variable (FL_DEFAULT_CHAR_MODE (&FLCHARM)) changes the default CHRMODE from STOP to SUBSTITUTE
* IBM Unicode Services is used if possible to lower memory and CPU utilization
** If none of the special features of FLAM character conversion module are requested, Unicode Serivces with HW support are used by default
** A new environment variable (FL_IBM_UNICODE_SERVICE (&FLIBMUS)) controls the use of Unicode Services
** See new chapter 'Use of IBM Unicode Services' for more information
* New parameter added to keep byte order marks (KEEPBOM) for faster character set conversions
** This brings BOM handling more in line with IBM Unicode services
** An invalid BOM will now cause an error with UTF-8 and UCS-1
*** Unicode services keep an invalid BOM in the middle of the data
*** To detect this and the wrong data behind it, the error is required
* Re-implementation of usage logging to support eligible SRB(zIIP) and zIIP times
* Support to dump elements in write.xml()
* New version of LZMA SDK (21.7) including x86 optimization
* zlib updated to version 1.2.12 (CVE-2018-25032)
* zstd updated to version 1.5.2
* libbzip2 updated to version 1.0.8
* A chapter regarding performance considerations was added to manuals
* Loading the environment occurs a little earlier than before due to character conversion optimizations
* Protect fliconv (FLCICV) interface with feature code (FLAC_FLCICV)
* Bugfixes
** Fix statistics values in FLAMFILE I/O (FIO-Time and Block-Count)
** Fix wrong handling of surrogates for UTF-16LE
*** Encoding of 4 byte characters (codepoints >= 64k) was wrong
*** Such a 4 byte encoding must be 2 times 16 bit little endian integers and not a 32 bit little endian integer
** Fix logging if self-signed PGP certificate are imported
** Correct handling of dirty delimiter (0x0D...0DC285) at text parsing
** Don't write log warning for attribute length mismatch more than once
** Correct pointer and space calculation after realloc in character conversion module (CNVCHR)
** Print info instead of warning on attribute length mismatch when reading in FMT.REC()
*** Print only one log message for attribute length mismatches and not for each record
** Correct sample program SOFLCICV and SPFLCICV if output buffer too small
** Fix a missing initialization in CNVREC
- 0000783: [2.2 Subprogram FLUC (CONV)] Add match of regular expression to string converter (Falk Reichbott) - resolved. - 0000962: [1.1 FLCL] Limit the maximal internal block size to prevent memory exhausting through expansion at decompression (Falk Reichbott) - resolved.
