FLAM® Issue Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000888FL53. Subsystem, Driver, Plugin, SPIpublic2017-08-29 10:532018-03-08 18:48
ReporterFalk Reichbott 
Assigned ToUlrich Schwab 
PlatformGeneralOSGeneralOS VersionGeneral
Product Version5.1.15 
Target Version5.1.17Fixed in Version5.1.17 
Summary0000888: Support an application transparent file conversion, compression and encryption solution like z/OS-Subsys for open systems
DescriptionCurrently FLAM supports subsystem functionality only on mainframe systems. To use FLAM as infrastructure it would be useful to support such an application transparent solution also on other platforms (UNIX, Windows).

Per logical file it must be possible to define read- und write-string for the physical representation of this file. This representation can be a converted, compressed, encrypted local or remote version of the logical file.

For example: An application (SAP) writes a file for credit card ordering which must be encrypted. A system programmer makes a definition in a hidden file, that this logical file must be represented as an OpenPGP file in phsical form and the FLUC I/O subsystems does this conversion without any change to the application.
Additional InformationFUSE (Filesystem in Userspace) could be a solution for UNIX systems
TagsNo tags attached.
Attached Files

- Relationships
related to 0000704resolvedFalk Reichbott Add a FLUC I/O subsystem to write transparent different formats for and to different platforms on z/OS 
related to 0000891assignedFalk Reichbott Make record and byte interface thread save 
related to 0000912resolvedFalk Reichbott Support multiple writes on the byte interface 

-  Notes
Falk Reichbott (administrator)
2017-08-29 11:04


https://www.ibm.com/developerworks/aix/library/au-spunix_newfs/index.html [^]
Falk Reichbott (administrator)
2017-08-31 09:23
edited on: 2017-08-31 12:23

Solution with FUSE

We implement a main (executable) for mount 'flucfuse' with standard fuse parameter (only write or read and write, only user or root or other, in forground or background, ...) and one optional file name with the write definitions in the CLP syntax below:

   substitute[(format="format.string" file="write/output.string" [readable])]

where substitute is an array for multiple writes (we use n times fcbopen() and do n times fcbpwrite() for each handle) and only one of them can have the readable flag set. If no central config file name is given, a hidden file (.flucfuse.write) is used in each directory. New write definitions might be added on the fly. If the readable flag is defined, then the inverse format (must be implemented) and read (must be changed from pfIvr to pdIvr) strings are written to a hidden file (.flucfuse.read) in each directory with the CLP syntax below:

   substitute(format="format.string" file="read/input.string")

This file is used for read requests in inverse form. The filesize for stat and other attributes are also stored in this directory extension.

Additionally we must finalise the chmod support, so that the user can define the access rights for the physical files in the "write/output.string".

The FLUC byte interface (FLCBYT) must be extended by the IVR functions to get the inverse strings for the read operation.

If multiple FIOs are defined in the "output.string" then only the first FIO definition must be used to build the inverse command, the other inverse strings must be comments, so that this implicit rule makes the read unique.

The byte interface gets 3 new function. First fcbpread() and fcbpwrite() which manage the offset on the FCB handle. The third will be an extension to the close function which provides the attribute string for the logical file. At each close of a converted file, the inverse entry is made if the readable flag is set and the read specification file is written.

Additionally all other FCB functions get a thread save version with errno, trcsiz and trcbuf as the first 3 parameter.

By linking the flucfuse main programm the LMAP must be checked to verfiy
that no static data area is present, in the implementation of the FLUC file system.

If a file don't fit with the match string a passthrough will be done. If a file matches, then the locical name is given in the state string to fcbopen() and will be converted to the pysiscal name based on the defined mapping rules in the "write/output.string". The physical name is given back in the inverse read string.

To automate reload of a central config file inotify should be used.
Changes to config files within the work tree are seen anyway and can be used

The implementation is limited to sequential read and write. If an application does seek and makes updates, an I/O error is flagged.

A special case is a seek_end, ftell and rewind. This cases are covered by a complete read (if possible) and a reopen of the file afterwards.

There will be another filesystem called flucfusecheck which makes passthrough, and logs per file all functions and check the offsets for sequential read and write. This file system can be mounted to figure out, which files of this mountpoint can be converted by a write specification and which files will fail.


Is a central config file for write operations useful?
The implementation would be more simple if the configuration is done as an extension to each directory only based on the file name, without any path. For example you mount the flucfuse on directory pgpfile in your home directory and you make the definition in the file pgpfile/.flucfuse.write.

   substitute(format="format.bin()" file="write.bin(encr.pgp(user='~'))"

In this case each file of this folder will be OpenPGP encrypted under the own user id.

Another solution could be that the config file at mount defines the conversion for the whole mountpoint. In this case there is no passthrough implemented. All files under this mount point are converted as defined at mount time. This requires a mount per write definition. Would this be more useful?

Falk Reichbott (administrator)
2017-08-31 10:23
edited on: 2017-08-31 12:26

For the read specification it makes more sense to work with a hidden directory where for each converted file a hidden file is stored with the inverse read specification.


This makes the access to the read specification easier.

Falk Reichbott (administrator)
2017-08-31 16:56
edited on: 2017-09-01 10:01

FUSE on AIX is not possible based on missing kernel support for it.

IBM recommend to use NFS on AIX in conjuction with a LINUX server where FUSE exists.

Falk Reichbott (administrator)
2017-08-31 17:06

A reposition to a previos location at read, can be simply implemented, see below:

if (uiOffset<psHdl->uiOffset) {
   psHdl=fcbreopen(psHdl);//makes close and open
} else if (uiOffset>psHdl->uiOffset) {

The function fcbreopen and fcbpos must be implemented.
Falk Reichbott (administrator)
2017-09-01 13:02
edited on: 2017-09-08 09:33

An additional hidden file containing the last read or write error must be added to the hidden folder described above.


Falk Reichbott (administrator)
2017-11-05 10:18
edited on: 2017-11-08 15:12

The internal flucFSgz (transparent compression of all files in the folder to GZIP files) works fine, now we start with the dynamisation of the conversion capabilities (making the write string variable) and the selection stuff (wildcard pattern matching) for the files.

Falk Reichbott (administrator)
2018-03-08 18:47

flucFS is using FUSE to mount an existing directory in some other folder. Everything written to the mounted folder is transparently converted, encrypted, compressed as defined in the configuration. If no match is found in its configuration, the data is passed through without any change. The configuration is done with the argument strings of the FLUC byte interface API. However, since flucFS is a filesystem and uses only the read and write functions of the API, the element access features of the API are not useable with flucFS.

flucFS2 provides the same functionality as flucFS but uses the older version 2 of FUSE. This should only be used on systems without support of FUSE version 3.

flucFSconf is a command line utility to view and modify the configuration of flucFS. A fast memory mapped database is used to store configuration and meta data. This database is stored in a subdirectory of fluc-root with the name .flucIndex. flucFSconf will check the formatString and fileString arguments for correct syntax before storing them. On error the exit code is set accordingly and a detailed error message is printed on stderr.

flucFSgui graphical user interface to view and modify flucFS configuration.

The fuse kernel module must be loaded in order to use any fuse filesystem. This kernel module is usually installed with the linux kernel.

- Issue History
Date Modified Username Field Change
2017-08-29 10:53 Falk Reichbott New Issue
2017-08-29 10:53 Falk Reichbott Status new => assigned
2017-08-29 10:53 Falk Reichbott Assigned To => Ulrich Schwab
2017-08-29 11:04 Falk Reichbott Note Added: 0001114
2017-08-31 09:23 Falk Reichbott Note Added: 0001116
2017-08-31 10:23 Falk Reichbott Note Added: 0001117
2017-08-31 12:23 Ulrich Schwab Note Edited: 0001116 View Revisions
2017-08-31 12:26 Ulrich Schwab Note Edited: 0001117 View Revisions
2017-08-31 16:56 Falk Reichbott Note Added: 0001119
2017-08-31 17:06 Falk Reichbott Note Added: 0001120
2017-09-01 10:01 Ulrich Schwab Note Edited: 0001119 View Revisions
2017-09-01 13:02 Falk Reichbott Note Added: 0001121
2017-09-07 14:31 Falk Reichbott Description Updated View Revisions
2017-09-08 09:33 Ulrich Schwab Note Edited: 0001121 View Revisions
2017-10-23 12:53 Falk Reichbott Relationship added related to 0000891
2017-11-05 10:18 Falk Reichbott Note Added: 0001131
2017-11-08 15:12 Ulrich Schwab Note Edited: 0001131 View Revisions
2017-11-10 09:36 Falk Reichbott Relationship added related to 0000704
2018-03-06 11:29 Ulrich Schwab Description Updated View Revisions
2018-03-06 11:29 Ulrich Schwab Additional Information Updated View Revisions
2018-03-08 18:47 Falk Reichbott Note Added: 0001151
2018-03-08 18:47 Falk Reichbott Status assigned => resolved
2018-03-08 18:47 Falk Reichbott Fixed in Version => 5.1.17
2018-03-08 18:47 Falk Reichbott Resolution open => fixed
2018-03-08 18:48 Falk Reichbott Relationship added related to 0000912

Copyright © 2000 - 2024 MantisBT Team
Powered by Mantis Bugtracker