FLAM Issue Tracker - FL5
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0000888FL53. Subsystem, Driver, Plugin, SPIpublic2017-08-29 10:532018-03-08 18:48
ReporterFalk Reichbott 
Assigned ToUlrich Schwab 
PriorityhighSeverityfeatureReproducibilityN/A
StatusresolvedResolutionfixed 
PlatformGeneralOSGeneralOS VersionGeneral
Product Version5.1.15 
Target Version5.1.17Fixed in Version5.1.17 
Summary0000888: Support an application transparent file conversion, compression and encryption solution like z/OS-Subsys for open systems
DescriptionCurrently FLAM supports subsystem functionality only on mainframe systems. To use FLAM as infrastructure it would be useful to support such an application transparent solution also on other platforms (UNIX, Windows).

Per logical file it must be possible to define read- und write-string for the physical representation of this file. This representation can be a converted, compressed, encrypted local or remote version of the logical file.

For example: An application (SAP) writes a file for credit card ordering which must be encrypted. A system programmer makes a definition in a hidden file, that this logical file must be represented as an OpenPGP file in phsical form and the FLUC I/O subsystems does this conversion without any change to the application.
Steps To Reproduce
Additional InformationFUSE (Filesystem in Userspace) could be a solution for UNIX systems
TagsNo tags attached.
Relationships
related to 0000704resolved Falk Reichbott Add a FLUC I/O subsystem to write transparent different formats for and to different platforms on z/OS 
related to 0000891assigned Falk Reichbott Make record and byte interface thread save 
related to 0000912resolved Falk Reichbott Support multiple writes on the byte interface 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2017-08-29 10:53Falk ReichbottNew Issue
2017-08-29 10:53Falk ReichbottStatusnew => assigned
2017-08-29 10:53Falk ReichbottAssigned To => Ulrich Schwab
2017-08-29 11:04Falk ReichbottNote Added: 0001114
2017-08-31 09:23Falk ReichbottNote Added: 0001116
2017-08-31 10:23Falk ReichbottNote Added: 0001117
2017-08-31 12:23Ulrich SchwabNote Edited: 0001116bug_revision_view_page.php?bugnote_id=1116#r372
2017-08-31 12:26Ulrich SchwabNote Edited: 0001117bug_revision_view_page.php?bugnote_id=1117#r374
2017-08-31 16:56Falk ReichbottNote Added: 0001119
2017-08-31 17:06Falk ReichbottNote Added: 0001120
2017-09-01 10:01Ulrich SchwabNote Edited: 0001119bug_revision_view_page.php?bugnote_id=1119#r376
2017-09-01 13:02Falk ReichbottNote Added: 0001121
2017-09-07 14:31Falk ReichbottDescription Updatedbug_revision_view_page.php?rev_id=378#r378
2017-09-08 09:33Ulrich SchwabNote Edited: 0001121bug_revision_view_page.php?bugnote_id=1121#r380
2017-10-23 12:53Falk ReichbottRelationship addedrelated to 0000891
2017-11-05 10:18Falk ReichbottNote Added: 0001131
2017-11-08 15:12Ulrich SchwabNote Edited: 0001131bug_revision_view_page.php?bugnote_id=1131#r398
2017-11-10 09:36Falk ReichbottRelationship addedrelated to 0000704
2018-03-06 11:29Ulrich SchwabDescription Updatedbug_revision_view_page.php?rev_id=412#r412
2018-03-06 11:29Ulrich SchwabAdditional Information Updatedbug_revision_view_page.php?rev_id=414#r414
2018-03-08 18:47Falk ReichbottNote Added: 0001151
2018-03-08 18:47Falk ReichbottStatusassigned => resolved
2018-03-08 18:47Falk ReichbottFixed in Version => 5.1.17
2018-03-08 18:47Falk ReichbottResolutionopen => fixed
2018-03-08 18:48Falk ReichbottRelationship addedrelated to 0000912

Notes
(0001114)
Falk Reichbott   
2017-08-29 11:04   
FUSE on AIX

https://www.ibm.com/developerworks/aix/library/au-spunix_newfs/index.html [^]
(0001116)
Falk Reichbott   
2017-08-31 09:23   
(edited on: 2017-08-31 12:23)
Solution with FUSE

We implement a main (executable) for mount 'flucfuse' with standard fuse parameter (only write or read and write, only user or root or other, in forground or background, ...) and one optional file name with the write definitions in the CLP syntax below:

fluc.write(file='matchstring*with*wildcards'
   substitute[(format="format.string" file="write/output.string" [readable])]

where substitute is an array for multiple writes (we use n times fcbopen() and do n times fcbpwrite() for each handle) and only one of them can have the readable flag set. If no central config file name is given, a hidden file (.flucfuse.write) is used in each directory. New write definitions might be added on the fly. If the readable flag is defined, then the inverse format (must be implemented) and read (must be changed from pfIvr to pdIvr) strings are written to a hidden file (.flucfuse.read) in each directory with the CLP syntax below:

fluc.read(file='logicalfilename'
   substitute(format="format.string" file="read/input.string")
   attributes(filesize=4711))

This file is used for read requests in inverse form. The filesize for stat and other attributes are also stored in this directory extension.

Additionally we must finalise the chmod support, so that the user can define the access rights for the physical files in the "write/output.string".

The FLUC byte interface (FLCBYT) must be extended by the IVR functions to get the inverse strings for the read operation.

If multiple FIOs are defined in the "output.string" then only the first FIO definition must be used to build the inverse command, the other inverse strings must be comments, so that this implicit rule makes the read unique.

The byte interface gets 3 new function. First fcbpread() and fcbpwrite() which manage the offset on the FCB handle. The third will be an extension to the close function which provides the attribute string for the logical file. At each close of a converted file, the inverse entry is made if the readable flag is set and the read specification file is written.

Additionally all other FCB functions get a thread save version with errno, trcsiz and trcbuf as the first 3 parameter.

By linking the flucfuse main programm the LMAP must be checked to verfiy
that no static data area is present, in the implementation of the FLUC file system.

If a file don't fit with the match string a passthrough will be done. If a file matches, then the locical name is given in the state string to fcbopen() and will be converted to the pysiscal name based on the defined mapping rules in the "write/output.string". The physical name is given back in the inverse read string.

To automate reload of a central config file inotify should be used.
Changes to config files within the work tree are seen anyway and can be used
instantly.

The implementation is limited to sequential read and write. If an application does seek and makes updates, an I/O error is flagged.

A special case is a seek_end, ftell and rewind. This cases are covered by a complete read (if possible) and a reopen of the file afterwards.

There will be another filesystem called flucfusecheck which makes passthrough, and logs per file all functions and check the offsets for sequential read and write. This file system can be mounted to figure out, which files of this mountpoint can be converted by a write specification and which files will fail.

Questions:

Is a central config file for write operations useful?
The implementation would be more simple if the configuration is done as an extension to each directory only based on the file name, without any path. For example you mount the flucfuse on directory pgpfile in your home directory and you make the definition in the file pgpfile/.flucfuse.write.

fluc.write(file='*'
   substitute(format="format.bin()" file="write.bin(encr.pgp(user='~'))"
              readable)

In this case each file of this folder will be OpenPGP encrypted under the own user id.

Another solution could be that the config file at mount defines the conversion for the whole mountpoint. In this case there is no passthrough implemented. All files under this mount point are converted as defined at mount time. This requires a mount per write definition. Would this be more useful?
(0001117)
Falk Reichbott   
2017-08-31 10:23   
(edited on: 2017-08-31 12:26)
For the read specification it makes more sense to work with a hidden directory where for each converted file a hidden file is stored with the inverse read specification.

/home/user/OpenPGPFiles/.flucfuse.write
/home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.read
/home/user/OpenPGPFiles/.flucfuse.read/.mysecondfile.txt.flucfuse.read
/home/user/OpenPGPFiles/myfirstfile.txt.pgp
/home/user/OpenPGPFiles/mysecondfile.txt.pgp

This makes the access to the read specification easier.
(0001119)
Falk Reichbott   
2017-08-31 16:56   
(edited on: 2017-09-01 10:01)
FUSE on AIX is not possible based on missing kernel support for it.

IBM recommend to use NFS on AIX in conjuction with a LINUX server where FUSE exists.
(0001120)
Falk Reichbott   
2017-08-31 17:06   
A reposition to a previos location at read, can be simply implemented, see below:

if (uiOffset<psHdl->uiOffset) {
   psHdl=fcbreopen(psHdl);//makes close and open
   fcbpos(psHdl,uiOffset);
} else if (uiOffset>psHdl->uiOffset) {
   fcbpos(psHdl,uiOffset);
}
fcbread(psHdl,uiLen,pcDat);

The function fcbreopen and fcbpos must be implemented.
(0001121)
Falk Reichbott   
2017-09-01 13:02   
(edited on: 2017-09-08 09:33)
An additional hidden file containing the last read or write error must be added to the hidden folder described above.

/home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.read.errortrace
/home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.write.errortrace
(0001131)
Falk Reichbott   
2017-11-05 10:18   
(edited on: 2017-11-08 15:12)
The internal flucFSgz (transparent compression of all files in the folder to GZIP files) works fine, now we start with the dynamisation of the conversion capabilities (making the write string variable) and the selection stuff (wildcard pattern matching) for the files.
(0001151)
Falk Reichbott   
2018-03-08 18:47   
flucFS is using FUSE to mount an existing directory in some other folder. Everything written to the mounted folder is transparently converted, encrypted, compressed as defined in the configuration. If no match is found in its configuration, the data is passed through without any change. The configuration is done with the argument strings of the FLUC byte interface API. However, since flucFS is a filesystem and uses only the read and write functions of the API, the element access features of the API are not useable with flucFS.

flucFS2 provides the same functionality as flucFS but uses the older version 2 of FUSE. This should only be used on systems without support of FUSE version 3.

flucFSconf is a command line utility to view and modify the configuration of flucFS. A fast memory mapped database is used to store configuration and meta data. This database is stored in a subdirectory of fluc-root with the name .flucIndex. flucFSconf will check the formatString and fileString arguments for correct syntax before storing them. On error the exit code is set accordingly and a detailed error message is printed on stderr.

flucFSgui graphical user interface to view and modify flucFS configuration.

The fuse kernel module must be loaded in order to use any fuse filesystem. This kernel module is usually installed with the linux kernel.