Persistence

Persistence

Persistence has been implemented for easy loading and storing of matrices and other data. The persistence framework would support all kinds of formats but only one binary format has been implemented yet.

Storing of data

Each object which is capable of writing itself to a stream implements the IMathPersistence interface:

IMathPersistence = interface
    ['{16A5F1D0-CA39-4BD0-BABE-E1E70B0045C3}']
    procedure SaveToFile(const FileName : string; Writer : TCustomMathPersistenceIOClass);
    procedure SaveToStream(stream : TStream; Writer : TCustomMathPersistenceIOClass);
  end;

For now there only exists the TBinaryReaderWriter and TJsonReaderWriter classes which has to be used as second parameter.

Loading of data

There are two functions to load an object from a file:

function ReadObjFromFile(const FileName : TFileName) : TBaseMathPersistence;
function ReadObjFromStream(stream : TStream) : TBaseMathPersistence;

The resulting object has to be casted to the expected class afterwards!

Note you need to have the BinaryReaderWriter unit included which automatically registers itself for the IO subsystem and is responsible for the data reading/writing and the compression.

Note that the loading and storing of data is independent of the file extension!

Examples

Storing a matrix to a file:

  mx := TDoubleMatrix.Create(1, 1);
  mx.SaveToFile('Test.dat', TBinaryReaderWriter);

or a more verbose example

procedure TestPersistence;
var dx, dy : Array[0..49] of double;
    i : Integer;
    mx, my : TDoubleMatrix;
begin
     for i := 0 to High(dx) do
     begin
          dx[i] := i + 1;
          dy[i] := i + 2;
     end;

     mx := TDoubleMatrix.Create;
     with TBinaryReaderWriter.Create do
     try
        mx.Assign(dx, 5, 10);
        SaveToFile(mx, 'matrixData1.txt');
        mx.Assign(dy, 5, 10);
        SaveToFile(mx, 'matrixData2.txt');
     finally
            Free;
     end;

     mx.Free;
     my.Free;
end;

Reading a matrix from a file

   // my is the matrix from the verbose example
   my := ReadObjFromFile('matrixData1.txt') as TDoubleMatrix;

Implemented data formats

Binary Reader and writer

The binary data format is the internal format and quite simple:

It consists of a very short header (9 bytes) containing of a type identifier and a version:

const cBinaryReaderHeader : Array[0..4] of AnsiChar = 'RMCLS';
      cBinaryReaderVersion : LongWord = 1;

followed by a set of a binary representation of a persisten object. An object consists of sections starting with the object identifier section type. A section is a variable length record starting with:

{$MINENUMSIZE 1}
type
  TBinarySectionType = (bsBinary, bsString, bsDouble, bsDoubleArr, bsIntArr, bsInteger, bsBeginList,
                        bsListIntArr, bsListDoubleArr, bsEndList, bsObjectDescription, bsObjectBegin, bsObjectEnd, bsEndaStream);

type
  TBinarySectionRec = packed record
    Len : LongInt;
    SectTpye : TBinarySectionType;
    NameLen : Word;
  end;

Len defines the complete length of the followed data. From this length one has to substract "NameLen" bytes interpreted as UTF8String which represents the objects property name. The bytes left are treated as data. Note that this simple data format actually can produce recursive object definitions. Arrays are encdoded within "bsBeginList bsEndList" qualifier (this can also be used for arrays of objects...).

JSON Reader and writer

The library contains a class to read and write JSON formatted data. Though JSON is clearly defined and it does not enforce any object property sorting order this library needs one since the data is processed in a one time stream. This enforces some properties to be written before others. Check out a complete example of a matrix written in JSON:

   {
    "Lib":"mrMath","Version":1,
    "obj":"MTX",
    "Name":"Test",
    "Width":5,
    "Height":2,
    "data":[2,3,4,5,6,7,8,9,10,11],
    "SubWidth":2,
    "SubHeight":2,
    "OffsetX":0,
    "OffsetY":0
   }

A mrMath JSON object descriptions always starts with {"Lib":"mrMath","Version":1, followed by the object type. When the JSON stream interpreter could determine the object type and created the object reading of the other objects properties takes place like here reading the data array and feeding some internal properties. There are also cases where some properties need to be defined before others e.g. the width and height values of the matrix are mandatory before the data list.

The plain pascal code equivalent to create this object would be:

	 mtx := TDoubleMatrix.Create( [2, 3, 4, 5, 6, 7, 8, 9, 10, 11], 5, 2);
	 mtx.Name := 'Test';
	 mtx.SetSubMatrix( 0, 0, 2, 2);
	 // now it may be used..

Implementing your own data format

To create your own data format one needs to subclass the class TCustomMathPersistenceIO which can be found in the unit BaseMathPersistence.pas. One has to implement at least the following skeleton:

   uses SysUtils, Classes, BaseMathPersistence, MathUtilFunc, types;
   
   type
     TYourDataFormatReaderWriter = class(TCustomMathPersistenceIO)
   private
   protected
      class function CanReadStream(aStream : TStream) : boolean; override;

      procedure WriteStreamHeader; override;
      procedure FinalizeStream; override;

      procedure WriteObjDescriptor(const Name : String; const Value : String); override;
      procedure WriteDoubleArr(const Name : String; const Value : Array of double); override;
      procedure WriteListBegin(const Name : String; count : integer); override;
      procedure WriteListEnd; override;
      procedure WriteBinaryProperty(const Name : String; const Value; size : integer); override;
      procedure WriteProperty(const Name : String; const Value : double); overload; override;
      procedure WriteProperty(const Name : String; const Value : String); overload; override;
      procedure WriteProperty(const Name : String; const Value : integer); overload; override;
      procedure WriteObject(Obj : TBaseMathPersistence); override;
      procedure WriteObject(const Name : String; Obj : TBaseMathPersistence); overload; override;
      procedure WriteListDoubleArr(const Value : Array of double); override;
      procedure WriteIntArr(const Name : String; const Value : Array of Integer); override;
      procedure WriteListIntArr(const Value : Array of Integer); override;
    public
      function LoadFromStream(aStream : TStream) : TBaseMathPersistence; override;

      constructor Create;
    end;
   
   implementattion
    
   initialization
     RegisterMathIOReader(TYourDataFormatReaderWriter);

Reading an object

In order to get the class working one needs to register the class in the IOReader list by issuing the RegisterMathIOReader call. The ioreader classes do not check for file extensions but rather inspect the file/stream header for that and finds the correct reader class by calling the CanReadStream class function on each registered reader/writer class. So all you have to do is to inspect that header with your expected file stream. E.g. the JSON readers checking procedure is:

   class function TJsonReaderWriter.CanReadStream(aStream: TStream): boolean;
   var buf : UTF8String;
   begin
        SetLength(buf, Length(cJSONHeader));
        Result := aStream.Read(buf[1], Length(cJSONHeader)) = Length(cJSONHeader);
        if Result then
           Result := (CompareStr(String(buf), String(cJSONHeader)) = 0);
   end;

The routine first checks if the stream has a minimum size and then compares the expected header strings.

After that the framework calls the method LoadFromStream where you may parse the stream. The stream reading itself actually consists of reading integers, doubles, strings, lists and objects.