Bug 799469 - Future Gnucash Object idea: Reconciliation Object

[christopherlam]

draft.

Upon a successful reconciliation, we can store the "statement" thus confirmed "correct" in an appropriate data store: an OFX file. This works if the <?xml version="1.0" encoding="UTF-8"?> is removed manually.

OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

<?xml version="1.0" encoding="UTF-8"?>
<OFX>
  <SIGNONMSGSRSV1>
    <SONRS>
      <STATUS>
        <CODE>0</CODE>
        <SEVERITY>INFO</SEVERITY>
        <MESSAGE>OK</MESSAGE>
      </STATUS>
      <DTSERVER>20241206104445</DTSERVER>
      <LANGUAGE>ENG</LANGUAGE>
    </SONRS>
  </SIGNONMSGSRSV1>
  <BANKMSGSRSV1>
    <STMTTRNRS>
      <STMTRS>
        <BANKTRANLIST>
          <STMTTRN>
            <TRNTYPE>CREDIT</TRNTYPE>
            <DTPOSTED>20240414185900</DTPOSTED>
            <TRNAMT>-52.40</TRNAMT>
            <MEMO>vibe</MEMO>
          </STMTTRN>
          <STMTTRN>
            <TRNTYPE>CREDIT</TRNTYPE>
            <DTPOSTED>20230518185900</DTPOSTED>
            <TRNAMT>-52.40</TRNAMT>
            <MEMO>vibe</MEMO>
          </STMTTRN>
          <STMTTRN>
            <TRNTYPE>CREDIT</TRNTYPE>
            <DTPOSTED>20230519185900</DTPOSTED>
            <TRNAMT>0.35</TRNAMT>
            <MEMO>Easypark</MEMO>
          </STMTTRN>
          <STMTTRN>
            <TRNTYPE>CREDIT</TRNTYPE>
            <DTPOSTED>20230518185900</DTPOSTED>
            <TRNAMT>-63.0</TRNAMT>
            <MEMO>Wesfarmers</MEMO>
          </STMTTRN>
        </BANKTRANLIST>
      </STMTRS>
    </STMTTRNRS>
  </BANKMSGSRSV1>
</OFX>

[jralls]

That's an interesting idea. Where would the file live? With the book, in GNC_DATA_DIR, or somewhere else?

[christopherlam]

Dunno... maybe a $datafile_data/ folder to hold all the backups, transaction logs, reconciliation ofx, and saved reports too?

For SQL maybe they may live in another table?

[gjanssens]

Storing the data in a file outside of the book brings extra challenges, similar to the ones we encounter with the gcm files. Assuming you store your ofx file(s) in a file or directory of which the name is similar to the data file they relate to. Then:

• If a user has two data files with the same name, but in different directories, you may get naming conflicts you have to resolve when saving ofx files for both of these data files
• if a user renames a data file outside of gnucash, just using the system's file manager for example, the connection between the data file and its ofx files is lost.
• deleting a data file via the file manager won't also delete the related ofx file(s), leaving cruft behind.

And as you already hinted at, for sql it gets worse as sql may be accessed from completely different computers.

[gjanssens]

I also think our engine code to save data to a table in sql is equivalent to using dedicated nodes in an xml file. I don't think we can do an separate sql table in case of sql and external files in case of xml backends, unless you want to write yet another data storage and retrieval engine.

IMO if this is to become a gnucash object, we really should stick to either extending our file/sql schema or bolt it on via kvp.

[christopherlam]

Ok point taken.

I still feel it may be acceptable to spin out these "second-class" objects whereby dataloss isn't critical.

Instead of saving the OFX into datafile_files/yymmdd_accountname we could save into rootaccount_guid/yymmdd_accountguid thereby establishing a 1-to-1 relationship between datafile and ofx.

On the other hand if we'd bolt onto kvp, would it be via an intricate tree of kvp objects, or save the whole OFX into a kvp string?

I quite like the availability of OFX whereby a reimport may use the import ui to fix cases whereby the user has deleted or damaged the transactions.

[jralls]

On the other hand if we'd bolt onto kvp, would it be via an intricate tree of kvp objects, or save the whole OFX into a kvp string?

It would depend on how you want to query the OFX. If you'd just retrieve the whole thing to reprocess it as an OFX file then it would make more sense to store it as a single string, while if you want to retrieve a piece of it with a KVP path then it would have to be broken up.

As an aside, the example you used is an OFX DirectConnect stream from AQBanking. The actual OFX starts at the <?xml> line you want to delete, and the reason you had to delete it in order to import it is that you tried to use Import OFX/QFX that uses libofx instead of importing with AQBanking. AQBanking can understand XML-formatted OFX but libofx understands only the older SGML-formatted OFX.

I quite like the availability of OFX whereby a reimport may use the import ui to fix cases whereby the user has deleted or damaged the transactions.

Not as it's written: OFX's STMTTRN requires a unique FITID element (missing from your example, making it ill-formed). GnuCash stores that in the transaction's online-id KVP element and the matcher ignores STMTTRNs with an FITID that matches an existing transaction's online-id. Some banks have gotten this wrong by reusing FITIDs; in those cases users have to preprocess their OFX downloads to tack a date string onto the FITIDs so that GnuCash will process new transactions.

Note also that neither AQBanking nor libofx can write so you'd need to implement that yourself.

[christopherlam]

The example ofx I copied above isn't a download, it's a direct output from my code above from my test datafile. From my experimentation it seems libofx handles both ofx1 and ofx2.2 just fine; but is more strict about ofx2.2 (eg it does require FITID).

Maybe we'll need to ask @benoitg whether ofx1 or ofx2.2 would be more appropriate

[gjanssens]

I can't help but wonder why you would want to store this as OFX inside of gnucash ? Can't we just store the parts we need in some format that is easier to integrate with the rest of gnucash ?

[christopherlam]

I can't help but wonder why you would want to store this as OFX inside of gnucash ? Can't we just store the parts we need in some format that is easier to integrate with the rest of gnucash ?

The idea is that OFX already has an elaborate and debugged importer, and any other kvp or first-class objects format would require building a whole UI to repair transactions.

And it's still unclear which "parts we need" actually consist of.

[jralls]

he idea is that OFX already has an elaborate and debugged importer, and any other kvp or first-class objects format would require building a whole UI to repair transactions.

We already have a much simpler way to re-create transactions, the transaction log.

[gjanssens]

I can't help but wonder why you would want to store this as OFX inside of gnucash ? Can't we just store the parts we need in some format that is easier to integrate with the rest of gnucash ?

The idea is that OFX already has an elaborate and debugged importer, and any other kvp or first-class objects format would require building a whole UI to repair transactions.

And it's still unclear which "parts we need" actually consist of.

I seem to be missing something here.

To me reconciliation means confirming transactions in your book appear on some external record. So to save that data in gnucash, one would need some id for that record, a few dates and a list of splits that are covered by the reconciliation.

You are now talking about importing, and transaction repair. How is that related ? Isn't that widening the scope quite a bit ?

[christopherlam]

To me reconciliation means confirming transactions in your book appear on some external record. So to save that data in gnucash, one would need some id for that record, a few dates and a list of splits that are covered by the reconciliation.

^ no disagreement here!

You are now talking about importing, and transaction repair. How is that related ? Isn't that widening the scope quite a bit ?

Agree... the scope is still ill-defined. The idea of storing a copy of successfully reconciled splits (alongside statement date, statement balance) is still useful IMHO. While coding storing reconciliation data into kvp, I that felt I was recreating an OFX file and recoded to output OFX.

kvp would offer a neater method to write a proper reconciliation report (i.e. show me the statement and its splits).

OFX offers the ability to reimport splits, hence repairing damaged reconciliations (e.g. via accidental deletes).

IMHO the Transaction Log is also neat but stores complete transactions rather than splits. Additionally I wager most users routinely delete these .log files.

Further debates welcome...

[jralls]

Maybe back up a bit and articulate the problem you're trying to solve. That's probably better done by closing the bug and this PR and starting over with a new bug report that states a problem rather than proposing a solution.