Rule Extensibility

Extensible Automation

Logic Bank automates transaction logic for databases - multi-table constraints and derivations, and actions such as sending messages or emails. It is based on Python and SQLAlchemy, an Object Relational Mapper (ORM).

Such backend logic is a significant element of database systems, often nearly half.

    Logic Bank can reduce 95% of your backend code by 40X,
    using a combination of spreadsheet-like Rules and (Python) code.

While 95% automation is impressive, it's not enough. Virtually all systems include complex database logic that cannot be automated, as well as non-database logic such as sending emails or messages.

It's therefore imperative to provide extensibility, using standard languages and paradigms. In this article, we'll discuss:

1. Spreadsheet-like rules - we'll briefly review these, but the main focus is on 2 levels of extensibility...

2. Event Handlers - Python event handlers (functions) that apply to:

   • a specific table ("ad hoc")

   • all tables ("generic")

3. Extensible Rules - learn how to define new rule types, providing

   • Reuse - such rules can be used in multiple applications

   • Discovery - such rules need to be discoverable by colleagues

1. Spreadsheet-like Rules Automate the "Cocktail Napkin Spec"

We’ve all seen how a clear specification - just a cocktail napkin spec - balloons into hundreds of lines of code. This leads to the key design objective for Logic Bank:

Introduce Spreadsheet-like Rules to
Automate the "Cocktail Napkin Spec"

The subsections below show how to declare, activate and execute rules.

Declare

Below is the implementation of our cocktail napkin spec (blue rectangle, upper right) to check credit:

In the diagram above, the rules are declared on lines 40-49 - 1:1 with the spec. These 5 rules replace several hundred lines of code (40X), as shown here.

Activate

Activate the rules while opening your database:

session_maker = sqlalchemy.orm.sessionmaker()
session_maker.configure(bind=engine)
session = session_maker()

LogicBank.activate(session=session, activator=declare_logic)

The activate function:

• invokes your declare_logic() function (shown in the screen shot above) to load your rules
• verifies them (e.g., check for cycles)
• installs the Logic Bank event handlers to listen to SQLAlchemy before_flush events

Execute

Rules are declarative: you do not need to call them, or order them. Internally, the Logic Bank event handlers call the Logic Bank rules engine to execute your logic. For more on declarative, click here.

2. Event Handlers - Ad Hoc Python Functions

While rules are powerful, they cannot automate everything. That leads to the second key design objective:

Rules must be complemented by code for extensibility,
and manageability (debugging, source control, etc). 

Python event handlers are straightforward. You declare your event handler function (line 51, above - congratulate_sales_rep), and the rules engine calls it (lines 32-38), providing arguments:

• row - an instance of a SQLAlchemy mapped class (here, Order)

  • Use code completion to access attributes (column values)
  • Observe SQLAlchemy rows provide access to related rows (e.g., sales_rep.Manager.FirstName)

• old_row - prior contents

• logic_row - wraps row and old_row, and includes useful state such as nest_level, ins_upd_dlt etc. It is also the rule engine executor for the wrapped row instance, via methods for insert / update and delete.

Python extensibility is shown on line 51, invoking the Python event-handler code on line 32:

def congratulate_sales_rep(row: Order, old_row: Order, logic_row: LogicRow):
    if logic_row.ins_upd_dlt == "ins":  # logic engine fills parents for insert
        sales_rep = row.SalesRep  # type : Employee
        if sales_rep is None:
            logic_row.log("no salesrep for this order")
        else:
            logic_row.log(f'Hi, {sales_rep.Manager.FirstName}, congratulate {sales_rep.FirstName} on their new order')

This is just a simple example that logs a message... a real event might generate a RESTful call, or send an email message.

The red dot in the left margin is a breakpoint, illustrating that you can use standard Python debugger services - breakpoints, variable inspection, single step, etc.

Event Types: Early, (normal) Event, Commit Event

There are actually 3 types of events. The Logic Bank rules engine invokes them at different points in logic execution, as follows.

SQLAlchemy before_flush events expose a list of rows altered by the client to the Logic Bank rule engine. For each row, Logic Bank creates a logic_row, and calls the insert / update / delete method as appropriate. In the course of rule execution, it invokes Early and normal events:

• Early Events run prior to rule execution
• Events run after rule execution

After all the rows have been processed, the rule engine cycles through them again, executing any CommitEvents. Since this is subsequent to row logic, all derivations have been performed, including multi-table rules. In particular, parent rows reflect child sums and counts.

Enough??

At this point, we can define rules for 95% of our logic, and code Python Event Handlers for the rest. For most people, that should be enough...

Not us.

There are still some tedious things we'd like to automate...

Generic Event Handlers

Imagine we wish to set date / time stamps. Our database follows naming conventions, so we want to set CreatedOn for all tables that have this column.

We could certainly write an event:

def audit_by_event(row: Employee, old_row: Employee, logic_row: LogicRow):
    tedious = False  # tedious code to repeat for every audited class
    if tedious:      # see instead the following rule extension - nw_copy_row
        if logic_row.are_attributes_changed([Employee.Salary, Employee.Title]):
            copy_to_logic_row = logic_row.new_logic_row(EmployeeAudit)
            copy_to_logic_row.link(to_parent=logic_row)
            copy_to_logic_row.set_same_named_attributes(logic_row)
            copy_to_logic_row.insert(reason="Manual Copy " + copy_to_logic_row.name)  # triggers rules...
            # logic_row.log("audit_by_event (Manual Copy) complete")

Rule.commit_row_event(on_class=Employee, calling=audit_by_event)

But we may have hundreds of tables, and we want to do this just once. The problem is, events and rules are "tied" to a particular table (mapped class).

To enable us to address this with a single event, Logic Bank provides a rule called early_row_event_all_classes. This is a Generic Event Handler that applies to all tables, typically driven by meta data.

Let's use it to address date / time stamps. In examples/nw/logic/logic.py, you will find:

def handle_all(logic_row: LogicRow):
    row = logic_row.row
    if logic_row.ins_upd_dlt == "ins" and hasattr(row, "CreatedOn"):
        row.CreatedOn = datetime.datetime.now()
        logic_row.log("early_row_event_all_classes - handle_all sets 'Created_on"'')

Rule.early_row_event_all_classes(early_row_event_all_classes=handle_all)

This illustrates you can provide reusable services (here, time / date stamping), based on your naming conventions.

Unalterable Derivations

Generic event handling can also be used to enforce other desired global behaviors, such as

ensure derived attributes cannot be altered by clients

We extend the technique above, here in examples/custom_exceptions/logic/rules_bank.py:

def handle_all(logic_row: LogicRow):
    row = logic_row.row
    if logic_row.ins_upd_dlt == "ins" and hasattr(row, "CreatedOn"):
        row.CreatedOn = datetime.datetime.now()
        logic_row.log("early_row_event_all_classes - handle_all sets 'Created_on"'')

    if logic_row.nest_level == 0:  # client updates should not alter derivations
        derived_attributes = logic_row._get_derived_attributes()
        if logic_row.are_attributes_changed(derived_attributes):
            # NOTE: this does not trigger constraint_event registered in activate
            raise ConstraintException("One or more derived attributes are changed")


Rule.early_row_event_all_classes(early_row_event_all_classes=handle_all)

3. Extended Rules

Ok, good, but what if we want a new rule altogether? After all, the trick of Logic Bank is to identify patterns, and then create rules to automate them. The fun should not be restricted to Logic Bank authors - it should be enabled for everybody.

Here's how.

In this example, we want to define a reusable audit rule that monitors Employee changes:

if the ```Salary``` or ```Title``` are changed,
insert a row into ```EmployeeAudit```,
initialized with like-named ```Employee``` attributes.

Logic Bank enables providers to create Rule extensions that extend the 3 Event classes, and publish them as new rule types. Here, we utilize such a rule called nw_copy_row:

NWRuleExtension.nw_copy_row(copy_from=Employee,
                            copy_to=EmployeeAudit,
                            copy_when=lambda logic_row:
                                logic_row.are_attributes_changed([Employee.Salary, Employee.Title]))

Our rule extension operates are summarized in the diagram:

1. Our logic.py rule declaration (upper left code window) calls...

2. nw.rule_extensions.py (lower left code window), which invokes the constructor of our extended rule...

3. nw_copy.py (upper right panel, above), which is the implementation of our extended rule

Let's review some key aspects of this technique in the sections below.

Rule Discovery

Creating an extended rule is of little value if colleagues don't find it. Emails get lost. The new rule needs to be integrated into their IDE.

nw.rule_extensions.py "publishes" the rule, and provides documentation, in conjunction with the Python / IDE (here, PyCharm). This enables you to build multiple rules, and make them all IDE-discoverable as shown below, just by adding these helper functions.

Rule Definition

The NWCopyRow constructor (__init__) creates a rule instance of our “audit” rule, and saves the rule parameters in the rule instance (here, copy_to etc).

Note this class extends Event.

It then calls the superclass constructor. This logs the rule into the “rule bank”, an internal data structure that stores the rules for subsequent rule execution on session.commit().

Rule Execution

Since our rule extends Event and been logged into the "rule bank", it's visible to the Logic Bank rule engine. That means its execute() instance method is called when events are executed, with the logic_row as a parameter.

Our code thus executes at runtime, with access to:

• with rule parameters available as instance variables
• the row (wrapped in logic_row)

For example, the NWCopyRow code below executes the lambda function self.copy_when (saved by the constructor in the rule instance), passing logic_row as an argument:

copy_from = logic_row
do_copy = self.copy_when(copy_from)

Powerful Example: Allocation

Copy was a pretty simple example. The objective was to illustrate the technique, so that you can detect your own patterns, and devise reusable solutions, and publish them to your team.

The Logic Bank runtime engine includes copy_row in the runtime. That means you have access to the audit service when you pip install logicbank:

RuleExtension.copy_row(copy_from=Employee,
                       copy_to=EmployeeAudit,
                       copy_when=lambda logic_row:
                          logic_row.are_attributes_changed([Employee.Salary, Employee.Title]))

The Logic Bank runtime also includes a much more powerful example: Allocation.

The entire implementation is in examples/payment_allocation/logic/rules_bank.py:

def unpaid_orders(provider: LogicRow):
    """ returns Payments' Customers' Orders, where AmountOwed > 0, by OrderDate """
    customer_of_payment = provider.row.Customer
    unpaid_orders_result = provider.session.query(Order)\
        .filter(Order.AmountOwed > 0, Order.CustomerId == customer_of_payment.Id)\
        .order_by(Order.OrderDate).all()
    return unpaid_orders_result


def declare_logic():

    Rule.sum(derive=Customer.Balance, as_sum_of=Order.AmountOwed)

    Rule.formula(derive=Order.AmountOwed, as_expression=lambda row: row.AmountTotal - row.AmountPaid)
    Rule.sum(derive=Order.AmountPaid, as_sum_of=PaymentAllocation.AmountAllocated)

    Rule.formula(derive=PaymentAllocation.AmountAllocated, as_expression=lambda row:
        min(Decimal(row.Payment.AmountUnAllocated), Decimal(row.Order.AmountOwed)))

    RuleExtension.allocate(provider=Payment,
                           recipients=unpaid_orders,
                           creating_allocation=PaymentAllocation)

Details are provided here, but to summarize... when we insert a payment:

• the last rule allocates the payment to a set of outstanding orders.
• the other rules determine the AmountAllocated, and adjust the Customer.Balance.
• unpaid_orders is a function that returns the recipient orders