Home

Introduction

dORM is a complete dynamic ORM( Object Relational Mapper ), that doesn't require to setup custom table structures per entity but is able to map dynamic entities on a database based on a couple of technical tables only.

What's the problem anyway...

Before we describe the solution, let's figure out what the problem is....

Typical applications use the normal ORMs that persist entities on table structures that are known at compile time predefined in the underlying dbms. While this is fine for most cases, there are some situations, where we need to configure entities dynamically. Think of a custom workflow definition ( e.g. with Camunda ) that wants to persist some complex internal state, which is not known upfront but will be configured during runtime.

Exactly this problem is solved by the current implementation.

Sample

Let's look a simple example first:

Assuming we have an injected ObjectManager which is responsible for transaction and object lifecycle management, we are able to specify an entity by defining the attributes including type and type constraint information

personDescriptor = manager.type("person")
   .add(attribute("name").type(string().length(100))) // string with length constraint
   .add(attribute("age").type(int().greaterThan(0)))
   .add(relation("parents").target("person").multiplicity(Multiplicity.ONE_OR_MANY).inverse("children"))
   .add(relation("children").target("person").multiplicity(Multiplicity.ZERO_OR_MANY).inverse("parents"))

   ...
   .register()

With this structural information - which is also persisted in the database - we can create and access DataObject instances, that carry the payload information

manager.begin()
try {
    val parent = manager.create(personDescriptor)

    // set some values by the custom get and set operators
    
    person["name"] = "Andi"
    person["age"] = 58

   val child = manager.create(personDescriptor)

    // set some values by the custom get and set operators
    
    child["name"] = "Nika"
    child["age"] = 14

    // add relation

    parent.relation("children").add(child) // will adjust the inverse parent relation automatically

}
finally {
    manager.commit() // will create!
}

Let's query all persons persisted so far...

val queryManager = manager.queryManager()
manager.begin()
try {
    val person = queryManager.from(personDescriptor)
    val query = queryManager
        .create()
        .select(person)
        .from(person)

    val queryResult = query.execute().getResultList()

    val name = queryResult[0]["name"]

    // let's modify values

    queryResult[0]["age"] = 30 // better!
}
finally {
    manager.commit() // will update all dirty objects
}

Since the objects are managed by the ObjectManager we can update values easily. The manager will know about any changes and will persist them at the end of the transaction.

Projections are possible as well, as seen here

val queryManager = manager.queryManager()
manager.begin()
try {
    val person = queryManager.from(personDescriptor)
    val children = person.join("children")
    val query = queryManager
        .create()
        .select(person.get("age"), person.get("name")
        .where(eq(children.get("name"), "Nika"))

    val tupleResult = query.execute().getResultList()

    val name = tupleResult[0][1]
}
finally {
    manager.commit()
}

In addition to a criteria api like query, we are of course able to specify hql like queries as well:

val queryManager = manager.queryManager()

manager.begin()
try {
    val query = manager.query<Any::class.java>("SELECT p.name FROM person AS p JOIN p.children as children WHERE children.name = :name")

    val queryResult = query.executor()
        .set("name", "Nika")
        .execute()
        .getResultList()

        ...
}
finally {
    manager.commit() // will update
}

Solution design

The solution is pretty straight forward. Entities are stored as a combination of three technical tables

• ENTITY a table referencing the entity definition and a generated primary key
• PROPERTY a table that will store single attributes of an entity
• RELATIONS a bridge table that stores relation

The property table defines the columns

• TYPE the id of the entity structure
• ENTITY the id of the corresponding entity
• NAME the property name

and a number of columns that are able to store payload data with respect to the supported low-level data types

• STRING_VALUE a string value
• INT_VALUE a long value ( stores boolean values well )
• DOUBLE_VALUE a floating point value

In order to model relations, the property table has a reflexive relation that expresses relationships stored in a bridge table RELATION.

As the definition of an entity is known, the engine will know which attributes are stored in which columns.

Let's look at a simple query, that will read a single person.

  select
        p.PROPERTY,
        p.ENTITY,
        p.DOUBLE_VALUE,
        p.INT_VALUE,
        p.STRING_VALUE,
        p.TYPE 
    from
        PROPERTY p 
    where
        p.ENTITY="person" and p.id=1

After reading the result set, the engine will create the appropriate DataObject instance and store the appropriate values in the correct places.

If we talk about queries, that code gets a little bit more complicated. Querying for an integer attribute "age" with the operator "=" and value 58 will result in something like

select
        p.ATTRIBUTE,
        p.ENTITY,
        p.DOUBLE_VALUE,
        p.INT_VALUE,
        p.STRING_VALUE,
        p.TYPE 
    from
        PROPERTY p 
    where
        p.ENTITY in (
            (select distinct p2.ENTITY 
               from PROPERTY p2 
              where
                    p2.TYPE="person" 
                    and p2.NAME="age"
                    and p2.INT_VALUE=58)) 
    order by
        p.ENTITY

Reference

Type Definition

Attribute definitions rely on a type concept that specifies base types ( e.g. integer vs. string ) including possible constraints.

The abstract base class

abstract class Type<T:Any>(val baseType: Class<T>) 

covers the information including a list of tests that will be executed, while validating a value. These tests are specified with a fluent syntax using methods of the base class and derived classes, which cover all Kotlin types, which are

• BooleanType
• CharacterType
• StringType
• ShortType
• IntType
• LongType
• FloatType
• DoubleType

Corresponding global functions - e.g. int(), string(), ... - are defined that instantiate a new instance.

Constraint Specification

• seal() disallows any additional constraint specifications.

Base Constraints

• required() specifies that nulls are not allowed
• optional() specifies that nulls are allowed

String Constraints

• fun length(length: Int) : StringType specifies the maximum length
• fun matches(expression: String) : StringType specifies a regular expression that needs to match

Boolean Constraints

• fun isTrue() : BooleanType only accepts true
• fun isFalse() : BooleanType only accepts false

Character Constraints

Numeric Constraints

• fun min(min: <Numeric>) : <Numeric>Type the minimum value
• fun max(max: <Numeric>) : <Numeric>Type the maximum value
• fun lessThan(min: <Numeric>) : <Numeric>Type the value needs to be < the passed value
• fun lessEqual(min: <Numeric>) : <Numeric>Type the value needs to be <= the passed value
• fun greaterThan(min: <Numeric>) : <Numeric>Type the value needs to be > the passed value
• fun greaterEqual(min: <Numeric>) : <Numeric>Type the value needs to be >= the passed value

Validation

Two main methods are implemented

• validate(obj: Any)
  validates the passed value and throws a ValidationError if at least one test fails.
• isValid(obj: Any) : Boolean
  returns true, if the value is valid, false otherwise

The ValidationError includes a list of violations, that contain

• type: Class<Any> the expected base type
• name: String the violated constraint which is always the same as the called methods ( e.g. ´lessThan´)
• parameter: Any? any parameter passed to the constraint, e.g. a max value
• value: Any the actual value

ObjectManager

The main class that controls the object meta-data and the lifecycle of objects is the class

Objectmanager

that is a regular spring component and can be injected.

Transaction Management

All operations - both meta-data and object related - need to be executed within a transaction. The ObjectManager offers the appropriate methods

• begin() start a new transaction
• commit() commits a running transaction and flushes all required database changes
• rollback() rollbacks a running transaction

Object Definition

Every object needs to be declared in the first step. The method

type(name: String)

creates a builder object, that is used to specify attributes and relations. A final register() call will finalize the specification.

Example:

   objectManager.begin()

   try {
      objectManager.type("person")
         .add(attribute("name").type(string()))
         .add(attribute("age").type(int()))

      objectManager.commit()
   }
   catch(error: Error) {
      objectManager.rollback() // can happen, for example if the object is already registered
   }

The add method is used to describe attributes or relations.

Attribute

The function attribute(name: String) will return a builder object that offers the methods

• type(type: Type<*>) specifies the type

Relation

The function relation(name: String) will return a builder object that offers the methods

• target(target: String) specifies the target object of this relation by name
• inverse(inverse: String) specifies the inverse relation name
• multiplicity(multiplicity: Multiplicity) specifies the multiplicity of the relation, which is an enum containing ZERO_OR_ONE, ONE, MANY, ZERO_OR_MANY
• cascade(cascade: Cascade) specifies cascading effect. Currently this is only the enum value DELETE which is required in case of an aggregation.

Three technical attributes are added automatically

• id a generated int id
• versionCounter a version counter used for optimistic locking
• entityStatus

The entity status in turn consists of

• created the date and time of creation ( a LocalDateTime )
• createdBy the user that created the object
• modified the date and time of modification ( a LocalDateTime )
• modifiedBy the user that modified the object

In order to reference the session user, a SessionContext has to be added to the spring context, that is initialized with the interface SessionProvider.

interface SessionContextProvider {
    fun getUser() : String
}

Example:

@Component
class DummySessionContext : SessionContext(DummySessionContextProvider("me")) {}

Lifecycle

Create

Objects are created via the ObjectManager method

create(descriptor: ObjectDescriptor)

the passed meta-data in turn can be fetched by calling

findDescriptor(name: String)

the newly constructed object is an instance of the class DataObject which has a generated long primary key ( with the attribute name "id" ) and is initialized by default values with respect to the datatypes

• String: empty string
• Boolean: false
• Numeric values: 0 ( or 0.0 )
• object reference: null
• relation: empty set object ( instance of MultiValuedRelation)

Access

The DataObject operator [] is overloaded in order to access and modify values.

Be aware that the set operator will directly validate the passed value.

Accessing single value relations is also achieved by the operator. Only for multi value relation we need another method

relation(name: String) : MultiValuedRelation

which casts to the internal relation object, which implements the Set interface.

Update

The operators are used to modify object values. Since the objects manager is aware of all managed objects, a commit-operation will flush the necessary changes to update the database accordingly.

In case of relations, all modifications will affect the inverse relations as well. For example, setting a value of a 1:n relation to null will remove the containing object from the inverse relation. ( Something that hibernate was obviously not able to do )

Delete

The DataObject method

delete() marks the object for deletion, which will be executed while committing the transaction. The delete operation may cascade on relations, if they are marked with the DELETE cascade mode.

Query

The framework offers two distinct query possibilities

• a functional query specification ( similar to the criteria api of JPA )
• a DSL based specification ( similar to HQL )

You may be pretty familiar with the offered functionality since it looks pretty much like JPA :-)

In both cases the framework will flush all objects to the database that may affect a query result!

API

Basis for query execution is the QueryManager that is able to create Query objects.

Example:

   val queryManager = objectManager.queryManager()
   val query = queryManager.create() // this will create a `Query<DataObject>` returning `DataObject`s

   ...

   val result = query.execute().getResultList()

Additional fluent methods will modify the query:

• fun select(vararg path: ObjectPath) : Query<T>
• fun from(root: AbstractFrom) : Query<T>
• fun where(expression: ObjectExpression) : Query<T>

The starting point for expressions and select targets are object references suitable for query expressions which are created by calling

queryManager.from(<object-descriptor>)

additional fluent methods can be used to reference

• attributes by calling get(<attribute-name>), or
• relations by calling join(<relation-name>)

Predicates are expressed by the global functions

• eq equals
• ne not equals
• lt less than
• le less or equal
• gt greater than
• ge greater for equals

and the logical predicates and and or

Let`s look at some examples

Example: Object query

val person = queryManager.from(objectManager.getDescriptor("person"))

val query = queryManager
              .create()
              .select(person)
              .from(person)
              .where(eq(person("name"), "Nika"))

Example: Tuple query

val person = queryManager.from(objectManager.getDescriptor("person"))

val query = queryManager
              .create(Array<Any>::class.java) // that's the generic return type
              .select(person.get("name“))
              .from(person)
              .where(eq(person.get("name"), "Nika"))

Example: join query

val person = queryManager.from(objectManager.getDescriptor("person"))
val children = person.join("children")

val query = queryManager
              .create(Array<Any>::class.java)
              .select(person.get("name"))
              .from(person)
              .where(eq(children.get("name"), "Nika"))

HQL

The DSL part is a subset of the JPA specification.

Example:

val query = objectManager.query<DataObject>(
                    "SELECT p " +
                    "  FROM person AS p " +
                    "  JOIN p.children child" +
                    "  WHERE child.name = :name")

val result = query.executor()
                .set("name", "Nika")
                .execute()
                .getResultList()

In case of parameters, we need an intermediate object - created with executor() - that is able to set the named parameters!

Missing query features

A lot of features are still missing, although some of them would be easy to implement

• no functions ( upper, lower, ...)
• no relations possible in the select clause ( e.g. "select p.children from ..." )
• no implicit joins ( e.g. "select p.children.name from ..." )
• ...

The task is not that hard, since we internally rely on jpa as well and in many cases would only have to add the missing functional parts ( as well as in the parser, which is based on the original jpa antler grammar, btw. ) and map them 1:1 on the criteria api.

I'll invest time as someone is using and requesting it :-)

JSON Support

The Jackson modules TypeModule, ObjectModule and ObjectDescriptorModule can be used to add JSON serialization and deserialization support for type and object meta-data und object instances.

Registering them with the object mapper will do the job.

    @Bean
    @Primary
    fun objectMapper(): ObjectMapper {
        return ObjectMapper()
            .registerModule(ObjectDescriptorModule())
            .registerModule(ObjectModule())
            .registerModule(TypeModule())
    }

Some remarks on the logic:

• the serializer requires a running transaction and is able to cope with cycles, and will insert { @ref: <ref> } objects.
• the deserializer will create its own transaction and will overwrite or create objects on the fly.

GraphQL

As i implemented the JSON serializer / deserializer i was not that happy with the handling of relations and thought i could give GraphQL a try. It turned out to be pretty straight forward with the help of the java framework GraphQL.

The maven module dorm/graphql sits on top of the dorm/core module and transforms all object definitions to a dynamic GraphQL schema.

With the integrated graphql endpoint it will serve all queries and mutations.

Schema Creation

As a first step, a number of technical schema artifacts are created.

type OperationResult {
  count: Int
}

input FloatFilter {
  eq: Float
  ge: Float
  gt: Float
  le: Float
  lt: Float
  ne: Float
}

input IntFilter {
  eq: Int
  ge: Int
  gt: Int
  le: Int
  lt: Int
  ne: Int
}

input DateFilter {
  eq: DateTime
  ge: DateTime
  gt: DateTime
  le: DateTime
  lt: DateTime
  ne: DateTime
}

input StringFilter {
  eq: String
  ne: String
}

input BooleanFilter {
  eq: Boolean
  ne: Boolean
}

For every found object definitions , it will create

type <object> {
    <property>: <type>
}

type <object>Input {
   <property>: <type>
}

type <object>Filter {
   and: [<object>Filter])
   or: [<object>Filter])

   <property>: <property-type>Filter
}

and as the overall result:

type Query {
  <object>(where: <objectFilter): [<object-type>]
}

type Mutation {
  create<object(input: <object>Input): <object>
  delete<object>s(where: <object>Filter): OperationResult
  update<object(input: <object>Input): <object>
  update<objects(input: <object>Input, where:<object>Filter): [<object>]
}

Example: Let's take the Person example

type Person {
  id: Int
  versionCounter: Int
  name: String
  age: Int
  children: [Person]
  father: Person
}

input PersonInput {
  id: Int
  name: String
  age: Int
  children: [PersonInput]
  father: PersonInput
}

input PersonFilter {
  and: [PersonFilter]
  or: [PersonFilter]

  id: IntFilter
  name: StringFilter
  age: IntFilter
  
  father: PersonFilter
}

type Query {
  Person(where: PersonFilter): [Person]
}

type Mutation {
  createPerson(input: PersonInput): Person
  deletePersons(where: PersonFilter): OperationResult
  updatePerson(input: PersonInput): Person
  updatePersons(input: PersonInput, where: PersonFilter): [Person]
}

Queries

As seen in the schema, you can query objects using all comparisons possible on all properties, which are either attributes or relations, including the possibility to add more complex and and or combinations.

Example: A query joining a relation

query sampleQuery {
   Person (
      where: {
         father: {name: {eq: "Andi"}}
      }
    ) {    
      id
      name
      father {
         id
         name
      }
   }
}

Example: and combination

query sampleQuery {
   Person (
      where: { and: [
        { father: {name: {eq: "Andi"}}}
        { age: {gt: 10}}
      ]}
    ) {    
      id
      name
      father {
         id
         name
      }
   }
}

Mutations

Create

Example:

mutation {
    createPerson(input: {
        age: 59, 
        name: "Sammy",
        father: {id: 1}
        }) {
        id
        name
        father {
            id
            name
        }
    }
}

Update

Example: Single update

mutation {
    updatePerson(
       input: { id: 1, age: 59, firstName: "Andi", name: "Ernst" }) {
        id
        name
        firstName
    }
}

Example: Bulk update

mutation {
    updatePersons(
       where: {age: {gt: 10}}, 
       input: { firstName: "Andi" }) {
        id
        name
        firstName
    }
}

Delete

Example:

mutation {
    deletePersons(where: {
        age: {gt: 50}
        }) {
        count
    }
}

Spring Setup

As both the core and the GraphQL part are simple spring beans, the setup consists more or less of providing the necessary spring environment for jpa and the web-part, and a minimal configuration consisting of adding the session context and referencing a configuration class.

Example:

@Component
class DummySessionContext : SessionContext(DummySessionContextProvider("me")) {}

@Configuration()
@ComponentScan
@Import(DORMConfiguration::class)
class ApplicationConfiguration {
    @Autowired
    lateinit var objectManager: ObjectManager

    @PostConstruct
    fun createData() {
        TestData(objectManager)
    }
}

@SpringBootApplication
class DORMApplication {
    companion object {
        @JvmStatic
        fun main(args: Array<String>) {
            SpringApplication.run(DORMApplication::class.java, *args)
        }
    }
}

If the graphql module is referenced, the graphql endpoint will also be created.

Check the code of the demo application