Reduce timing log chatter

What's Changed

• 90%: Migrate build to circleci by @rsinger in #139
• Change timingLog level from INFO to DEBUG by @astilla in #138

New Contributors

• @astilla made their first contribution in #138

Full Changelog: 0.19.0...0.20.0

Pin composer version to 1.10.17

PLAT-4051 pin composer version to 1.10.17 (#136)

• pin composer version to 1.10.17 in build.xml

Remove ref to getStoreStatPath

Remove ref to getStoreStatPath (#132)

* Remove ref to getStoreStatPath

* Remove test refs

* Updated syntax

StatsD class should use non-blocking UDP connections

The implementation of our StatsD class which sends statistics via a UDP socket was being opened by default in blocking mode, which forces client code to wait. This release corrects this behaviour and ensures the socket is opened in non-blocking mode.

Optionally return cursor for table `results` value

This release introduces a couple of new options for \Tripod\Mongo\Composites\Tables->getTableRows():

• returnCursor => true: will return a \MongoDB\Driver\Cursor as the value of the results property.
• includeCount => false: will skip getting a total hit count - avoids a performance hit if returning all results via a cursor - the actual individual results returned are identical to returning them as an array
• documentType => '{some BSONUnserializable class}': gives the ability to rewrite the table row documents as returned from the cursor, so no post-processing is necessary.

Batch Composite Generation

Previously, Tripod composites (Tables, Views, Search Documents) would send each individual matching CBD document as an ApplyOperation job, despite the fact that ApplyOperation was designed to handle this in bulk. This could cause bottlenecks by queuing massive amounts of Resque jobs, each doing one operation for one subject.

This release allows the composites to be generated in configurable batches: Views, by default, are set to a much lower threshold than Tables, for instance, since Views can be much more intensive to build.

Config generators and skinny Resque jobs

This release brings a significant change, but should be backwards compatible and shouldn't rely on any changes to upgrade.

That said, you will see some deprecation notices.

The main change is the introduction of 'Config Generators'. Before explaining what these are, it helps to understand what motivated their need.

Previously, Tripod's configuration was managed via Tripod\Mongo\Config, which took a fully fleshed out array of configuration options and loaded it as a singleton which was reused throughout the lifecycle of the session.

However, it's necessary that the asynchronous workers also have access to the same config. To accomplish that simply and independently (i.e. workers didn't need to know anything about the environment that created them), Tripod passed the entire configuration as a job argument. Given that Tripod requires a lot of configuration to work, this led to extremely enormous jobs (our average Resque job size was ~46KB) with massively redundant data (while we have about 100 different configurations we pass to Tripod, they rarely change) which then requires considerably larger memory overhead in Redis.

Instead, we've now provided the option to replace \Tripod\Mongo\Config with your own objects that implement \Tripod\Mongo\IConfigInstance. These classes need a serialize instance method and deserialize class method that can provide all the context needed to fill out the config.

This is probably easiest to visualize using an example from the test suite. We have created an extremely simple config generator that gets its config from a static JSON file and extends \Tripod\Mongo\Config.

<?php
class FileBasedConfigGenerator extends \Tripod\Mongo\Config
{
    protected $fileName;
    private function __construct()
    {
    }
    public function serialize()
    {
        return ['class' => get_class($this), 'filename' => $this->fileName];
    }
    public static function deserialize(array $config)
    {
        $instance = new self();
        $instance->fileName = $config['filename'];
        $cfg = json_decode(file_get_contents($config['filename']), true);
        $instance->loadConfig($cfg);
        return $instance;
    }
}

This basically uses all of the plumbing that \Tripod\Mongo\Config has already laid out, but the footprint in the jobs would be:

{
    'class' => 'FileBasedConfigGenerator',
    'filename' => '/path/to/your/config.json'
}

Then, assuming that class FileBasedConfigGenerator is available to Tripod and its workers, you are ready to go.

There are some changes to Tripod that were required for this, however. Previously, you would set your Tripod config like this:

$conf = some_way_to_get_configuration_array();
\Tripod\Mongo\Config::setConfig($conf);

And access the configuration instance like this:

\Tripod\Mongo\Config::getInstance()

This will still work for the time being, but you will get warning log lines with a deprecation notice.

Instead, you will now do something more like:

$conf = ['class' => 'MyConfigGenerator']; // although you can still use the exact same method as above
\Tripod\Config::setConfig($conf); // Note the change of namespace

\Tripod\Config::getInstance();

The last line here will call \Tripod\ConfigFactory and instantiate your config instance if it hasn't already been created.

Since we were using our own experience earlier for reference, to show the effect of this, the same Resque job size mentioned that was ~46KB went down to ~1.5KB, a nearly 97% decrease.

Allow non-expiring views

This release introduces the notion of 'non-expiring' views by setting a ttl property of -1 in the view's spec.

The purpose of this enhancement is to allow generation of views that will never change directly, and thus have no need for an impact index (but also have no need to expire).

This change does have an effect on existing views with ttl properties: if the RDF type of the view's resource changes, it will regenerate the view's graph, regardless of whether or not the TTL has been met.

Cleanup composites on bulk generation

This release introduces a mechanism (via \Tripod\Mongo\JobGroup) to track bulk composite generation jobs and will delete any stale composites when all jobs are complete.

It does this by adding a _cts property to the composite documents (identical to what's always existed on the CBD documents) to track when a composite was created.

Fix pass by reference errors in PHP 7.0

Some of the methods in ExtendedGraph had code that needed to be split out to not generate errors in PHP 7.0.