Redesign handling of pipeline-state/history

[flosell]

Problem

Currently, the complete build history is saved to disk and pulled into memory (and kept there) at startup. This is neither scalable nor efficient and needs to change.

Objective

• Implement handling of pipeline state that can scale, by only keeping what's needed in memory and accessing everything else from disk.
• Provide a clean API that can support this and is usable for a variety of use-cases

Context

Users of Pipeline State:

• UIs (lambdacd, sroidl/lambda-ui)
• lambdacd-cctray
• SimonMonecke/lambdacd-mongodb
• retriggering

Related issues

• Keep history of pipeline structure Keep history of the pipeline structure #6
• Step Metadata Introduce a concept of step-metadata #86
• Build Metadata Introduce the concept of build-metadata #138

Potential Solution

API

• Updating State:
  • consume-step-result-update [state build-number step-id step-results]
  • consume-pipeline-structure [state build-number pipeline-structure]
  • _Metadata (done in Introduce the concept of build-metadata #138)
• Querying State
  • next-build-number [state] (can be arbitrary but must be sortable and ascending)
  • get-build [state build-number] (returns structure which contains everything known about this build)
  • build-numbers [state] (returns stored build-numbers, e.g. for building a history)

Supporting functions

• Querying the build-structure
  • Get step-result
  • Get pipeline structure
  • Get step results?
  • TBD Get metadata (done in Introduce the concept of build-metadata #138)

Default Persistence

• Decouple state updates from flushing to disk -> Persistence: Decouple step-updates from flushing to disk to improve performance/resource usage #137
• Each build gets one directory containing
  • state of each build step
  • metadata (tbd)
  • structure of pipeline (needs to include function metadata; use the same data for UI so we can be sure UI also works for older pipelines read from disk)

Assumptions

• Information about one build is not too big so we can fully load it into memory even if we just need the metadata, or one particular step
• It is efficient enough to query a couple of builds, e.g. to build the history

Additional tasks

• Implement contract-test suite for persistence components? (optional)
• Implement new protocols for default pipeline state persistence
• Basic Documentation: https://github.com/flosell/lambdacd/wiki/Pipeline-State

[flosell]

Basic implementation done, metadata and optimizations are handled in separate issues.