docs: update docs

Signed-off-by: Alex Collins <[email protected]>

README.md

@@ -56,29 +56,31 @@ Read in order:
 Beginner:
 
 * [Quick start](docs/QUICK_START.md)
+* [Concepts](docs/CONCEPTS.md)
+* [Sources](docs/SOURCES.md)
+* [Processors](docs/PROCESSORS.md)
+* [Sinks](docs/SINKS.md)
 * [Examples](docs/EXAMPLES.md)
-* [Steps](docs/STEPS.md)
-* [Sources](docs/SOURCES.md) and [Sinks](docs/SINKS.md)
-* [Garbage collection](docs/GC.md)
 
 Intermediate:
 
 * [Handlers](docs/CODE.md)
 * [Git usage](docs/GIT.md)
-* [Reliability](docs/RELIABILITY.md)
+* [Expression syntax](docs/EXPRESSIONS.md)
+* [Garbage collection](docs/GC.md)
 * [Scaling](docs/SCALING.md)
 * [Command line](docs/CLI.md)
-* [Expression syntax](docs/EXPRESSIONS.md)
-* [Meta-data](docs/META.md)
-* [Workflow interop](docs/WORKFLOW_INTEROP.md)
-* [Events interop](docs/EVENTS_INTEROP.md)
 * [Kubectl](docs/KUBECTL.md)
+* [Events interop](docs/EVENTS_INTEROP.md)
+* [Workflow interop](docs/WORKFLOW_INTEROP.md)
+* [Meta-data](docs/META.md)
 
 Advanced
 
 * [Configuration](docs/CONFIGURATION.md)
 * [Features](docs/FEATURES.md)
 * [Limitations](docs/LIMITATIONS.md)
+* [Reliability](docs/RELIABILITY.md)
 * [Metrics](docs/METRICS.md)
 * [Image contract](docs/IMAGE_CONTRACT.md)
 * [Jaeger tracing](docs/JAEGER.md)

docs/CONCEPTS.md

@@ -0,0 +1,47 @@
+# Concepts
+
+## Pipelines & Steps
+
+A **pipeline** is specified by a Kubernetes custom resource and consists of one or more **steps**. Each step is
+specified as a
+**processor**, one or more **sources**, and one or more **sinks**. Messages are read from the source, sent to the
+processor, and then maybe written the output to the sink.
+
+## Sources
+
+A source is somewhere to get messages from, e.g.:
+
+* HTTP - a HTTP endpoint, the HTTP body is the message,
+* Kafka - a Kafka topic
+* Jetstream - a Jetstream subject
+
+[Learn more](SINKS.md)
+
+## Processors
+
+A processor is a function that processes messages:
+
+* Built-in:
+    * Filter - filter out messages based on an expression
+    * Map - map messages to new messages
+* Code - run Golang or Python function
+* Git - checkout a function from git and run it.
+* Container - run a container image to process the function
+
+All processors are actually containers that just listen on http://localhost:8080/messages and messages are sent in the
+body of each request to that endpoint. If the endpoint returns a body, then it is sent to the sink.
+
+You can write a processor in any language that can receive HTTP requests, and we provide SDKs so you don't even need to
+write that. Dataflow takes care of the messaging for you.
+
+[Learn more](PROCESSORS.md)
+
+## Sinks
+
+A sink is somewhere a message can be sent to, e.g.:
+
+* Log - write the message to a log file
+* HTTP - make a HTTP request where the message is the body
+* Kafka - write to a Kafka topic
+
+[Learn more](SOURCES.md)

docs/FEATURES.md

@@ -21,7 +21,7 @@ alpha/beta/stable.
 | Expression based scaling | v0.0.90 | v0.0.128 | |
 | Filter step | v0.0.59 | v0.0.70 | |
 | FMEA tests | | v0.0.59 | |
-| [Generator step](STEPS.md#Generator-step) | v0.0.59 | | |
+| [Generator step](PROCESSORS.md#Generator-step) | v0.0.59 | | |
 | Graceful step termination | v0.0.59 | v0.0.128 | |
 | Group step | v0.0.59 | | |
 | [Git step](GIT.md) | v0.0.59 | v0.0.70 | |

docs/STEPS.md → docs/PROCESSORS.md

@@ -1,8 +1,8 @@
-# Steps
+# Processors
 
-A step is the execution unit of a Dataflow pipeline.
+A processor is the execution unit of a Dataflow pipeline.
 
-## Built-in Steps
+## Built-in 
 
 There are a number of built in steps. These are tested for reliability and performance and so should be you first
 choice: