From 93b74d0b75a02a857a3f98b49da0de2c0116d24c Mon Sep 17 00:00:00 2001 From: Luca Pette Date: Wed, 27 Dec 2023 11:40:02 +0100 Subject: [PATCH] docs: restructure the roadmap (#87) --- .github/ISSUE_TEMPLATE/bug_report.md | 22 +++-- .github/ISSUE_TEMPLATE/feature_request.md | 19 +++++ .github/ISSUE_TEMPLATE/support_request.md | 7 ++ docs/docs/reference/language/spec.md | 6 ++ docs/docs/roadmap.md | 98 ++++++++++++++++------- 5 files changed, 110 insertions(+), 42 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 .github/ISSUE_TEMPLATE/support_request.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index e2c24c1..682f5f7 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,28 +1,26 @@ --- name: Bug report about: Create a report to help us improve -title: '' +title: "" labels: bug -assignees: '' - +assignees: "" --- -**Describe the bug** +## What's wrong? A clear and concise description of what the bug is. -**To Reproduce** - -Steps to reproduce the behavior: - -**Expected behavior** +## Expected behavior A clear and concise description of what you expected to happen. -**Screenshots** +## Steps to reproduce + +Steps to reproduce the behavior: -If applicable, add screenshots to help explain your problem. +- Go to '...' +- Run '....' -**Additional context** +## Additional context Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..8d50ee0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,19 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: "" +labels: enhancement +assignees: "" +--- + +## What + +A clear and concise description of what you want this feature to do. + +## Motivation + +A clear and concise description of the goals of the feature. + +## Open questions + +If any, list open questions here. diff --git a/.github/ISSUE_TEMPLATE/support_request.md b/.github/ISSUE_TEMPLATE/support_request.md new file mode 100644 index 0000000..68f0705 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/support_request.md @@ -0,0 +1,7 @@ +--- +name: Support request +about: Ask a question about this project +title: "" +labels: question +assignees: "" +--- diff --git a/docs/docs/reference/language/spec.md b/docs/docs/reference/language/spec.md index 4eeea40..24270bb 100644 --- a/docs/docs/reference/language/spec.md +++ b/docs/docs/reference/language/spec.md @@ -1,5 +1,11 @@ # Specification +:::warning + +This document is a work in progress. It's not complete and it's not final. + +::: + The purpose of this document is to spec `TypeStream` language features. While `TypeStream` syntax is inspired by diff --git a/docs/docs/roadmap.md b/docs/docs/roadmap.md index 7814cfe..98c69de 100644 --- a/docs/docs/roadmap.md +++ b/docs/docs/roadmap.md @@ -2,42 +2,80 @@ A loose outlook of where `TypeStream` is headed. -## Short term +There's no particular order to this list of lists at the moment. This reflects +the current state of ideas we may or may not implement in the near feature. -- Language - - introduce functions and expression statements like: - - `answers=$(cat questions | grep answers); grep $answers 42 > result_42; grep $answers 24 > result_24` -- Commands (these imply supporting more types) - - find - - tr -- Serialization - - extend support for `protobuf` - - Add JSON schema encoding support +To get a better sense of the upcoming features, check out the +[issues](https://github.com/typestreamio/typestream/issues). -## Medium term +## Language -- Introduce new sources: - - pulsar - - postgresql -- Introduce naming strategies for: graph nodes, `TypeStream` apps. +Refer to the [specs](reference/language/spec.md) for the current state of the language. -## Long term +- introduce functions and expression statements like: + - `answers=$(cat questions | grep answers); grep $answers 42 > result_42; grep $answers 24 > result_24` +- support for `*` expansion +- support list values (it's necessary for the `*` expansion and commands like find and tr) +- support for DataStream Values (commands like find and ls imply this) -- Introduce a [job planner](#job-planner). -- Virtual filesystem - - Introduce setting for displaying topics with conventions (vs code uses - patterns for folders and maybe it's more flexible). It has to be cluster - specific. - - `/media` filesystem so we can expose data like `/media/websocket/server1` +## New commands -### Job planner +Since it's early days, some commands will require work at the language level but +we still have a separate section for them so it's clearer what commands we know +we want to support. -It would be interesting to have a streaming job planner. Here's an example: +- find +- tr +- [tee](https://github.com/typestreamio/typestream/issues/78) +- mount (see [Filesystem](#filesystem)) -```sh -grep topic answer > result # first job -grep topic answer > another.topic # second job -``` +## Filesystem -obviously it feels like we can reuse `grep topic answer`. Can we? If so, the -planner would be responsible to answer this question. +The virtual filesystem is a core component of `TypeStream`. It's how the +metaphor "everything is a stream" is implemented. + +- `/media` filesystem (aka "mounting") + - websocket server + - http server + - jdbc server +- There's no way to get the output of a long running program. We want to make + this work [How to view the output of a running process in another bash + session?](https://unix.stackexchange.com/questions/58550/how-to-view-the-output-of-a-running-process-in-another-bash-session) + at filesystem level + +## CI + +We could always have more tests... right? 😃 + +- integration tests (defined as cli + server interactions. The whole stack only + basically) + - kubernetes features + - config map + - need to test the server and worker at integration level + - This test suite must be fully integrated with the release process (as we + want to run it on release branches automatically. Should also be possible to + run manually or trigger in pull requests) + +## Server + +- Failure handling strategy + - we have to restart watchers if they fail (retry until a certain threshold + and then crash the server?) + - if the catalog fails, we can't type check but the errors are unclear (as we + just say we can't find the type) + - Add an exception handler for kafka streams applications. You don't want them + to blow up but to log stuff + +## Encoding + +- schema enum isn't really efficient (as it carries the type in each instance, we need a container type for this) +- [Avro](https://avro.apache.org) + - support duration + - some types metadata isn't carried over (see logical types with precision and scale for example) +- [Protocol Buffers](https://protobuf.dev/) + - Introduce "SmokeType" first so we can support: + - google imports + - nested types + - multiple messages in the same definition +- [json schema](https://json-schema.org) +- [cloud events](https://cloudevents.io)