Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow configuring per-stream persistability #207

Merged
merged 11 commits into from
May 5, 2023
Merged

feat: allow configuring per-stream persistability #207

merged 11 commits into from
May 5, 2023

Conversation

de-sh
Copy link
Member

@de-sh de-sh commented Apr 21, 2023 •
edited
Loading

Closes #

Changes

Why?

The user must be able to choose if a stream should be persisted or the data it carries is not important enough and it's loss can be ignored.

Trials Performed

  1. Run uplink with imu stream configured to be non-persistent.
  2. Attach simulator to uplink such that it generates imu, gps, device_shadow, etc.
  3. Stop uplink and verify backup file contains all but for the imu stream with backup deserializer utility.

Uplink logs

2023-04-21T16:56:47.775976Z  INFO uplink::base::serializer: Switching to slow eventloop mode!!

  2023-04-21T16:56:47.777330Z  INFO uplink::base::serializer:              slow: batches = 10  errors = 0 lost = 0 disk_files = 0   write_memory = 0 B read_memory = 0 B

  2023-04-21T16:56:47.879427Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:48.671691Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 3   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:56:49.127637Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:50.377976Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:51.618219Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:51.674543Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 4   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:56:52.663594Z  INFO uplink::collector::utils::streams:                  bms: points = 217   batches = 2     latency = 4123

  2023-04-21T16:56:52.663691Z  INFO uplink::collector::utils::streams:     peripheral_state: points = 58    batches = 0     latency = 0

  2023-04-21T16:56:52.663709Z  INFO uplink::collector::utils::streams:                motor: points = 219   batches = 2     latency = 4088

  2023-04-21T16:56:52.663725Z  INFO uplink::collector::utils::streams:        device_shadow: points = 58    batches = 58    latency = 7007

  2023-04-21T16:56:52.663739Z  INFO uplink::collector::utils::streams:                  imu: points = 483   batches = 4     latency = 7087

  2023-04-21T16:56:52.663756Z  INFO uplink::collector::utils::streams:                  gps: points = 58    batches = 0     latency = 0

  2023-04-21T16:56:52.867780Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:54.112080Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:54.678228Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 5   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:56:55.349182Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:56.604641Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:57.682486Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 6   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:56:57.777557Z  INFO uplink::base::serializer:              slow: batches = 117 errors = 0 lost = 0 disk_files = 0   write_memory = 685.39 kB read_memory = 0 B

  2023-04-21T16:56:57.841411Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:56:59.076817Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:00.316933Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:00.686288Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 7   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:57:01.559450Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:02.664670Z  INFO uplink::collector::utils::streams:                  bms: points = 361   batches = 3     latency = 5052

  2023-04-21T16:57:02.664753Z  INFO uplink::collector::utils::streams:     peripheral_state: points = 98    batches = 1     latency = 4226

  2023-04-21T16:57:02.664764Z  INFO uplink::collector::utils::streams:                motor: points = 365   batches = 3     latency = 4956

  2023-04-21T16:57:02.664777Z  INFO uplink::collector::utils::streams:        device_shadow: points = 98    batches = 98    latency = 4995

  2023-04-21T16:57:02.664793Z  INFO uplink::collector::utils::streams:                  imu: points = 806   batches = 8     latency = 4551

  2023-04-21T16:57:02.664803Z  INFO uplink::collector::utils::streams:                  gps: points = 98    batches = 1     latency = 4213

  2023-04-21T16:57:02.787696Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:03.688620Z ERROR uplink::base::mqtt:                        disconnected: reconnects = 8   publishes = 0   pubacks = 0   pingreqs = 0   pingresps = 0  

  2023-04-21T16:57:04.033854Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:05.279754Z  WARN uplink::base::serializer: Not persisting stream: imu

  2023-04-21T16:57:06.013841Z  INFO disk: Flushing data to disk!! "/tmp/uplink/backup@0"

❯ ../deserialize-backup -s 256000 -d /tmp/uplink
╭──────────────────┬──────────────────────┬───────┬──────────────┬───────────┬────────────┬─────────────────┬───────────────┬──────────────╮
│ stream_name      │ serialization_format │ count │ message_rate │ data_size │ data_rate  │ start_timestamp │ end_timestamp │ milliseconds │
├──────────────────┼──────────────────────┼───────┼──────────────┼───────────┼────────────┼─────────────────┼───────────────┼──────────────┤
│ gps              │ jsonarray            │ 100   │ 9.874593 /s  │ 8.1 KiB   │ 816.9 B/s  │ 1682096206749   │ 1682096216876 │ 10127        │
│ device_shadow    │ jsonarray            │ 178   │ 9.8201475 /s │ 33 KiB    │ 1.8 KiB/s  │ 1682096207877   │ 1682096226003 │ 18126        │
│ motor            │ jsonarray            │ 700   │ 36.52682 /s  │ 167.9 KiB │ 8.8 KiB/s  │ 1682096206674   │ 1682096225838 │ 19164        │
│ bms              │ jsonarray            │ 700   │ 36.201904 /s │ 910.8 KiB │ 47.1 KiB/s │ 1682096206674   │ 1682096226010 │ 19336        │
│ peripheral_state │ jsonarray            │ 100   │ 9.861933 /s  │ 20.5 KiB  │ 2 KiB/s    │ 1682096206749   │ 1682096216889 │ 10140        │
╰──────────────────┴──────────────────────┴───────┴──────────────┴───────────┴────────────┴─────────────────┴───────────────┴──────────────╯
NOTE: timestamps are relative to UNIX epoch and in milliseconds and data_rate is in units of points/second

Aggregated values
╭─────────────┬──────────────────────┬───────┬──────────────┬───────────┬───────────┬─────────────────┬───────────────┬──────────────╮
│ stream_name │ serialization_format │ count │ message_rate │ data_size │ data_rate │ start_timestamp │ end_timestamp │ milliseconds │
├─────────────┼──────────────────────┼───────┼──────────────┼───────────┼───────────┼─────────────────┼───────────────┼──────────────┤
│ total       │ jsonarray            │ 1778  │ 91.952835 /s │ 1.1 MiB   │ 59 KiB/s  │ 1682096206674   │ 1682096226010 │ 19336        │
╰─────────────┴──────────────────────┴───────┴──────────────┴───────────┴───────────┴─────────────────┴───────────────┴──────────────╯

de-sh
de-sh commented Apr 26, 2023
View reviewed changes
uplink/src/base/bridge/mod.rs Outdated Show resolved Hide resolved
@de-sh de-sh requested a review from tekjar April 28, 2023 13:02
@de-sh de-sh changed the base branch from main to next May 1, 2023 13:01
@de-sh de-sh merged commit bd800ce into next May 5, 2023
de-sh added a commit that referenced this pull request May 15, 2023
@tekjar @de-sh
feat: per-stream compression and optional disk persistence (#231)
4f748fd 
* feat: implement lz4 compression (#204)

* feat: implement lz4 compression

* fix: compression should be opt-in

* feat: allow configuring compressibility per-stream (#209)

* feat: configuring per-stream compressibility

* doc: explain `is_compressible` config

* feat: compression as an enum

* fix: remove global `enable_compression`

* refactor: `compress` ~> `lz4_compress`

* fix: broken merge

* feat: allow configuring per-stream persistability (#207)

* feat: allow configuring non-persistent streams

* doc: add example config for non-persistent stream

* fix: serializer test

* fix: `is_persistable` ~> `persistance`

* fix: normal ~> slow even if non-persistable data

* fix: logs are always persisted

* Merge branch 'next' into non-persist

* fix: use compressed stream topic as is from config (#230)

* feat!: per-stream persistence to disk (#229)

* Guard log config in mac

* Extract some methods into persistence struct

* Finish storage module with optional persistence

* feat: use `storage` instead of `disk` (#223)

* fix: ensure test run successfully

* feat: `use storage` not `disk`

* feat: restructure persistence config and use

* fix: serializer tests

* refactor: remove old `disk`

* feat: trace lost segment with debug log

* style: clippy suggestion

* fix: load previously persisted files

* fix: comment out unused code

* refactor: simplify serializer test

* feat: add test to ensure files removed post flush (#228)

* fix: add partial read file presence check

* feat: completely per-stream persistance

* fix: serializer metrics

* doc: explain persist/compress configs

* refactor: stream topic

* fix: ensure all backlogs are cleared on catchup

* refactor: unnecessary result

* fix: default to `MAX_BUFFER_SIZE`

* fix: add missing import

* fix: append stream name to path in persistence

* refactor: `StorageHandler`

* fix: finish merge

* doc: persistence config change

* fix: remove unncessarily included config variable

* fix: metrics updation

* refactor: `StorageHandler.next()` returns `Option`

---------

Co-authored-by: tekjar <[email protected]>

* style: clippy suggestion

---------

Co-authored-by: Devdutt Shenoi <[email protected]>
@de-sh de-sh deleted the non-persist branch September 6, 2023 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tekjar tekjar Awaiting requested review from tekjar

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@de-sh