Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: tool to analyze backlog files in persistence #208

Merged
merged 12 commits into from
Apr 28, 2023
Merged

Conversation

de-sh
Copy link
Member

@de-sh de-sh commented Apr 21, 2023

Closes #

Changes

Why?

Files from backlog in persistence can give the user important information of the kind of data being persisted to disk by uplink

Trials Performed

  1. Run uplink with persistence enabled
  2. Attach simulator to generate gps, device_shadow, etc. data
  3. ensure network is down and uplink is persisting to disk
  4. stop uplink and run the utility to analyze contents of persistence

example output generated with simulator generating data on streams gps, device_shadow and peripheral_state every 100ms, on motor and bms streams every 25ms:

❯ cargo run -- -s 256000 -d /tmp/uplink
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/deserialize-backup -s 256000 -d /tmp/uplink`
╭──────────────────┬──────────────────────┬───────┬──────────────┬───────────┬────────────┬─────────────────┬───────────────┬──────────────╮
│ stream_name      │ serialization_format │ count │ message_rate │ data_size │ data_rate  │ start_timestamp │ end_timestamp │ milliseconds │
├──────────────────┼──────────────────────┼───────┼──────────────┼───────────┼────────────┼─────────────────┼───────────────┼──────────────┤
│ peripheral_state │ jsonarray            │ 100   │ 9.861933 /s  │ 20.5 KiB  │ 2 KiB/s    │ 1682096206749   │ 1682096216889 │ 10140        │
│ bms              │ jsonarray            │ 700   │ 36.201904 /s │ 910.8 KiB │ 47.1 KiB/s │ 1682096206674   │ 1682096226010 │ 19336        │
│ device_shadow    │ jsonarray            │ 178   │ 9.8201475 /s │ 33 KiB    │ 1.8 KiB/s  │ 1682096207877   │ 1682096226003 │ 18126        │
│ motor            │ jsonarray            │ 700   │ 36.52682 /s  │ 167.9 KiB │ 8.8 KiB/s  │ 1682096206674   │ 1682096225838 │ 19164        │
│ gps              │ jsonarray            │ 100   │ 9.874593 /s  │ 8.1 KiB   │ 816.9 B/s  │ 1682096206749   │ 1682096216876 │ 10127        │
╰──────────────────┴──────────────────────┴───────┴──────────────┴───────────┴────────────┴─────────────────┴───────────────┴──────────────╯
NOTE: timestamps are relative to UNIX epoch and in milliseconds and message_rate is in units of points/second

Aggregated values
╭───────┬──────────────┬───────────┬───────────┬─────────────────┬───────────────┬──────────────╮
│ count │ message_rate │ data_size │ data_rate │ start_timestamp │ end_timestamp │ milliseconds │
├───────┼──────────────┼───────────┼───────────┼─────────────────┼───────────────┼──────────────┤
│ 1778  │ 91.952835 /s │ 1.1 MiB   │ 59 KiB/s  │ 1682096206674   │ 1682096226010 │ 19336        │
╰───────┴──────────────┴───────────┴───────────┴─────────────────┴───────────────┴──────────────╯

@de-sh
Copy link
Member Author

de-sh commented Apr 26, 2023

-s should be optional

@de-sh de-sh merged commit 57750b3 into main Apr 28, 2023
@de-sh de-sh deleted the deserialize-backup branch April 28, 2023 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant