Tasklist

[mariusandra]

If you'd like to give a helping hand, pick any of the unfinished items and submit a PR. Don't ask for permission, everything is available and there are no issues to assign.

Old archived tasks

July to November 2023. Python proof of concept

• Backend + frontend for control panel running
• SSH connection to frames
• Deploy and launch client software
• Launch as a daemon (via tmux for now)
• API connection back
• Streaming SSH logs to client
• UI to add frames
• Get live images back
• Reload button in FCP UI
• Show image as it's updating
• Sync frame updating state to control panel
• Sync button presses to control panel
• Add migrations
• Send logs sequentially
• Configure image via UI
• Kiosk mode URL
• Create new frame installation instructions (from rpi)
• Add apps
• Draw stuff on images
• OpenAI DallE2
• Rotate images
• Scaling mode as part of the frame, not app
• App for scaling with cover modes
• Edit via a diagram UI
• Require user/pass to see interface
• Close a tab, fix bugs with tabs
• Handle saving modified apps
• Virtualize logs (list gets too big)
• Settings page with credential storage (HA, OpenAI, Github)
• Save scenes (one or many) as templates
• Template .zip export
• Template .zip import
• Github template repositories
• Support making screenshots of web pages (on 64bit devices only)
• Apps/devices must preselect the settings/secrets they are using, only those are copied over
• SSH keys
• Python and JSON errors inline in editor (red if file is obviously wrong)
• Editing text in app config jumps to end
• Document the coding environment and the available apps
• Get CPU, temp and other metrics from the frames
• Add Sentry support
• Better login screen
• If/else branches in diagram editor
• Horizontal/vertical split (grid mode)

December to January 2024. Nim rewrite. Blog post. HN thread

• Explore migrating from PIL to pixie-python make sense? We'd get great font rendering for free, minus emoji support sadly. --> Might be great, but putting energy into the next point instead.
• Experiment if it makes any sense to rewrite the FrameOS on-device part in a compiled language. Conclusion: yes it does. We're doing real time graphics on low power devices. A compiled language brings 10x improvements in speed and/or performance. Migrating some parts to a compiled language is inevitable. After looking around a lot, nim seems like the best choice. People praise it. It's friendlier than rust/zig/go ("If you imagine native-compiled type-annotated Python where nearly 100% of your code is business logic with no cruft, you're getting close to the Nim experience."), it's easy to pick up from examples, fast, it's used for games (it's fast), and comes with an amazing image handling library and other goodies. None of the other languages hit this sweet spot, so let's go.
• Rewrite FrameOS in nim.
  • Proof of concept in the nim branch
  • Deploy the nim code onto a raspberry with on-device compilation
  • Caching for on device compilation
  • Support for apps and the execution context
  • Support numbers, color and other types of fields
  • Support template strings in config -> inline code nodes for vars
  • Support edited custom apps
  • Convert to async server with schedules, etc
  • Scheduler
  • Inky impression display support? No C bindings? Only python available. Will call python code for now.
  • Rotated display support
  • System of hardware specific drivers, and devices that group them
  • Port over the main apps
    • Unsplash
    • Text
    • Code
    • Clock
    • Download URL
    • FrameOS Gallery
    • Color
    • Gradient
    • HA Sensor
    • If
    • Split
    • Resize
    • Rotate
    • OpenAI
    • Break if rendering
  • Log batching in a separate thread
  • Rendering in a separate thread
  • HDMI support (double buffering, frame info, different color modes)
  • Sanitize app fields
  • Display on/off controls
  • Skip most logs in high fps mode
  • Metrics logger
  • Waveshare eink support (using C libraries!)
  • Touch and click handlers
  • Ignore render when rendering, run all events after
• 60fps mode (max 2fps with Python now) -> got 60fps+ (more like 100) on a rpi5 with a 720p display (pikvm). So it's possible, even with simple "software rendering". Once we have the basics covered, I can investigate something like boxy to increase performance even more.
• Stability and hours of use (been running an inky impression + rpi zero w 1 and a hyperpixel round + zero w 2 at 24fps for days now without downtime)
• Show errors when editing nim apps in the interface
• Update docs Update for the nim rewrite frameos-docs#1
• Update the Dockerfile
• Merge the PR: Rewrite in nim #5
• Add B/W and B/W/R waveshare drivers
• Full screen mode in web server
• Investigate timeouts hanging the logger
• Delete old build folders on deploy
• Any way to lock/secure dependencies
• Global last image lock
• Cache deps in docker
• Show frame metric charts
• Memory hole?? Memory is not freed when requesting the image over http.
• Add dithering for 2-color waveshare devices
• Add dithering for the rest
• Add waveshare drivers
  • Black/White
  • 4-Grayscale
  • 7-color
  • 4-color Black/White/Red/Yellow
  • Black/White/Yellow
  • Black/White/Red
• Compile lgpio from scratch if no package
• Swap from string node ID-s in compiled scenes to integers (should speed it up a bit and reduce memory usage)
• GPIO button handlers
• Outgoing event nodes
• Disable swap (saves SD card from death if we have a memory leak) --> revert? messes up compilation on W2
• Faster compilation (was: local buildchains. Decided against it for now)
• Inky impressions native driver. Went with faster python redeploys instead. The driver itself works fine as is, so not replacing today.
• Logger crashes again, and takes its thread to 100%. Let's try again...
• Debug toggle
• Double check all the gcsafe hacks
• General polish

Rest of 2024. Scenes, apps, data nodes, state nodes, caching, HA addon, 12.48" video

• App: QR code
• App: GitHub star count
• App: GPT text
• Scene state
• Scene state updates and persistence
• Scene state updates with built in web server
• Auth for frame API access
• Don't drag from within textfields
• Sometimes 404 error when opening frame and clicking back only during rebuilds in dev
• Multiple scenes
• Move parts of frame config to scene config/state
• Revamped scenes and templates
• New default repository, easier sharing of scenes via repositories
• Visual regression tests for apps and scenes
• Tests for scene generation
• Update docs and screenshots
• Update docs about building apps
• Easier split tool
• Initial scene wizard when setting up a new frame
• Improve path dragging/removing in the editor
• Persist last open scene for restarts
• Schedule a restart
• Faster frontend builds
• Code nodes with arguments
• Universal caching controls
• Async data nodes (download url)
• Images as data (?)
• New nodes by dragging to the diagram
• Drag to empty space -> new node
• Data apps, universal caching, refactors all old apps, context menus
• Cursor jumps to the end of the text field
• Unsplash app
• Update all templates with data apps
• Control render/sleep durations programmatically
• Clean up nim app code
• Fix inputs cache type
• Logger leaks memory (stopped buffering, got better, added workaround: nightly restarts)
• iCal support
• Simple timezone support
• Save OpenAI images to disk
• Edit apps with multiple files
• State nodes
• Document building scenes
• Build rich text app
• Build agenda app
• Set system timezone
• Assets browser
• XKCD scene
• Push edge when adding node
• Show scene.nim errors
• Video intro (Nimconf 2024)
• Flask to FastAPI
• Package as a home assistant or Hacs addon
• Adding variable/argument to code node is flakey
• Wrap up 12.48" project (upload stl files, record video, etc)

In 2025. Case maker, fonts, scheduler, captive portal setup, agent connection, NixOS

• Create a clean case for the 12.48" frame
• Create a 3d printed case system
• Store diagram/edit state in URL
• More fonts
• Upload images (better asset browser)
• Scheduler
• Scenes in scenes
• Customizable GPIO buttons
• RTSP snapshot
• Browser screenshot
• Captive portal setup for WiFi credentials
• FrameOS agent that runs on device, connects to the server and manages updates and rollbacks
• Basic asset management (move/delete/upload/etc)
• Pallette editing support on dithered displays (works for spectra now)
• Download sd card image from the backend
• Swap RPi OS for NixOS?
• Asset image thumbnails
• SSH terminal (pretty basic, but works)
• Pimoroni 2025 dithering via frameos + palettes

Todos in no specific order

• Layout scenes
• Compile apps, scenes and core separately
• Dynamically loaded modules
• Versioning and dependencies
• Shared app and scene repositories
• Some way to "adopt" a frame into your account after connecting it to wifi
• Multi tenant support in the backend
• Logging via websockets, like agent
• Encrypted communication between frames and controllers when over HTTP
• Copy/paste nodes
• Undo/redo support
• Drag to select / select multiple in the diagram
• Better scene selector in the control URL
• Build apps
  • Calendar
  • Graphs and charts
  • Google Photos
  • Image transitions / screensavers (e.g time based smooth zoom)
• Show image metadata
• Custom event types
• Visual, grid, layout, placement, etc editor for apps
• UI (via lvgl?)
• Errors and exceptions in the diagram
• Release 1.0
• Clean up edges that are not connected
• Automatically update timezones
• Autocomplete for nim code
• Monaco for code nodes
• Backups
• Live preview via JS/WASM
• Font with emoji support
• Verify support for all other inky frames
• Unified background data fetching to not slow down real time mode
• Throttle if CPU is hot
• Slow down or turn off rendering if display is off
• Multiple SSH keys
• Events (e.g. GPIO buttons) as home assistant events
• Investigate GPU rendering via boxy
• Resize image data app: center properly if source is smaller than target

[andiohn]

Hey! I LOVE THIS PROJECT SO MUCH!

Good work man. I am very inspired by this.

Ideas:

1. Simple http served frame for crappy devices (kobo aura HD etc) with really old browsers.
2. "touch targets" for those crappy devices (so you can pretend to interact with Home Assistant toggles.

Thank you for this. I'm going to try to use it and try to contribute.