Rewrite encoderd in Python

PLAN.md

@@ -0,0 +1,28 @@
+**Goal**
+- Rewrite `system/loggerd/encoderd.cc` as `system/loggerd/encoderd.py` with near 1:1 behavior: read VisionIPC frames, encode per stream, publish messages, rotate segments, and send periodic thumbnails.
+
+**Design (Keep It 1:1)**
+- Orchestrate in Python; keep encoder logic equivalent to C++.
+- PC path (FFmpeg): use existing `av` (PyAV) to encode HEVC/H.264 with the same settings (bitrate, gop, no B-frames). Extract keyframe flags and headers from packets.
+- QCOM path (V4L2 M2M): implement direct `ioctl` + `mmap` in Python (stdlib `fcntl`, `ctypes`), mirroring `v4l_encoder.cc` (formats, request/queue/dequeue, stream on/off, timestamps, flags).
+- JPEG thumbnails: generate from NV12 using NumPy downscale (quarter-res) and encode with `jpeglib` (Python libjpeg bindings) to mirror C++ `jpeg_encoder.cc`; publish same schema.
+- No Cython. No new dependencies beyond `av` (already available). No change to `pyproject.toml` unless requested.
+
+**Behavior Parity**
+- Startup sync across encoders using shared `start_frame_id` (same logic and margins).
+- Rotation at `frames_per_seg = SEGMENT_LENGTH * MAIN_FPS` since `start_frame_id`.
+- Message fields and topics identical to C++ (`EncodeIdx` data, headers on keyframes, width/height, lengths, flags, encodeId monotonic per encoder).
+- CLI parity: default main encoders; `--stream` uses livestream encoders. Apply realtime priority and core affinity on device (best-effort).
+- “Lagging” check (buf.get_frame_id vs extra.frame_id): not exposed in Python; acceptable to omit or replace with best-effort logging.
+
+**Implementation Steps**
+1) Skeleton `encoderd.py` + CLI (`--stream`).
+2) Dataclasses for `EncoderInfo`/`LogCameraInfo` mirroring `loggerd.h`.
+3) `EncoderdState` + `sync_encoders` direct port.
+4) VisionIPC integration: discover streams, connect, per-stream threads.
+5) FFmpeg encoder (PyAV) with matching settings and packet publish.
+6) V4L2 M2M encoder via `ioctl`/`mmap` ported from `v4l_encoder.cc`.
+7) JPEG cadence and publish via `jpeglib` (YUV420 input, libjpeg settings matching C++ defaults).
+8) Segment rotation reopen logic.
+9) Lint: run `tools/op.sh lint` and fix any issues.
+10) Run existing tests (`system/loggerd/tests/test_encoder.py`).

pyproject.toml

@@ -50,6 +50,8 @@ dependencies = [
   "pyzmq",
   "sentry-sdk",
   "xattr", # used in place of 'os.getxattr' for macos compatibility
+  "av",
+  "jpeglib",
 
   # athena
   "PyJWT",
@@ -101,7 +103,6 @@ testing = [
 ]
 
 dev = [
-  "av",
   "azure-identity",
   "azure-storage-blob",
   "dbus-next", # TODO: remove once we moved everything to jeepney

system/loggerd/SConscript

@@ -4,22 +4,17 @@ libs = [common, messaging, visionipc,
         'avformat', 'avcodec', 'avutil',
         'yuv', 'OpenCL', 'pthread', 'zstd']
 
-src = ['logger.cc', 'zstd_writer.cc', 'video_writer.cc', 'encoder/encoder.cc', 'encoder/v4l_encoder.cc', 'encoder/jpeg_encoder.cc']
-if arch != "larch64":
-  src += ['encoder/ffmpeg_encoder.cc']
+src = ['logger.cc', 'zstd_writer.cc', 'video_writer.cc']
 
 if arch == "Darwin":
   # fix OpenCL
   del libs[libs.index('OpenCL')]
   env['FRAMEWORKS'] = ['OpenCL']
-  # exclude v4l
-  del src[src.index('encoder/v4l_encoder.cc')]
 
 logger_lib = env.Library('logger', src)
 libs.insert(0, logger_lib)
 
 env.Program('loggerd', ['loggerd.cc'], LIBS=libs)
-env.Program('encoderd', ['encoderd.cc'], LIBS=libs + ["jpeg"])
 env.Program('bootlog.cc', LIBS=libs)
 
 if GetOption('extras'):

system/loggerd/encoder/encoder.py

@@ -0,0 +1,33 @@
+from dataclasses import dataclass
+
+import cereal.messaging as messaging
+
+
+@dataclass
+class EncoderSettings:
+  encode_type: str
+  bitrate: int
+  gop_size: int
+  b_frames: int = 0
+
+
+class VideoEncoder:
+  def __init__(self, encoder_info, in_width: int, in_height: int):
+    self.encoder_info = encoder_info
+    self.in_width = in_width
+    self.in_height = in_height
+    self.out_width = encoder_info.frame_width if encoder_info.frame_width > 0 else in_width
+    self.out_height = encoder_info.frame_height if encoder_info.frame_height > 0 else in_height
+    self.pm = messaging.PubMaster([encoder_info.publish_name])
+    self.encode_cnt = 0
+    self.segment_num = -1
+    self.counter = 0
+
+  def open(self) -> None:
+    raise NotImplementedError
+
+  def close(self) -> None:
+    raise NotImplementedError
+
+  def encode_frame(self, buf, frame_id: int, timestamp_sof: int, timestamp_eof: int) -> int:
+    raise NotImplementedError