OAM Uploader Workplan

[cgiovando]

UC Berkeley Code For Good - OAM Uploader Workplan

Background

OpenAerialMap recently underwent a backend modernization moving from MongoDB to a STAC-compliant architecture using pgSTAC, FastAPI, and TiTiler.

• New API: https://api.imagery.hotosm.org/
• Frontend prototype: https://imagery.hotosm.org/
• OIN Metadata Spec: https://github.com/openimagerynetwork/oin-metadata-spec
• STAC Extension for OAM: https://github.com/hotosm/stactools-hotosm

The current uploader is unreliable for large files and lacks modern resilience features. This project focuses on building a new imagery uploader that integrates with the modernized STAC-based backend.

Code For Good

UC Berkeley Code For Good runs a program where software engineering students contribute to open-source projects. A student team is working on the OAM Uploader for the Spring 2026 semester (10 weeks), alongside a separate team working on osm-tagger.

Objectives

Build a modern, resilient web application and API that enables drone pilots and imagery providers to upload large orthomosaics to OpenAerialMap:

1. Reliable large file uploads - Handle files up to 100GB with resume capability, using S3 multipart upload with presigned URLs (direct browser-to-S3)
2. Cloud-native processing - Convert imagery to COG format using Kubernetes-based Argo Workflows
3. STAC integration - Automatically register imagery with the pgSTAC backend
4. User authentication - OAuth 2.0 via HOT and Google login
5. Progress tracking - Real-time status updates for uploads and processing
6. Programmatic API - RESTful API for uploads from external software (e.g. WebODM) and scripts

Technical Architecture

Agreed approach (as of March 2026)

• Frontend: Simple HTML/JS (no React/TypeScript overhead needed for this use case)
  • Upload form with OIN metadata fields
  • User dashboard showing upload history and status
• Upload path: Direct browser-to-S3 multipart upload via presigned URLs (no temporary server storage)
  • Better for scaling during disaster events with multiple concurrent uploads
  • Avoids resource constraints on processing nodes
• Backend: Python (FastAPI or Litestar)
  • Exposed service: upload API, OAuth, S3 multipart orchestration, HTML serving
  • Internal service: job management and processing coordination
• Processing pipeline: Argo Workflows on Kubernetes
  • Stateless containers: pull from S3, process, push back to S3
  • No message queue needed - Argo handles job orchestration natively
  • Job status via polling the Kubernetes API (simple approach preferred over event-driven for this use case)

Processing steps

1. Validate GeoTIFF (check georeferencing, band count, bit depth)
2. Convert to Cloud-Optimized GeoTIFF (COG) using GDAL
3. Extract metadata (GSD, bbox, footprint, projection) via gdalinfo
4. Generate thumbnail
5. Upload COG to OIN S3 bucket
6. Create STAC Item and register with pgSTAC
7. Update upload status and notify user

Raw data preservation (planned)

• Option to keep original files alongside COGs in S3 (separate directory structure)
• Important for disaster response and scientific analysis where multispectral data is lost during COG conversion
• Tracked via STAC metadata

Nice-to-haves / future considerations

• Browser-based COG conversion using GDAL WASM for smaller files (check client machine specs, fall back to server if needed)
• Google Drive / Dropbox integration
• Email notifications

Milestones

Phase 1: Foundation (Weeks 1-3)

• Environment setup, repository structure, CI/CD
• Authentication flow (OSM OAuth)
• Basic upload form UI with OIN metadata fields
• S3 multipart upload integration
• Initial API structure with OpenAPI documentation

Phase 2: Core Processing Pipeline (Weeks 4-6)

• Argo Workflows configuration for processing jobs
• GeoTIFF validation and COG conversion pipeline
• Metadata extraction and STAC item creation
• Upload status tracking and progress display

Phase 3: Enhanced Features (Weeks 7-8)

• User dashboard (view/manage uploads)
• API key generation for programmatic access
• Error handling and retry logic
• Raw data preservation option

Phase 4: Polish & Deployment (Weeks 9-10)

• Kubernetes deployment manifests
• End-to-end testing (UI and API)
• Documentation (user guide, developer docs, API reference)
• Handoff and demo

Deliverables

1. Source code - Open source (AGPL-3.0), to be integrated into hotosm/openaerialmap
2. Documentation - README, API docs (OpenAPI/Swagger), deployment guide
3. Deployment artifacts - Docker images, Kubernetes manifests
4. Demo - Working deployment on HOT staging infrastructure

[spwoodcock]

Argo is great for our infrastructure and a serious production install.

Based on #248, I'm definitely keen to keep a pathway open for a very simple install too, via docker compose.

This would mean we probably want two execution modes behind the same API interface (via an abstraction / user selection at install):

• ProcessingBackend interface.
• Inherits ArgoWorker or LocalWorker options.
• LocalWorker processing runs a worker container (runs GDAL pipeline directly)
• To save complexity polling the Argo k8s interface, and for compatibility with local processing, I think the final step of the workflow should call the backend API we create, e.g. POST /uploads/{id}/complete.
• This would update the processing status, for say the frontend polling to realise the job is done.

---

I would start with the LocalWorker, then if we have time / team willingness we can implement the ArgoWorker afterwards =)

[spwoodcock]

Also, I'd really love to see this implemented in HTMX (pretty easy when using LiteStar).

While its always tempting to reach for a full React frontend or similar, this site is basically just an upload form and maybe a profile page.

HTMX (using web components and hotosm/ui) would significantly simplify future maintenance. It would also make ops very easy, requiring a single 'backend' container to run everything self contained.

There is already a great example and structure ready to copy (including layout, bundled libraries, and included HOT header we can reuse):
https://github.com/hotosm/field-tm/tree/dev/src%2Fbackend%2Fapp

---

Sorry if this seems a tall order considering team experience! Let me know what you think & maybe we can compromise somewhere 🙏

But at the end of the day HTMX is just a light wrapper on standard HTML/CSS and is pretty simple web dev stuff 😃