Skip to content

Commit 6bd5a79

Browse files
spwoodcockspwoodcock
committed
docs: add info about future tech plans
1 parent 7c737ce commit 6bd5a79

File tree

3 files changed

+147
-13
lines changed

3 files changed

+147
-13
lines changed

‎docs/modules/backend.md‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,18 +5,30 @@
55
A way to consistently implement OSM login (via underlying OAuth2) in
66
our applications.
77

8-
## [OSM Raw Data](https://hotosm.github.io/osm-rawdata)
8+
## [PG Nearest City](https://hotosm.github.io/pg-nearest-city)
99

10-
OMS Raw Data supports the creation of data extracts into GeoJson or a
11-
Postgres database.
10+
A very simple tool for reverse-geocoding points to the nearest
11+
city over 1000 population. Requires no network requests and
12+
runs entirely in an attached PostGIS instance.
13+
14+
## [GeoJSON AOI Parser](https://hotosm.github.io/geojson-aoi-parser)
15+
16+
Consistent parsing of user GeoJSON upload, for processing by tools
17+
such as raw-data-api, FieldTM, Drone-TM, etc, using an underlying
18+
PostGIS database.
19+
20+
## [Raw Data API Py](https://hotosm.github.io/raw-data-api-py)
21+
22+
A Python wrapper for HOT's raw-data-api. This allows for downloading
23+
of frequently updated OSM data very conveniently.
1224

1325
## [OSM Fieldwork](https://hotosm.github.io/osm-fieldwork)
1426

1527
Various utility programs useful for field data collection using ODK
16-
Collect. These modules are used extensively in FMTM for all the
28+
Central. These modules are used extensively in Field for all the
1729
backend data processing.
1830

19-
## [FMTMSplitter](https://hotosm.github.io/fmtm-splitter)
31+
## [FieldTM Splitter](https://hotosm.github.io/fmtm-splitter)
2032

2133
A splitting algorithm using PostGIS to divide an idea into task areas,
2234
factoring in prominent map features (roads, rivers, etc).
@@ -26,8 +38,3 @@ The division is configurable via various parameters.
2638
## [fAIr Utilities](https://github.com/hotosm/fAIr-utilities)
2739

2840
Various machine learning utils used within the fAIr backend.
29-
30-
## [GeoJSON AOI Parser](https://hotosm.github.io/geojson-aoi-parser)
31-
32-
Consistent parsing of user GeoJSON upload, for processing by tools
33-
such as raw-data-api, FMTM, Drone-TM, etc.

‎docs/projects/future-plans.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Future Plans / Ideas
2+
3+
This page describes various things we **want** at HOT, and are either
4+
in the process of achieving, or have marked as potential future
5+
projects.
6+
7+
If you are software developer and wish to contribute to something new,
8+
fun, and exciting, feel free to reach out about one of these ideas.
9+
10+
## Kubernetes
11+
12+
- Every HOT tool should be hosted in a Kubernetes cluster.
13+
- This means every tool needs to be containerized and follow
14+
a [12factor](https://12factor.net) app development approach.
15+
- Tools should be deployed using a GitOps approach, via ArgoCD.
16+
This means the container repositories for each tool are scanned,
17+
and when a new version is uploaded, it is automatically deployed.
18+
- Load is shared amongst tools on cluster nodes, reducing overall
19+
costs of running multiple separate servers.
20+
- Autoscaling capabilities when high load is detected for a tool,
21+
automatically spawning new containers as needed.
22+
23+
## Shared UI Components
24+
25+
See [this page](../modules/frontend.md) for more details.
26+
27+
- We don't have the resources to implement a very in-depth design
28+
system across all of our tools.
29+
- We also have existing tech debt due to development velocity and
30+
no centralised design from the start (a need to move quickly...).
31+
- The proposed solution is to converse around a single **web component**
32+
library, such as [Web Awesome](https://backers.webawesome.com).
33+
- Web Awesome is free to use / open-source, has strong backers, and
34+
would allow for a consistent style across all tools, regardless of
35+
framework used (framework agnostic, web standards).
36+
- It should also improve maintainability, as part of the promise of web
37+
components, is minimal requirement to upgrade into the future, as the
38+
World Wide Web Consortium (W3C) does not break existing web functionality.
39+
- We would also avoid the framework churn, by having our **underlying
40+
building blocks** as web components, while building our wrapper apps in
41+
whichever technology we choose.
42+
- A few additional **helper** components will be designed and made available
43+
as part of the [HOTOSM UI](https://github.com/hotosm/ui) library. For
44+
example, a consistent header, footer, and sidebar for every tool.
45+
46+
## Shared Map Components
47+
48+
See [this page](../modules/frontend.md) for more details.
49+
50+
- Historically we may have used OpenLayers and Leaflet in some places.
51+
This should be consolidated around MapLibre.
52+
- We have many implementation of MapLibre in our different tools. Ideally
53+
we can use the same wrapper implementation, improving developer experience
54+
and speed of development.
55+
- **Option 1**: contribute to an existing established tool, that can compile
56+
to plain JavaScript. This JS could be wrapped in a web-component for use in
57+
any web app, any framework.
58+
A top candidate would be
59+
[svelte-maplibre](https://github.com/dimfeld/svelte-maplibre).
60+
- **Option 2**: build a web component wrapper for MapLibre from scratch. It
61+
is anticipated that community support would be high for this, being entirely
62+
framework agnostic and embeddable anywhere.
63+
64+
## Python Modules (Backend)
65+
66+
See [this page](../modules/backend.md) for more details.
67+
68+
We have a lot of duplicated logic that could be centralised:
69+
70+
- Parsing of user uploaded GeoJSON AOIs (no as simple as it
71+
sounds to do right).
72+
[geojson-aoi-parser](https://github.com/hotosm/geojson-aoi-parser)
73+
- Download of OSM data via raw-data-api. Every tool doing this should
74+
use the [raw-data-api-py](https://github.com/hotosm/raw-data-api-py)
75+
wrapper.
76+
- A bunch of OpenStreetMap-specific XLSForm surveys have been started
77+
in [osm-fieldwork](https://hotosm.github.io/osm-fieldwork), but this
78+
work needs some good community testing and peer review. The idea is to
79+
plug the XLSForms into FieldTM, for easy OSM-centric mapping of an area
80+
(to improve the map tags available, then push to OSM).
81+
82+
## AI / LLMs
83+
84+
- OSM tag extraction from images or free text. Some work has started
85+
on this in [osm-tagger](https://github.com/hotosm/osm-tagger).
86+
- A geospatial query chatbot, based on LLMs. Users want data for an
87+
area. The LLM can provide commands / ways to get this data, either
88+
via raw-data-api, overpass, or other available tools.
89+
90+
## Easy OSM Conflation
91+
92+
- Various tools in our stack need to contribute data back to OSM, merging
93+
it with the existing data present there (conflation).
94+
- There are two possible approaches we could take, listed below.
95+
96+
### Conflation Web Component
97+
98+
- A standalone 'conflation' web component that could be embedded in any
99+
tool.
100+
- It will load various dataset sources: OSM, custom data, Overture, etc.
101+
- Various imagery sources can be loaded in. Ideally high-resolution drone
102+
or recent open satellite imagery.
103+
- The UI allows the user to select the best geometry / tags for a given
104+
feature, based on the available imagery.
105+
- The feature is validated by others (optional), then uploaded to OSM
106+
as the new source of truth.
107+
- If features came from FieldTM are are **field verified** they should
108+
take precedence, and will be tagged as such in OSM.
109+
110+
### Tasking Manager Conflation
111+
112+
- Use Tasking Manager as a collaborative conflation tool.
113+
- Create a new 'conflation' project type.
114+
- Users load the imagery, and datasets required.
115+
- Tasking allow the conflation process to be subdivided amongst
116+
the community, and validated.
117+
- Final data is uploaded to OSM.
118+
119+
## Other Ideas
120+
121+
- Now we have [raw-data-api-py](https://github.com/hotosm/raw-data-api-py),
122+
it would be quite simple to make a QGIS plugin for this. Users of QGIS
123+
could download the latest OSM data directly into their project.
124+
- We could also have a QGIS plugin to easily access data via the
125+
[Humanitarian Data Exchange](https://data.humdata.org)

‎mkdocs.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ nav:
108108
- Docs: https://hotosm.github.io/raw-data-api/
109109
- Roadmap: https://github.com/hotosm/raw-data-api/tree/master?tab=readme-ov-file#product-roadmap
110110
- Development: https://github.com/orgs/hotosm/projects/29
111+
- Future Plans / Ideas:
111112
- Modularization:
112113
- Info: modules.md
113114
- Backend: modules/backend.md
@@ -116,10 +117,11 @@ nav:
116117
- UI Components: https://hotosm.github.io/ui/
117118
- Python Modules:
118119
- OSM Fieldwork: https://hotosm.github.io/osm-fieldwork/
119-
- OSM RawData: https://hotosm.github.io/osm-rawdata/
120-
- OSM Merge: https://hotosm.github.io/osm-merge/
120+
- raw-data-api-py: https://hotosm.github.io/raw-data-api-py/
121+
- pg-nearest-city: https://hotosm.github.io/pg-nearest-city/
122+
- geojson-aoi-parser: https://hotosm.github.io/geojson-aoi-parser/
121123
- OSM Login: https://hotosm.github.io/osm-login-python/
122-
- FMTM Splitter: https://hotosm.github.io/fmtm-splitter/
124+
- FieldTM Splitter: https://hotosm.github.io/fmtm-splitter/
123125
- fAIr Utilities: https://hotosm.github.io/fAIr-utilities/
124126
- Developer Guide:
125127
- Intro: dev-guide/intro.md

0 commit comments

Comments
 (0)