map.transformResponse

[stevage]

Motivation

It is a very common need to fetch data from somewhere then turn it into GeoJSON, before adding it into the map. Typically the source is either CSV or JSON.

Currently this has some downsides:

• having to use a 3rd party library like D3-request to fetch (extra dependencies - ok, not strictly required, but generally what one does)
• having to manage waiting processes (wait for map to load, wait for data to load), then waiting for both to finish. This is awkward because map.on('load') is an event (not a promise), while the request is probably a promise, or maybe a callback. (And of course loaded() and on('load') event do not work as expected #6707)
• not being able to take advantage of the simplicity of map.addSource('mydata', { type: 'geojson', source: 'http://example.com/not-yet-a-geo.json')

Design Alternatives

Custom Sources may solve this, but I suspect will be much more complicated. And they're not available yet, right? :)

Design

map = new Mapboxgl.map({
    ...,
    transformResponse((url, response) => {
        if (url.match('example.com/')) {
            const json = JSON.parse(response);
            return {
                type: 'FeatureCollection',
                features: json.items.map(item => ({
                    type: 'Point',
                    coordinates: [item.x, item.y],
                    propertes: {
                        name: item.name
                    }
                }))
            }
        }
    })
})

What are the advantages of this design?

It's very simple and easy to understand for the user. It neatly parallels transformRequest.

There might be some performance benefits? (The docs say that using GeoJSON via url is more efficient than fetching it yourself and calling setData() but I don't know why.)

What are some potential drawbacks of this design?

There is no direct connection between the source and the transformResponse function. If there were multiple different sources that required transformation, that function would start to get complex.

I think there is some overlap with #5104 but I don't entirely understand that issue.

(Partly inspired by This SO question)

[jfirebaugh]

Thanks for the suggestion @stevage.

Tile responses happen on the Web Worker, and functions cannot be transferred to a worker, so unfortunately it can't be as simple as a function callback.

Custom sources are our intended solution for this, and we aim to make them as convenient as possible within the constraints we have. Let's keep discussion on that issue.

[stevage]

Thanks. Can you explain what you mean by "tile responses"? This wouldn't be per-tile, just once per source, right?

[jfirebaugh]

Well, I meant to say "tile requests", but it also applies to GeoJSON requests -- both of those are made from the worker, and the expectation for a transformResponse API would be that it can transform responses to all the same requests as transformRequest. (We designed transformRequest so that it could be run on the main thread, and then the result send to the worker as part of the "loadTile" command, but that's not feasible with transformResponse.)

[andrewharvey]

having to manage waiting processes (wait for map to load, wait for data to load), then waiting for both to finish. This is awkward because map.on('load') is an event (not a promise), while the request is probably a promise, or maybe a callback. (And of course

I'm running into this too. I'm using a semaphore to deal with the fact that the data may be ready before the map loads or after, but I agree it's clunky.

[stevage]

I tend to use this pattern:

map.onLoad = cb => map.loaded() ? cb () : map.on('load', cb);

d3.csv(...).then(rows => {
  map.onLoad(() => map.addSource(rowsToSource(rows));
});

[andrewharvey]

@stevage Thanks that much nicer!