Create scaper for pulling new rows out of data.dc.gov

[mkalish]

The dataset is too big to be proxied and will not be queryable. Currently, the data has been pushed through 1/1/2017 but a scaper needs to be written to do this regularly.

Tasks:

• Write a scaper that will pull data off the dc data site limited to the last date recorded in data.codefordc.org
• Push that data to the the data portal

Upload idead using existing tools from esri to geojson to csv to data portal
https://www.npmjs.com/package/esri-dump
https://www.npmjs.com/package/json2csv
https://www.npmjs.com/package/ckan

Get the last imported data in data portal
Check if esri data exist beyond the last imported data
If data exist, attempt to get a dump from that start point to the end
Since the data in ckan is current csv, convert the data
Upload data to data portal

This script could then run to sync data portal info with esri.

[romoy]

This seems to be a pre requisite, so researching this one instead.

[romoy]

http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Public_Service_WebMercator/MapServer/34 - Campaign Contributions

http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Public_Service_WebMercator/MapServer/35 - Campaign Expenditures

Use of https://github.com/openaddresses/pyesridump to grab the latest data.

esri2geojson http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Public_Service_WebMercator/MapServer/34 ocf-contributions.geojson

esri2geojson http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Public_Service_WebMercator/MapServer/35 ocf-expenditures.geojson

[romoy]

Added ocf-expenditures.geojson to http://data.codefordc.org/dataset/dc-campaign-expenditures-ocf

[romoy]

Attempt an add of ocf-contributions but failed with 413 response; will try to split the file and upload.

[mkalish]

Looking good. I would take a look at the datastore API that can more gracefully handle pushing a lot of rows

[romoy]

http://maps2.dcgis.dc.gov/dcgis/sdk/rest/index.html#//02ss0000000r000000

[romoy]

Upload idea using existing tools from esri to geojson to csv to data portal
https://www.npmjs.com/package/esri-dump
https://www.npmjs.com/package/json2csv
https://www.npmjs.com/package/ckan

[romoy]

http://www.convertcsv.com/geojson-to-csv.htm

[mkalish]

Does that have an API?

[romoy]

https://github.com/koopjs/koop-opendata

[romoy]

http://opendata.arcgis.com/datasets/DCGIS::campaign-financial-expenditures/geoservice

[romoy]

http://opendata.arcgis.com/datasets/DCGIS::campaign-financial-contributions/geoservice