Skip to content

A Python script that ingests Figshare's API data and transforms it into data suitable for loading into a recommendation engine.

License

Notifications You must be signed in to change notification settings

gittyeric/figshare-recommender-etl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

figshare-recommender-etl

A Python script that ingests Figshare's API data and transforms it into data suitable for loading into a recommendation engine. In our case, we save the recommendation data in Kafka but you can easily change this in ingest.py.

Installation

First make sure you have your Python dependencies:

pip3 install kafka-python

Change your Kafka connection settings under kafka_output.py. You may also want to change your app token in api.py.

Running

Simply run the ingest.py file:

python3 ingest.py

You'll need to give the script access to write a small local file "bookmark.json" to keep track of ingestion over multiple runs. Delete this file if you'd like to start again from scratch.

About

A Python script that ingests Figshare's API data and transforms it into data suitable for loading into a recommendation engine.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages