Skip to content

Automated creation of a new Versioned Dataset

Jump to bottom
timrdf edited this page Jan 8, 2012 · 84 revisions

Background

(for distinction among Abstract Dataset, Versioned Dataset, and Layer Dataset, see Springer LOGD book chapter (in press)).

Directory structure as described in Conversion process phase: retrieve.

Introduction

Creating a new Versioned Dataset from a Google spreadsheet

Going into a dataset's version/ directory:

$ cd /source/twc-rpi-edu/instance-hub-us-states-and-territories/version/
$ ls

we see a directory for each version of abstract dataset http://logd.tw.rpi.edu/source/twc-rpi-edu/dataset/instance-hub-us-states-and-territories:

2011-Apr-01/
2011-Apr-09/
2011-Mar-31/

We can set up future versions by creating a script retrieve.sh with contents:

#!/bin/bash
#
#3> <>
#3> rdfs:comment 
#3> "Script to retrieve and convert a new version of the dataset.";
#3>
#3> rdfs:seeAlso 
#3> <https://github.com/timrdf/csv2rdf4lod-automation/wiki/Automated-creation-of-a-new-Versioned-Dataset>,
#3> <https://github.com/timrdf/csv2rdf4lod-automation/wiki/tic-turtle-in-comments>;
#3> .

export CSV2RDF4LOD_CONVERT_OMIT_RAW_LAYER="true"
$CSV2RDF4LOD_HOME/bin/util/google2source.sh -w t9QH44S-_D6-4FQPOCM81BA auto

Running google2source.sh will describe it's usage. The -w indicates to actually create the version directory (instead of a dry run), t9QH44S-_D6-4FQPOCM81BA is the Google spreadsheet key that can be copied from the URL when viewing it, and auto says to use a default name for the local file created when retrieving the spreadsheet.

Remember to chmod +x retrieve.sh the first time, then run:

./retrieve.sh

whenever you want to create a new versioned dataset by retrieving another copy of the Google spreadsheet. When doing so, the initial raw conversion will be run automatically and any enhancement conversions will be run if the [global enhancement parameters are in place](Reusing enhancement parameters for multiple versions or datasets) (e.g., /source/twc-rpi-edu/instance-hub-us-states-and-territories/version/).

If global enhancement parameters are established and the raw layer is useless to you, include CSV2RDF4LOD_CONVERT_OMIT_RAW_LAYER:

If Google returns an empty row before the header, use conversion:HeaderRow.

Creating a new Versioned Dataset from any URL

In the twc-lobd svn, going to a dataset's version/ directory:

$ cd /source/ncbi-nlm-nih-gov/gene2ensembl/version/
$ ls

we see a directory for each version of abstract dataset http://health.tw.rpi.edu/source/ncbi-nlm-nih-gov/dataset/gene2ensembl:

2011-Apr-16/

We can set up future versions by creating a script retrieve.sh with contents:

#!/bin/bash
#
#3> @prefix doap:    <http://usefulinc.com/ns/doap#> .
#3> @prefix dcterms: <http://purl.org/dc/terms/> .
#3> @prefix rdfs:    <http://www.w3.org/2000/01/rdf-schema#> .
#3> 
#3> <#>
#3>    a doap:Project; 
#3>    dcterms:description 
#3>      "Script to retrieve and convert a new version of the dataset.";
#3>    rdfs:seeAlso 
#3>      <https://github.com/timrdf/csv2rdf4lod-automation/wiki/Automated-creation-of-a-new-Versioned-Dataset>;
#3> .

export CSV2RDF4LOD_CONVERT_OMIT_RAW_LAYER="true"
$CSV2RDF4LOD_HOME/bin/cr-create-versioned-dataset-dir.sh cr:auto                                               \
                                                        'ftp://ftp.ncbi.nlm.nih.gov/gene/DATA/gene2ensembl.gz' \
                                                       --comment-character '#'                                 \
                                                       --header-line        0                                  \
                                                       --delimiter         '\t'

Running cr-create-versioned-dataset-dir.sh will describe it's usage:

$ cr-create-versioned-dataset-dir.sh 
usage: cr-create-versioned-dataset-dir.sh version-identifier URL [--comment-character char]
                                                                 [--header-line        row]
                                                                 [--delimiter         char]
   version-identifier: conversion:version_identifier for the VersionedDataset to create (use cr:auto for default)
   URL               : URL to retrieve the data file.

Remember to chmod +x retrieve.sh the first time, then run:

./retrieve.sh

whenever you want to create a new versioned dataset by retrieving the data file URL. When doing so, the initial raw conversion will be run automatically and any enhancement conversions will be run if the [global enhancement parameters are in place](Reusing enhancement parameters for multiple versions or datasets) (e.g., /source/ncbi-nlm-nih-gov/gene2ensembl/version/gene2ensembl.e1.params.ttl).

Creating a new Versioned Dataset using custom code

Use the retrieve.sh template.

cd source/contactingthecongress/directory-for-the-112th-congress/version
cp $CSV2RDF4LOD_HOME/bin/cr-create-versioned-dataset-dir.sh retrieve.sh

Codifying intermediate tweaks with a global 2manual.sh

#!/bin/bash
#
# Script to prepare data retrieved in source/ for conversion (results in manual/).
# Script also constructs the conversion trigger.
# Invoked by global retrieve.sh with the conversion cockpit as the current working directory.
#
# See https://github.com/timrdf/csv2rdf4lod-automation/wiki/Automated-creation-of-a-new-Versioned-Dataset
#     https://github.com/timrdf/csv2rdf4lod-automation/wiki/Conversion-trigger
#     https://github.com/timrdf/csv2rdf4lod-automation/wiki/Conversion-cockpit

See also

Clone this wiki locally