Collection of ideas/issues (to be refined)

[ktk]

Ideas

Some of these things might already exist or did exist in an older version of barnard59. Some might never happen.

• npm generic pipelines (with manifests) #85
• load from more than one ttl file #93
• Facilitate bootstrapping pipelines
  • bootstrap metadata step #92
• prov-step (gitlab CI, repo name, etc)
  • New Component: Automatically add metadata in dataset. #35
  • New Component: Collect statistics of a created dataset. #34
• prov-metadata
• shacl step (auto-create shacl from passed data)
• void-step (check what we have and make this useful, use/provide shacl for validation as we do for some projects). Good examples:
  • https://sparql.uniprot.org/.well-known/void
  • https://sparql.rhea-db.org/.well-known/void
  • https://beta.sparql.swisslipids.org/.well-known/void
• debug description/tutorial (as I did in vs code)
• re-think tdb step (https://github.com/zazuko/barnard59-tdb)
  • we use this for "inference" and/or rules
  • either make sure it still works (didn't test)
  • or come up with alternative ideas (reasoners like EYE JS or RoXi?)
• generic swagger API step?
• s3 sink step
• JSONata, related to Add datatype to access variable with JSONPath #61
  • Node module: https://docs.jsonata.org/using-nodejs
• "pv" like for triples/sec?
• make https://github.com/zazuko/barnard59-pipeline-validation useful again
• make sure we have manifests everywhere
• auto-generate docs from manifests
• There should be an output in the logging for the "close" event. As in it logs when it's done. Like this we see exactly how long the pipeline was running, which is not the case if the last step takes longer.
• Valid IRI step. As far as I know it is still possible to create IRIs that are invalid & would fail when loading to a store. I don't think we catch that right now.

Can (probably) be solved with better templates/examples/docs

• Generic file/directory processing: Archetypes for file processing #30
• Add static & concat: Add static files to the stream #27
• some API fetch example step?
• gzip read/write step (probably easy but not for me)
  • related? https://github.com/zazuko/barnard59-compression

Probably done

• Concat Add sequence step #28
• SHACL Validation Implement a SHACL step #26
• Loader Schema Resolver #2

[tpluscode]

npm generic pipelines (with manifests)

By this you mean so that they are easily reused? I might also add a template package to easily start an extends project. For that to use copier

load from more than one ttl file (required for above?)

re #65 and check pipeline.ts in museumplus-pipeline which follows a predicate to combine multiple RDF files

[ktk]

@tpluscode yes for re-use. We have standard pipelines that are used everywhere by now so we could include them instead and semantic version them.

[cristianvasquez]

Regarding Prov:

We have steps to add metadata and count subjects and properties using files as templates.

Append metadata
zazuko/barnard59-rdf#18

Count subjects and properties
zazuko/barnard59-rdf#21

These can be testes in the playground
https://barnard59-steps-playground.zazukoians.org/

IMO there is a lot of room for improvement, specifically for data produced via XRM, to generate this data automatically, and having declarative PROV metadata. This could be tested while upgrading flux-pipelines.