Skip to content

not everything can be a blob

coldnebo edited this page Feb 26, 2013 · 1 revision

While the ability to store metadata in DataHut as a blob and autobox any Ruby type is attractive, it's probably not a good idea for general extract and transform. Why?

Because metadata is explicitly keyed. You know what you want, you get it. It's just names or values, or some other structure you already know.

Core data isn't structured in a way you know, otherwise you wouldn't be analyzing it, would you? So if I allow blobs in core data, it means that all the powerful functions and indexing of Sequel are crippled (i.e. no avg, min, max on Ruby structures). I mean, I could imagine extending that functionality into DataHut, but now we're talking about creating an ObjectStore, rather than leveraging Sequel... that's a bit out of scope. (I suppose along that thread DataHut could be adapted to any NoSQL db, Mongo, CouchDB, etc.)

The other direction is a hybrid. Support the basic types fully via Sequel filters and the Ruby marshalled types not at all, or loosely as strings -- this is kind of hokey... I'm not sure how hokey, but I'm leaning towards not supporting blobs at this time unless something else changes my mind.

Clone this wiki locally