Data Provider Skeleton Mockup #75
Conversation
| assert.Equal(t, skeleton.SkeletonDataSourceId, dataSourceId) | ||
|
|
||
| sourceSpecificConfig, err := skeleton.GetSourceSpecificConfig(sourceConfig) | ||
| assert.NoError(t, err) |
There was a problem hiding this comment.
generate tests to make sure the config is valid and can be loaded, that we can extract the data source id from it, and that we can extract the SkeletonConfig.
The user just has to fill in a valid config string and write some asserts
| import "github.com/Stork-Oracle/stork-external/apps/lib/data_provider/types" | ||
|
|
||
| type SkeletonConfig struct { | ||
| DataSource types.DataSourceId `json:"dataSource"` // required for all Data Provider Sources |
There was a problem hiding this comment.
All configs need to have a DataSource field
The user just needs to add configuration details specific to their dataSource
| } | ||
|
|
||
| func newSkeletonDataSource(sourceConfig types.DataProviderSourceConfig) *skeletonDataSource { | ||
| skeletonConfig, err := GetSourceSpecificConfig(sourceConfig) |
There was a problem hiding this comment.
This is a little opinionated, but it seems like any data source would need to be initialized using the data source's config.
Note that as written this skeleton code won't even compile, since this variable is unused. It might be a little nicer to have skeleton code which compiles cleanly but fails at runtime - this way seemed a little more readable to the developer though.
| } | ||
|
|
||
| func (r skeletonDataSource) RunDataSource(updatesCh chan types.DataSourceUpdateMap) { | ||
| // TODO: Write all logic to fetch datapoints and add to updatesCh |
There was a problem hiding this comment.
super open ended
| ) | ||
|
|
||
| func TestSkeletonDataSource(t *testing.T) { | ||
| // TODO: write some unit tests for your data source |
There was a problem hiding this comment.
very open ended - didn't want to auto-generate unit tests that actually run the integration in case the integration hits external services
| @@ -0,0 +1,30 @@ | |||
| package skeleton | |||
There was a problem hiding this comment.
this file is fully auto-generated - user doesn't need to touch it
There was a problem hiding this comment.
maybe we can add a comment at the top of the file saying that it is autogenerated and to please not touch - similar to mockery
There was a problem hiding this comment.
hmm I'm not sure if we want to actually stop people from changing this file if they want to - it wouldn't be crazy for some advanced user to make changes to the factory code.
There was a problem hiding this comment.
i'm not 100% sure what i should be looking for but it seems like a good skeleton. should we provide an example implementation to make it even clearer?
also, what do you think about providing an explicit function in the flow for transformations? too opinionated?
| @@ -0,0 +1,30 @@ | |||
| package skeleton | |||
There was a problem hiding this comment.
maybe we can add a comment at the top of the file saying that it is autogenerated and to please not touch - similar to mockery
| @@ -0,0 +1,12 @@ | |||
| { | |||
There was a problem hiding this comment.
removed the //TODO comments from here because it's invalid json and was causing some framework tests to fail. We should probably explicitly say in the docs that users will need to edit this file once they write their config object.
If they try to configure anything other than data source id with this default schema, validation will fail. So hopefully that will force users to update this file even without the TODOs
We'd like to make adding a new Data Provider source easier by auto-generating some skeleton code which the user can then fill in so they don't have to think too much about the framework.
In this PR I've manually written some suggested skeleton code for a new data source called
skeleton, which can hopefully serve as a starting model for our templatingI think the ideal flow for a user would be a CLI-like tool where we ask them to give us the name of their dataSource in
TitleCase. We would then automaticallyskeleton. We can figure outcamelCase,snake_caseandflatcaseversions of the dataSource name automatically and insert the appropriate one into the template.python3 ./apps/scripts/update_shared_data_provider_code.pyto update a couple of shared framework filesThen the user just fills in the TODOs, gets their tests passing and submits the PR.