3.4. Processor

Processor

A processor implements IProcessor interface and is responsible of creating and executing ETL jobs. Each job is composed by one extractor, one transformer and one loader.

After all jobs have completed their execution, the processor will call Dispose() in all extractors, transformers and loaders that owns and implement IDisposable. To achieve this, we need to call Dispose() processor methos.

Create a processor

To create a processor we need to use ProcessorBuilder object. This object will create a processor that will handle all ETL jobs with a common DestinationType data type.

Example: creating a processor

IExtractor<IEnumerable<string>> extractorString = //create extractor;
IExtractor<IEnumerable<Item>> extractorItem = //create extractor;

ITransformer<IEnumerable<string>, IEnumerable<Item>> transformerString = //create transformer;

IEnumerable<ILoader<IEnumerable<Item>>> loaders = //create loaders;

IProcessor processor =
    new ProcessorBuilder<IEnumerable<Item>>(loaders)
    .AddSource(extractorString, transformerString)
    .AddSource(extractorItem) //does not need transformation. result from Extractor will be provided to loaders.
    .Build();

Execute jobs

After a processor has been created, we have to call Process method. This will return an IEnumerable<Task<JobResult>>.

IEnumerable<Task<JobResult>> tasks = processor.Process();
Task.WaitAll(new List<Task<JobResult>>(tasks).ToArray());//Wait for all tasks to complete