-
Notifications
You must be signed in to change notification settings - Fork 135
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1880 from Danielle9897/RDoc-2882-EtlToAQS
RDoc-2882 ETL to Azure Queue Storage
- Loading branch information
Showing
83 changed files
with
6,390 additions
and
135 deletions.
There are no files selected for viewing
42 changes: 24 additions & 18 deletions
42
Documentation/4.1/Raven.Documentation.Pages/server/ongoing-tasks/.docs.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,25 @@ | ||
| [ | ||
| { | ||
| "Path": "general-info.markdown", | ||
| "Name": "General Info", | ||
| "DiscussionId": "d24b385e-5eae-4171-b35d-277157b8e933", | ||
| "Mappings": [] | ||
| }, | ||
| { | ||
| "Path": "/etl", | ||
| "Name": "ETL", | ||
| "Mappings": [] | ||
| }, | ||
| { | ||
| "Path": "backup-overview.markdown", | ||
| "Name": "Backup Overview", | ||
| "DiscussionId": "2b3b34b3-04f0-4c1a-bfcf-40a9ed314f8a", | ||
| "Mappings": [] | ||
| } | ||
| ] | ||
| { | ||
| "Path": "general-info.markdown", | ||
| "Name": "General Info", | ||
| "LastSupportedVersion": "4.1", | ||
| "DiscussionId": "d24b385e-5eae-4171-b35d-277157b8e933", | ||
| "Mappings": [ | ||
| { | ||
| "Version": 4.2, | ||
| "Key": "studio/database/tasks/ongoing-tasks/general-info" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "Path": "/etl", | ||
| "Name": "ETL", | ||
| "Mappings": [] | ||
| }, | ||
| { | ||
| "Path": "backup-overview.markdown", | ||
| "Name": "Backup Overview", | ||
| "DiscussionId": "2b3b34b3-04f0-4c1a-bfcf-40a9ed314f8a", | ||
| "Mappings": [] | ||
| } | ||
| ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
52 changes: 52 additions & 0 deletions
52
...Documentation.Pages/client-api/operations/maintenance/etl/add-etl.java.markdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Operations: How to Add ETL | ||
|
|
||
| You can add ETL task by using **AddEtlOperation**. | ||
|
|
||
| ## Syntax | ||
|
|
||
| {CODE:java add_etl_operation@ClientApi\Operations\AddEtl.java /} | ||
|
|
||
| | Parameters | | | | ||
| | ------------- | ----- | ---- | | ||
| | **configuration** | `EtlConfiguration<T>` | ETL configuration where `T` is connection string type | | ||
|
|
||
| ## Example - Add Raven ETL | ||
|
|
||
| {NOTE: Secure servers} | ||
| To [connect secure RavenDB servers](../../../../server/security/authentication/certificate-management#enabling-communication-between-servers:-importing-and-exporting-certificates) | ||
| you need to | ||
|
|
||
| 1. Export the server certificate from the source server. | ||
| 2. Install it as a client certificate on the destination server. | ||
|
|
||
| This can be done in the RavenDB Studio -> Server Management -> [Certificates view](../../../../server/security/authentication/certificate-management#studio-certificates-management-view). | ||
| {NOTE/} | ||
|
|
||
|
|
||
| {CODE:java add_raven_etl@ClientApi\Operations\AddEtl.java /} | ||
|
|
||
| ## Example - Add Sql ETL | ||
|
|
||
| {CODE:java add_sql_etl@ClientApi\Operations\AddEtl.java /} | ||
|
|
||
| ## Example - Add OLAP ETL | ||
|
|
||
| {CODE:java add_olap_etl@ClientApi\Operations\AddEtl.java /} | ||
|
|
||
| ## Related Articles | ||
|
|
||
| ### ETL | ||
|
|
||
| - [Basics](../../../../server/ongoing-tasks/etl/basics) | ||
| - [RavenDB ETL](../../../../server/ongoing-tasks/etl/raven) | ||
| - [SQL ETL](../../../../server/ongoing-tasks/etl/sql) | ||
|
|
||
| ### Studio | ||
|
|
||
| - [RavenDB ETL Task](../../../../studio/database/tasks/ongoing-tasks/ravendb-etl-task) | ||
|
|
||
| ### Connection Strings | ||
|
|
||
| - [Add](../../../../client-api/operations/maintenance/connection-strings/add-connection-string) | ||
| - [Get](../../../../client-api/operations/maintenance/connection-strings/get-connection-string) | ||
| - [Remove](../../../../client-api/operations/maintenance/connection-strings/remove-connection-string) |
41 changes: 41 additions & 0 deletions
41
...n.Documentation.Pages/client-api/operations/maintenance/etl/add-etl.js.markdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Operations: How to Add ETL | ||
|
|
||
| You can add ETL task by using **AddEtlOperation**. | ||
|
|
||
| ## Usage | ||
|
|
||
| {CODE:nodejs add_etl_operation@client-api\Operations\addEtl.js /} | ||
|
|
||
| | Parameters | | | | ||
| | ------------- | ----- | ---- | | ||
| | **configuration** | `EtlConfiguration<T>` | ETL configuration where `T` is connection string type | | ||
|
|
||
| ## Example - Add Raven ETL | ||
|
|
||
| {CODE:nodejs add_raven_etl@client-api\Operations\addEtl.js /} | ||
|
|
||
| ## Example - Add Sql ETL | ||
|
|
||
| {CODE:nodejs add_sql_etl@client-api\Operations\addEtl.js /} | ||
|
|
||
| ## Example - Add OLAP ETL | ||
|
|
||
| {CODE:nodejs add_olap_etl@client-api\Operations\addEtl.js /} | ||
|
|
||
| ## Related Articles | ||
|
|
||
| ### ETL | ||
|
|
||
| - [Basics](../../../../server/ongoing-tasks/etl/basics) | ||
| - [RavenDB ETL](../../../../server/ongoing-tasks/etl/raven) | ||
| - [SQL ETL](../../../../server/ongoing-tasks/etl/sql) | ||
|
|
||
| ### Studio | ||
|
|
||
| - [RavenDB ETL Task](../../../../studio/database/tasks/ongoing-tasks/ravendb-etl-task) | ||
|
|
||
| ### Connection Strings | ||
|
|
||
| - [Add](../../../../client-api/operations/maintenance/connection-strings/add-connection-string) | ||
| - [Get](../../../../client-api/operations/maintenance/connection-strings/get-connection-string) | ||
| - [Remove](../../../../client-api/operations/maintenance/connection-strings/remove-connection-string) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 1 addition & 2 deletions
3
....Documentation.Pages/studio/database/tasks/ongoing-tasks/olap-etl-task.markdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,5 @@ | ||
| # OLAP ETL Task | ||
|
|
||
| -- | ||
| --- | ||
|
|
||
| {NOTE: } | ||
|
|
||
|
|
||
122 changes: 122 additions & 0 deletions
122
Documentation/5.3/Samples/java/src/test/java/net/ravendb/ClientApi/Operations/AddEtl.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| package net.ravendb.ClientApi.Operations; | ||
|
|
||
| import net.ravendb.client.documents.DocumentStore; | ||
| import net.ravendb.client.documents.IDocumentStore; | ||
| import net.ravendb.client.documents.operations.etl.*; | ||
| import net.ravendb.client.documents.operations.etl.olap.OlapConnectionString; | ||
| import net.ravendb.client.documents.operations.etl.olap.OlapEtlConfiguration; | ||
| import net.ravendb.client.documents.operations.etl.sql.SqlConnectionString; | ||
| import net.ravendb.client.documents.operations.etl.sql.SqlEtlConfiguration; | ||
| import net.ravendb.client.documents.operations.etl.sql.SqlEtlTable; | ||
|
|
||
| import java.util.Arrays; | ||
|
|
||
| public class AddEtl { | ||
|
|
||
| private interface IFoo<T> { | ||
| /* | ||
| //region add_etl_operation | ||
| public AddEtlOperation(EtlConfiguration<T> configuration); | ||
| //endregion | ||
| */ | ||
| } | ||
|
|
||
| public AddEtl() { | ||
| try (IDocumentStore store = new DocumentStore()) { | ||
| //region add_raven_etl | ||
| RavenEtlConfiguration configuration = new RavenEtlConfiguration(); | ||
| configuration.setName("Employees ETL"); | ||
| Transformation transformation = new Transformation(); | ||
| transformation.setName("Script #1"); | ||
| transformation.setScript("loadToEmployees ({\n" + | ||
| " Name: this.FirstName + ' ' + this.LastName,\n" + | ||
| " Title: this.Title\n" + | ||
| "});"); | ||
|
|
||
| configuration.setTransforms(Arrays.asList(transformation)); | ||
| AddEtlOperation<RavenConnectionString> operation = new AddEtlOperation<>(configuration); | ||
| AddEtlOperationResult result = store.maintenance().send(operation); | ||
| //endregion | ||
| } | ||
|
|
||
| try (IDocumentStore store = new DocumentStore()) { | ||
| //region add_sql_etl | ||
| SqlEtlConfiguration configuration = new SqlEtlConfiguration(); | ||
| SqlEtlTable table1 = new SqlEtlTable(); | ||
| table1.setTableName("Orders"); | ||
| table1.setDocumentIdColumn("Id"); | ||
| table1.setInsertOnlyMode(false); | ||
|
|
||
| SqlEtlTable table2 = new SqlEtlTable(); | ||
| table2.setTableName("OrderLines"); | ||
| table2.setDocumentIdColumn("OrderId"); | ||
| table2.setInsertOnlyMode(false); | ||
|
|
||
| configuration.setSqlTables(Arrays.asList(table1, table2)); | ||
| configuration.setName("Order to SQL"); | ||
| configuration.setConnectionStringName("sql-connection-string-name"); | ||
|
|
||
| Transformation transformation = new Transformation(); | ||
| transformation.setName("Script #1"); | ||
| transformation.setCollections(Arrays.asList("Orders")); | ||
| transformation.setScript("var orderData = {\n" + | ||
| " Id: id(this),\n" + | ||
| " OrderLinesCount: this.Lines.length,\n" + | ||
| " TotalCost: 0\n" + | ||
| "};\n" + | ||
| "\n" + | ||
| " for (var i = 0; i < this.Lines.length; i++) {\n" + | ||
| " var line = this.Lines[i];\n" + | ||
| " orderData.TotalCost += line.PricePerUnit;\n" + | ||
| "\n" + | ||
| " // Load to SQL table 'OrderLines'\n" + | ||
| " loadToOrderLines({\n" + | ||
| " OrderId: id(this),\n" + | ||
| " Qty: line.Quantity,\n" + | ||
| " Product: line.Product,\n" + | ||
| " Cost: line.PricePerUnit\n" + | ||
| " });\n" + | ||
| " }\n" + | ||
| " orderData.TotalCost = Math.round(orderData.TotalCost * 100) / 100;\n" + | ||
| "\n" + | ||
| " // Load to SQL table 'Orders'\n" + | ||
| " loadToOrders(orderData)"); | ||
|
|
||
| configuration.setTransforms(Arrays.asList(transformation)); | ||
|
|
||
| AddEtlOperation<SqlConnectionString> operation = new AddEtlOperation<>(configuration); | ||
|
|
||
| AddEtlOperationResult result = store.maintenance().send(operation); | ||
| //endregion | ||
| } | ||
|
|
||
| try (IDocumentStore store = new DocumentStore()) { | ||
| //region add_olap_etl | ||
| OlapEtlConfiguration configuration = new OlapEtlConfiguration(); | ||
|
|
||
| configuration.setName("Orders ETL"); | ||
| configuration.setConnectionStringName("olap-connection-string-name"); | ||
|
|
||
| Transformation transformation = new Transformation(); | ||
| transformation.setName("Script #1"); | ||
| transformation.setCollections(Arrays.asList("Orders")); | ||
| transformation.setScript("var orderDate = new Date(this.OrderedAt);\n"+ | ||
| "var year = orderDate.getFullYear();\n"+ | ||
| "var month = orderDate.getMonth();\n"+ | ||
| "var key = new Date(year, month);\n"+ | ||
| "loadToOrders(key, {\n"+ | ||
| " Company : this.Company,\n"+ | ||
| " ShipVia : this.ShipVia\n"+ | ||
| "})" | ||
| ); | ||
|
|
||
| configuration.setTransforms(Arrays.asList(transformation)); | ||
|
|
||
| AddEtlOperation<OlapConnectionString> operation = new AddEtlOperation<OlapConnectionString>(configuration); | ||
|
|
||
| AddEtlOperationResult result = store.maintenance().send(operation); | ||
| //endregion | ||
| } | ||
|
|
||
| } | ||
| } |
Oops, something went wrong.