diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.dotnet.markdown new file mode 100644 index 0000000000..89b05997e0 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.dotnet.markdown @@ -0,0 +1,178 @@ +# Data Subscriptions: Consumption API Overview + +--- + +{NOTE: } + +* In this page: + * [Subscription worker generation](../../../client-api/data-subscriptions/consumption/api-overview#subscription-worker-generation) + * [SubscriptionWorkerOptions](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworkeroptions) + * [Running subscription worker](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker) + * [SubscriptionBatch<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch) + * [SubscriptionWorker<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworker) + +{NOTE/} + +--- + +{PANEL:Subscription worker generation} + +Subscription worker generation is accessible through the `DocumentStore`'s `Subscriptions` Property, of type `DocumentSubscriptions`: +{CODE subscriptionWorkerGeneration@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +| Parameters | | | +| ------------- | ------------- | ----- | +| **subscriptionName** | `string` | The subscription's name. This parameter appears in more simple overloads allowing to start processing without creating a `SubscriptionCreationOptions` instance, relying on the default values | +| **options** | `SubscriptionWorkerOptions` | Contains subscription worker, affecting the interaction of the specific worker with the subscription, but does not affect the subscription's definition | +| **database** | `string` | Name of the database to look for the data subscription. If `null`, the default database configured in DocumentStore will be used. | + +| Return value | | +| ------------- | ----- | +| `SubscriptionWorker` | A created data subscription worker. When returned, the worker is Idle and it will start working only when the `Run` function is called. | + + +{PANEL/} + +{PANEL:SubscriptionWorkerOptions} + +{NOTE The only mandatory parameter for SubscriptionWorkerOptions creation is the subscription's name. /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **SubscriptionName** | `string` | Returns the subscription name passed to the constructor. This name will be used by the server side to identify the subscription in question. | +| **TimeToWaitBeforeConnectionRetry** | `TimeSpan` | Time to wait before reconnecting, in the case of non-aborting failure during the subscription processing. Default: 5 seconds. | +| **IgnoreSubscriberErrors** | `bool` | If true, will not abort subscription processing if client code, passed to the `Run` function, throws an unhandled exception. Default: false. | +| **Strategy** | `SubscriptionOpeningStrategy`
(enum) | Sets the way the server will treat current and/or other clients when they will try to connect. See [Workers interplay](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#worker-interplay). Default: `OpenIfFree`. | +| **MaxDocsPerBatch** | `int` | Maximum amount of documents that the server will try sending in a batch. If the server will not find "enough" documents, it won't wait and send the amount it found. Default: 4096. | +| **CloseWhenNoDocsLeft** | `bool` | If true, it performs an "ad-hoc" operation that processes all possible documents, until the server can't find any new documents to send. At that moment, the task returned by the `Run` function will fail and throw a `SubscriptionClosedException` exception. Default: false. | +| **SendBufferSizeInBytes** | `int` | The size in bytes of the TCP socket buffer used for _sending_ data.
Default: 32,768 (32 KiB) | +| **ReceiveBufferSizeInBytes** | `int` | The size in bytes of the TCP socket buffer used for _receiving_ data.
Default: 32,768 (32 KiB) | + +{PANEL/} + +{PANEL:Running subscription worker} + +After [generating](../../../client-api/data-subscriptions/consumption/api-overview#subscription-worker-generation) a subscription worker, the subscription worker is still not processing any documents. SubscriptionWorker's `Run` function allows you to start processing worker operations. +The `Run` function receives the client-side code as a delegate that will process the received batches: + +{CODE subscriptionWorkerRunning@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + + +| Parameters | | | +| ------------- | ------------- | ----- | +| **processDocuments** | `Action>` | Delegate for sync batches processing | +| **processDocuments** | `Func, Task>` | Delegate for async batches processing | +| **ct** | `CancellationToken` | Cancellation token used in order to halt the worker operation | + +| Return value | | +| ------------- | ----- | +| `Task` | Task that is alive as long as the subscription worker is processing or tries processing. If the processing is aborted, the task exits with an exception | + +{PANEL/} + + +{PANEL:SubscriptionBatch<T>} + +| Member | Type | Description | +|--------|:-----|-------------| +| **Items** | `List.Item>` | Batch's items list. | +| **NumberOfItemsInBatch** | `int` | Amount of items in the batch. | + +| Method Signature | Return value | Description | +|--------|:-------------|-------------| +| **OpenSession()** | `IDocumentSession` | New document session, that tracks all items and included items of the current batch. | +| **OpenAsyncSession()** | `IDocumentSession` | New asynchronous document session, that tracks all items and included items of the current batch. | + + +{NOTE:Subscription Worker Connectivity} + +As long as there is no exception, the worker will continue addressing the same +server that the first batch was received from. +If the worker fails to reach that node, it will try to +[failover](../../../client-api/configuration/load-balance/overview) to another node +from the session's topology list. +The node that the worker succeeded connecting to, will inform the worker which +node is currently responsible for data subscriptions. + +{NOTE/} + + + +{INFO:SubscriptionBatch<T>.Item} + +{NOTE if T is `BlittableJsonReaderObject`, no deserialization will take place /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **Result** | `T` | Current batch item. | +| **ExceptionMessage** | `string` | Message of the exception thrown during current document processing in the server side. | +| **Id** | `string` | Current batch item's underlying document ID. | +| **ChangeVector** | `string` | Current batch item's underlying document change vector of the current document. | +| **RawResult** | `BlittableJsonReaderObject` | Current batch item before serialization to `T`. | +| **RawMetadata** | `BlittableJsonReaderObject` | Current batch item's underlying document metadata. | +| **Metadata** | `IMetadataDictionary` | Current batch item's underlying metadata values. | + + +{WARNING Usage of `RawResult`, `RawMetadata`, and `Metadata` values outside of the document processing delegate are not supported /} + + +{INFO/} + +{PANEL/} + +{PANEL:SubscriptionWorker<T>} + +{NOTE:Methods} + +| Method Signature| Return Type | Description | +|--------|:-----|-------------| +| **Dispose()** | `void` | Aborts subscription worker operation ungracefully by waiting for the task returned by the `Run` function to finish running. | +| **DisposeAsync()** | `Task` | Async version of `Dispose()`. | +| **Dispose(bool waitForSubscriptionTask)** | `void` | Aborts the subscription worker, but allows deciding whether to wait for the `Run` function task or not. | +| **DisposeAsync(bool waitForSubscriptionTask)** | `void` | Async version of `DisposeAsync(bool waitForSubscriptionTask)`. | +| **Run (multiple overloads)** | `Task` | Starts the subscription worker work of processing batches, receiving the batch processing delegates (see [above](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker)). | + +{NOTE/} + +{NOTE:Events} + +| Event | Type\Return type | Description | +|--------|:-----|-------------| +| **AfterAcknowledgment** | `AfterAcknowledgmentAction` (event) | Event that is risen after each time the server acknowledges batch processing progress. | +| **OnSubscriptionConnectionRetry** | `Action` (event) | Event that is fired when the subscription worker tries to reconnect to the server after a failure. The event receives as a parameter the exception that interrupted the processing. | +| **OnDisposed** | `Action>` (event) | Event that is fired after the subscription worker was disposed. | + +{INFO:AfterAcknowledgmentAction} + +| Parameters | | | +| ------------- | ------------- | ----- | +| **batch** | `SubscriptionBatch<T>` | The batch process which was acknowledged | + +| Return value | | +| ------------- | ----- | +| `Task` | Task for which the worker will wait for the event processing to be finished (for async functions, etc.) | + +{INFO/} + +{NOTE/} + + + +{NOTE:Properties} + +| Member | Type\Return type | Description | +|--------|:-----|-------------| +| **CurrentNodeTag** | `string` | Returns current processing RavenDB server's node tag. | +| **SubscriptionName** | `string` | Returns processed subscription's name. | + +{NOTE/} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.java.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.java.markdown new file mode 100644 index 0000000000..4b21372701 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.java.markdown @@ -0,0 +1,155 @@ +# Data Subscriptions: Consumption API Overview + +--- + +{NOTE: } + +* In this page: + * [Subscription worker generation](../../../client-api/data-subscriptions/consumption/api-overview#subscription-worker-generation) + * [SubscriptionWorkerOptions](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworkeroptions) + * [Running subscription worker](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker) + * [SubscriptionBatch<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch) + * [SubscriptionWorker<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworker) + +{NOTE/} + +--- + +{PANEL:Subscription worker generation} + +Subscription worker generation is accessible through the `DocumentStore`'s `subscriptions()` method, of type `DocumentSubscriptions`: +{CODE:java subscriptionWorkerGeneration@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +| Parameters | | | +| ------------- | ------------- | ----- | +| **subscriptionName** | `String` | The subscription's name. This parameter appears in more simple overloads allowing to start processing without creating a `SubscriptionCreationOptions` instance, relying on the default values | +| **options** | `SubscriptionWorkerOptions` | Contains subscription worker, affecting the interaction of the specific worker with the subscription, but does not affect the subscription's definition | +| **database** | `String` | Name of the database to look for the data subscription. If `null`, the default database configured in DocumentStore will be used. | + +| Return value | | +| ------------- | ----- | +| `SubscriptionWorker` | A created data subscription worker. When returned, the worker is Idle and it will start working only when the `run` function is called. | + + +{PANEL/} + +{PANEL:SubscriptionWorkerOptions} + +{NOTE The only mandatory parameter for SubscriptionWorkerOptions creation is the subscription's name. /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **subscriptionName** | `String` | Returns the subscription name passed to the constructor. This name will be used by the server side to identify the subscription in question. | +| **timeToWaitBeforeConnectionRetry** | `Duration` | Time to wait before reconnecting, in the case of non-aborting failure during the subscription processing. Default: 5 seconds. | +| **ignoreSubscriberErrors** | `boolean` | If true, will not abort subscription processing if client code, passed to the `run` function, throws an unhandled exception. Default: false. | +| **strategy** | `SubscriptionOpeningStrategy`
(enum) | Sets the way the server will treat current and/or other clients when they will try to connect. See [Workers interplay](how-to-consume-data-subscription#workers-interplay). Default: `OPEN_IF_FREE`. | +| **maxDocsPerBatch** | `int` | Maximum amount of documents that the server will try sending in a batch. If the server will not find "enough" documents, it won't wait and send the amount it found. Default: 4096. | +| **closeWhenNoDocsLeft** | `boolean` | If true, it performs an "ad-hoc" operation that processes all possible documents, until the server can't find any new documents to send. At that moment, the task returned by the `Run` function will fail and throw a `SubscriptionClosedException` exception. Default: false. | +| **sendBufferSizeInBytes** | `int` | The size in bytes of the TCP socket buffer used for _sending_ data.
Default: 32,768 (32 KiB) | +| **receiveBufferSizeInBytes** | `int` | The size in bytes of the TCP socket buffer used for _receiving_ data.
Default: 32,768 (32 KiB) | + +{PANEL/} + +{PANEL:Running subscription worker} + +After receiving a subscription worker, the subscription worker is still not processing any documents. SubscriptionWorker's `run` function allows you to start processing worker operations. +The `run` function receives the client-side code as a consumer that will process the received batches: + +{CODE:java subscriptionWorkerRunning@ClientApi\DataSubscriptions\DataSubscriptions.java /} + + +| Parameters | | | +| ------------- | ------------- | ----- | +| **processDocuments** | `Consumer>` | Delegate for sync batches processing | + +| Return value | | +| ------------- | ----- | +| `CompletableFuture` | Task that is alive as long as the subscription worker is processing or tries processing. If the processing is aborted, the future exits with an exception | + +{PANEL/} + + +{PANEL:SubscriptionBatch<T>} + +| Member | Type | Description | +|--------|:-----|-------------| +| **items** | `List.Item>` | Batch's items list. | +| **numberOfItemsInBatch** | `int` | Amount of items in the batch. | + +| Method Signature | Return value | Description | +|--------|:-------------|-------------| +| **openSession()** | `IDocumentSession` | New document session, that tracks all items and included items of the current batch. | + + +{NOTE:Subscription Worker Connectivity} + +As long as there is no exception, the worker will continue addressing the same +server that the first batch was received from. +If the worker fails to reach that node, it will try to failover to another node +from the session's topology list. +The node that the worker succeeded connecting to, will inform the worker which +node is currently responsible for data subscriptions. + +{NOTE/} + + +{INFO:SubscriptionBatch<T>.Item} + +{NOTE if T is `ObjectNode`, no deserialization will take place /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **result** | `T` | Current batch item. | +| **exceptionMessage** | `String` | Message of the exception thrown during current document processing in the server side. | +| **id** | `String` | Current batch item's underlying document ID. | +| **changeVector** | `String` | Current batch item's underlying document change vector of the current document. | +| **rawResult** | `ObjectNode` | Current batch item before serialization to `T`. | +| **rawMetadata** | `ObjectNode` | Current batch item's underlying document metadata. | +| **metadata** | `IMetadataDictionary` | Current batch item's underlying metadata values. | + +{INFO/} + +{PANEL/} + +{PANEL:SubscriptionWorker<T>} + +{NOTE:Methods} + +| Method Signature| Return Type | Description | +|--------|:-----|-------------| +| **close()** | `void` | Aborts subscription worker operation ungracefully by waiting for the task returned by the `run` function to finish running. | +| **run (multiple overloads)** | `CompletableFuture` | Starts the subscription worker work of processing batches, receiving the batch processing delegates (see [above](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker)). | + +{NOTE/} + +{NOTE:Events} + +| Event | Type\Return type | Description | +|--------|:-----|-------------| +| **addAfterAcknowledgmentListener** | `Consumer>` (event) | Event that is risen after each the server acknowledges batch processing progress. | +| **onSubscriptionConnectionRetry** | `Consumer` (event) | Event that is fired when the subscription worker tries to reconnect to the server after a failure. The event receives as a parameter the exception that interrupted the processing. | +| **onClosed** | `Consumer>` (event) | Event that is fired after the subscription worker was disposed. | + +{NOTE/} + + + +{NOTE:Properties} + +| Member | Type\Return type | Description | +|--------|:-----|-------------| +| **currentNodeTag** | `String` | Returns current processing RavenDB server's node tag. | +| **subscriptionName** | `String` | Returns processed subscription's name. | + +{NOTE/} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.python.markdown new file mode 100644 index 0000000000..46e64b9004 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/api-overview.python.markdown @@ -0,0 +1,155 @@ +# Data Subscriptions: Consumption API Overview + +--- + +{NOTE: } + +* In this page: + * [Subscription worker generation](../../../client-api/data-subscriptions/consumption/api-overview#subscription-worker-generation) + * [`SubscriptionWorkerOptions`](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworkeroptions) + * [Running subscription worker](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker) + * [`SubscriptionBatch[_T]`](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch[_t]) + * [`SubscriptionWorker[_T]`](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionworker[_t]) + +{NOTE/} + +--- + +{PANEL:Subscription worker generation} + +Create a subscription worker using `get_subscription_worker` or `get_subscription_worker_by_name`. + +* Use the `get_subscription_worker` method to specify the subscription options while creating the worker. +* Use the `get_subscription_worker_by_name` method to create the worker using the default options. + +{CODE:python subscriptionWorkerGeneration@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Parameters | | | +| ------------- | ------------- | ----- | +| **options** | `SubscriptionWorkerOptions` | Subscription worker options (affecting the interaction of the specific worker with the subscription, but not the subscription's definition) | +| **object_type** (Optional) | `Type[_T]` | Defines the object type (class) for the items that will be included in the received `SubscriptionBatch` object. | +| **database** (Optional) | `str` | Name of the database to look for the data subscription. If `None`, the default database configured in DocumentStore will be used. | +| **subscription_name** (Optional) | `str` | The subscription's name. Used when the worker is generated without creating a `SubscriptionCreationOptions` instance, relying on the default values. | + +| Return value | | +| ------------- | ----- | +| `SubscriptionWorker` | A created data subscription worker. When returned, the worker is Idle. It will start working only when its `run` function is called. | + +{PANEL/} + +{PANEL:`SubscriptionWorkerOptions`} + +{NOTE The only mandatory parameter for the ceation of `SubscriptionWorkerOptions` is the subscription **name**. /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **subscription_name** | `str` | Returns the subscription name passed to the constructor. This name will be used by the server side to identify the subscription. | +| **time_to_wait_before_connection_retry** | `timedelta` | Time to wait before reconnecting, in the case of non-aborting failure during the subscription processing.
Default: 5 seconds. | +| **ignore_subscriber_errors** | `bool` | If `True`, will not abort subscription processing if client code, passed to the `run` function, throws an unhandled exception.
Default: `False` | +| **strategy** | `SubscriptionOpeningStrategy`
(enum) | Sets the way the server will treat current and/or other clients when they try to connect. See [Workers interplay](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#worker-interplay).
Default: `OPEN_IF_FREE`. | +| **max_docs_per_batch** | `int` | Max number of documents the server will try to send in a batch. If the server doesn't find as many documents as specified, it will not wait and send those it did find.
Default: 4096. | +| **close_when_no_docs_left** | `bool` | If `True`, an "ad-hoc" operation processes all possible documents, until the server finds no new documents to send. When this happens, the task returned by the `run` function will fail and throw a `SubscriptionClosedException` exception.
Default: `False` | +| **send_buffer_size_in_bytes** | `int` | Size, in bytes, of the TCP socket buffer used for _sending_ data.
Default: 32,768 (32 KiB) | +| **receive_buffer_size_in_bytes** | `int` | Size, in bytes, of the TCP socket buffer used for _receiving_ data.
Default: 32,768 (32 KiB) | + +{PANEL/} + +{PANEL:Running subscription worker} + +After [generating](../../../client-api/data-subscriptions/consumption/api-overview#subscription-worker-generation) +a subscription worker, the worker doesn't process any documents yet. +Processing starts when the `run` function is called. +The `run` function receives the client-side code as a function that will process the retrieved batches. +{CODE:python subscriptionWorkerRunning@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Parameters | | | +| ------------- | ------------- | ----- | +| **process_documents** (Optional) | `[Callable[[SubscriptionBatch[_T]], Any]]` | Delegate to sync batch processing | + +{PANEL/} + + +{PANEL:`SubscriptionBatch[_T]`} + +| Member | Type | Description | +|--------|:-----|-------------| +| **items** | `SubscriptionBatch[_T].Item` array | Batch items list | +| **number_of_items_in_batch** | `int` | Number of items in the batch | + +{CODE:python number_of_items_in_batch_definition@ClientApi\DataSubscriptions\DataSubscriptions.py /} + + +{NOTE:Subscription Worker Connectivity} +As long as there is no exception, the worker will continue addressing the same +server that the first batch was received from. +If the worker fails to reach that node, it will try to +[failover](../../../client-api/configuration/load-balance/overview) to another node +from the session's topology list. +The node that the worker succeeds connecting to, will inform the worker which +node is currently responsible for data subscriptions. +{NOTE/} + +{CODE:python Item_definition@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE:python SubscriptionBatch_definition@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| `SubscriptionBatch[_T].item` Member | Type | Description | +|--------|:-----|-------------| +| **\_result** (Optional) | `_T_Item` | Current batch item | +| **\_exception_message** (Optional) | `str` | Message of the exception thrown during current document processing in the server side | +| **\_key** (Optional) | `str` | Current batch item underlying document ID | +| **\_change_vector** (Optional) | `str` | Current batch item underlying document change vector of the current document | +| **\_projection** (Optional) | `bool` | indicates whether the value id a projection | +| **raw_result** (Optional) | `Dict` | Current batch item before serialization to `T` | +| **raw_metadata** (Optional) | `Dict` | Current batch item underlying document metadata | +| **\_metadata** (Optional) | `MetadataAsDictionary` | Current batch item underlying metadata values | + +{WARNING: } +Usage of `raw_result`, `raw_metadata`, and `_metadata` values outside of the document processing delegate +is not supported. +{WARNING/} + +{PANEL/} + +{PANEL:`SubscriptionWorker[_T]`} + +--- + +### Methods: + +| Method | Return Type | Description | +|--------|:-----|-------------| +| `close(bool wait_for_subscription_task = True)` | `void` | Aborts subscription worker operation ungracefully by waiting for the task returned by the `run` function to finish running. | +| `run` | `Future[None]` | Starts the subscription worker work of processing batches, receiving the batch processing delegates (see [above](../../../client-api/data-subscriptions/consumption/api-overview#running-subscription-worker)). | + +--- + +### Events: + +| Event | Type\Return type | Description | +|--------|:-----|-------------| +| **after\_acknowledgment** | `Callable[[SubscriptionBatch[_T]], None]` | Event invoked after each time the server acknowledges batch processing progress. | + +| `after_acknowledgment` Parameters | | | +| ------------- | ------------- | ----- | +| **batch** | `SubscriptionBatch[_T]` | The batch process which was acknowledged | + +| Return value | | +| ------------- | ----- | +| `Future[None]` | The worker waits for the task to finish the event processing | + +### Properties: + +| Member | Type\Return type | Description | +|--------|:-----|-------------| +| **current_node_tag** | `str` | Returns the node tag of the current processing RavenDB server | +| **subscription_name** | `str` | Returns processed subscription name | + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.dotnet.markdown new file mode 100644 index 0000000000..2ad0ca7e99 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.dotnet.markdown @@ -0,0 +1,114 @@ +# Data Subscriptions: Subscription Consumption Examples + +--- + +{NOTE: } + +* In this page: + * [Subscription workers with failover on other nodes](../../../client-api/data-subscriptions/consumption/examples#subscription-workers-with-failover-on-other-nodes) + * [Worker with a specified batch size](../../../client-api/data-subscriptions/consumption/examples#worker-with-a-specified-batch-size) + * [Client with full exception handling and processing retries](../../../client-api/data-subscriptions/consumption/examples#client-with-full-exception-handling-and-processing-retries) + * [Subscription that ends when no documents are left](../../../client-api/data-subscriptions/consumption/examples#subscription-that-ends-when-no-documents-are-left) + * [Worker that processes dynamic objects](../../../client-api/data-subscriptions/consumption/examples#worker-that-processes-dynamic-objects) + * [Subscription that works with a session](../../../client-api/data-subscriptions/consumption/examples#subscription-that-works-with-a-session) + * [Subscription that uses included documents](../../../client-api/data-subscriptions/consumption/examples#subscription-that-uses-included-documents) + * [Subscription that works with lowest level API](../../../client-api/data-subscriptions/consumption/examples#subscription-that-works-with-lowest-level-api) + * [Subscription workers with a primary and a secondary node](../../../client-api/data-subscriptions/consumption/examples#subscription-workers-with-a-primary-and-a-secondary-node) + +{NOTE/} + +--- + +{PANEL:Subscription workers with failover on other nodes} + +In this configuration, any available node will create a worker. +If the worker fails, another available node will take over. + +{CODE waitforfree@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Worker with a specified batch size} + +Here we create a worker, specifying the maximum batch size we want to receive. + +{CODE subscription_worker_with_batch_size@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Client with full exception handling and processing retries} + +Here we implement a client that treats exceptions thrown by a worker, and retries creating the worker if an exception is recoverable. + +{CODE reconnecting_client@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription that ends when no documents are left} + +Here we create a subscription client that runs only up to the point there are no more new documents left to process. + +This is useful for an ad-hoc single-use processing that the user wants to be sure is performed completely. + +{CODE single_run@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + + +{PANEL:Worker that processes dynamic objects} + +Here we create a worker that processes received data as dynamic objects. + +{CODE dynamic_worker@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription that works with a session} + +Here we create a worker that receives all orders without a shipping date, lets the shipment mechanism handle it, and updates the `ShippedAt` field value. + +{CODE subscription_with_open_session_usage@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription that uses included documents} + +Here we create a subscription utilizing the includes feature, by processing `Order` documents and including all `Product`s of each order. +When processing the subscription, we create a session using the [SubscriptionBatch<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch) object, +and for each order line, we obtain the `Product` document and process it alongside with the `Order`. + +{CODE subscription_with_includes_path_usage@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription that works with lowest level API} + +Here we create a subscription that works with blittable document representation that can be useful in extreme high-performance scenarios, +but it may be dangerous due to the direct usage of unmanaged memory. + +{CODE blittable_worker@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription workers with a primary and a secondary node} + +Here we create two workers: + +* The primary worker, set with a `TakeOver` strategy, will take the lead over the secondary worker. +* The secondary worker, set with a `WaitForFree` strategy, will take over if the primary worker fails (e.g. due to a machine failure). + +The primary worker: +{CODE waiting_subscription_1@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +The secondary worker: +{CODE waiting_subscription_2@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.java.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.java.markdown new file mode 100644 index 0000000000..a8d0fbd774 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.java.markdown @@ -0,0 +1,95 @@ +# Data Subscriptions: Subscription Consumption Examples + +--- + +{NOTE: } + +* this page: + * [Worker with a specified batch size](../../../client-api/data-subscriptions/consumption/examples#worker-with-a-specified-batch-size) + * [Client with full exception handling and processing retries](../../../client-api/data-subscriptions/consumption/examples#client-with-full-exception-handling-and-processing-retries) + * [Subscription that ends when no documents left](../../../client-api/data-subscriptions/consumption/examples#subscription-that-ends-when-no-documents-left) + * [Subscription that processes ObjectNode objects](../../../client-api/data-subscriptions/consumption/examples#subscription-that-processes-objectnode-objects) + * [Subscription that works with a session](../../../client-api/data-subscriptions/consumption/examples#subscription-that-works-with-a-session) + * [Subscription that uses included documents](../../../client-api/data-subscriptions/consumption/examples#subscription-that-uses-included-documents) + * [Two subscription workers that are waiting for each other](../../../client-api/data-subscriptions/consumption/examples#two-subscription-workers-that-are-waiting-for-each-other) + +{NOTE/} + +--- + +{PANEL:Worker with a specified batch size} + +Here we create a worker, specifying the maximum batch size we want to receive. + +{CODE:java subscription_worker_with_batch_size@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +{PANEL:Client with full exception handling and processing retries} + +Here we implement a client that treats exceptions thrown by worker, and retries creating the worker if an exception is recoverable. + +{CODE:java reconnecting_client@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +{PANEL:Subscription that ends when no documents left} + +Here we create a subscription client that runs only up to the point there are no more new documents left to process. + +This is useful for an ad-hoc single use processing that the user wants to be sure is performed completely. + +{CODE:java single_run@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +{PANEL:Worker that processes raw objects} + +Here we create a worker that processes received data as ObjectNode objects. + +{CODE:java dynamic_worker@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +{PANEL:Subscription that works with a session} + +Here we create a worker that receives all orders without a shipping date, lets the shipment mechanism to handle it and updates the `ShippedAt` field value. + +{CODE:java subscription_with_open_session_usage@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +{PANEL:Subscription that uses included documents} + +Here we create a subscription utilizing the includes feature, by processing `Order` documents and including all `Product`s of each order. +When processing the subscription, we create a session using the [SubscriptionBatch<T>](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch) object, +and for each order line, we obtain the `Product` document and process it alongside with the `Order`. + +{CODE:java subscription_with_includes_path_usage@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + + +{PANEL:Two subscription workers that are waiting for each other} + +Here we create two workers: +* The main worker with the `TAKE_OVER` strategy that will take over the other one and will take the lead +* The secondary worker that will wait for the first one fail (due to machine failure etc.) + +The main worker: + +{CODE:java waiting_subscription_1@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +The secondary worker: + +{CODE:java waiting_subscription_2@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.python.markdown new file mode 100644 index 0000000000..7e7f0cf7e6 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/examples.python.markdown @@ -0,0 +1,104 @@ +# Data Subscriptions: Subscription Consumption Examples + +--- + +{NOTE: } + +* In this page: + * [Subscription workers with failover on other nodes](../../../client-api/data-subscriptions/consumption/examples#subscription-workers-with-failover-on-other-nodes) + * [Worker with a specified batch size](../../../client-api/data-subscriptions/consumption/examples#worker-with-a-specified-batch-size) + * [Client with full exception handling and processing retries](../../../client-api/data-subscriptions/consumption/examples#client-with-full-exception-handling-and-processing-retries) + * [Subscription that ends when no documents are left](../../../client-api/data-subscriptions/consumption/examples#subscription-that-ends-when-no-documents-are-left) + * [Worker that processes dynamic objects](../../../client-api/data-subscriptions/consumption/examples#worker-that-processes-dynamic-objects) + * [Subscription that uses included documents](../../../client-api/data-subscriptions/consumption/examples#subscription-that-uses-included-documents) + * [Subscription that works with a session](../../../client-api/data-subscriptions/consumption/examples#subscription-that-works-with-a-session) + * [Subscription workers with a primary and a secondary node](../../../client-api/data-subscriptions/consumption/examples#subscription-workers-with-a-primary-and-a-secondary-node) + +{NOTE/} + +--- + +{PANEL:Subscription workers with failover on other nodes} + +In this configuration, any available node will create a worker. +If the worker fails, another available node will take over. + +{CODE:python waitforfree@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Worker with a specified batch size} + +Here we create a worker, specifying the maximum batch size we want to receive. + +{CODE:python subscription_worker_with_batch_size@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Client with full exception handling and processing retries} + +Here we implement a client that treats exceptions thrown by a worker, and retries creating the worker if an exception is recoverable. + +{CODE:python reconnecting_client@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Subscription that ends when no documents are left} + +Here we create a subscription client that runs only up to the point there are no more new documents left to process. + +This is useful for an ad-hoc single-use processing that the user wants to be sure is performed completely. + +{CODE:python single_run@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + + +{PANEL:Worker that processes dynamic objects} + +Here we create a worker that processes received data as dynamic objects. + +{CODE:python dynamic_worker@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Subscription that works with a session} + +Here we create a worker that receives all orders without a shipping date, lets the shipment mechanism handle it, and updates the `ShippedAt` field value. + +{CODE:python subscription_with_open_session_usage@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Subscription that uses included documents} + +Here we create a subscription utilizing the includes feature, by processing `Order` documents and including all `Product`s of each order. +When processing the subscription, we create a session using the [SubscriptionBatch[\_T]](../../../client-api/data-subscriptions/consumption/api-overview#subscriptionbatch) object, +and for each order line, we obtain the `Product` document and process it alongside the `Order`. + +{CODE subscription_with_includes_path_usage@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +{PANEL:Subscription workers with a primary and a secondary node} + +Here we create two workers: + +* The primary worker, set with a `TAKE_OVER` strategy, will take the lead over the secondary worker. +* The secondary worker, set with a `WAIT_FOR_FREE` strategy, will take over if the primary worker fails (e.g. due to a machine failure). + +The primary worker: +{CODE:python waiting_subscription_1@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +The secondary worker: +{CODE:python waiting_subscription_2@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.dotnet.markdown new file mode 100644 index 0000000000..2f0ccb98af --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.dotnet.markdown @@ -0,0 +1,188 @@ +# Data Subscriptions: How to Consume a Data Subscription + +--- + +{NOTE: } + +* Batches of documents sent from a Subscription Task defined on the server are + consumed and processed by a subscription worker client. +* The `SubscripionWorker` object, defined on the client, manages the communication + between the server and the client and processes the document batches sent from the server. +* There are several ways to create and configure the SubscriptionWorker - see `SubscriptionWorkerOptions`. + +* In this page: + * [SubscriptionWorker lifecycle](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#subscriptionworker-lifecycle) + * [Error handling](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#error-handling) + * [Worker interplay](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#worker-interplay) + * [Available Worker Strategies](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#available-worker-strategies) + * [Determining Which Workers a Subscription Will Serve](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#determining-which-workers-a-subscription-will-serve) + +{NOTE/} + +--- + +{PANEL:SubscriptionWorker lifecycle} +A `SubscriptionWorker` object starts its life from being generated by the `DocumentsStore.Subscriptions`: +{CODE subscription_open_simple@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +At this point, the worker has only got its configuration. No connection or processing happens at this moment. +To start processing, the `Run` method should be called. The Run method receives the batch processing logic that should be performed: +{CODE subscription_run_simple@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +From this point on, the subscription worker will start processing batches. +If processing is aborted for any reason, the returned task (`subscriptionRuntimeTask`) +will be finished with an exception. + +{PANEL/} + + +{PANEL:Error handling} +There are two categories of errors that may occur during subscription processing: + +{INFO:Internal mechanism errors} +Internal mechanism errors occur during the normal server-client communication between the worker and the server. +If an unexpected error occurs, the worker will try to reconnect to the server. +There are conditions in which the worker will cease its operation and will not try to reconnect: + +* The subscription does not exist or was deleted +* Another worker took over the subscription (see connection strategy) +* The worker could not connect to any of the servers +* The worker could not receive the node in charge of the task (this can happen when there is no leader) +* Authorization exception +* Exception during connection establishment + +{INFO/} + +{INFO:User's batch processing logic unhandled exception} +Example: +{CODE throw_during_user_logic@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +If an exception is thrown, the worker will abort the current batch process. +A worker can be configured to treat the thrown exception in either of the following two ways: + +* By default, the worker will wrap the thrown exception with a `SubscriberErrorException` exception and rethrow it, + terminating the subscription execution without acknowledging progress or retrying. The task returned by the `Run` function will + be terminated with an erroneous state, throwing a `SubscriberErrorException` exception. + +* If `SubscriptionWorkerOptions`'s value `IgnoreSubscriberErrors` is set to true, the erroneous batch will get acknowledged without retrying and the next batches will continue processing. +{INFO/} + +{INFO: Reconnecting} +In the cases above, we described situations in which a worker will try to reconnect with the server. There are two key `SubscriptionWorkerOptions` fields controlling this state: + +* `TimeToWaitBeforeConnectionRetry` - The time that the worker will 'sleep' before trying to reconnect. +* `MaxErroneousPeriod` - The maximum time in which the worker is allowed to be in an erroneous state. After that time passes, the worker will stop trying to reconnect. +{INFO/} + +{INFO: Timing out} +A worker will time out after losing its connectivity with the server for a given time period. + +* The timeout period can be set using the `ConnectionStreamTimeout` option. + E.g. + {CODE worker_timeout_minimal_sample@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +* Default timeout period: 30 second +{INFO/} + + +{INFO: OnUnexpectedSubscriptionError} +`OnUnexpectedSubscriptionError` is the event raised when a connection failure occurs +between the subscription worker and the server and it throws an unexpected exception. +When this occurs, the worker will automatically try to reconnect again. This event is +useful for logging these unexpected exceptions. +{INFO/} + +{PANEL/} + +{PANEL: Worker interplay} + +* Subscription workers are configured with a **strategy**, that determines whether + they can connect their subscription **concurrently**, or only one at a time. +* If a one-at-a-time strategy is chosen, it also determines how the workers + interact with each other to resolve which will connect the subscription. + +{NOTE: } + +The strategy is configured by the `SubscriptionWorkerOptions` `Strategy` field. +The strategy field is the enum `SubscriptionOpeningStrategy`. + +| `SubscriptionOpeningStrategy` | | +| ------------- | ------------- | +| `OpenIfFree` | Connect if no other worker is connected | +| `WaitForFree` | Wait for currently connected worker to disconnect | +| `TakeOver` | Take over the connection | +| `Concurrent` | Connect concurrently | + +{NOTE/} + +## Available Worker Strategies + +--- + +### One Worker Per Subscription Strategies + +The following three strategies allow only a **single worker to connect +the subscription at any given time**, and determine what happens when one +worker is connected and another tries to connect. + +* `SubscriptionOpeningStrategy.OpenIfFree` + The server will allow the worker to connect only if there is currently + no other connected worker. + If there is an existing connection, the incoming worker will throw a + `SubscriptionInUseException`. +* `SubscriptionOpeningStrategy.WaitForFree` + If the worker currently cannot open the subscription because it is used + by another worker, it will wait for the previous worker to disconnect and + connect only then. + This is useful in worker failover scenarios, where there is one active + worker and another already waits to take its place. +* `SubscriptionOpeningStrategy.TakeOver` + The server will allow an incoming connection to overthrow an existing one, + according to the existing connection strategy. + * If the existing connection **does not** have a `TakeOver` strategy: + The incoming connection will take over, causing the existing + connection to throw a `SubscriptionInUseException`. + * If the existing connection **has** a `TakeOver` strategy: + The incoming connection will throw a `SubscriptionInUseException` exception. + +--- + +### Concurrent Strategy + +* `SubscriptionOpeningStrategy.Concurrent` + Multiple workers of the same subscription are allowed to connect it simultaneously. + Read more about concurrent subscriptions [here](../../../client-api/data-subscriptions/concurrent-subscriptions). + +{PANEL/} + +{PANEL: Determining Which Workers a Subscription Will Serve} + +{NOTE: } +The **strategy used by the first worker connecting to a subscription** will determine +which additional workers this subscription will be able to serve until all worker connections +are dropped. +{NOTE/} + +* A subscription that serves one or more [Concurrent](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#concurrent-strategy) + workers, **can serve only other concurrent workers** until all worker connections are dropped. + If a worker that uses a [One Worker Per Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#one-worker-per-subscription-strategies) + strategy attempts to connect it - + * The connection attempt will be rejected. + * `SubscriptionInUseException` will be thrown. + +* A subscription that serves a worker that uses a + [One Worker Per Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#one-worker-per-subscription-strategies) + strategy, **cannot** serve + [Concurrent](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#concurrent-strategy) + workers until the worker's connection is dropped. + If a concurrent worker attempts to connect it - + * The connection attempt will be rejected. + * `SubscriptionInUseException` will be thrown. + +{PANEL/} + +## Related Articles + +### Data Subscriptions + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.java.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.java.markdown new file mode 100644 index 0000000000..de813573ac --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.java.markdown @@ -0,0 +1,96 @@ +# Data Subscriptions: How to Consume a Data Subscription + +--- + +{NOTE: } + +Subscriptions are consumed by processing batches of documents received from the server. +A `SubscriptionWorker` object manages the documents processing and the communication between the client and the server according to a set of configurations received upon it's creation. +We've introduced several ways to create and configure a SubscriptionWorker, starting from just giving a subscription name, and ending with a detailed configuration object - `SubscriptionWorkerOptions`. + +* In this page: + * [SubscriptionWorker lifecycle](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#subscriptionworker-lifecycle) + * [Error handling](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#error-handling) + * [Workers interplay](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#workers-interplay) + +{NOTE/} + +--- + +{PANEL:SubscriptionWorker lifecycle} +A `SubscriptionWorker` object starts its life from being generated by the `DocumentsStore.subscriptions`: + +{CODE:java subscription_open_simple@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +At this point, the worker has only got its configuration. No connection or processing happens at this moment. +In order to start processing, the `run` method should be called. The `run` method receives the batch processing logic that should be performed: + +{CODE:java subscription_run_simple@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +From this point on, the subscription worker will start processing batches. If for any reason, the processing is aborted, and the returned task (`subscriptionRuntimeTask`) will be finished with an exception. + +{PANEL/} + + +{PANEL:Error handling} +There are two categories of errors that may occur during subscription processing: + +{INFO:Internal mechanism errors} +Those errors occur during the normal server-client communication between the worker and the server. +If an unexpected error occurs, the worker will try to reconnect to the server. There are conditions in which the worker will cease its operation and will not try to reconnect: + +* The subscription does not exist or was deleted +* Another worker took over the subscription (see connection strategy) +* The worker could not connect to any of the servers +* The worker could not receive the node in charge of the task (this can happen when there is no leader) +* Authorization exception +* Exception during connection establishment + +{INFO/} + +{INFO:User's batch processing logic unhandled exception} +Example: +{CODE:java throw_during_user_logic@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +If an exception is thrown, the worker will abort the current batch process. +A worker can be configured to treat the thrown exception by either of the following two ways: + +* By default, the worker will wrap the thrown exception with a `SubscriberErrorException` exception and rethrow it, + terminating the subscription execution without acknowledging progress or retrying. The task returned by the `Run` function will + be terminated with an erroneous state, throwing a `SubscriberErrorException` exception. + +* If `SubscriptionWorkerOptions`'s value `ignoreSubscriberErrors` is set to true, the erroneous batch will get acknowledged without retrying and the next batches will continue processing. +{INFO/} + +{INFO: Reconnecting} +In the cases above, we described situations in which a worker will try to reconnect with the server. There are two key `SubscriptionWorkerOptions` fields controlling this state: + +* `timeToWaitBeforeConnectionRetry` - The time that the worker will 'sleep' before trying to reconnect. +* `maxErroneousPeriod` - The maximum time in which the worker is allowed to be in erroneous state. After that time passes, the worker will stop trying to reconnect +{INFO/} + + +{PANEL/} + +{PANEL: Workers interplay} +There can only be one active subscription worker working on a subscription. +Nevertheless, there are scenarios where it is required to interact between an existing subscription worker and one that tries to connect. +This relationship and interoperation is configured by the `SubscriptionConnectionOptions` `Strategy` field. +The strategy field is an enum, having the following values: + +* `OPEN_IF_FREE` - the server will allow the worker to connect only if there isn't any other currently connected workers. + If there is a existing connection, the incoming worker will throw a SubscriptionInUseException. +* `WAIT_FOR_FREE` - If the client currently cannot open the subscription because it is used by another client, it will wait for the previous client to disconnect and only then will connect. + This is useful in client failover scenarios where there is one active client and another one already waiting to take its place. +* `TAKE_OVER` - the server will allow an incoming connection to overthrow an existing one. It will behave according to the existing connection strategy: + * The existing connection has a strategy that is not `TAKE_OVER`. In this case, the incoming connection will take over it causing the existing connection to throw a SubscriptionInUseException exception. + * The existing connection has a strategy that is `TAKE_OVER`. In this case, the incoming connection will throw a SubscriptionInUseException exception. +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.python.markdown new file mode 100644 index 0000000000..7a3e6f6331 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/consumption/how-to-consume-data-subscription.python.markdown @@ -0,0 +1,182 @@ +# Data Subscriptions: How to Consume a Data Subscription + +--- + +{NOTE: } + +* Batches of documents sent from a Subscription Task defined on the server are + consumed and processed by a subscription worker client. +* The `subscription_worker` object, defined on the client, manages the communication + between the server and the client and processes the document batches sent from the server. +* There are several ways to create and configure the SubscriptionWorker - see `SubscriptionWorkerOptions`. + +* In this page: + * [`subscription_worker` lifecycle](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#subscription_worker-lifecycle) + * [Error handling](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#error-handling) + * [Worker interplay](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#worker-interplay) + * [Available Worker Strategies](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#available-worker-strategies) + * [Determining Which Workers a Subscription Will Serve](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#determining-which-workers-a-subscription-will-serve) + +{NOTE/} + +--- + +{PANEL:`subscription_worker` lifecycle} +A `subscription_worker` object starts its life from being generated by the `store.subscriptions`: +{CODE:python subscription_open_simple@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +At this point, the worker has only got its configuration. No connection or processing happens at this moment. +To start processing, the `run` method should be called. The Run method receives the batch processing logic that should be performed: +{CODE:python subscription_run_simple@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +From this point on, the subscription worker will start processing batches. +If processing is aborted for any reason, the returned task (`subscription_runtime_task`) +will be finished with an exception. + +{PANEL/} + + +{PANEL:Error handling} +There are two categories of errors that may occur during subscription processing: + +{INFO:Internal mechanism errors} +Internal mechanism errors occur during the normal server-client communication between the worker and the server. +If an unexpected error occurs, the worker will try to reconnect to the server. +There are conditions in which the worker will cease its operation and will not try to reconnect: + +* The subscription does not exist or was deleted +* Another worker took over the subscription (see connection strategy) +* The worker could not connect to any of the servers +* The worker could not receive the node in charge of the task (this can happen when there is no leader) +* Authorization exception +* Exception during connection establishment + +{INFO/} + +{INFO:User's batch processing logic unhandled exception} +Example: +{CODE:python throw_during_user_logic@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +If an exception is thrown, the worker will abort the current batch process. +A worker can be configured to treat the thrown exception in either of the following two ways: + +* By default, the worker will wrap the thrown exception with a `SubscriberErrorException` exception + and rethrow it, terminating the subscription execution without acknowledging progress or retrying. + The task returned by the `run` function will be terminated with an erroneous state, throwing + a `SubscriberErrorException` exception. + +* If `SubscriptionWorkerOptions`'s value `ignore_subscriber_errors` is set to True, the erroneous + batch will get acknowledged without retrying and the next batches will continue processing. +{INFO/} + +{INFO: Reconnecting} +The above cases describe situations in which a worker tries to reconnect with the server. +There are two key `SubscriptionWorkerOptions` fields controlling this state: + +* `time_to_wait_before_connection_retry` - Worker 'sleep' period before trying to reconnect. +* `max_erroneous_period` - Maximum time that the worker is allowed to be in an erroneous state. + When this time elapses, the worker will stop trying to reconnect. +{INFO/} + +{INFO: `on_unexpected_subscription_error`} +`on_unexpected_subscription_error` is the event raised when a connection failure occurs +between the subscription worker and the server and it throws an unexpected exception. +When this occurs, the worker will automatically try to reconnect again. This event is +useful for logging these unexpected exceptions. +{INFO/} + +{PANEL/} + +{PANEL: Worker interplay} + +* Subscription workers are configured with a **strategy**, that determines whether + they can connect their subscription **concurrently**, or only one at a time. +* If a one-at-a-time strategy is chosen, it also determines how the workers + interact with each other to resolve which will connect the subscription. + +{NOTE: } + +The strategy is configured by the `SubscriptionWorkerOptions` `strategy` field. +The strategy field is the `SubscriptionOpeningStrategy` enum. + +| `SubscriptionOpeningStrategy` | | +| ------------- | ------------- | +| `OPEN_IF_FREE` | Connect if no other worker is connected | +| `WAIT_FOR_FREE` | Wait for currently connected worker to disconnect | +| `TAKE_OVER` | Take over the connection | +| `CONCURRENT` | Connect concurrently | + +{NOTE/} + +## Available Worker Strategies + +--- + +### One Worker Per Subscription Strategies + +The following three strategies allow only a **single worker to connect +the subscription at any given time**, and determine what happens when one +worker is connected and another tries to connect. + +* `SubscriptionOpeningStrategy.OPEN_IF_FREE` + The server will allow the worker to connect only if there is currently + no other connected worker. + If there is an existing connection, the incoming worker will throw a + `SubscriptionInUseException`. +* `SubscriptionOpeningStrategy.WAIT_FOR_FREE` + If the worker currently cannot open the subscription because it is used + by another worker, it will wait for the previous worker to disconnect and + connect only then. + This is useful in worker failover scenarios, where there is one active + worker and another already waits to take its place. +* `SubscriptionOpeningStrategy.TAKE_OVER` + The server will allow an incoming connection to overthrow an existing one, + according to the existing connection strategy. + * If the existing connection **does not** have a `TAKE_OVER` strategy: + The incoming connection will take over, causing the existing + connection to throw a `SubscriptionInUseException`. + * If the existing connection **has** a `TAKE_OVER` strategy: + The incoming connection will throw a `SubscriptionInUseException` exception. + +--- + +### Concurrent Strategy + +* `SubscriptionOpeningStrategy.CONCURRENT` + Multiple workers of the same subscription are allowed to connect it simultaneously. + Read more about concurrent subscriptions [here](../../../client-api/data-subscriptions/concurrent-subscriptions). + +{PANEL/} + +{PANEL: Determining Which Workers a Subscription Will Serve} + +{NOTE: } +The **strategy used by the first worker connecting to a subscription** will determine +which additional workers this subscription will be able to serve until all worker connections +are dropped. +{NOTE/} + +* A subscription that serves one or more [CONCURRENT](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#concurrent-strategy) + workers, **can serve only other concurrent workers** until all worker connections are dropped. + If a worker that uses a [One Worker Per Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#one-worker-per-subscription-strategies) + strategy attempts to connect it - + * The connection attempt will be rejected. + * `SubscriptionInUseException` will be thrown. + +* A subscription that serves a worker that uses a + [One Worker Per Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#one-worker-per-subscription-strategies) + strategy, **cannot** serve + [CONCURRENT](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription#concurrent-strategy) + workers until the worker's connection is dropped. + If a concurrent worker attempts to connect it - + * The connection attempt will be rejected. + * `SubscriptionInUseException` will be thrown. + +{PANEL/} + +## Related Articles + +### Data Subscriptions + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.dotnet.markdown new file mode 100644 index 0000000000..c53a2ce15a --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.dotnet.markdown @@ -0,0 +1,170 @@ +# Data Subscriptions: Creation and Update API Overview + +--- + +{NOTE: } + +* In this page: + * [Subscription Creation](../../../client-api/data-subscriptions/creation/api-overview#subscription-creation) + * [SubscriptionCreationOptions](../../../client-api/data-subscriptions/creation/api-overview#subscriptioncreationoptions) + * [SubscriptionCreationOptions<T>](../../../client-api/data-subscriptions/creation/api-overview#subscriptioncreationoptions) + * [Update Subscription](../../../client-api/data-subscriptions/creation/api-overview#update-subscription) + * [SubscriptionUpdateOptions](../../../client-api/data-subscriptions/creation/api-overview#subscriptionupdateoptions) + * [Subscription query](../../../client-api/data-subscriptions/creation/api-overview#subscription-query) + +{NOTE/} + +--- + +{PANEL:Subscription Creation} + +Subscription creation is accessible through `DocumentStore`'s `Subscriptions` Property, of type `DocumentSubscriptions`: +{CODE subscriptionCreationOverloads@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + + | Parameter | Type | Description | + | ------------- | ------------- | ----- | +| **predicate** | `Expression>` | Predicate that returns a boolean, describing filter of the subscription documents | +| **options** | `SubscriptionCreationOptions` | Contains subscription creation options | +| **database** | `string` | Name of database to create a data subscription. If `null`, default database configured in DocumentStore will be used. | +| **token** | `CancellationToken` | Cancellation token used in order to halt the subscription creation process. | + +| Return value | Description | +| ------------- | ----- | +| `string` | Created data subscription name. If Name was provided in SubscriptionCreationOptions, it will be returned, otherwise, a unique name will be generated by server. | + +{PANEL/} + +{PANEL:SubscriptionCreationOptions} + +Non generic version of the class, relies on user's full knowledge of the RQL query structure + +| Member | Type | Description | +|--------|:-----|-------------| +| **Name** | `string` | User defined name of the subscription: allows to have a human readable identification of a subscription. The name must be unique in the database. | +| **Query** | `string` | **Required.** RQL query that describes the subscription. That RQL comes with additional support to JavaScript clause inside the 'Where' statement and special semantics for subscriptions on documents revisions. | +| **ChangeVector** | `string` | Allows to define a change vector, from which the subscription will start processing. It might be useful for ad-hoc processes that need to process only recent changes in data, for that specific use, the field may receive a special value: "LastDocument", that will take the latest change vector in the machine. | +| **MentorNode** | `string` | Allows to define a specific node in the cluster that we want to treat the subscription. That's useful in cases when one server is preffered over other, either because of stronger hardware or closer geographic proximity to clients etc. | + +{PANEL/} + +{PANEL:SubscriptionCreationOptions<T>} + +An RQL statement will be built based on the fields. + +{CODE sub_create_options_strong@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **<T>** | type | Type of the object, from which the collection will be derived. | +| **Name** | `string` | User defined name of the subscription: allows to have a human readable identification of a subscription. The name must be unique in the database. | +| **Filter** | `Expression>` | Lambda describing filter logic for the subscription. Will be translated to a JavaScript function. | +| **Projection** | `Expression>` | Lambda describing the projection of returned documents. Will be translated to a JavaScript function. | +| **Includes** | `Action>` | Action with an [ISubscriptionIncludeBuilder](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-include-statement) parameter that allows you to define an include clause for the subscription. Methods can be chained to include documents as well as [counters](../../../client-api/data-subscriptions/creation/examples#including-counters). | +| **ChangeVector** | `string` | Allows to define a change vector, from which the subscription will start processing. It might be useful for ad-hoc processes that need to process only recent changes in data, for that specific use, the field may receive a special value: "LastDocument", that will take the latest change vector in the machine. | +| **MentorNode** | `string` | Allows to define a specific node in the cluster that we want to treat the subscription. That's useful in cases when one server is preffered over other, either because of stronger hardware or closer geographic proximity to clients etc. | + +{PANEL/} + +{PANEL: Update Subscription} + +Modifies an existing data subscription. These methods are accessible at `DocumentStore.Subscriptions`. + +{CODE updating_subscription@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +| Parameter | Type | Description | +| - | - | - | +| **options** | `SubscriptionUpdateOptions` | A subscription update options object | +| **database** | `string` | Name of database to create a data subscription. If `null`, default database configured in DocumentStore will be used. | +| **token** | `CancellationToken` | Cancellation token used in order to halt the updating process. | + +| Return value | Description | +| ------------- | ----- | +| `string` | The updated data subscription's name. | + +{PANEL/} + +{PANEL: SubscriptionUpdateOptions} + +Inherits from `SubscriptionCreationOptions` and has all the same fields (see [above](../../../client-api/data-subscriptions/creation/api-overview#subscriptioncreationoptions)) plus the two additional fields described below: + +{CODE sub_update_options@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +| Parameter | Type | Description | +| - | - | - | +| **Id** | `long?` | Unique server-side ID of the data subscription, see description of the [subscription state object](../../../client-api/data-subscriptions/advanced-topics/maintenance-operations#getting-subscription-status). `Id` can be used instead of the subscription update options `Name` field, and takes precedence over it. This allows you to change the subscription's name: submit a subscription's ID, and submit a different name in the `Name` field. | +| **CreateNew** | `bool` | If set to `true`, and the specified subscription does not exist, the subscription is created. If set to `false`, and the specified subscription does not exist, an exception is thrown. | + +{PANEL/} + +{PANEL: Subscription query} + +All subscriptions, are eventually translated to an RQL-like statement. These statements has four parts: + +* Functions definition part, like in ordinary RQL. Those functions can contain any JavaScript code, + and also supports `load` and `include` operations. + +* From statement, defining the documents source, ex: `from Orders`. The from statement can only address collections, therefore, indexes are not supported. + +* Where statement describing the criteria according to which it will be decided to either +send the documents to the worker or not. Those statements supports either RQL like `equality` operations (`=`, `==`) , +plain JavaScript expressions or declared function calls, allowing to perform complex filtering logic. +The subscriptions RQL does not support any of the known RQL searching keywords. + +* Select statement, that defines the projection to be performed. +The select statements can contain function calls, allowing complex transformations. + +* Include statement allowing to define include path in document. + +{INFO: Keywords} +Although subscription's query syntax has an RQL-like structure, it supports only the `declare`, `select` and `where` keywords, usage of all other RQL keywords is not supported. +Usage of JavaScript ES5 syntax is supported. +{INFO/} + +{INFO: Paths} +Paths in subscriptions RQL statements are treated as JavaScript indirections and not like regular RQL paths. +It means that a query that in RQL would look like: + +``` +from Orders as o +where o.Lines[].Product = "products/1-A" +``` + +Will look like that in subscriptions RQL: + +``` +declare function filterLines(doc, productId) +{ + if (!!doc.Lines){ + return doc.Lines.filter(x=>x.Product == productId).length >0; + } + return false; +} + +from Orders as o +where filterLines(o, "products/1-A") +``` +{INFO/} + +{INFO: Revisions} +In order to define a data subscription that uses documents revisions, there first has to be revisions configured for the specific collection. +The subscription should be defined in a special way: +* In case of the generic API, the `SubscriptionCreationOptions<>` generic parameter should be of the generic type `Revision<>`, while it's generic parameter correlates to the collection to be processed. Ex: `new SubscriptionCreationOptions>()` +* For RQL syntax, the `(Revisions = true)` statement should be concatenated to the collection to be queries. Ex: `From orders(Revisions = true) as o` + +{INFO/} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) + + diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.python.markdown new file mode 100644 index 0000000000..01306bbb97 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/api-overview.python.markdown @@ -0,0 +1,143 @@ +# Data Subscriptions: Creation and Update API Overview + +--- + +{NOTE: } + +* In this page: + * [Subscription Creation](../../../client-api/data-subscriptions/creation/api-overview#subscription-creation) + * [SubscriptionCreationOptions](../../../client-api/data-subscriptions/creation/api-overview#subscriptioncreationoptions) + * [Update Subscription](../../../client-api/data-subscriptions/creation/api-overview#update-subscription) + * [SubscriptionUpdateOptions](../../../client-api/data-subscriptions/creation/api-overview#subscriptionupdateoptions) + * [Subscription query](../../../client-api/data-subscriptions/creation/api-overview#subscription-query) + +{NOTE/} + +--- + +{PANEL:Subscription Creation} + +Subscriptions can be created using the `create_for_options` and `create_for_class` methods. +{CODE:python subscriptionCreationOverloads@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Parameter | Type | Description | +| ------------- | ------------- | ----- | +| **options** | `SubscriptionCreationOptions` | Contains subscription creation options | +| **database** (Optional) | `[str]` | Name of database to create a data subscription. If `None`, default database configured in DocumentStore will be used. | +| **object_type** | `Type[_T]` | Predicate describing the subscription documents filter | + +| Return value | Description | +| ------------- | ----- | +| `str` | Created data subscription name. If the name was provided in `SubscriptionCreationOptions`, it will be returned. Otherwise, a unique name will be generated by the server. | + +{PANEL/} + +{PANEL:SubscriptionCreationOptions} + +An RQL statement will be built based on the fields. +{CODE:python sub_create_options@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Member | Type | Description | +|--------|:-----|-------------| +| **name** (Optional) | `str` | User-defined name of the subscription: allows to have a human readable identification of a subscription. The name must be unique in the database. | +| **query** (Optional) | `str` | RQL query that describes the subscription. This RQL comes with additional support to JavaScript clause inside the `where` statement and special semantics for subscriptions on documents revisions. | +| **change_vector** (Optional) | `str` | Allows to define a change vector, from which the subscription will start processing. It might be useful for ad-hoc processes that need to process only recent changes in data, for that specific use, the field may receive a special value: "LastDocument", that will take the latest change vector in the machine. | +| **mentor_node** (Optional) | `str` | Allows to define a specific node in the cluster that we want to treat the subscription. That's useful in cases when one server is preffered over other, either because of stronger hardware or closer geographic proximity to clients etc. | +| **includes** (Optional) | `[Callable[[SubscriptionIncludeBuilder]` | Action with a [SubscriptionIncludeBuilder](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-include-statement) parameter that allows you to define an include clause for the subscription. Methods can be chained to include documents as well as [counters](../../../client-api/data-subscriptions/creation/examples#including-counters). | + +{PANEL/} + +{PANEL: Update Subscription} + +Modifies an existing data subscription. These methods are accessible at `DocumentStore.Subscriptions`. + +{CODE:python updating_subscription@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Parameter | Type | Description | +| - | - | - | +| **options** | `SubscriptionUpdateOptions` | A subscription update options object | +| **database** (Optional) | `str` | Name of database to create a data subscription. If `None`, default database configured in DocumentStore will be used. | + +| Return value | Description | +| ------------- | ----- | +| `str` | The updated data subscription's name. | + +{PANEL/} + +{PANEL: SubscriptionUpdateOptions} + +Inherits from `SubscriptionCreationOptions` and has all the same fields (see [above](../../../client-api/data-subscriptions/creation/api-overview#subscriptioncreationoptions)) plus the two additional fields described below: + +{CODE:python sub_update_options@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +| Parameter | Type | Description | +| - | - | - | +| **key** (Optional) | `int` | Unique server-side ID of the data subscription. `key` can be used instead of the subscription update options `name` field, and takes precedence over it. This allows you to change the subscription's name: submit a subscription's ID, and submit a different name in the `name` field. | +| **create_new** (Optional) | `bool` | If set to `True`, and the specified subscription does not exist, the subscription is created. If set to `False`, and the specified subscription does not exist, an exception is thrown. | + +{PANEL/} + +{PANEL: Subscription query} + +All subscriptions, are eventually translated to an RQL-like statement. These statements has four parts: + +* Functions definition part, like in ordinary RQL. Those functions can contain any JavaScript code, + and also supports `load` and `include` operations. + +* From statement, defining the documents source, ex: `from Orders`. The from statement can only address collections, therefore, indexes are not supported. + +* Where statement describing the criteria according to which it will be decided to either +send the documents to the worker or not. Those statements supports either RQL like `equality` operations (`=`, `==`) , +plain JavaScript expressions or declared function calls, allowing to perform complex filtering logic. +The subscriptions RQL does not support any of the known RQL searching keywords. + +* Select statement, that defines the projection to be performed. +The select statements can contain function calls, allowing complex transformations. + +* Include statement allowing to define include path in document. + +{INFO: Keywords} +Although subscription's query syntax has an RQL-like structure, it supports only the `declare`, `select` and `where` keywords, usage of all other RQL keywords is not supported. +Usage of JavaScript ES5 syntax is supported. +{INFO/} + +{INFO: Paths} +Paths in subscriptions RQL statements are treated as JavaScript indirections and not like regular RQL paths. +It means that a query that in RQL would look like: + +``` +from Orders as o +where o.Lines[].Product = "products/1-A" +``` + +Will look like that in subscriptions RQL: + +``` +declare function filterLines(doc, productId) +{ + if (!!doc.Lines){ + return doc.Lines.filter(x=>x.Product == productId).length >0; + } + return false; +} + +from Orders as o +where filterLines(o, "products/1-A") +``` +{INFO/} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) + + diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.dotnet.markdown new file mode 100644 index 0000000000..1932d649de --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.dotnet.markdown @@ -0,0 +1,145 @@ +# Data Subscriptions: Common Data Subscription Creation Examples + +--- + +{NOTE: } + +* In this page: + * [Create subscription on all documents in a collection](../../../client-api/data-subscriptions/creation/examples#create-subscription-on-all-documents-in-a-collection) + * [Create subscription with filtering](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-filtering) + * [Create subscription with filtering and projection](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-filtering-and-projection) + * [Create subscription with load document in filter projection](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-load-document-in-filter-projection) + * [Create subscription with include statement](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-include-statement) + * [Including counters](../../../client-api/data-subscriptions/creation/examples#including-counters) + * [Create revisions enabled subscription](../../../client-api/data-subscriptions/creation/examples#create-revisions-enabled-subscription) + * [Update existing subscription](../../../client-api/data-subscriptions/creation/examples#update-existing-subscription) +{NOTE/} + +--- + +{PANEL:Create subscription on all documents in a collection} + +Here we create a plain subscription on the Orders collection, without any constraint or transformation. +{CODE-TABS} +{CODE-TAB:csharp:Generic-syntax create_whole_collection_generic_with_name@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-syntax create_whole_collection_RQL@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Create subscription with filtering} + +Here we create a subscription on Orders collection, which total order revenue is greater than 100. +{CODE-TABS} +{CODE-TAB:csharp:Generic-syntax create_filter_only_generic@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-syntax create_filter_only_RQL@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Create subscription with filtering and projection} + +Here we create a subscription on Orders collection, which total order revenue is greater than 100, and return only ID and total revenue. +{CODE-TABS} +{CODE-TAB:csharp:Generic-syntax create_filter_and_projection_generic@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-syntax create_filter_and_projection_RQL@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Create subscription with load document in filter projection} + +Here we create a subscription on Orders collection, which total order revenue is greater than 100, and return ID, total revenue, shipping address and responsible employee name. +{CODE-TABS} +{CODE-TAB:csharp:Generic-syntax create_filter_and_load_document_generic@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-syntax create_filter_and_load_document_RQL@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Create subscription with include statement} + +Here we create a subscription on the collection Orders, which returns the orders and brings along all products mentioned in the order as included documents. +See the usage example [here](../../../client-api/data-subscriptions/consumption/examples#subscription-that-uses-included-documents). + +Include statements can be added to a subscription in the raw RQL, or with the **`ISubscriptionIncludeBuilder`**. + +The subscription include builder is assigned to the option **Includes** in `SubscriptionCreationOptions` +(see [subscription API overview](../../../client-api/data-subscriptions/creation/api-overview)). It +supports methods for including documents as well as counters. These methods can be chained. + +In raw RQL, include statements come in two forms, like in any other RQL statements: +1. Include statement in the end of the query, starting with the `include` keyword, followed by paths to the field containing the IDs of the documents to include. +If projection is performed, the mechanism will look for the paths in the projected result, rather then the original document. +It is recommended to prefer this approach when possible both because of clarity of the query and slightly better performance. +2. Include function call inside a 'declared' function. + +{CODE-TABS} +{CODE-TAB:csharp:Builder-syntax create_subscription_with_includes_strongly_typed@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-path-syntax create_subscription_with_includes_rql_path@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-javascript-syntax create_subscription_with_includes_rql_javascript@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +--- + +#### Including Counters + +`ISubscriptionIncludeBuilder` has three methods for including counters: + +{CODE:csharp include_builder_counter_methods@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +`IncludeCounter` is used to specify a single counter, and `IncludeCounters` for multiple counters. `IncludeAllCounters` +retrieves all counters from all subscribed documents. + +| Parameters | Type | Description | +| - | - | - | +| **name** | `string` | The name of a counter. The subscription will include all counters with this name that are contained in the documents the subscription retrieves. | +| **names** | `string[]` | Array of counter names. | + +In this example, we create a subscription that uses all three methods to include counters. This demonstrates +how the methods can be chained (needless to say, calling `IncludeAllCounters()` makes the other two methods +redundant). + +{CODE:csharp create_subscription_include_counters_builder@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + + +{PANEL:Create revisions enabled subscription} + +Here we create a subscription on Orders collection, which returns current and previous version of the subscriptions. +Please see the [page](../../../client-api/data-subscriptions/advanced-topics/subscription-with-revisioning) dedicated to subscriptions with revisions for more details and examples. + +{CODE-TABS} +{CODE-TAB:csharp:Generic-syntax create_simple_revisions_subscription_generic@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TAB:csharp:RQL-syntax create_simple_revisions_subscription_RQL@ClientApi\DataSubscriptions\DataSubscriptions.cs /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Update existing subscription} + +Here we update the filter query of an existing data subscription named "my subscription". + +{CODE:csharp update_subscription_example_0@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +In addition to names, subscriptions also have a **subscription ID** on the server side. The +ID can be used to identify the subscription instead of using its name. This allows use to change +an existing subscription's name by specifying the subscription with the ID, and submitting +a new string in the `Name` field of the `SubscriptionUpdateOptions`. + +{CODE:csharp update_subscription_example_1@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.python.markdown new file mode 100644 index 0000000000..dd0f03e58d --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/examples.python.markdown @@ -0,0 +1,127 @@ +# Data Subscriptions: Common Data Subscription Creation Examples + +--- + +{NOTE: } + +* In this page: + * [Create subscription on all documents in a collection](../../../client-api/data-subscriptions/creation/examples#create-subscription-on-all-documents-in-a-collection) + * [Create subscription with filtering](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-filtering) + * [Create subscription with filtering and projection](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-filtering-and-projection) + * [Create subscription with load document in filter projection](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-load-document-in-filter-projection) + * [Create subscription with include statement](../../../client-api/data-subscriptions/creation/examples#create-subscription-with-include-statement) + * [Including counters](../../../client-api/data-subscriptions/creation/examples#including-counters) + * [Update existing subscription](../../../client-api/data-subscriptions/creation/examples#update-existing-subscription) +{NOTE/} + +--- + +{PANEL:Create subscription on all documents in a collection} + +Here we create a plain subscription on the Orders collection, without any constraint or transformation. +{CODE-TABS} +{CODE-TAB:python:Generic-syntax create_whole_collection_generic_with_name@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE-TAB:python:RQL-syntax create_whole_collection_RQL@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE-TABS/} + +{PANEL/} + +{PANEL:Create subscription with filtering} + +Here we create a subscription on the Orders collection, for total order revenues greater than 100. +{CODE:python create_filter_only_RQL@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Create subscription with filtering and projection} + +Here we create a subscription on the Orders collection, for total order revenues greater than 100, +and return only the ID and total revenue. +{CODE:python create_filter_and_projection_RQL@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Create subscription with load document in filter projection} + +Here we create a subscription on the Orders collection, for total order revenue greater than 100, +and return the ID, total revenue, shipping address and responsible employee name. +{CODE:python create_filter_and_load_document_RQL@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +{PANEL:Create subscription with include statement} + +Here we create a subscription on the Orders collection, which returns the orders and brings along +all products mentioned in the order as included documents. + +{CODE-TABS} +{CODE-TAB:python:Builder-syntax create_subscription_with_includes_strongly_typed@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE-TAB:python:RQL-path-syntax create_subscription_with_includes_rql_path@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE-TAB:python:RQL-javascript-syntax create_subscription_with_includes_rql_javascript@ClientApi\DataSubscriptions\DataSubscriptions.py /} +{CODE-TABS/} + +Include statements can be added to a subscription in the raw RQL, or using `SubscriptionIncludeBuilder`. +The subscription include builder is assigned to the `includes` option of `SubscriptionCreationOptions` +(see [subscription API overview](../../../client-api/data-subscriptions/creation/api-overview)). +It supports methods for including documents as well as counters. These methods can be chained. + +In raw RQL, include statements come in two forms, like in any other RQL statements: + +1. Include statement in the end of the query, starting with the `include` keyword, followed by paths to the + field containing the IDs of the documents to include. + If projection is performed, the mechanism will look for the paths in the projected result, rather then the + original document. + It is recommended to prefer this approach when possible both because of clarity of the query and slightly + better performance. +2. Include function call inside a 'declared' function. + +## Including Counters + +`SubscriptionIncludeBuilder` has three methods for including counters: + +{CODE:python include_builder_counter_methods@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +`include_counter` is used to specify a single counter. +`include_counters` is used to specify multiple counters. +`include_all_counters` retrieves all counters from all subscribed documents. + +| Parameters | Type | Description | +| - | - | - | +| **name** | `str` | The name of a counter. The subscription will include all counters with this name that are contained in the documents the subscription retrieves. | +| **\*names** | `str` | Array of counter names. | + +In the below example we create a subscription that uses all three methods to include counters, demonstrating +how the methods can be chained (stating the obvious, calling `include_all_counters` makes the other two methods +redundant). + +{CODE:python create_subscription_include_counters_builder@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + + +{PANEL:Update existing subscription} + +Here we update the filter query of an existing data subscription named "my subscription". + +{CODE:python update_subscription_example_0@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +In addition to names, subscriptions also have a **subscription ID** on the server side. The +ID can be used to identify the subscription instead of using its name. This allows use to change +an existing subscription's name by specifying the subscription with the ID, and submitting +a new string in the `name` field of the `SubscriptionUpdateOptions`. + +{CODE:python update_subscription_example_1@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Create a Data Subscription](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.dotnet.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.dotnet.markdown new file mode 100644 index 0000000000..4244373652 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.dotnet.markdown @@ -0,0 +1,74 @@ +# Data Subscriptions: How to Create a Data Subscription + +--- + +{NOTE: } + +Subscription tasks are created by performing a request to the server with certain subscription creations parameters, see [Creation API summary](../../../client-api/data-subscriptions/creation/api-overview#subscription-creation). +Once created, its definition and progress will be stored on the cluster, and not in a single server. +Upon subscription creation, the cluster will choose a preferred node that will run the subscription (unless the client has stated a responsible node). +From that point and on, clients that will connect to a server in order to consume the subscription will be redirected to the node mentioned above. + +* In this page: + * [Subscription creation prerequisites](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-creation-prerequisites) + * [Subscription name](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-name) + * [Responsible node](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#responsible-node) + +{NOTE/} + +--- + +{PANEL:Subscription creation prerequisites} + +Data subscription is a batch processing mechanism that sends documents that answer specific criteria to connected clients. +In order to create a data subscription, we first need to define the criteria. The minimum is to provide the collection to which the data subscription belongs. +However, the criteria can be a complex RQL-like expression defining JavaScript functions for the filtering and the projections. See a simple example: + +{CODE create_whole_collection_generic1@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +Fore more complex usage examples see [examples](../../../client-api/data-subscriptions/creation/examples) + +{PANEL/} + +{PANEL:Subscription name} + +Another important, but not mandatory subscription property is its name: + +{CODE create_whole_collection_generic_with_name@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +In order to consume a data subscription, a subscription name is required to identify it. +By default, the server can generate a subscription name on its own, but it's also possible to pass a custom name. +A dedicated name can be useful for use cases like dedicated, long-running batch processing mechanisms, where it'll be more comfortable to use a human-readable +name in the code and even use the same name between different environments (as long as subscription creation is taken care of upfront). + +{INFO:Uniqueness} +Note that the subscription name is unique and it will not be possible to create two subscriptions with the same name in the same database. +{INFO/} + +{PANEL/} + +{PANEL:Responsible node} + +As stated above, upon creation, the cluster will choose a node that will be responsible for the data subscription server-side processing. +Once chosen, that node will be the only node to process the subscription. There is an enterprise license level feature +that supports subscription (and any other ongoing task) +failover between nodes, but eventually, as long as the originally assigned node is online, it will be the one to process the data subscription. +Nevertheless, there is an option to manually decide the node that will be responsible for the subscription processing, it's called the `MentorNode`: + +{CODE create_whole_collection_generic_with_mentor_node@ClientApi\DataSubscriptions\DataSubscriptions.cs /} + +The mentor node receives the responsible node tag, as can be seen in the subscription topology. +Setting that node manually can help manually choosing a more fitting server from resources, client proximity, or any other point of view. + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.java.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.java.markdown new file mode 100644 index 0000000000..a49251964a --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.java.markdown @@ -0,0 +1,73 @@ +# Data Subscriptions: How to Create a Data Subscription + +--- + +{NOTE: } + +Subscriptions are created by performing a request to the server with certain subscription creations parameters, see [Creation API summary](api-overview#create-and-createasync-overloads-summary). +Once created, it's definition and progress will be stored on the cluster, and not in a single server. +Upon subscription creation, the cluster will choose a preferred node that will run the subscription (unless client has stated a mentor node). +From that point and on, clients that will connect to a server in order to consume the subscription will be redirected to the node mentioned above. + +* In this page: + * [Subscription creation prerequisites](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-creation-prerequisites) + * [Subscription name](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-name) + * [Responsible node](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#responsible-node) + +{NOTE/} + +--- + +{PANEL:Subscription creation prerequisites} + +Data subscription is a batch processing mechanism, that send the clients documents that answer a specific criteria. +In order to create a data subscription, we first need to define the criteria. The minimum is to provide the collection to which the data subscription belongs. +However, the criteria can be a complex RQL-like expression defining JavaScript functions for the filtering and the projections. See a simple example: + +{CODE:java create_whole_collection_generic1@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +Fore more complex usage examples see [examples](../../../client-api/data-subscriptions/creation/examples) + +{PANEL/} + +{PANEL:Subscription name} + +Another important, but not mandatory subscription property is its name: + +{CODE:java create_whole_collection_generic_with_name@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +In order to consume a data subscription, a subscription name is required to identify it. +By default, the server can generate a subscription name by its own, but it's also possible to pass a custom name. +A dedicated name can be useful for use cases like dedicated, long running batch processing mechanisms, where it'll be more comfortable to use a human readable +name in the code and even use the same name between different environments (as long as subscription creation is taken care of up front). + +{INFO:Uniqueness} +Note that subscription name is unique and it will not be possible to create two subscriptions with the same name in the same database. +{INFO/} + +{PANEL/} + +{PANEL:Responsible node} + +As stated above, upon creation, the cluster will choose a node that will be responsible for the data subscription server side processing. Once chosen, that node +will be the node that will the only node to process the subscription. There is an enterprise license level feature that support subscription (and any other ongoing task) +failover between nodes, but eventually, as long as the originally assigned node is online, it will be the one to process the data subscription. +Nevertheless, there is an option to manually decide the node that will be responsible for the subscription processing, it's called the `MentorNode`: + +{CODE:java create_whole_collection_generic_with_mentor_node@ClientApi\DataSubscriptions\DataSubscriptions.java /} + +The mentor node receives the responsible node tag, as can be seen in the subscription topology. +Setting that node manually can help manually choosing a more fitting server from resources, client proximity or any other point of view. + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.js.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.js.markdown new file mode 100644 index 0000000000..08a89d07c9 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.js.markdown @@ -0,0 +1,73 @@ +# Data Subscriptions: How to Create a Data Subscription + +--- + +{NOTE: } + +Subscriptions are created by performing a request to the server with certain subscription creations parameters, see [Creation API summary](api-overview#create-and-createasync-overloads-summary). +Once created, it's definition and progress will be stored on the cluster, and not in a single server. +Upon subscription creation, the cluster will choose a preferred node that will run the subscription (unless client has stated a mentor node). +From that point and on, clients that will connect to a server in order to consume the subscription will be redirected to the node mentioned above. + +* In this page: + * [Subscription creation prerequisites](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-creation-prerequisites) + * [Subscription name](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-name) + * [Responsible node](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#responsible-node) + +{NOTE/} + +--- + +{PANEL:Subscription creation prerequisites} + +Data subscription is a batch processing mechanism, that send the clients documents that answer a specific criteria. +In order to create a data subscription, we first need to define the criteria. The minimum is to provide the collection to which the data subscription belongs. +However, the criteria can be a complex RQL-like expression defining JavaScript functions for the filtering and the projections. See a simple example (in which collection name is determined based on the entity type): + +{CODE:nodejs create_whole_collection_generic1@client-api\dataSubscriptions\dataSubscriptions.js /} + +Fore more complex usage examples see [examples](../../../client-api/data-subscriptions/creation/examples) + +{PANEL/} + +{PANEL:Subscription name} + +Another important, but not mandatory subscription property is its name: + +{CODE:nodejs create_whole_collection_generic_with_name@client-api\dataSubscriptions\dataSubscriptions.js /} + +In order to consume a data subscription, a subscription name is required to identify it. +By default, the server can generate a subscription name by its own, but it's also possible to pass a custom name. +A dedicated name can be useful for use cases like dedicated, long running batch processing mechanisms, where it'll be more comfortable to use a human readable +name in the code and even use the same name between different environments (as long as subscription creation is taken care of up front). + +{INFO:Uniqueness} +Note that subscription name is unique and it will not be possible to create two subscriptions with the same name in the same database. +{INFO/} + +{PANEL/} + +{PANEL:Responsible node} + +As stated above, upon creation, the cluster will choose a node that will be responsible for the data subscription server side processing. Once chosen, that node +will be the node that will the only node to process the subscription. There is an enterprise license level feature that support subscription (and any other ongoing task) +failover between nodes, but eventually, as long as the originally assigned node is online, it will be the one to process the data subscription. +Nevertheless, there is an option to manually decide the node that will be responsible for the subscription processing, it's called the `MentorNode`: + +{CODE:nodejs create_whole_collection_generic_with_mentor_node@client-api\dataSubscriptions\dataSubscriptions.js /} + +The mentor node receives the responsible node tag, as can be seen in the subscription topology. +Setting that node manually can help manually choosing a more fitting server from resources, client proximity or any other point of view. + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.python.markdown b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.python.markdown new file mode 100644 index 0000000000..5d95c87b01 --- /dev/null +++ b/Documentation/5.4/Raven.Documentation.Pages/client-api/data-subscriptions/creation/how-to-create-data-subscription.python.markdown @@ -0,0 +1,75 @@ +# Data Subscriptions: How to Create a Data Subscription + +--- + +{NOTE: } + +Subscription tasks are created by performing a request to the server with certain subscription creations parameters, +see [Creation API summary](../../../client-api/data-subscriptions/creation/api-overview#subscription-creation). +Once created, its definition and progress will be stored on the cluster, and not in a single server. +Upon subscription creation, the cluster will choose a preferred node that will run the subscription (unless the client has stated a responsible node). +From that point and on, clients that will connect to a server in order to consume the subscription will be redirected to the node mentioned above. + +* In this page: + * [Subscription creation prerequisites](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-creation-prerequisites) + * [Subscription name](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#subscription-name) + * [Responsible node](../../../client-api/data-subscriptions/creation/how-to-create-data-subscription#responsible-node) + +{NOTE/} + +--- + +{PANEL:Subscription creation prerequisites} + +Data subscription is a batch processing mechanism that sends documents that answer specific criteria to connected clients. +In order to create a data subscription, we first need to define the criteria. The minimum is to provide the collection to which the data subscription belongs. +However, the criteria can be a complex RQL-like expression defining JavaScript functions for the filtering and the projections. See a simple example: + +{CODE:python create_whole_collection_generic1@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +Fore more complex usage examples see [examples](../../../client-api/data-subscriptions/creation/examples) + +{PANEL/} + +{PANEL:Subscription name} + +Another important, but not mandatory subscription property is its name: + +{CODE:python create_whole_collection_generic_with_name@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +In order to consume a data subscription, a subscription name is required to identify it. +By default, the server can generate a subscription name on its own, but it's also possible to pass a custom name. +A dedicated name can be useful for use cases like dedicated, long-running batch processing mechanisms, where it'll be more comfortable to use a human-readable +name in the code and even use the same name between different environments (as long as subscription creation is taken care of upfront). + +{INFO:Uniqueness} +Note that the subscription name is unique and it will not be possible to create two subscriptions with the same name in the same database. +{INFO/} + +{PANEL/} + +{PANEL:Responsible node} + +As stated above, upon creation, the cluster will choose a node that will be responsible for the data subscription server-side processing. +Once chosen, that node will be the only node to process the subscription. There is an enterprise license level feature +that supports subscription (and any other ongoing task) +failover between nodes, but eventually, as long as the originally assigned node is online, it will be the one to process the data subscription. +Nevertheless, there is an option to manually decide the node that will be responsible for the subscription processing, it's called the `mentor_node`: + +{CODE:python create_whole_collection_generic_with_mentor_node@ClientApi\DataSubscriptions\DataSubscriptions.py /} + +The mentor node receives the responsible node tag, as can be seen in the subscription topology. +Setting that node manually can help manually choosing a more fitting server from resources, client proximity, or any other point of view. + +{PANEL/} + +## Related Articles + +**Data Subscriptions**: + +- [What are Data Subscriptions](../../../client-api/data-subscriptions/what-are-data-subscriptions) +- [How to Consume a Data Subscription](../../../client-api/data-subscriptions/consumption/how-to-consume-data-subscription) + +**Knowledge Base**: + +- [JavaScript Engine](../../../server/kb/javascript-engine) diff --git a/Documentation/5.4/Samples/csharp/Raven.Documentation.Samples/ClientApi/DataSubscriptions/DataSubscriptions.cs b/Documentation/5.4/Samples/csharp/Raven.Documentation.Samples/ClientApi/DataSubscriptions/DataSubscriptions.cs new file mode 100644 index 0000000000..755d138e15 --- /dev/null +++ b/Documentation/5.4/Samples/csharp/Raven.Documentation.Samples/ClientApi/DataSubscriptions/DataSubscriptions.cs @@ -0,0 +1,1058 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Linq.Expressions; +using System.Threading; +using System.Threading.Tasks; +using Raven.Client.Documents; +using Raven.Client.Documents.Queries; +using Raven.Client.Documents.Session.Loaders; +using Raven.Client.Documents.Subscriptions; +using Raven.Client.Exceptions.Database; +using Raven.Client.Exceptions.Documents.Subscriptions; +using Raven.Client.Exceptions.Security; +using Raven.Documentation.Samples.Orders; +using Sparrow.Json; +using Sparrow.Logging; +using Xunit; + +namespace Raven.Documentation.Samples.ClientApi.DataSubscriptions +{ + public class DataSubscriptions + { + private interface ISubscriptionCreationOveloads + { + #region subscriptionCreationOverloads + string Create(SubscriptionCreationOptions options, string database = null); + string Create(Expression> predicate = null, + SubscriptionCreationOptions options = null, string database = null); + string Create(SubscriptionCreationOptions options, string database = null); + + Task CreateAsync(SubscriptionCreationOptions options, + string database = null, + CancellationToken token = default); + + Task CreateAsync(Expression> predicate = null, + SubscriptionCreationOptions options = null, + string database = null, + CancellationToken token = default); + + Task CreateAsync(SubscriptionCreationOptions options, + string database = null, + CancellationToken token = default); + #endregion + } + + private interface IUpdatingSubscription + { + #region updating_subscription + string Update(SubscriptionUpdateOptions options, string database = null); + + Task UpdateAsync(SubscriptionUpdateOptions options, string database = null, + CancellationToken token = default); + #endregion + } + + private interface ISubscriptionCreationOptions + { + #region sub_create_options + public class SubscriptionCreationOptions + { + public string Name { get; set; } + public string Query { get; set; } + public string ChangeVector { get; set; } + public string MentorNode { get; set; } + } + #endregion + + #region sub_create_options_strong + public class SubscriptionCreationOptions + { + public string Name { get; set; } + public Expression> Filter { get; set; } + public Expression> Projection { get; set; } + public Action> Includes { get; set; } + public string ChangeVector { get; set; } + public string MentorNode { get; set; } + } + #endregion + + #region sub_update_options + public class SubscriptionUpdateOptions : SubscriptionCreationOptions + { + public long? Id { get; set; } + public bool CreateNew { get; set; } + } + #endregion + } + + public interface ISubscriptionConsumptionOverloads + { + #region subscriptionWorkerGeneration + SubscriptionWorker GetSubscriptionWorker(string subscriptionName, string database = null); + SubscriptionWorker GetSubscriptionWorker(SubscriptionWorkerOptions options, string database = null); + SubscriptionWorker GetSubscriptionWorker(string subscriptionName, string database = null) where T : class; + SubscriptionWorker GetSubscriptionWorker(SubscriptionWorkerOptions options, string database = null) where T : class; + #endregion + } + + public interface ISubscriptionWorkerRunning + { + #region subscriptionWorkerRunning + Task Run(Action> processDocuments, CancellationToken ct = default(CancellationToken)); + Task Run(Func, Task> processDocuments, CancellationToken ct = default(CancellationToken)); + #endregion + + } + + #region subscriptions_example + public async Task Worker(IDocumentStore store, CancellationToken cancellationToken) + { + string subscriptionName = await store.Subscriptions.CreateAsync(x => x.Company == "companies/11"); + SubscriptionWorker subscription = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + Task subscriptionTask = subscription.Run(x => + x.Items.ForEach(item => + Console.WriteLine($"Order #{item.Result.Id} will be shipped via: {item.Result.ShipVia}")), + cancellationToken); + + await subscriptionTask; + } + #endregion + + + interface FakeDocumentSubscriptions + { + + } + interface FakeStore + { + + } + + #region creation_api + + #endregion + + + + [Fact] + public async Task CreationExamples() + { + string name; + IDocumentStore store = new DocumentStore(); + + #region create_whole_collection_generic_with_name + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions + { + Name = "OrdersProcessingSumbscription" + }); + + #endregion + + + #region create_whole_collection_generic_with_mentor_node + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions + { + MentorNode = "D" + }); + + #endregion + + + #region create_whole_collection_generic1 + + name = await store.Subscriptions.CreateAsync(); + + #endregion + + #region create_whole_collection_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = "From Orders" + }); + + #endregion + + #region create_filter_only_generic + + name = await store.Subscriptions.CreateAsync(x => + x.Lines.Sum(line => line.PricePerUnit * line.Quantity) > 100); + + #endregion + + #region create_filter_only_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @" + declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + From Orders as o + Where getOrderLinesSum(o) > 100" + }); + + #endregion + + #region create_filter_and_projection_generic + + name = store.Subscriptions.Create( + new SubscriptionCreationOptions() + { + Filter = x => x.Lines.Sum(line => line.PricePerUnit * line.Quantity) > 100, + Projection = x => new + { + Id = x.Id, + Total = x.Lines.Sum(line => line.PricePerUnit * line.Quantity) + } + }); + + #endregion + + #region create_filter_and_projection_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + + declare function projectOrder(doc){ + return { + Id: order.Id, + Total: getOrderLinesSum(order) + }; + } + + From Orders as o + Where getOrderLinesSum(o) > 100 + Select projectOrder(o)" + }); + + #endregion + + #region create_filter_and_load_document_generic + + name = store.Subscriptions.Create( + new SubscriptionCreationOptions() + { + Filter = x => x.Lines.Sum(line => line.PricePerUnit * line.Quantity) > 100, + Projection = x => new + { + Id = x.Id, + Total = x.Lines.Sum(line => line.PricePerUnit * line.Quantity), + ShipTo = x.ShipTo, + EmployeeName = RavenQuery.Load(x.Employee).FirstName + " " + + RavenQuery.Load(x.Employee).LastName + } + }); + + #endregion + + #region create_filter_and_load_document_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + + declare function projectOrder(doc){ + var employee = load(doc.Employee); + return { + Id: order.Id, + Total: getOrderLinesSum(order), + ShipTo: order.ShipTo, + EmployeeName: employee.FirstName + ' ' + employee.LastName + + }; + } + + From Orders as o + Where getOrderLinesSum(o) > 100 + Select projectOrder(o)" + }); + + #endregion + + + #region create_simple_revisions_subscription_generic + + name = store.Subscriptions.Create( + new SubscriptionCreationOptions>()); + + #endregion + + #region create_simple_revisions_subscription_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"From Orders (Revisions = true)" + }); + + #endregion + + + + + #region use_simple_revision_subscription_generic + + SubscriptionWorker> revisionWorker = store.Subscriptions.GetSubscriptionWorker>(name); + + await revisionWorker.Run((SubscriptionBatch> x) => + { + foreach (var documentsPair in x.Items) + { + var prev = documentsPair.Result.Previous; + var current = documentsPair.Result.Current; + + ProcessOrderChanges(prev, current); + } + } + ); + + #endregion + + void ProcessOrderChanges(Order prev, Order cur) + { + + } + + + #region create_projected_revisions_subscription_generic + + name = store.Subscriptions.Create( + new SubscriptionCreationOptions>() + { + Filter = tuple => tuple.Current.Lines.Count > tuple.Previous.Lines.Count, + Projection = tuple => new + { + PreviousRevenue = tuple.Previous.Lines.Sum(x => x.PricePerUnit * x.Quantity), + CurrentRevenue = tuple.Current.Lines.Sum(x => x.PricePerUnit * x.Quantity) + } + }); + + #endregion + + #region create_projected_revisions_subscription_RQL + + name = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + + From Orders (Revisions = true) + Where getOrderLinesSum(this.Current) > getOrderLinesSum(this.Previous) + Select + { + PreviousRevenue: getOrderLinesSum(this.Previous), + CurrentRevenue: getOrderLinesSum(this.Current) + }" + }); + + #endregion + + #region consume_revisions_subscription_generic + + SubscriptionWorker revenuesComparisonWorker = store.Subscriptions.GetSubscriptionWorker(name); + + await revenuesComparisonWorker.Run(x => + { + foreach (var item in x.Items) + { + Console.WriteLine($"Revenue for order with Id: {item.Id} grown from {item.Result.PreviousRevenue} to {item.Result.CurrentRevenue}"); + } + }); + + #endregion + + SubscriptionWorker subscription; + var cancellationToken = new CancellationTokenSource().Token; + #region consumption_0 + var subscriptionName = await store.Subscriptions.CreateAsync(x => x.Company == "companies/11"); + + subscription = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + var subscriptionTask = subscription.Run(x => + x.Items.ForEach(item => + Console.WriteLine($"Order #{item.Result.Id} will be shipped via: {item.Result.ShipVia}")), + cancellationToken); + + await subscriptionTask; + #endregion + #region open_1 + subscription = store.Subscriptions.GetSubscriptionWorker(name); + #endregion + + #region open_2 + subscription = store.Subscriptions.GetSubscriptionWorker(new SubscriptionWorkerOptions(name) + { + Strategy = SubscriptionOpeningStrategy.WaitForFree + }); + #endregion + + #region open_3 + subscription = store.Subscriptions.GetSubscriptionWorker(new SubscriptionWorkerOptions(name) + { + Strategy = SubscriptionOpeningStrategy.WaitForFree, + MaxDocsPerBatch = 500, + IgnoreSubscriberErrors = true + }); + #endregion + + #region create_subscription_with_includes_strongly_typed + store.Subscriptions.Create(new SubscriptionCreationOptions() + { + Includes = builder => builder + .IncludeDocuments(x => x.Lines.Select(y => y.Product)) + }); + #endregion + #region create_subscription_with_includes_rql_path + store.Subscriptions.Create(new SubscriptionCreationOptions() + { + Query = @"from Orders include Lines[].Product" + }); + #endregion + + #region create_subscription_with_includes_rql_javascript + store.Subscriptions.Create(new SubscriptionCreationOptions() + { + Query = @" + declare function includeProducts(doc) + { + doc.IncludedFields=0; + doc.LinesCount = doc.Lines.length; + for (let i=0; i< doc.Lines.length; i++) + { + doc.IncludedFields++; + include(doc.Lines[i].Product); + } + return doc; + } + from Orders as o select includeProducts(o)" + }); + #endregion + + /* + #region include_builder_counter_methods + ISubscriptionIncludeBuilder IncludeCounter(string name); + + ISubscriptionIncludeBuilder IncludeCounters(string[] names); + + ISubscriptionIncludeBuilder IncludeAllCounters(); + #endregion + */ + + #region create_subscription_include_counters_builder + store.Subscriptions.Create(new SubscriptionCreationOptions() + { + Includes = builder => builder + .IncludeCounter("numLines") + .IncludeCounters(new[] { "pros", "cons" }) + .IncludeAllCounters() + }); + #endregion + + #region update_subscription_example_0 + store.Subscriptions.Update(new SubscriptionUpdateOptions() + { + Name = "my subscription", + Query = "From Orders" + }); + #endregion + + #region update_subscription_example_1 + SubscriptionState mySubscription = store.Subscriptions.GetSubscriptionState("my subscription"); + + long subscriptionId = mySubscription.SubscriptionId; + + store.Subscriptions.Update(new SubscriptionUpdateOptions() + { + Id = subscriptionId, + Name = "new name" + }); + #endregion + } + + public interface IMaintainanceOperations + { + #region interface_subscription_deletion + void Delete(string name, string database = null); + Task DeleteAsync(string name, string database = null, CancellationToken token = default); + #endregion + + #region interface_subscription_dropping + void DropConnection(string name, string database = null); + Task DropConnectionAsync(string name, string database = null, CancellationToken token = default); + #endregion + + #region interface_subscription_enabling + void Enable(string name, string database = null); + Task EnableAsync(string name, string database = null, CancellationToken token = default); + #endregion + + #region interface_subscription_disabling + void Disable(string name, string database = null); + Task DisableAsync(string name, string database = null, CancellationToken token = default); + #endregion + + #region interface_subscription_state + SubscriptionState GetSubscriptionState(string subscriptionName, string database = null); + Task GetSubscriptionStateAsync(string subscriptionName, string database = null, CancellationToken token = default); + #endregion + } + public async Task SubscriptionMaintainance() + { + string subscriptionName = string.Empty; + using (var store = new DocumentStore()) + { + #region subscription_enabling + store.Subscriptions.Enable(subscriptionName); + #endregion + + #region subscription_disabling + store.Subscriptions.Disable(subscriptionName); + #endregion + + #region subscription_deletion + store.Subscriptions.Delete(subscriptionName); + #endregion + + #region connection_dropping + store.Subscriptions.DropConnection(subscriptionName); + #endregion + + #region subscription_state + var subscriptionState = store.Subscriptions.GetSubscriptionState(subscriptionName); + #endregion + } + } + + public class UnsupportedCompanyException : Exception + { + public UnsupportedCompanyException(string message, Exception ex = null) + { + + } + } + + public class Logger + { + public static void Error(string text) { } + + public static void Error(string text, Exception ex) { } + } + + public class OrderRevenues + { + public int PreviousRevenue { get; set; } + public int CurrentRevenue { get; set; } + } + + public class OrderAndCompany + { + public string OrderId; + public Company Company; + } + + [Fact] + public async Task OpeningExamples() + { + string name; + IDocumentStore store = new DocumentStore(); + SubscriptionWorker subscriptionWorker; + Task subscriptionRuntimeTask; + string subscriptionName = null; + CancellationToken cancellationToken = new CancellationToken(); + + #region subscription_open_simple + subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + #endregion + + #region subscription_run_simple + subscriptionRuntimeTask = subscriptionWorker.Run(batch => + { + // your logic here + }); + #endregion + + #region subscription_worker_with_batch_size + var workerWBatch = store.Subscriptions.GetSubscriptionWorker( + new SubscriptionWorkerOptions(subscriptionName) + { + MaxDocsPerBatch = 20 + }); + _ = workerWBatch.Run(x => { /* custom logic */ }); + #endregion + + #region throw_during_user_logic + _ = workerWBatch.Run(x => throw new Exception()); + #endregion + + + #region reconnecting_client + + while (true) + { + var options = new SubscriptionWorkerOptions(subscriptionName); + + // here we configure that we allow a down time of up to 2 hours, and will wait for 2 minutes for reconnecting + options.MaxErroneousPeriod = TimeSpan.FromHours(2); + options.TimeToWaitBeforeConnectionRetry = TimeSpan.FromMinutes(2); + + subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(options); + + try + { + // here we are able to be informed of any exception that happens during processing + subscriptionWorker.OnSubscriptionConnectionRetry += exception => + { + Logger.Error("Error during subscription processing: " + subscriptionName, exception); + }; + + await subscriptionWorker.Run(async batch => + { + foreach (var item in batch.Items) + { + // we want to force close the subscription processing in that case + // and let the external code decide what to do with that + if (item.Result.Company == "companies/832-A") + throw new UnsupportedCompanyException("Company Id can't be 'companies/832-A', you must fix this"); + await ProcessOrder(item.Result); + } + }, cancellationToken); + + // Run will complete normally if you have disposed the subscription + return; + } + catch (Exception e) + { + Logger.Error("Failure in subscription: " + subscriptionName, e); + + if (e is DatabaseDoesNotExistException || + e is SubscriptionDoesNotExistException || + e is SubscriptionInvalidStateException || + e is AuthorizationException) + throw; // not recoverable + + + if (e is SubscriptionClosedException) + // closed explicitly by admin, probably + return; + + if (e is SubscriberErrorException se) + { + // for UnsupportedCompanyException type, we want to throw an exception, otherwise + // we continue processing + if (se.InnerException != null && se.InnerException is UnsupportedCompanyException) + { + throw; + } + + continue; + } + + // handle this depending on subscription + // open strategy (discussed later) + if (e is SubscriptionInUseException) + continue; + + return; + } + finally + { + subscriptionWorker.Dispose(); + } + } + + #endregion + + while (true) + { + #region worker_timeout_minimal_sample + var options = new SubscriptionWorkerOptions(subscriptionName); + + // Set the worker's timeout period + options.ConnectionStreamTimeout = TimeSpan.FromSeconds(45); + #endregion + } + + while (true) + { + #region worker_timeout + var options = new SubscriptionWorkerOptions(subscriptionName); + + // Set the worker's timeout period + options.ConnectionStreamTimeout = TimeSpan.FromSeconds(45); + + subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(options); + + try + { + subscriptionWorker.OnSubscriptionConnectionRetry += exception => + { + Logger.Error("Error during subscription processing: " + subscriptionName, exception); + }; + + await subscriptionWorker.Run(async batch => + { + foreach (var item in batch.Items) + { + //... + } + }, cancellationToken); + + // Run will complete normally if you have disposed the subscription + return; + } + catch (Exception e) + { + Logger.Error("Error during subscription process: " + subscriptionName, e); + } + finally + { + subscriptionWorker.Dispose(); + } + #endregion + } + + async Task ProcessOrder(Order o) + { + + } + + + } + + private static async Task SingleRun(DocumentStore store) + { + + var subsId = store.Subscriptions.Create( + new SubscriptionCreationOptions + { + Filter = order => order.Lines.Sum(line => line.PricePerUnit * line.Quantity) > 10000, + Projection = order => new OrderAndCompany + { + OrderId = order.Id, + Company = RavenQuery.Load(order.Company) + } + }); + + #region single_run + var highValueOrdersWorker = store.Subscriptions.GetSubscriptionWorker( + new SubscriptionWorkerOptions(subsId) + { + // Here we ask the worker to stop when there are no documents left to send. + // Will throw SubscriptionClosedException when it finishes it's job + CloseWhenNoDocsLeft = true + }); + + try + { + await highValueOrdersWorker.Run(async batch => + { + foreach (var item in batch.Items) + { + await SendThankYouNoteToEmployee(item.Result); + } + }); + } + catch (SubscriptionClosedException) + { + // that's expected + } + #endregion + async Task SendThankYouNoteToEmployee(OrderAndCompany oac) + { + + } + } + + public async Task DynamicWorkerSubscription(DocumentStore store) + { + #region dynamic_worker + var subscriptionName = "My dynamic subscription"; + await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Name = "My dynamic subscription", + Projection = order => new { DynanamicField_1 = "Company: " + order.Company + " Employee: " + order.Employee } + }); + + var subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + _ = subscriptionWorker.Run(async batch => + { + foreach (var item in batch.Items) + { + await RaiseNotification(item.Result.DynanamicField_1); + } + }); + #endregion + + async Task RaiseNotification(string message) + { + + } + } + + public async Task SubscriptionWithIncludesPath(DocumentStore store) + { + #region subscription_with_includes_path_usage + var subscriptionName = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"from Orders include Lines[].Product" + }); + + var subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + _ = subscriptionWorker.Run(async batch => + { + using (var session = batch.OpenAsyncSession()) + { + foreach (var order in batch.Items.Select(x => x.Result)) + { + foreach (var orderLine in order.Lines) + { + // this line won't generate a request, because orderLine.Product was included + var product = await session.LoadAsync(orderLine.Product); + await RaiseNotification(order, product); + } + + } + } + }); + #endregion + + async Task RaiseNotification(Order modifiedOrder, Product productInOrder) + { + + } + } + + public async Task SubscriptionsWithOpenSession(DocumentStore store) + { + #region subscription_with_open_session_usage + var subscriptionName = await store.Subscriptions.CreateAsync(new SubscriptionCreationOptions() + { + Query = @"from Orders as o where o.ShippedAt = null" + }); + + var subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(subscriptionName); + _ = subscriptionWorker.Run(async batch => + { + using (var session = batch.OpenAsyncSession()) + { + foreach (var order in batch.Items.Select(x => x.Result)) + { + await TransferOrderToShipmentCompanyAsync(order); + order.ShippedAt = DateTime.UtcNow; + + } + + // we know that we have at least one order to ship, + // because the subscription query above has that in it's WHERE clause + await session.SaveChangesAsync(); + } + }); + #endregion + + async Task TransferOrderToShipmentCompanyAsync(Order modifiedOrder) + { + } + } + + + public async Task BlittableWorkerSubscription(DocumentStore store, string subscriptionId) + { + #region blittable_worker + await store.Subscriptions.CreateAsync( + new SubscriptionCreationOptions + { + Projection = x => new + { + x.Employee + } + }); + + var subscriptionWorker = store.Subscriptions.GetSubscriptionWorker(subscriptionId); + _ = subscriptionWorker.Run(async batch => + { + foreach (var item in batch.Items) + { + await RaiseNotification(item.Result["Employee"].ToString()); + } + }); + #endregion + + async Task RaiseNotification(string message) + { + + } + } + + + public async Task WaitForFreeSubscription(DocumentStore store, string subscriptionName) + { + #region waitforfree + var worker = store.Subscriptions.GetSubscriptionWorker + (new SubscriptionWorkerOptions(subscriptionName) + { + Strategy = SubscriptionOpeningStrategy.WaitForFree + }); + #endregion + } + + public async Task TwoSubscriptions(DocumentStore store, string subscriptionName) + { + #region waiting_subscription_1 + var primaryWorker = store.Subscriptions.GetSubscriptionWorker(new SubscriptionWorkerOptions(subscriptionName) + { + Strategy = SubscriptionOpeningStrategy.TakeOver + }); + + while (true) + { + try + { + await primaryWorker.Run(x => + { + // your logic + }); + } + catch (Exception) + { + // retry + } + } + #endregion + + #region waiting_subscription_2 + var secondaryWorker = store.Subscriptions.GetSubscriptionWorker(new SubscriptionWorkerOptions(subscriptionName) + { + Strategy = SubscriptionOpeningStrategy.WaitForFree + }); + + while (true) + { + try + { + await secondaryWorker.Run(x => + { + // your logic + }); + } + catch (Exception) + { + // retry + } + } + #endregion + } + public DataSubscriptions() + { + IDocumentStore store = new DocumentStore(); + + //{ + + // #region open_2 + // var orders = store.Subscriptions.Open(id, new SubscriptionConnectionOptions() + // { + // BatchOptions = new SubscriptionBatchOptions() + // { + // MaxDocCount = 16*1024, + // MaxSize = 4*1024*1024, + // AcknowledgmentTimeout = TimeSpan.FromMinutes(3) + // }, + // IgnoreSubscribersErrors = false, + // ClientAliveNotificationInterval = TimeSpan.FromSeconds(30) + // }); + // #endregion + + // #region open_3 + // orders.Subscribe(x => + // { + // GenerateInvoice(x); + // }); + + // orders.Subscribe(x => + // { + // if(x.RequireAt > DateTime.Now) + // SendReminder(x.Employee, x.Id); + // }); + // #endregion + + // #region open_4 + // var subscriber = orders.Subscribe(x => { }); + + // subscriber.Dispose(); + // #endregion + + // #region delete_2 + // store.Subscriptions.Delete(id); + // #endregion + + // #region get_subscriptions_2 + // var configs = store.Subscriptions.GetSubscriptions(0, 10); + // #endregion + + // #region release_2 + // store.Subscriptions.Release(id); + // #endregion + //} + } + + private void SendReminder(string employee, string id) + { + } + + public void GenerateInvoice(Order o) + { + + } + + //private interface IFoo + //{ + + // #region open_1 + // Subscription Open(long id, SubscriptionConnectionOptions options, string database = null); + + // Subscription Open(long id, SubscriptionConnectionOptions options, string database = null) + // #endregion + // where T : class; + + // #region delete_1 + // void Delete(long id, string database = null); + // #endregion + + // #region get_subscriptions_1 + // List GetSubscriptions(int start, int take, string database = null); + // #endregion + + // #region release_1 + // void Release(long id, string database = null); + // #endregion + //} + + //#region events + //public delegate void BeforeBatch(); + + //public delegate bool BeforeAcknowledgment(); + + //public delegate void AfterAcknowledgment(Etag lastProcessedEtag); + + //public delegate void AfterBatch(int documentsProcessed); + //#endregion + + + } +} diff --git a/Documentation/5.4/Samples/java/src/test/java/net/ravendb/ClientApi/DataSubscriptions/DataSubscriptions.java b/Documentation/5.4/Samples/java/src/test/java/net/ravendb/ClientApi/DataSubscriptions/DataSubscriptions.java new file mode 100644 index 0000000000..5b19dd3cac --- /dev/null +++ b/Documentation/5.4/Samples/java/src/test/java/net/ravendb/ClientApi/DataSubscriptions/DataSubscriptions.java @@ -0,0 +1,637 @@ +package net.ravendb.ClientApi.DataSubscriptions; + +import com.fasterxml.jackson.databind.node.ObjectNode; +import net.ravendb.client.documents.DocumentStore; +import net.ravendb.client.documents.IDocumentStore; +import net.ravendb.client.documents.session.IDocumentSession; +import net.ravendb.client.documents.subscriptions.*; +import net.ravendb.client.exceptions.database.DatabaseDoesNotExistException; +import net.ravendb.client.exceptions.documents.subscriptions.*; +import net.ravendb.client.exceptions.security.AuthorizationException; +import net.ravendb.client.primitives.ExceptionsUtils; +import net.ravendb.northwind.Order; +import net.ravendb.northwind.OrderLine; +import net.ravendb.northwind.Product; +import org.apache.commons.logging.Log; + +import java.time.Duration; +import java.util.Date; +import java.util.concurrent.CompletableFuture; +import java.util.function.Consumer; + +public class DataSubscriptions { + + private interface ISubscriptionCreationOveloads { + //region subscriptionCreationOverloads + String create(SubscriptionCreationOptions options); + + String create(SubscriptionCreationOptions options, String database); + + String create(Class clazz); + + String create(Class clazz, SubscriptionCreationOptions options); + + String create(Class clazz, SubscriptionCreationOptions options, String database); + + String createForRevisions(Class clazz); + + String createForRevisions(Class clazz, SubscriptionCreationOptions options); + + String createForRevisions(Class clazz, SubscriptionCreationOptions options, String database); + //endregion + } + + public interface ISubscriptionConsumptionOverloads { + //region subscriptionWorkerGeneration + SubscriptionWorker getSubscriptionWorker(SubscriptionWorkerOptions options); + SubscriptionWorker getSubscriptionWorker(SubscriptionWorkerOptions options, String database); + + SubscriptionWorker getSubscriptionWorker(String subscriptionName); + SubscriptionWorker getSubscriptionWorker(String subscriptionName, String database); + + SubscriptionWorker getSubscriptionWorker(Class clazz, SubscriptionWorkerOptions options); + SubscriptionWorker getSubscriptionWorker(Class clazz, SubscriptionWorkerOptions options, String database); + + SubscriptionWorker getSubscriptionWorker(Class clazz, String subscriptionName); + SubscriptionWorker getSubscriptionWorker(Class clazz, String subscriptionName, String database); + + SubscriptionWorker> getSubscriptionWorkerForRevisions(Class clazz, SubscriptionWorkerOptions options); + SubscriptionWorker> getSubscriptionWorkerForRevisions(Class clazz, SubscriptionWorkerOptions options, String database); + + SubscriptionWorker> getSubscriptionWorkerForRevisions(Class clazz, String subscriptionName); + SubscriptionWorker> getSubscriptionWorkerForRevisions(Class clazz, String subscriptionName, String database); + //endregion + } + + public interface ISubscriptionWorkerRunning { + //region subscriptionWorkerRunning + CompletableFuture run(Consumer> processDocuments); + //endregion + } + + //region subscriptions_example + public void worker(IDocumentStore store) { + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("from Orders where Company = 'companies/11'"); + + String subscriptionName = store.subscriptions().create(Order.class, options); + SubscriptionWorker subscription = store + .subscriptions().getSubscriptionWorker(Order.class, subscriptionName); + subscription.run(x -> { + for (SubscriptionBatch.Item item : x.getItems()) { + System.out.println("Order #" + + item.getResult().getId() + + " will be shipped via: " + item.getResult().getShipVia()); + } + }); + } + //endregion + + interface FakeDocumentSubscriptions { + + } + + interface FakeStore { + + } + + public void creationExamples() { + String name; + IDocumentStore store = new DocumentStore(); + + { + //region create_whole_collection_generic_with_name + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setName("OrdersProcessingSubscription"); + name = store.subscriptions().create(Order.class, options); + //endregion + } + + { + //region create_whole_collection_generic_with_mentor_node + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setMentorNode("D"); + name = store.subscriptions().create(Order.class, options); + //endregion + } + + { + //region create_whole_collection_generic1 + name = store.subscriptions().create(Order.class); + //endregion + } + + { + //region create_whole_collection_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("from Orders"); + name = store.subscriptions().create(options); + //endregion + } + + { + //region create_filter_only_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("declare function getOrderLinesSum(doc) {" + + " var sum = 0;" + + " for (var i in doc.Lines) { sum += doc.Lines[i]; }" + + " return sum;" + + "}" + + "from Orders as o " + + "where getOrderLinesSum(o) > 100 "); + + name = store.subscriptions().create(options); + //endregion + } + + { + //region create_filter_and_projection_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery(" declare function getOrderLinesSum(doc) {" + + " var sum = 0; " + + " for (var i in doc.Lines) { sum += doc.Lines[i]; }" + + " return sum;" + + "}" + + "" + + " declare function projectOrder(doc) {" + + " return {" + + " Id: order.Id," + + " Total: getOrderLinesSum(order)" + + " }" + + " }" + + " from order as o " + + " where getOrderLinesSum(o) > 100 " + + " select projectOrder(o)"); + + name = store.subscriptions().create(options); + //endregion + } + + { + //region create_filter_and_load_document_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery(" declare function getOrderLinesSum(doc) {" + + " var sum = 0; " + + " for (var i in doc.Lines) { sum += doc.Lines[i]; }" + + " return sum;" + + "}" + + "" + + " declare function projectOrder(doc) {" + + " var employee = LoadDocument(doc.Employee); " + + " return {" + + " Id: order.Id," + + " Total: getOrderLinesSum(order)," + + " ShipTo: order.ShipTo," + + " EmployeeName: employee.FirstName + ' ' + employee.LastName " + + " }" + + " }" + + " from order as o " + + " where getOrderLinesSum(o) > 100 " + + " select projectOrder(o)"); + + name = store.subscriptions().create(options); + //endregion + } + + { + //region create_simple_revisions_subscription_generic + name = store.subscriptions().createForRevisions(Order.class); + //endregion + } + + { + //region create_simple_revisions_subscription_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("from orders (Revisions = true)"); + name = store.subscriptions().createForRevisions(Order.class, options); + //endregion + } + + { + //region use_simple_revision_subscription_generic + SubscriptionWorker> revisionWorker = store + .subscriptions().getSubscriptionWorkerForRevisions(Order.class, name); + revisionWorker.run(x -> { + for (SubscriptionBatch.Item> documentsPair : x.getItems()) { + + Order prev = documentsPair.getResult().getPrevious(); + Order current = documentsPair.getResult().getCurrent(); + + processOrderChanges(prev, current); + } + }); + //endregion + } + + { + //region create_projected_revisions_subscription_RQL + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("declare function getOrderLinesSum(doc) {" + + " var sum = 0;" + + " for (var i in doc.Lines) { sum += doc.Lines[i]; } " + + " return sum;" + + "}" + + "" + + " from orders (Revisions = true) " + + " where getOrderLinesSum(this.Current) > getOrderLinesSum(this.Previous) " + + " select {" + + " previousRevenue: getOrderLinesSum(this.Previous)," + + " currentRevenue: getOrderLinesSum(this.Current)" + + "}"); + + name = store.subscriptions().create(options); + //endregion + } + + { + //region consume_revisions_subscription_generic + SubscriptionWorker revenuesComparisonWorker = store + .subscriptions().getSubscriptionWorker(OrderRevenues.class, name); + + revenuesComparisonWorker.run(x -> { + for (SubscriptionBatch.Item item : x.getItems()) { + System.out.println("Revenue for order with Id: " + + item.getId() + " grown from " + + item.getResult().getPreviousRevenue() + + " to " + item.getResult().getCurrentRevenue()); + } + }); + //endregion + } + } + + void processOrderChanges(Order prev, Order cur) { + + } + + public interface IMaintainanceOperations { + //region interface_subscription_deletion + void delete(String name); + + void delete(String name, String database); + //endregion + + //region interface_subscription_dropping + void dropConnection(String name); + + void dropConnection(String name, String database); + //endregion + + //region interface_subscription_state + SubscriptionState getSubscriptionState(String subscriptionName); + + SubscriptionState getSubscriptionState(String subscriptionName, String database); + //endregion + } + + public void subscriptionMaintainance() { + String subscriptionName = ""; + try (IDocumentStore store = new DocumentStore()) { + //region subscription_deletion + store.subscriptions().delete(subscriptionName); + //endregion + + //region connection_dropping + store.subscriptions().dropConnection(subscriptionName); + //endregion + + //region subscription_state + SubscriptionState subscriptionState = store.subscriptions().getSubscriptionState(subscriptionName); + //endregion + } + } + + public static class OrderRevenues { + private int previousRevenue; + private int currentRevenue; + + public int getPreviousRevenue() { + return previousRevenue; + } + + public void setPreviousRevenue(int previousRevenue) { + this.previousRevenue = previousRevenue; + } + + public int getCurrentRevenue() { + return currentRevenue; + } + + public void setCurrentRevenue(int currentRevenue) { + this.currentRevenue = currentRevenue; + } + } + + public static class OrderAndCompany { + private String orderId; + private Company company; + + public String getOrderId() { + return orderId; + } + + public void setOrderId(String orderId) { + this.orderId = orderId; + } + + public Company getCompany() { + return company; + } + + public void setCompany(Company company) { + this.company = company; + } + } + + private class Company { + } + + public void openingExamples() throws Exception { + String name; + DocumentStore store = new DocumentStore(); + SubscriptionWorker subscriptionWorker; + CompletableFuture subscriptionRuntimeTask; + String subscriptionName = null; + + //region subscription_open_simple + subscriptionWorker = store.subscriptions().getSubscriptionWorker(Order.class, subscriptionName); + //endregion + + //region subscription_run_simple + subscriptionRuntimeTask = subscriptionWorker.run(batch -> { + // your logic here + }); + //endregion + + { + //region subscription_worker_with_batch_size + SubscriptionWorkerOptions options = new SubscriptionWorkerOptions(subscriptionName); + options.setMaxDocsPerBatch(20); + SubscriptionWorker workerWBatch = store.subscriptions().getSubscriptionWorker(Order.class, options); + workerWBatch.run(x -> { /* custom logic */}); + //endregion + } + + { + SubscriptionWorker workerWBatch = null; + //region throw_during_user_logic + workerWBatch.run(x -> { + throw new RuntimeException(); + }); + //endregion + } + + Log logger = null; + + //region reconnecting_client + while (true) { + SubscriptionWorkerOptions options = new SubscriptionWorkerOptions(subscriptionName); + // here we configure that we allow a down time of up to 2 hours, + // and will wait for 2 minutes for reconnecting + + options.setMaxErroneousPeriod(Duration.ofHours(2)); + options.setTimeToWaitBeforeConnectionRetry(Duration.ofMinutes(2)); + + subscriptionWorker = store.subscriptions().getSubscriptionWorker(Order.class, options); + + try { + // here we are able to be informed of any exception that happens during processing + subscriptionWorker.addOnSubscriptionConnectionRetry(exception -> { + logger.error("Error during subscription processing: " + subscriptionName, exception); + }); + + subscriptionWorker.run(batch -> { + for (SubscriptionBatch.Item item : batch.getItems()) { + // we want to force close the subscription processing in that case + // and let the external code decide what to do with that + if ("Europe".equalsIgnoreCase(item.getResult().getShipVia())) { + throw new IllegalStateException("We cannot ship via Europe"); + } + processOrder(item.getResult()); + } + }).get(); + + + // Run will complete normally if you have disposed the subscription + return; + } catch (Exception e) { + logger.error("Failure in subscription: " + subscriptionName, e); + + e = ExceptionsUtils.unwrapException(e); + if (e instanceof DatabaseDoesNotExistException || + e instanceof SubscriptionDoesNotExistException || + e instanceof SubscriptionInvalidStateException || + e instanceof AuthorizationException) { + throw e; // not recoverable + } + + if (e instanceof SubscriptionClosedException) { + // closed explicitly by admin, probably + return; + } + + if (e instanceof SubscriberErrorException) { + SubscriberErrorException se = (SubscriberErrorException) e; + // for IllegalStateException type, we want to throw an exception, otherwise + // we continue processing + if (se.getCause() != null && se.getCause() instanceof IllegalStateException) { + throw e; + } + + continue; + } + + // handle this depending on subscription + // open strategy (discussed later) + if (e instanceof SubscriptionInUseException) { + continue; + } + + return; + } finally { + subscriptionWorker.close(); + } + } + //endregion + } + + private void processOrder(Order result) { + } + + private static void singleRun() { + IDocumentStore store = null; + String subsId = null; + //region single_run + SubscriptionWorkerOptions options = new SubscriptionWorkerOptions(subsId); + + // Here we ask the worker to stop when there are no documents left to send. + // Will throw SubscriptionClosedException when it finishes it's job + options.setCloseWhenNoDocsLeft(true); + SubscriptionWorker highValueOrdersWorker = store + .subscriptions().getSubscriptionWorker(OrderAndCompany.class, options); + + try { + highValueOrdersWorker.run(batch -> { + for (SubscriptionBatch.Item item : batch.getItems()) { + sendThankYouNoteToEmployee(item.getResult()); + } + }); + } catch (SubscriptionClosedException e) { + //that's expected + } + //endregion + } + + private static void sendThankYouNoteToEmployee(OrderAndCompany oac) { + // empty + } + + public void twoSubscription1(DocumentStore store, String subscriptionName) { + //region waiting_subscription_1 + SubscriptionWorkerOptions options1 = new SubscriptionWorkerOptions(subscriptionName); + options1.setStrategy(SubscriptionOpeningStrategy.TAKE_OVER); + SubscriptionWorker worker1 = store.subscriptions().getSubscriptionWorker(Order.class, options1); + + + while (true) { + try { + worker1 + .run(x -> { + // your logic + }); + } catch (Exception e) { + // retry + } + } + //endregion + } + public void twoSubscription2(DocumentStore store, String subscriptionName) { + //region waiting_subscription_2 + SubscriptionWorkerOptions options2 = new SubscriptionWorkerOptions(subscriptionName); + options2.setStrategy(SubscriptionOpeningStrategy.WAIT_FOR_FREE); + SubscriptionWorker worker2 = store.subscriptions().getSubscriptionWorker(Order.class, options2); + + while (true) { + try { + worker2 + .run(x -> { + // your logic + }); + } catch (Exception e) { + // retry + } + } + //endregion + } + + + public void createSubscriptionWithIncludeStatement(DocumentStore store) { + { + //region create_subscription_with_includes_rql_path + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("from Orders include Lines[].Product"); + store.subscriptions().create(options); + //endregion + } + + { + //region create_subscription_with_includes_rql_javascript + SubscriptionCreationOptions options = new SubscriptionCreationOptions(); + options.setQuery("declare function includeProducts(doc) " + + " {" + + " doc.IncludedFields=0;" + + " doc.LinesCount = doc.Lines.length;" + + " for (let i=0; i< doc.Lines.length; i++)" + + " {" + + " doc.IncludedFields++;" + + " include(doc.Lines[i].Product);" + + " }" + + " return doc;" + + " }" + + " from Orders as o select includeProducts(o)"); + store.subscriptions().create(options); + //endregion + } + + { + //region dynamic_worker + String subscriptionName = "My dynamic subscription"; + + SubscriptionCreationOptions subscriptionCreationOptions = new SubscriptionCreationOptions(); + subscriptionCreationOptions.setName("My dynamic subscription"); + subscriptionCreationOptions.setQuery("from Orders as o \n" + + "select { \n" + + " DynamicField_1: 'Company:' + o.Company + ' Employee: ' + o.Employee \n" + + "}"); + + SubscriptionWorker worker = store.subscriptions().getSubscriptionWorker(subscriptionName); + worker.run(x -> { + for (SubscriptionBatch.Item item : x.getItems()) { + ObjectNode result = item.getResult(); + raiseNotification(result.get("DynamicField_1")); + } + }); + //endregion + } + + } + + public void raiseNotification(Object str) { + } + + public void consumeSubscriptionWithIncludeStatement(DocumentStore store) { + //region subscription_with_open_session_usage + SubscriptionCreationOptions subscriptionCreationOptions = new SubscriptionCreationOptions(); + subscriptionCreationOptions.setQuery("from Orders as o where o.ShippedAt = null"); + String subscriptionName = store.subscriptions().create(subscriptionCreationOptions); + + SubscriptionWorker subscriptionWorker = store.subscriptions().getSubscriptionWorker(Order.class, subscriptionName); + + subscriptionWorker.run(batch -> { + try (IDocumentSession session = batch.openSession()) { + for (SubscriptionBatch.Item orderItem : batch.getItems()) { + transferOrderToShipmentCompany(orderItem.getResult()); + orderItem.getResult().setShippedAt(new Date()); + } + + // we know that we have at least one order to ship, + // because the subscription query above has that in it's WHERE clause + session.saveChanges(); + } + }); + //endregion + } + + private void transferOrderToShipmentCompany(Order order) { + + } + + public void consumeSubscriptionWithIncludeStatements(DocumentStore store) { + //region subscription_with_includes_path_usage + SubscriptionCreationOptions subscriptionCreationOptions = new SubscriptionCreationOptions(); + subscriptionCreationOptions.setQuery("from Orders include Lines[].Product"); + + + String subscriptionName = store.subscriptions().create(subscriptionCreationOptions); + + SubscriptionWorker subscriptionWorker = store.subscriptions().getSubscriptionWorker(Order.class, subscriptionName); + + subscriptionWorker.run(batch -> { + try (IDocumentSession session = batch.openSession()) { + for (SubscriptionBatch.Item orderItem : batch.getItems()) { + Order order = orderItem.getResult(); + for (OrderLine orderLine : order.getLines()) { + // this line won't generate a request, because orderLine.Product was included + Product product = session.load(Product.class, orderLine.getProduct()); + raiseProductNotification(order, product); + } + } + } + }); + //endregion + } + + private void raiseProductNotification(Order order, Product product) { + } + +} diff --git a/Documentation/5.4/Samples/nodejs/client-api/dataSubscriptions/dataSubscriptions.js b/Documentation/5.4/Samples/nodejs/client-api/dataSubscriptions/dataSubscriptions.js new file mode 100644 index 0000000000..1969448d98 --- /dev/null +++ b/Documentation/5.4/Samples/nodejs/client-api/dataSubscriptions/dataSubscriptions.js @@ -0,0 +1,478 @@ +import * as assert from "assert"; +import { DocumentStore } from "ravendb"; + +let database, options, clazz, subscriptionName; + +class InvalidOperationException extends Error {} + +const store = new DocumentStore(); +const session = store.openSession(); + +{ + //region subscriptionCreationOverloads + store.subscriptions.create(clazz, [database]); + store.subscriptions.create(options, [database]); + + store.subscriptions.createForRevisions(clazz, [database]); + store.subscriptions.createForRevisions(options, [database]); + //endregion +} + + async function example() { + const subscription = await store.subscriptions.create({}); + //region subscriptionWorkerGeneration + store.subscriptions.getSubscriptionWorker(options, [database]); + store.subscriptions.getSubscriptionWorker(subscriptionName, [database]); + + store.subscriptions.getSubscriptionWorkerForRevisions(options, [database]); + store.subscriptions.getSubscriptionWorkerForRevisions(subscriptionName, [database]); + //endregion + } + + { + let subscriptionWorker; + //region subscriptionWorkerRunning + subscriptionWorker.on("batch", (batch, callback) => { }); + subscriptionWorker.on("error", (error) => {}); + subscriptionWorker.on("end", () => {}); + //endregion + } + + //region subscriptions_example + async function worker() { + const subscriptionName = await store.subscriptions.create({ + query: "from Orders where Company = 'companies/11'" + }); + const subscription = store.subscriptions.getSubscriptionWorker(subscriptionName); + subscription.on("batch", (batch, callback) => { + for (const item of batch.items) { + console.log(`Order #${item.result.Id} will be shipped via: ${item.result.ShipVia}`); + } + + callback(); + }); + } + //endregion + + async function createExamples() { + + class Order {} + + { + //region create_whole_collection_generic_with_name + const name = await store.subscriptions.create({ + name: "OrdersProcessingSubscription", + documentType: Order + }); + //endregion + } + + { + //region create_whole_collection_generic_with_mentor_node + const name = await store.subscriptions.create({ + documentType: Order, + mentorNode: "D" + }); + //endregion + } + + { + //region create_whole_collection_generic1 + store.subscriptions.create(Order); + //endregion + } + + { + //region create_whole_collection_RQL + store.subscriptions.create({ query: "from Orders" }); + //endregion + } + + { + //region create_filter_only_RQL + const query = `declare function getOrderLinesSum(doc) { + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + from Orders as o + where getOrderLinesSum(o) > 100`; + + const name = await store.subscriptions.create({ query }); + //endregion + } + + { + //region create_filter_and_projection_RQL + const query = + `declare function getOrderLinesSum(doc) { + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + + declare function projectOrder(doc) { + return { + Id: order.Id, + Total: getOrderLinesSum(order) + } + } + from order as o + where getOrderLinesSum(o) > 100 + select projectOrder(o)`; + + const name = await store.subscriptions.create({ query }); + //endregion + } + + { + //region create_filter_and_load_document_RQL + const query = + `declare function getOrderLinesSum(doc) { + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + + declare function projectOrder(doc) { + var employee = LoadDocument(doc.Employee); + return { + Id: order.Id, + Total: getOrderLinesSum(order), + ShipTo: order.ShipTo, + EmployeeName: employee.FirstName + ' ' + employee.LastName + } + } + from order as o + where getOrderLinesSum(o) > 100 + select projectOrder(o)`; + + const name = await store.subscriptions.create({ query }); + //endregion + } + + { + //region create_simple_revisions_subscription_generic + const name = await store.subscriptions.createForRevisions(Order); + //endregion + } + + { + //region create_simple_revisions_subscription_RQL + const name = await store.subscriptions.createForRevisions({ + query: "from orders (Revisions = true)" + }); + //endregion + } + + { + //region use_simple_revision_subscription_generic + const revisionWorker = store.subscriptions + .getSubscriptionWorkerForRevisions({ + documentType: Order, + name + }); + revisionWorker.on("batch", (batch, callback) => { + for (const documentsPair of batch.items) { + const prev = documentsPair.result.previous; + const current = documentsPair.result.current; + processOrderChanges(prev, current); + callback(); + } + }); + //endregion + } + + { + //region create_projected_revisions_subscription_RQL + const query = + `declare function getOrderLinesSum(doc) { + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + + from orders(Revisions = true) + where getOrderLinesSum(this.Current) > getOrderLinesSum(this.Previous) + select { + PreviousRevenue: getOrderLinesSum(this.Previous), + CurrentRevenue: getOrderLinesSum(this.Current) + }`; + + const name = await store.subscriptions.create({ query }); + //endregion + } + + { + //region consume_revisions_subscription_generic + const revenuesComparisonWorker = store + .subscriptions.getSubscriptionWorker({ + subscriptionName: name + }); + + revenuesComparisonWorker.on("batch", (batch, callback) => { + for (const item of batch.items) { + console.log("Revenue for order with Id: " + + item.id + " grown from " + + item.result.PreviousRevenue + + " to " + item.result.CurrentRevenue); + } + + callback(); + }); + //endregion + } + } + + function processOrderChanges() { } + + { + //region interface_subscription_deletion + store.subscriptions.delete(name, [database]); + //endregion + + //region interface_subscription_dropping + store.subscriptions.dropConnection(name, [database]); + //endregion + + //region interface_subscription_state + store.subscriptions.getSubscriptionState(subscriptionName, [database]); + //endregion + } + + { + const subscriptionName = ""; + //region subscription_deletion + store.subscriptions.delete(subscriptionName); + //endregion + + //region connection_dropping + store.subscriptions.dropConnection(subscriptionName); + //endregion + + //region subscription_state + store.subscriptions.getSubscriptionState(subscriptionName); + //endregion + } + + class OrderRevenues { } + + class OrderAndCompany { } + + class Company { } + + async function openingExamples() { + let name; + let subscriptionName; + + //region subscription_open_simple + const subscriptionWorker = store.subscriptions.getSubscriptionWorker({ subscriptionName }); + //endregion + + //region subscription_run_simple + subscriptionWorker.on("batch", (batch, callback) => { + // your logic here + + // report batch processing error passing it to callback + // callback(err) + callback(); + }); + + subscriptionWorker.on("error", error => { + // handle errors + }); + //endregion + + { + //region subscription_worker_with_batch_size + const options = { + subscriptionName, + maxDocsPerBatch: 20 + }; + + const workerWBatch = store.subscriptions.getSubscriptionWorker(options); + workerWBatch.on("batch", (batch, callback) => { /* custom logic */}); + //endregion + } + + { + let workerWBatch = null; + //region throw_during_user_logic + workerWBatch.on("batch", (batch, callback) => { + callback(new Error("Error during processing batch.")); + }); + //endregion + } + + async function reconnecting() { + //region reconnecting_client + + // here we configure that we allow a down time of up to 2 hours, + // and will wait for 2 minutes for reconnecting + const options = { + subscriptionName, + maxErroneousPeriod: 2 * 3600 * 1000, + timeToWaitBeforeConnectionRetry: 2 * 60 * 1000 + }; + + setupReconnectingSubscription(options); + + function setupReconnectingSubscription(subscriptionWorkerOptions) { + let subscriptionWorker; + + reconnect(); + + function closeWorker(worker) { + worker.removeAllListeners(); + worker.on("error", () => {}); // ignore errors from old connection + worker.dispose(); + } + + function reconnect() { + if (subscriptionWorker) { + closeWorker(); + } + + subscriptionWorker = store.subscriptions.getSubscriptionWorker(subscriptionWorkerOptions); + + // here we are able to be informed of any exception that happens during processing + subscriptionWorker.on("connectionRetry", error => { + console.error( + "Error during subscription processing: " + subscriptionName, error); + }); + + subscriptionWorker.on("batch", (batch, callback) => { + for (const item of batch.items) { + // we want to force close the subscription processing in that case + // and let the external code decide what to do with that + if (item.result.shipVia + && "Europe" === item.result.shipVia) { + callback(new InvalidOperationException("We cannot ship via Europe.")); + return; + } + + processOrder(item.result); + } + }); + + subscriptionWorker.on("error", error => { + console.error("Failure in subscription: " + subscriptionName, error); + + if (error.name === "DatabaseDoesNotExistException" || + error.name === "SubscriptionDoesNotExistException" || + error.name === "SubscriptionInvalidStateException" || + error.name === "AuthorizationException") { + throw error; + } + + if (error.name === "SubscriptionClosedException") { + // closed explicitly by admin, probably + return closeWorker(subscriptionWorker); + } + + if (error.name === "SubscriberErrorException") { + // for InvalidOperationException type, we want to throw an exception, otherwise + // we continue processing + // RavenDB client uses VError - it can nest errors and keep track of inner errors + // under cause property + if (error.cause && error.cause.name === "InvalidOperationException") { + throw error; + } + + return reconnect(); + } + + // handle this depending on subscription + // open strategy (discussed later) + if (error.name === "SubscriptionInUseException") { + return reconnect(); + } + + return reconnect(); + }); + + subscriptionWorker.on("end", () => { + closeWorker(subscriptionWorker); + }); + } + } + //endregion + } + + function processOrder(result) { } + + function singleRun() { + let subsId; + //region single_run + const options = { + subscriptionName: subsId, + // Here we ask the worker to stop when there are no documents left to send. + // Will throw SubscriptionClosedException when it finishes it's job + closeWhenNoDocsLeft: true + }; + + const highValueOrdersWorker = store + .subscriptions.getSubscriptionWorker(options); + + highValueOrdersWorker.on("batch", async (batch, callback) => { + for (const item of batch.items) { + await sendThankYouNoteToEmployee(item.result); + } + + callback(); + }); + + highValueOrdersWorker.on("error", err => { + if (err.name === "SubscriptionClosedException"){ + //that's expected + } + }); + //endregion + } + + async function sendThankYouNoteToEmployee(oac) { } + + class Order {} + + async function twoSubscription1() { + //region waiting_subscription_1 + const options1 = { + subscriptionName, + strategy: "TakeOver", + documentType: Order + }; + + const worker1 = store.subscriptions.getSubscriptionWorker(options1); + + worker1.on("batch", (batch, callback) => { + // your logic + callback(); + }); + + worker1.on("error", err => { + // retry + }); + //endregion + } + + async function twoSubscription2() { + //region waiting_subscription_2 + const options2 = { + subscriptionName, + strategy: "WaitForFree", + documentType: Order + }; + + const worker2 = store.subscriptions.getSubscriptionWorker(options2); + + worker2.on("batch", (batch, callback) => { + // your logic + callback(); + }); + + worker2.on("error", err => { + // retry + }); + //endregion + } +} diff --git a/Documentation/5.4/Samples/python/ClientApi/DataSubscriptions/DataSubscriptions.py b/Documentation/5.4/Samples/python/ClientApi/DataSubscriptions/DataSubscriptions.py new file mode 100644 index 0000000000..99a7655356 --- /dev/null +++ b/Documentation/5.4/Samples/python/ClientApi/DataSubscriptions/DataSubscriptions.py @@ -0,0 +1,739 @@ +import logging +from concurrent.futures import Future +from datetime import timedelta, datetime +from typing import Optional, Type, TypeVar, Callable, Any, Dict, List + +from ravendb import DocumentStore +from ravendb.documents.session.loaders.include import SubscriptionIncludeBuilder +from ravendb.documents.subscriptions.options import ( + SubscriptionCreationOptions, + SubscriptionUpdateOptions, + SubscriptionWorkerOptions, + SubscriptionOpeningStrategy, +) +from ravendb.documents.subscriptions.state import SubscriptionState +from ravendb.documents.subscriptions.worker import SubscriptionWorker, SubscriptionBatch +from ravendb.exceptions.exceptions import ( + DatabaseDoesNotExistException, + SubscriptionDoesNotExistException, + SubscriptionInvalidStateException, + AuthorizationException, + SubscriptionClosedException, + SubscriberErrorException, + SubscriptionInUseException, +) +from ravendb.serverwide.operations.common import DeleteDatabaseOperation + +from examples_base import Order, ExampleBase, Company + +_T = TypeVar("_T") + + +class OrderRevenues: + def __init__(self, previous_revenue: int = None, current_revenue: int = None): + self.previous_revenue = previous_revenue + self.current_revenue = current_revenue + + +class Foo: + # region subscriptionCreationOverloads + def create_for_options(self, options: SubscriptionCreationOptions, database: Optional[str] = None) -> str: ... + + def create_for_class( + self, + object_type: Type[_T], + options: Optional[SubscriptionCreationOptions] = None, + database: Optional[str] = None, + ) -> str: ... + + # endregion + # region updating_subscription + def update(self, options: SubscriptionUpdateOptions, database: Optional[str] = None) -> str: ... + + # endregion + + # region sub_create_options + class SubscriptionCreationOptions: + def __init__( + self, + name: Optional[str] = None, + query: Optional[str] = None, + includes: Optional[Callable[[SubscriptionIncludeBuilder], None]] = None, + change_vector: Optional[str] = None, + mentor_node: Optional[str] = None, + ): + self.name = name + self.query = query + self.includes = includes + self.change_vector = change_vector + self.mentor_node = mentor_node + + # endregion + + # region sub_update_options + + class SubscriptionUpdateOptions(SubscriptionCreationOptions): + def __init__( + self, + name: Optional[str] = None, + query: Optional[str] = None, + includes: Optional[Callable[[SubscriptionIncludeBuilder], None]] = None, + change_vector: Optional[str] = None, + mentor_node: Optional[str] = None, + key: Optional[int] = None, + create_new: Optional[bool] = None, + ): ... + + # endregion + # region subscriptionWorkerGeneration + def get_subscription_worker( + self, options: SubscriptionWorkerOptions, object_type: Optional[Type[_T]] = None, database: Optional[str] = None + ) -> SubscriptionWorker[_T]: ... + + def get_subscription_worker_by_name( + self, + subscription_name: Optional[str] = None, + object_type: Optional[Type[_T]] = None, + database: Optional[str] = None, + ) -> SubscriptionWorker[_T]: ... + + # endregion + + # region subscriptionWorkerRunning + def run(self, process_documents: Optional[Callable[[SubscriptionBatch[_T]], Any]]) -> Future: ... + + # endregion + + # region subscriptions_example + def worker(self, store: DocumentStore) -> Future: + subscription_name = store.subscriptions.create_for_class( + Order, SubscriptionCreationOptions(query="from Orders where Company = " "") + ) + subscription = store.subscriptions.get_subscription_worker_by_name(subscription_name) + + def __subscription_callback(subscription_batch: SubscriptionBatch[Order]): + for item in subscription_batch.items: + print(f"Order #{item.result.key}") + + subscription_task = subscription.run(__subscription_callback) + + return subscription_task + + # endregion + + # region creation_api + # endregion + + +class SubscriptionExamples(ExampleBase): + def setUp(self): + super().setUp() + + def tearDown(self): + with self.embedded_server.get_document_store("Manager") as store: + parameters = DeleteDatabaseOperation.Parameters(["SubscriptionsExamples"], True) + store.maintenance.server.send(DeleteDatabaseOperation(parameters=parameters)) + + def test_subscriptions(self): + with self.embedded_server.get_document_store("SubscriptionsExamples") as store: + # region create_whole_collection_generic_with_name + name = store.subscriptions.create_for_class( + Order, SubscriptionCreationOptions(name="OrdersProcessingSubscription") + ) + # endregion + + # region create_whole_collection_generic_with_mentor_node + name = store.subscriptions.create_for_class(Order, SubscriptionCreationOptions(mentor_node="D")) + # endregion + + # region create_whole_collection_generic1 + name = store.subscriptions.create_for_class(Order) + # endregion + + # region create_whole_collection_RQL + name = store.subscriptions.create_for_options(SubscriptionCreationOptions(query="From Orders")) + # endregion + all_name = name + + # region create_filter_only_RQL + name = store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=( + "declare function getOrderLinesSum(doc){" + " var sum = 0;" + " for (var i in doc.Lines) { sum += doc.Lines[i]; }" + " return sum;" + "}" + "From Orders as o " + "Where getOrderLinesSum(o) > 100 " + ) + ), + ) + # endregion + + # region create_filter_and_projection_RQL + name = store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=""" + declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + + declare function projectOrder(doc){ + return { + Id: order.Id, + Total: getOrderLinesSum(order) + }; + } + + From Orders as o + Where getOrderLinesSum(o) > 100 + Select projectOrder(o) + """ + ) + ) + # endregion + + # region create_filter_and_load_document_RQL + name = store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=""" + declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i]; } + return sum; + } + + declare function projectOrder(doc){ + return { + Id: order.Id, + Total: getOrderLinesSum(order) + }; + } + + From Orders as o + Where getOrderLinesSum(o) + """ + ) + ) + # endregion + + # region create_filter_and_load_document_RQL + name = store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=""" + declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + + declare function projectOrder(doc){ + var employee = load(doc.Employee); + return { + Id: order.Id, + Total: getOrderLinesSum(order), + ShipTo: order.ShipTo, + EmployeeName: employee.FirstName + ' ' + employee.LastName + + }; + } + + From Orders as o + Where getOrderLinesSum(o) > 100 + Select projectOrder(o) + """ + ) + ) + # endregion + + # region create_projected_revisions_subscription_RQL + name = store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=""" + declare function getOrderLinesSum(doc){ + var sum = 0; + for (var i in doc.Lines) { sum += doc.Lines[i];} + return sum; + } + + From Orders (Revisions = true) as o + Where getOrderLinesSum(o.Current) > getOrderLinesSum(o.Previous) + Select + { + previous_revenue: getOrderLinesSum(o.Previous), + current_revenue: getOrderLinesSum(o.Current) + } + """ + ) + ) + # endregion + + # region consume_revisions_subscription_generic + revenues_comparison_worker = store.subscriptions.get_subscription_worker_by_name(name, OrderRevenues) + + def _revenues_callback(batch: SubscriptionBatch[OrderRevenues]): + for item in batch.items: + print( + f"Revenue for order with Id: {item.key} grown from {item.result.previous_revenue} to {item.result.current_revenue}" + ) + + revenues_comparison_worker.run(_revenues_callback) + # endregion + """ + # region consumption_0 + subscription_name = store.subscriptions.create_for_class( + Order, SubscriptionCreationOptions(query='From Orders Where Company = "companies/11"') + ) + + subscription = store.subscriptions.get_subscription_worker_by_name(subscription_name, Order) + + def _orders_callback(batch: SubscriptionBatch[Order]): + for item in batch.items: + print(f"Order {item.result.key} will be shipped via: {item.result.ship_via}") + + subscription_task = subscription.run(_orders_callback) + + subscription_task.result() # Optionally wrap it in an asyncio.Future + # endregion + """ + # region open_1 + subscription = store.subscriptions.get_subscription_worker_by_name(name, Order) + # endregion + + # region open_2 + subscription = store.subscriptions.get_subscription_worker( + SubscriptionWorkerOptions(name, strategy=SubscriptionOpeningStrategy.WAIT_FOR_FREE) + ) + # endregion + + # region open_3 + subscription = store.subscriptions.get_subscription_worker( + SubscriptionWorkerOptions( + name, + strategy=SubscriptionOpeningStrategy.WAIT_FOR_FREE, + max_docs_per_batch=500, + ignore_subscriber_errors=True, + ) + ) + # endregion + + # region create_subscription_with_includes_strongly_typed + store.subscriptions.create_for_class( + Order, + SubscriptionCreationOptions(includes=lambda builder: builder.include_documents("Lines[].Product")), + ) + # endregion + + # region create_subscription_with_includes_rql_path + store.subscriptions.create_for_options( + SubscriptionCreationOptions(query="from Orders include Lines[].Product") + ) + # endregion + + # region create_subscription_with_includes_rql_javascript + store.subscriptions.create_for_options( + SubscriptionCreationOptions( + query=""" + declare function includeProducts(doc) + { + doc.IncludedFields=0; + doc.LinesCount = doc.Lines.length; + for (let i=0; i< doc.Lines.length; i++) + { + doc.IncludedFields++; + include(doc.Lines[i].Product); + } + return doc; + } + from Orders as o select includeProducts(o) + """ + ) + ) + + # endregion + + # region include_builder_counter_methods + def include_counter(self, name: str) -> SubscriptionIncludeBuilder: ... + + def include_counters(self, *names: str) -> SubscriptionIncludeBuilder: ... + + def include_all_counters(self) -> SubscriptionIncludeBuilder: ... + + # endregion + + """ + # region create_subscription_include_counters_builder + store.subscriptions.create_for_class( + Order, + SubscriptionCreationOptions( + includes=lambda builder: builder.include_counter("numLines") + .include_counters("pros", "cons") + .include_all_counters() + ), + ) + # endregion + + + # region update_subscription_example_0 + store.subscriptions.update(SubscriptionUpdateOptions(name="My subscription", query="From Orders")) + # endregion + + + # region update_subscription_example_1 + my_subscription = store.subscriptions.get_subscription_state("my subscription") + + subscription_id = my_subscription.subscription_id + + store.subscriptions.update(SubscriptionUpdateOptions(key=subscription_id, name="new name")) + + # endregion + """ + # region interface_subscription_deletion + def delete(self, name: str, database: Optional[str] = None) -> None: ... + + # endregion + + # region interface_connection_dropping + def drop_connection(self, name: str, database: Optional[str] = None) -> None: ... + + # endregion + + # region interface_subscription_enabling + def enable(self, name: str, database: Optional[str] = None) -> None: ... + + # endregion + + # region interface_subscription_disabling + def disable(self, name: str, database: Optional[str] = None) -> None: ... + + # endregion + + # region interface_subscription_state + def get_subscription_state( + self, subscription_name: str, database: Optional[str] = None + ) -> SubscriptionState: ... + + # endregion + + subscription_name = "" + """ + # region subscription_enabling + store.subscriptions.enable(subscription_name) + # endregion + + # region subscription_disabling + store.subscriptions.disable(subscription_name) + # endregion + # region subscription_deletion + store.subscriptions.delete(subscription_name) + # endregion + + # region connection_dropping + store.subscriptions.drop_connection(subscription_name) + # endregion + + # region subscription_state + store.subscriptions.get_subscription_state(subscription_name) + # endregion + + + # region subscription_open_simple + subscription_worker = store.subscriptions.get_subscription_worker_by_name(subscription_name, Order) + # endregion + + # region subscription_run_simple + subscription_runtime_task = subscription_worker.run( + process_documents=lambda batch: ... + ) # Pass your method that takes SubscriptionBatch[_T] as an argument, with your logic in it + # endregion + + # region subscription_worker_with_batch_size + worker_w_batch = store.subscriptions.get_subscription_worker( + SubscriptionWorkerOptions(subscription_name, max_docs_per_batch=20), Order + ) + + _ = worker_w_batch.run( + process_documents=lambda batch: ... + ) # Pass your method that takes SubscriptionBatch[_T] as an argument, with your logic in it + + # endregion + + # region throw_during_user_logic + def _throw_exception(batch: SubscriptionBatch): + raise Exception() + + _ = worker_w_batch.run(_throw_exception) + # endregion + + logger = logging.Logger("my_logger") + + class UnsupportedCompanyException(Exception): ... + + process_order = lambda x: ... + subscription_name = all_name + # region reconnecting_client + while True: + options = SubscriptionWorkerOptions(subscription_name) + + # here we configure that we allow a down time of up to 2 hours, and will wait for 2 minutes for reconnecting + options.max_erroneous_period = timedelta(hours=2) + options.time_to_wait_before_connection_retry = timedelta(minutes=2) + + subscription_worker = store.subscriptions.get_subscription_worker(options, Order) + + try: + # here we are able to be informed of any exceptions that happens during processing + subscription_worker.add_on_subscription_connection_retry( + lambda exception: logger.error( + f"Error during subscription processing: {subscription_name}", exc_info=exception + ) + ) + + def _process_documents_callback(batch: SubscriptionBatch[Order]): + for item in batch.items: + # we want to force close the subscription processing in that case + # and let the external code decide what to do with that + if item.result.company == "companies/832-A": + raise UnsupportedCompanyException( + "Company Id can't be 'companies/832-A', you must fix this" + ) + process_order(item.result) + + # Run will complete normally if you have disposed the subscription + return + + # Pass the callback to worker.run() + subscription_worker.run(_process_documents_callback) + + except Exception as e: + logger.error(f"Failure in subscription: {subscription_name}", exc_info=e) + exception_type = type(e) + if ( + exception_type is DatabaseDoesNotExistException + or exception_type is SubscriptionDoesNotExistException + or exception_type is SubscriptionInvalidStateException + or exception_type is AuthorizationException + ): + raise # not recoverable + + if exception_type is SubscriptionClosedException: + # closed explicitely by admin, probably + return + + if exception_type is SubscriberErrorException: + # for UnsupportedCompanyException type, we want to throw an exception, otherwise + # we continue processing + if e.args[1] is not None and type(e.args[1]) is UnsupportedCompanyException: + raise + + continue + + # handle this depending on subscription + # open strategy (discussed later) + if e is SubscriptionInUseException: + continue + + return + finally: + subscription_worker.close(False) + # endregion + + # region worker_timeout + options = SubscriptionWorkerOptions(subscription_name) + + subscription_worker = store.subscriptions.get_subscription_worker(options, Order) + + try: + subscription_worker.add_on_subscription_connection_retry( + lambda exception: logger.error( + f"Error during subscription processing: {subscription_name}", exc_info=exception + ) + ) + + subscription_worker.run(lambda batch: [... for item in batch.items]) + + # Run will complete normally if you have disposed the subscription + return + except Exception as e: + logger.error(f"Error during subscription process: {subscription_name}", exc_info=e) + finally: + subscription_worker.__exit__() + # endregion + """ + subs_id = store.subscriptions.create_for_class( + Order, + SubscriptionCreationOptions( + query=""" + declare function getOrderLinesTotal(doc){ + var total = 0; + for (var i in doc.Lines) { + total += (doc.Lines[i].PricePerUnit * doc.Lines[i].Quantity); + } + return total; + } + + From Orders as o + Where getOrderLinesTotal(o) > 10000 + Select + { + order_id: id(o), + company: this.LoadDocument(o.Company, "Companies") + } + """ + ), + ) + + class OrderAndCompany: + def __init__(self, order_id: str = None, company: Company = None): + self.order_id = order_id + self.company = company + + @classmethod + def from_json(cls, json_dict: Dict[str, Any]): + return cls(json_dict["order_id"], Company.from_json(json_dict["company"])) + + send_thank_you_note_to_employee = lambda x: ... + """ + # region single_run + high_value_orders_worker = store.subscriptions.get_subscription_worker( + SubscriptionWorkerOptions( + subs_id, + # Here we ask the worker to stop when there are no documents left to send. + # Will throw SubscriptionClosedException when it finishes its job + close_when_no_docs_left=True, + ), + OrderAndCompany, + ) + + try: + + def _subscription_batch_callback(batch: SubscriptionBatch[OrderAndCompany]): + for item in batch.items: + send_thank_you_note_to_employee(item.result) + + high_value_orders_worker.run(_subscription_batch_callback) + except SubscriptionClosedException: + # that's expected + ... + # endregion + """ + raise_notification = lambda x: ... + # region dynamic_worker + subscription_name = "My dynamic subscription" + store.subscriptions.create_for_class( + Order, + SubscriptionCreationOptions( + subscription_name, + query=""" + From Orders as o + Select + { + dynamic_field_1: "Company: " + o.Company + " Employee: " + o.Employee, + } + """, + ), + ) + + subscription_worker = store.subscriptions.get_subscription_worker_by_name(subscription_name) + + def _raise_notification_callback(batch: SubscriptionBatch[Order]): + for item in batch.items: + raise_notification(item.result.dynamic_field_1) + + _ = subscription_worker.run(_raise_notification_callback) + + # endregion + transfer_order_to_shipment_company = lambda x: ... + # region subscription_with_open_session_usage + subscription_name = store.subscriptions.create_for_options( + SubscriptionCreationOptions(query="from Orders as o where o.ShippedAt = null") + ) + + subscription_worker = store.subscriptions.get_subscription_worker_by_name(subscription_name, Order) + + def _transfer_order_callback(batch: SubscriptionBatch[Order]): + with batch.open_session() as session: + for order in (item.result for item in batch.items): + transfer_order_to_shipment_company(order) + order.shipped_at = datetime.utcnow() + + # we know that we have at least one order to ship, + # because the subscription query above has that in it's WHERE clause + session.save_changes() + + _ = subscription_worker.run(_transfer_order_callback) + # endregion + + # region waitforfree + worker = store.subscriptions.get_subscription_worker( + SubscriptionWorkerOptions(subscription_name, strategy=SubscriptionOpeningStrategy.WAIT_FOR_FREE), Order + ) + # endregion + """ + # region waiting_subscription_1 + primary_worker = store.subscriptions.get_subscription_worker(SubscriptionWorkerOptions(subscription_name, strategy=SubscriptionOpeningStrategy.TAKE_OVER), Order) + + while True: + try: + run_future = primary_worker.run(lambda batch: ...) # your logic + except Exception: + ... # retry + # endregion + + # region waiting_subscription_2 + secondary_worker = store.subscriptions.get_subscription_worker(SubscriptionWorkerOptions(subscription_name), strategy=SubscriptionOpeningStrategy.WAIT_FOR_FREE) + + while True: + try: + run_future = secondary_worker.run(lambda batch: ...) # your logic + except Exception: + ... # retry + # endregion + """ + + # region Item_definition + class Item(Generic[_T_Item]): + """ + Represents a single item in a subscription batch results. + This class should be used only inside the subscription's run delegate, + using it outside this scope might cause unexpected behavior. + """ + # endregion + + # region number_of_items_in_batch_definition + def number_of_items_in_batch(self) -> int: + return 0 if self.items is None else len(self.items) + # endregion + + # region SubscriptionBatch_definition + class SubscriptionBatch(Generic[_T]): + + def __init__(self): + self._result: Optional[_T_Item] = None + self._exception_message: Optional[str] = None + self._key: Optional[str] = None + self._change_vector: Optional[str] = None + self._projection: Optional[bool] = None + self._revision: Optional[bool] = None + self.raw_result: Optional[Dict] = None + self.raw_metadata: Optional[Dict] = None + self._metadata: Optional[MetadataAsDictionary] = None + # endregion + + # region get_subscriptions_1 + def get_subscriptions( + self, start: int, take: int, database: Optional[str] = None + ) -> List[SubscriptionState]: ... + + # endregion + # region delete_1 + def delete(self, name: str, database: Optional[str] = None) -> None: ... + + # endregion + + # region events + self._after_acknowledgment: List[Callable[[SubscriptionBatch[_T]], None]] = [] + # endregion