MQTT Service private preview user documentation

content/device-integration/mqtt-service-bundle/faq.md

@@ -0,0 +1,16 @@
+---
+weight: 90
+layout: redirect
+title: FAQ
+---
+
+Q: How to obtain device credential?<br/>
+A: MQTT Service does not support a device bootstrap process yet. Instead, follow the [Integration life cycle](/device-integration/mqtt/#integration-life-cycle)
+to bootstrap the device and obtain device credentials. Once the device credentials are obtained, ensure that they have `Mqtt Service` `ADMIN` permission.
+You can achieve this for all devices by granting this permission to the `devices` global role.
+
+Q: Does MQTT Service support the SmartREST 2.0 protocol?<br/>
+A: Not yet, support for SmartREST 2.0 will be added in the future.
+
+Q: Why does MQTT Service not use standard MQTT ports 1883 and 8883?<br/>
+A: Those ports are already used by {{< product-c8y-iot >}} MQTT. Both endpoints are working together MQTT Service must use different ports.

content/device-integration/mqtt-service-bundle/implementation.md

@@ -0,0 +1,178 @@
+---
+weight: 20
+title: MQTT protocol implementation
+layout: redirect
+---
+
+This section lists the implementation details for the MQTT Service. The MQTT Service implementation supports MQTT Version 3.1.1, support for 5.0 is planned.
+
+### Connecting via MQTT {#connecting-via-mqtt}
+
+MQTT Service is supported via TCP. Use your tenant domain as the URL.
+
+Available ports:
+
+|   | TCP |
+|:-----|:----|
+| TLS | 9883 |
+| no TLS | 2883 |
+
+Port 9883 is enabled by default. It currently supports one-way SSL meaning that only the client validates the server certificate to ensure its identity.
+The client is authenticated by the server via standard username and password credentials.
+To enable port 2883 please contact [Product support](/additional-resources/contacting-support/).
+
+### Topic {#topic}
+
+MQTT Service topics are mapped to the Messaging Service subscriptions with identical names, including additional URL encoding.
+The Messaging Service subscriptions reliably store the topic messages for asynchronous processing.
+The messages stored on these subscriptions can be consumed using a dedicated [Java Client](/device-integration/mqtt-service#java-client).
+
+#### Topic restrictions {#topic-restrictions}
+
+MQTT Service does not impose any topic structure. There are just a few topic names which are reserved for historic purposes and future use, namely:
+* All [SmartREST 2.0](/smartrest/smartrest-two) related topics
+* `error`
+* `devicecontrol/notifications`
+
+Other than that you are free to use any topic name which is compatible with the [MQTT specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718106).
+
+{{< c8y-admon-info >}}
+Wildcard topics (`+`, `#`) and system topics starting with `$` are not supported.
+{{< /c8y-admon-info >}}
+
+#### Topic limit {#topic-limit}
+
+MQTT Service has the ability to limit the total number of topics that a single tenant can create. The current default is no limit.
+When the creation of a new topic, either by creating it via the client publishing a message or subscribing to a non-existent topic, would breach the topic limit
+the delivery of the packet is prevented.
+
+The different MQTT protocols provide the following feedback.
+
+MQTT 5 clients:
+
+* Have access to the reason code and reason string describing the failure when using QoS 1 with acknowledgements,
+reason code being `QUOTA_EXCEEDED: 0x97`.
+
+MQTT 3.1 and 3.1.1 clients:
+
+* Clients only have access to the reason code describing the failure when using QoS 1 with acknowledgements and only
+for the SUBSCRIBE packets, where the reason code is `0x80`.
+* For the PUBLISH packets, the client will be disconnected with no further information as per the MQTT specification.
+
+#### Error Topic {#error-topic}
+
+MQTT Service provides clients the ability to review errors through messages received by subscribing to the error topic, `$debug/$error`.
+When subscribing to the topic it will act as a per-client topic, meaning the client will only receive messages exclusively related to their client ID. For example,
+if a client was attempting to subscribe to a new topic, and the creation of the topic would exceed the topic limit, only that client would receive an error.
+
+According to the MQTT 3.1.1 specification, if either the server or the client encounters a protocol violation, it must close the network connection on
+which it received the control packet which caused the violation.
+
+In such instances MQTT clients must reconnect to be able to receive error messages from the error topic via the subscription. Error messages received after this reconnection
+are from the previous session. This can lead to confusion when attempting corrective actions. Therefore, we highly recommend you to build a microservice which uses
+the MQTT Service SDK to consume error messages, or use MQTT 5 for clients and make use of the reason codes feature.
+
+#### Topic cleanup {#topic-cleanup}
+
+The MQTT service will automatically remove topics which are no longer active. Topics are recognized as inactive when there are no subscriptions and
+the internal publisher to the topic is closed. The publisher is responsible for publishing the modified MQTT service messages to the correct topic.
+The publishers live within a cache, where the publisher expires after one hour. Due to this it can take up to an hour after removing all subscriptions from a topic
+for it to be automatically deleted.
+
+### Payload {#payload}
+
+The original MQTT messages are re-packed into MQTT Service message format which includes the original payload and additional metadata fields.
+Assuming Java types, the packed message structure looks as follows:
+
+`MqttServiceMessage`
+| Field name | Type                | Description                    |
+|:-----------|:--------------------|:-------------------------------|
+| payload    | byte[]              | MQTT payload                   |
+| metadata   | MqttServiceMetadata | Metadata from the MQTT message |
+
+`MqttServiceMetadata`
+| Field name             | Type    | Description                                                             |
+|:-----------------------|:--------|:------------------------------------------------------------------------|
+| clientId               | String  | Unique MQTT client identifier, usually used as an external identifier   |
+| messageId              | int     | Unique MQTT message ID per client, available only with QoS 1 and 2      |
+| dupFlag                | boolean | Indicates this message is a resend by the MQTT client                   |
+| userProperties         | Map     | Reserved for future use of MQTT 5.0 features                            |
+| payloadFormatIndicator | enum    | Reserved for future use of MQTT 5.0 features                            |
+| contentType            | String  | Reserved for future use of MQTT 5.0 features                            |
+| correlationData        | byte[]  | Reserved for future use of MQTT 5.0 features                            |
+| responseTopic          | String  | Reserved for future use of MQTT 5.0 features                            |
+| topic                  | String  | The name of the MQTT topic that the message was published by the client |
+
+The [Java Client](/device-integration/mqtt-service#java-client) contains classes representing the above model.
+
+#### Payload restrictions {#payload-restrictions}
+
+MQTT Service doesn't force you to use any specific payload format. 
+All the incoming MQTT messages must meet the specification in terms of fixed and variable headers, but the payload for published messages is unrestricted.
+Just keep in mind that you will receive exactly the same set of bytes which was sent from the device in your custom microservice
+and you have to convert them to {{< product-c8y-iot >}} compatible format.
+
+{{< c8y-admon-info >}}
+For all MQTT connections to the platform, the maximum accepted payload size is 1048576 bytes (1 MiB), which includes
+both message header and body. The header size varies, but its minimum is 2 bytes.
+{{< /c8y-admon-info >}}
+
+### Features {#features}
+
+#### Authentication and authorization {#authentication-and-authorization}
+
+Authentication types supported by MQTT Service are:
+
+*   Username and password: The MQTT username must include the tenant ID and username in the format `<tenantID>/<username>`.
+*   Device certificates: Not yet supported. This will be added in a future release.
+
+#### ClientId {#client-id}
+
+The **MQTT ClientID** field identifies the connected client. **ClientID** may consist of up to 128 alphanumeric characters.
+Each client connecting to MQTT Service must have a unique client identifier, connecting a second client with the same identifier will result in the previous client's disconnection.
+
+#### Quality of Service (QoS) {#quality-of-service-qos}
+
+The {{< product-c8y-iot >}} implementation supports two levels of MQTT QoS:
+
+* QoS 0: At most once:
+    - The client just sends the message once (fire and forget).
+    - No response from the server.
+    - No guarantee that subscribers will receive the message.
+* QoS 1: At least once:
+    - The client awaits server acknowledgment for each published message.
+    - The client should re-send the message if there was no acknowledgement from the server.
+    - It is guaranteed that subscribers will receive a message that was acknowledged by the server.
+    - Subscribers may receive more than one copy of a message.
+* QoS 2: Exactly once:
+          - not supported
+
+For subscriptions, MQTT Service will deliver all messages in the QoS that the client defined when subscribing to the topic.
+
+#### Clean session {#clean-session}
+
+MQTT Service requires clean session to be set to "1" (true). We cannot guarantee that disabling clean session will work reliably, hence we recommend you to always enable clean session.
+
+#### Retained flag {#retained-flag}
+
+Retained flag is ignored. Publishing data with the retained flag on the topic is allowed but has no practical difference to sending it without the flag.
+
+#### Last will {#last-will}
+
+In MQTT, the "last will" is a message that is specified at connection time and that is executed when the client loses the connection. 
+Last will is fully supported by MQTT Service and like with any other publish messages you can use any unreserved topic and any payload.
+
+### Return codes {#return-codes}
+
+MQTT Service follows the MQTT specification for server responses. For example, if invalid credentials are sent in the `CONNECT` message,
+the server response `CONNACK` message contains the `0x05` return code.
+The return code can be treated similarly to REST API HTTP codes, such as 401.
+
+### MQTT 5.0 features {#mqtt-50-features}
+
+Support for MQTT 5.0 features will be added in the near future.
+
+### MQTT TLS certificates {#mqtt-tls-certificates}
+
+MQTT Service uses the certificates which are assigned to the main environment domain. It always sends these certificates during TLS handshake to devices.
+Moreover, {{< enterprise-tenant >}}s are not able to customize those certificates via the SSL Management feature.

content/device-integration/mqtt-service-bundle/index.html

@@ -0,0 +1,4 @@
+---
+title: MQTT service
+headless: true
+---

content/device-integration/mqtt-service-bundle/java-client.md

@@ -0,0 +1,77 @@
+---
+weight: 30
+layout: redirect
+title: Java Client
+---
+
+The MQTT Service Java Client library provides the classes necessary to interact with MQTT Service.
+The following operations are supported by the client:
+* Publishing messages to the MQTT Service via WebSocket protocol.
+* Subscribing to messages from the MQTT Service via WebSocket protocol
+
+#### Repositories and dependencies {#repositories-and-dependencies}
+
+Follow the [Microservice SDK](/microservice-sdk/java/#add-repositories-and-dependencies) documentation for guidance on how to configure Maven repositories.
+To include MQTT Service Java Client into your project, add the following dependency inside the `<dependencies>` node:
+```xml
+<dependency>
+    <groupId>com.cumulocity.sdk.mqtt</groupId>
+    <artifactId>mqtt-service-ws</artifactId>
+    <version>${c8y.version}</version>
+</dependency>
+```
+
+#### Example {#example}
+Example of publishing messages to the MQTT Service via WebSocket:
+```java
+// Message to be sent
+final String payload = "Hello World";
+
+// Construct a new MqttServiceMessage and set the payload
+final MqttServiceMessage message = new MqttServiceMessage();
+message.setPayload(payload.getBytes());
+
+// Create an instance of MqttServiceApi by specifying the server URI to connect to along with TokenApi
+final MqttServiceApi mqttServiceApi = MqttServiceApi.webSocket()
+        .url(webSocketBaseUrl)
+        .tokenApi(tokenApi)
+        .build();
+
+// Build PublisherConfig with topic to which the message is to be sent
+final PublisherConfig config = PublisherConfig.publisherConfig().topic(topic).build();
+
+// Build Publisher and publish MqttServiceMessage. Close the resource either by using a [try-with-resources block](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) as below or by calling publisher.close() explicitly
+try (final Publisher publisher = mqttServiceApi.buildPublisher(config)) {
+    publisher.publish(message);
+} catch (Exception e) {
+    log.error("Could not sent message to {}", topic, e);
+}
+mqttServiceApi.close();
+```
+
+Example of subscribing to messages from the MQTT Service via WebSocket:
+```java
+// Create an instance of MqttServiceApi by specifying the server URI to connect to along with TokenApi
+final MqttServiceApi mqttServiceApi = MqttServiceApi.webSocket()
+        .url(webSocketBaseUrl)
+        .tokenApi(tokenApi)
+        .build();
+
+// Build SubscriberConfig with topic and subscriber name
+final SubscriberConfig config = SubscriberConfig.subscriberConfig().topic(topic).subscriber(subscriberName).build();
+
+// Build Subscriber
+final Subscriber subscriber = mqttServiceApi.buildSubscriber(config);
+
+// Subscribe by passing implementation of MessageListener to handle messages from the MQTT Service.
+subscriber.subscribe(new MessageListener() {
+    @Override
+    public void onMessage(MqttServiceMessage message) {
+        log.info("Message Received: {}", new String(message.getPayload()));
+    }
+});
+
+// Close the resources after usage
+subscriber.close();
+mqttServiceApi.close();
+```

content/device-integration/mqtt-service-bundle/overview.md

@@ -0,0 +1,64 @@
+---
+weight: 10
+layout: redirect
+title: Overview
+---
+
+{{< c8y-admon-preview >}}
+This feature is in Private Preview, that is, it is not enabled by default and maybe subject to change in the future.
+Please reach out to the [product support](/additional-resources/contacting-support/) if you are interested in this feature.
+{{< /c8y-admon-preview >}}
+
+{{< c8y-admon-req >}}
+To work with the MQTT Service, the following requirements have to be met:
+* The {{< product-c8y-iot >}} Messaging Service must be available on your {{< product-c8y-iot >}} platform.
+* MQTT Service must be enabled for your tenant.
+{{< /c8y-admon-req >}}
+
+MQTT Service is the new MQTT endpoint implementation of {{< product-c8y-iot >}} which provides the following benefits:
+
+* Support for publishing and subscribing arbitrary payloads on any MQTT topic. The topics that are used by the existing {{< product-c8y-iot >}} MQTT implementation are not permitted for use with MQTT Service.
+* Support for subscribing to the data stream from MQTT Service and mapping it to {{< product-c8y-iot >}} compatible one or routing them to a different service.
+* Multi-tenancy support: A single endpoint serves multiple tenants.
+* Full horizontal scalability.
+
+This section does not describe the basics of MQTT communication. If you are unfamiliar with MQTT, we recommend 
+you to consult one of the numerous introductions on the internet. Some references can be found on the [MQTT website](https://mqtt.org/mqtt-specification/).
+
+### Architecture {#architecture}
+
+MQTT Service works together with the Messaging Service to provide a framework for highly customizable and flexible MQTT message processing solutions.
+The diagram below illustrates how a message flows, starting from the device, through the Messaging Service, 
+then to the microservice where it is converted to the final {{< product-c8y-iot >}} REST request.
+
+![MQTT Service send](/images/mqtt-service/mqtt-service-send.svg)
+
+All MQTT messages coming to MQTT Service are forwarded to the Messaging Service, where they are persisted, waiting to be consumed.
+A custom microservice that understands the topic and payload structure can, with the help of [Java Client](/device-integration/mqtt-service#java-client), 
+consume the MQTT messages, translate them to the {{< product-c8y-iot >}} format, and then use the [Microservice SDK](/microservice-sdk/java) to push them into {{< product-c8y-iot >}}.
+
+Similarly, messages can be sent to devices, as shown in the diagram below.
+
+![MQTT Service push](/images/mqtt-service/mqtt-service-push.svg)
+
+Like with the message coming from the device the same solution can be also applied when trying to communicate with the device.
+Given the MQTT client ID and the topic, a microservice can push any MQTT message to a device via [Java Client](/device-integration/mqtt-service#java-client).
+
+### MQTT Service vs {{< product-c8y-iot >}} MQTT {#mqtt-service-vs-cumulocity-iot-mqtt}
+
+The table below presents a basic comparison between the standard {{< product-c8y-iot >}} MQTT functionality and that of MQTT Service.
+
+|                              | {{< product-c8y-iot >}} MQTT                            | MQTT Service                                                                      |
+|:-----------------------------|:--------------------------------------------------------|:----------------------------------------------------------------------------------|
+| QoS                          | 0, 1, 2                                                 | 0, 1                                                                              |
+| Clean session                | Starting with clean session is recommended              | Starting with clean session is recommended                                        |
+| Retained flag                | Not supported                                           | Not supported                                                                     |
+| Last will                    | Supported                                               | Supported                                                                         |
+| MQTT 5.0 features            | Not supported                                           | Support is planned                                                                |
+| Authentication               | Basic and device certificate                            | Basic authentication is supported, device certificate support is planned          |
+| Scalability                  | Horizontal                                              | Horizontal                                                                        |
+| Topic format                 | Determined by the SmartREST 2.0 protocol                | Unrestricted, SmartREST topic names are reserved and cannot currently be used     |
+| Payload                      | Determined by the SmartREST 2.0 protocol                | Unrestricted, maximum message size is 1048576 bytes (1 MiB) including all headers |
+| Extensibility                | Limited by SmartREST 2.0 custom templates               | Custom mapping microservices can support arbitrary MQTT-based protocols           |
+| Message processors/consumers | Build-in message processor for each SmartREST 2.0 topic | Custom mapping microservices can support multiple processors for a topic          |
+| JSON via MQTT                | Limited feature set                                     | Custom mapping microservices can support arbitrary JSON payloads                  |

content/device-integration/mqtt-service.md

@@ -0,0 +1,7 @@
+---
+weight: 26
+title: MQTT Service
+layout: bundle
+section:
+  - device_management
+---