Add OpenTelemetry distributed tracing

.gitignore

@@ -16,3 +16,4 @@ target/
 
 # Branch switching
 generated/
+/otel/splunk-otel-javaagent-all.jar

README.md

@@ -1,28 +1,40 @@
-# Distributed version of the Spring PetClinic Sample Application built with Spring Cloud 
+# Spring PetClinic 
 
 [![Build Status](https://travis-ci.org/spring-petclinic/spring-petclinic-microservices.svg?branch=master)](https://travis-ci.org/spring-petclinic/spring-petclinic-microservices/) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 
+Distributed version of the Sample Application built with Spring Cloud. Instrumented with the Splunk distribution of the OpenTelemetry Java Instrumentation and accompanied by OpenTelemetry Collector.
+
+
 This microservices branch was initially derived from [AngularJS version](https://github.com/spring-petclinic/spring-petclinic-angular1) to demonstrate how to split sample Spring application into [microservices](http://www.martinfowler.com/articles/microservices.html).
 To achieve that goal we use Spring Cloud Gateway, Spring Cloud Circuit Breaker, Spring Cloud Config, Spring Cloud Sleuth, Resilience4j, Micrometer 
 and the Eureka Service Discovery from the [Spring Cloud Netflix](https://github.com/spring-cloud/spring-cloud-netflix) technology stack.
 
-## Starting services locally without Docker
+## Starting the application with instrumentation
+
+In order to start the application as configured for Splunk OpenTelemetry distribution, simply run `start.sh`. The script will:
+- download the agent
+- build Docker images
+- run all microservices using Docker compose
 
-Every microservice is a Spring Boot application and can be started locally using IDE or `../mvnw spring-boot:run` command. Please note that supporting services (Config and Discovery Server) must be started before any other application (Customers, Vets, Visits and API).
-Startup of Tracing server, Admin server, Grafana and Prometheus is optional.
 If everything goes well, you can access the following services at given location:
 * Discovery Server - http://localhost:8761
 * Config Server - http://localhost:8888
 * AngularJS frontend (API Gateway) - http://localhost:8080
 * Customers, Vets and Visits Services - random port, check Eureka Dashboard 
+* Otel Collector - http://localhost:55680
 * Tracing Server (Zipkin) - http://localhost:9411/zipkin/ (we use [openzipkin](https://github.com/openzipkin/zipkin/tree/master/zipkin-server))
 * Admin Server (Spring Boot Admin) - http://localhost:9090
 * Grafana Dashboards - http://localhost:3000
 * Prometheus - http://localhost:9091
 
-You can tell Config Server to use your local Git repository by using `native` Spring profile and setting
-`GIT_REPO` environment variable, for example:
-`-Dspring.profiles.active=native -DGIT_REPO=/projects/spring-petclinic-microservices-config`
+## Downloading OpenTelemetry Java Agent
+
+Download the [latest release](https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent-all.jar)
+   of the Splunk Distribution of OpenTelemetry Java Instrumentation. For example use: 
+   ```bash
+   $ # download the newest version of the agent
+   $ sudo curl -vsSL -o /opt/splunk-otel-javaagent-all.jar 'https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent-all.jar'
+   ```
 
 ## Starting services locally with docker-compose
 In order to start entire infrastructure using Docker, you have to build images by executing `./mvnw clean install -P buildDocker` 
@@ -60,33 +72,6 @@ Our issue tracker is available here: https://github.com/spring-petclinic/spring-
 ## Database configuration
 
 In its default configuration, Petclinic uses an in-memory database (HSQLDB) which gets populated at startup with data.
-A similar setup is provided for MySql in case a persistent database configuration is needed.
-Dependency for Connector/J, the MySQL JDBC driver is already included in the `pom.xml` files.
-
-### Start a MySql database
-
-You may start a MySql database with docker:
-
-```
-docker run -e MYSQL_ROOT_PASSWORD=petclinic -e MYSQL_DATABASE=petclinic -p 3306:3306 mysql:5.7.8
-```
-or download and install the MySQL database (e.g., MySQL Community Server 5.7 GA), which can be found here: https://dev.mysql.com/downloads/
-
-### Use the Spring 'mysql' profile
-
-To use a MySQL database, you have to start 3 microservices (`visits-service`, `customers-service` and `vets-services`)
-with the `mysql` Spring profile. Add the `--spring.profiles.active=mysql` as programm argument.
-
-By default, at startup, database schema will be created and data will be populated.
-You may also manually create the PetClinic database and data by executing the `"db/mysql/{schema,data}.sql"` scripts of each 3 microservices. 
-In the `application.yml` of the [Configuration repository], set the `initialization-mode` to `never`.
-
-If you are running the microservices with Docker, you have to add the `mysql` profile into the (Dockerfile)[docker/Dockerfile]:
-```
-ENV SPRING_PROFILES_ACTIVE docker,mysql
-```
-In the `mysql section` of the `application.yml` from the [Configuration repository], you have to change 
-the host and port of your MySQL JDBC connection string. 
 
 ## Custom metrics monitoring
 

docker-compose.yml

@@ -5,82 +5,151 @@ services:
     image: springcommunity/spring-petclinic-config-server
     container_name: config-server
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=config-server
     ports:
-     - 8888:8888
+      - 8888:8888
 
   discovery-server:
     image: springcommunity/spring-petclinic-discovery-server
     container_name: discovery-server
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=discovery-server
     depends_on:
       - config-server
-    entrypoint: ["./dockerize","-wait=tcp://config-server:8888","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+    entrypoint: ["./dockerize","-wait=tcp://config-server:8888","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-     - 8761:8761
+      - 8761:8761
 
   customers-service:
     image: springcommunity/spring-petclinic-customers-service
     container_name: customers-service
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=customers-service
     depends_on:
-     - config-server
-     - discovery-server
-    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+      - config-server
+      - discovery-server
+    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-    - 8081:8081
+      - 8081:8081
 
   visits-service:
     image: springcommunity/spring-petclinic-visits-service
     container_name: visits-service
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=visits-service
     depends_on:
-     - config-server
-     - discovery-server
-    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+      - config-server
+      - discovery-server
+    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-     - 8082:8082
+      - 8082:8082
 
   vets-service:
     image: springcommunity/spring-petclinic-vets-service
     container_name: vets-service
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=vets-service
     depends_on:
-     - config-server
-     - discovery-server
-    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+      - config-server
+      - discovery-server
+    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-     - 8083:8083
+      - 8083:8083
 
   api-gateway:
     image: springcommunity/spring-petclinic-api-gateway
     container_name: api-gateway
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=api-gateway
     depends_on:
-     - config-server
-     - discovery-server
-    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+      - config-server
+      - discovery-server
+    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-     - 8080:8080
+      - 8080:8080
 
   tracing-server:
-    image: openzipkin/zipkin
+    image: otel/opentelemetry-collector
     container_name: tracing-server
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    depends_on:
+      - zipkin-server
+    command: --config /otel/collector.yaml
+    ports:
+      - 55679:55679
+      - 55680:55680
+
+  zipkin-server:
+    image: openzipkin/zipkin
+    container_name: zipkin-server
+    mem_limit: 512M
     environment:
-    - JAVA_OPTS=-XX:+UnlockExperimentalVMOptions -Djava.security.egd=file:/dev/./urandom
+      - JAVA_OPTS=-XX:+UnlockExperimentalVMOptions -Djava.security.egd=file:/dev/./urandom
     ports:
-     - 9411:9411
+      - 9411:9411
 
   admin-server:
     image: springcommunity/spring-petclinic-admin-server
     container_name: admin-server
     mem_limit: 512M
+    volumes:
+      - ./otel:/otel
+    environment:
+      JAVA_TOOL_OPTIONS: -javaagent:/otel/splunk-otel-javaagent-all.jar
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_METRICS_EXPORTER: none
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tracing-server:55680
+      OTEL_RESOURCE_ATTRIBUTES: service.name=admin-server
     depends_on:
-     - config-server
-     - discovery-server
-    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+      - config-server
+      - discovery-server
+    entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=300s","--","java", "-jar","/app.jar"]
     ports:
-     - 9090:9090
+      - 9090:9090
 
   ## Grafana / Prometheus
 
@@ -89,11 +158,11 @@ services:
     container_name: grafana-server
     mem_limit: 256M
     ports:
-    - 3000:3000
+      - 3000:3000
 
   prometheus-server:
     build: ./docker/prometheus
     container_name: prometheus-server
     mem_limit: 256M
     ports:
-    - 9091:9090
+      - 9091:9090

docker/Dockerfile

@@ -16,4 +16,10 @@ EXPOSE ${EXPOSED_PORT}
 
 ENV SPRING_PROFILES_ACTIVE docker
 
-ENTRYPOINT ["java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
+ENTRYPOINT ["java", \
+            "-XX:+UnlockExperimentalVMOptions", \
+            "-XX:+UseCGroupMemoryLimitForHeap", \
+            "-Djava.security.egd=file:/dev/./urandom", \
+            "-jar", \
+            "/app.jar" \
+]

otel/collector.yaml

@@ -0,0 +1,33 @@
+extensions:
+  health_check:
+  zpages:
+    endpoint: 0.0.0.0:55679
+
+receivers:
+  otlp:
+    protocols:
+        grpc:
+
+processors:
+  batch:
+  queued_retry:
+  attributes/newenvironment:
+    actions:
+    - key: environment
+      value: "petclinic"
+      action: insert
+
+exporters:
+  logging:
+    logLevel: debug
+  zipkin:
+    endpoint: "http://zipkin-server:9411/api/v2/spans"
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch, attributes/newenvironment, queued_retry]
+      exporters: [logging, zipkin]
+
+  extensions: [health_check, zpages]

spring-petclinic-customers-service/src/main/resources/application.properties

@@ -0,0 +1,4 @@
+spring.datasource.schema=classpath*:db/hsqldb/schema.sql
+spring.datasource.data=classpath*:db/hsqldb/data.sql
+spring.datasource.driver-class-name=org.hsqldb.jdbc.JDBCDriver
+spring.datasource.url=jdbc:hsqldb:mem:testdb;DB_CLOSE_DELAY=-1

spring-petclinic-vets-service/src/main/resources/application.properties

@@ -1,2 +1,6 @@
 spring.profiles.active=production
 spring.cache.cache-names=vets
+spring.datasource.schema=classpath*:db/hsqldb/schema.sql
+spring.datasource.data=classpath*:db/hsqldb/data.sql
+spring.datasource.driver-class-name=org.hsqldb.jdbc.JDBCDriver
+spring.datasource.url=jdbc:hsqldb:mem:testdb;DB_CLOSE_DELAY=-1

spring-petclinic-visits-service/src/main/resources/application.properties

@@ -0,0 +1,4 @@
+spring.datasource.schema=classpath*:db/hsqldb/schema.sql
+spring.datasource.data=classpath*:db/hsqldb/data.sql
+spring.datasource.driver-class-name=org.hsqldb.jdbc.JDBCDriver
+spring.datasource.url=jdbc:hsqldb:mem:testdb;DB_CLOSE_DELAY=-1

start-instrumented.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+curl -vsSL -o ./otel/splunk-otel-javaagent-all.jar 'https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent-all.jar'
+./mvnw clean install -P buildDocker
+docker-compose up