This repository provides XML code examples that use SAS Event Stream Processing to process real-time streaming data. Project code is written to run in SAS Event Stream Processing Studio. Use these examples with SAS Event Stream Processing 2020.1 and later, unless stated otherwise.
To obtain XML code examples that run with earlier releases of SAS Event Stream Processing, visit the SAS Event Stream Processing Code Examples download page.
Beginning with SAS Event Stream Processing 2023.09, you can install some examples directly from SAS Event Stream Processing Studio:
- In SAS Event Stream Processing Studio, do one the following:
- On the Projects page, click View example projects.
- On any page, click
and select Examples.
- In the SAS Event Stream Processing Examples window, select an example with the Install example button in the right pane.
- Click Install example. A project package ZIP file is imported from the relevant example's directory in this GitHub. The project XML file in the project package has connectors that point to input files in the product package. As a result, you do not need to configure connectors.
- Run the imported example in test mode. For more information, see Running a Test.
For more information about how the example works, return to the SAS Event Stream Processing Examples window, select the example and click View README in GitHub. Beginning in SAS Event Stream Processing 2024.08, when you have installed an example, you can also access its README file from the Project Package pane. When you are viewing an example project in SAS Event Stream Processing Studio, you can also select each window in turn in the workspace to read brief descriptions of those windows in the right pane.
When the Install example button is not available but the example's directory contains a project package ZIP file, you can manually import the project package to SAS Event Stream Processing Studio, but only when you are using SAS Event Stream Processing 2023.09 or later. For more information, see Import a Project.
For more information about project packages in general, see Project Package.
There are some circumstances when you might need to import examples manually:
- You are using SAS Event Stream Processing 2023.08 or earlier.
- You are using SAS Event Stream Processing 2023.09 or later, but the functionality for project packages is not enabled in your environment.
- An example's directory does not include a project package ZIP file.
To import an example manually:
- Save the contents of the example's directory to a convenient location on your computer.
- Import the project XML file to SAS Event Stream Processing Studio. For more information, see Import a Project.
- Open the project in SAS Event Stream Processing Studio and examine its contents. You can select each window in turn in the workspace to read brief descriptions of those windows in the right pane.
- To pass events from input files into the project, do one of the following:
- Configure connectors: Place the input files on the Kubernetes persistent volume and adjust any existing connectors to point to the location of the input files. If the project does not contain any connectors, you must add connectors yourself. For more information, see Configure a File and Socket Connector.
- Publish events directly from an input file: Deactivate any existing connectors. After you run the project in test mode (see step 5), use the Publish button to send events from the input files into Source windows. Using the Publish button enables you to send events into a running project without placing files on the Kubernetes persistent volume and without configuring connectors. For more information, see Publish Events from a CSV file.
- Run the project in test mode and view the test results. For more information, see Running a Test.
Note:
- You cannot use the Publish button to publish events to multiple Source windows simultaneously. For projects that contain multiple Source windows, you must publish events for each Source window separately.
- When connector orchestration is used (for example, in the
geofence,join,sailing, andunionexamples), it is not appropriate to use the Publish button. Instead, configure connectors for the project.
The GettingStarted directory contains a set of projects to help you get started quickly. Window code contains a description that describes its purpose.
| Example | Description |
|---|---|
| aggregate | Apply aggregation functions to a group of stock market trades. When new events arrive in the Aggregate window, they are placed into aggregate groups. After a new event enters the event stream, the aggregation functions are called. The functions then modify the aggregation groups accordingly. Aggregate field calculation functions or expressions that are registered to the Aggregate window must appear in non-key fields, which in this project are totalQuant and maxPrice. |
| compute | Use a Compute window to match people from the city of Apex. You can use Compute windows to project input fields from one event to a new event and to augment this event with fields that result from a calculation. |
| fitstat | Use the FitStat (fit statistics) algorithm. The goodness-of-fit of a statistical model describes how well a model fits a set of data. The measures summarize the difference between observed values and predicted values of the model under consideration. |
| join | Join stock market trades with the corresponding traders. A Join window receives events from a left input window and a right input window. It produces a single output stream of joined events. Joined events are created according to a join type and join conditions. |
| splitter | Use a splitter. You can use expressions to define window-output splitter-slot calculations (for example, you can use an expression to evaluate where to send a generated event). In this project, the splitter works with user-defined functions to calculate the slot number to decide which Copy window the event goes to. |
The Basic directory contains projects that demonstrate how to use a few windows. See also GettingStarted.
| Example | Description |
|---|---|
| lua_compute | Use Lua code to modify user data. The Lua window in this example is used as an alternative to a Compute window. Note: Use this example with SAS Event Stream Processing 2021.2.1 and later. |
| lua_connector | Use a Lua connector. This example contains a Source window with a Lua connector that reads RSS feeds from scientific sites and publishes them into the project. Note: Use this example with SAS Event Stream Processing 2023.04 and later. |
| lua_module | Use a Lua module. This example demonstrates how you can define useful functions in a Lua module at project level and reference those functions from other Lua code in the project. Note: Use this example with SAS Event Stream Processing 2023.12 and later. |
| lua_parse | Use Lua code to parse JSON data and generate multiple events. This example contains two projects. The Lua window in these projects is used as an alternative to a Functional window. The Basic Example Project contains step-by-step instructions about how to run the project. The Advanced Example Project is for demonstration purposes. Note: Use these examples with SAS Event Stream Processing 2022.10 and later. |
| lua_pattern | Use Lua code to identify increases in a stock's price over time. The Pattern window in this example uses Lua code to define events of interest (EOI) to be matched. Note: Use this example with SAS Event Stream Processing 2024.03 and later. |
| lua_state | Use Lua code to maintain and use event state to generate events. The Lua window in this project generates events every time a new stock symbol price exceeds the current maximum price for that stock symbol. Note: Use these examples with SAS Event Stream Processing 2022.10 and later. |
| python_compute | Use Python code to modify user data. The Python window in this example is used as an alternative to a Compute window. Note: Use this example with SAS Event Stream Processing 2024.03 and later. |
| python_connector | Use a Python connector. This example contains a Source window with a Python connector that calls a JSON API to read Star Wars data and publish that data into the project every five seconds. Note: Use this example with SAS Event Stream Processing 2023.12 and later. |
| python_module | Use a Python module. This example demonstrates how you can define useful functions in a Python module at project level and reference those functions from other Python code in the project. Note: Use this example with SAS Event Stream Processing 2024.02 and later. |
| python_pattern | Use Python code to identify decreases in a stock's price over time. The Pattern window in this example uses Python code to define events of interest (EOI) to be matched. Note: Use this example with SAS Event Stream Processing 2024.03 and later. |
| python_snippet | Use a Python snippet. This example provides a simple demonstration of how to define a Python snippet and reference it. Note: Use this example with SAS Event Stream Processing 2024.02 and later. |
| removestate | Transition a model from stateful to stateless. This example demonstrates how to facilitate the transition of a stateful part of a model to a stateless part of a model. |
| union | Merge three event streams of stock market trades together. A Union window unites two or more event streams using a strict policy or a loose policy. |
The Advanced directory contains projects that demonstrate the use of a larger number of windows together, or that are otherwise more complex than examples in the Basic or GettingStarted directories.
| Example | Description |
|---|---|
| activitytracker | Use a job template to deploy a project to an edge server. This example processes GPS data from activity tracker devices worn by players in a football match. The example demonstrates how you can publish a project in SAS Event Stream Processing Studio to make it available in SAS Event Stream Manager, and then use a job template to deploy the project to an edge server. |
| geofence | Display a list of wanted vehicles found near critical infrastructure sites. This example uses Join, Geofence, and Filter windows to match wanted vehicles that are in close proximity to critical infrastructure sites. Note: Use this example with SAS Event Stream Processing 2024.01 and later. |
| lua_snippet | Use a Lua snippet. This example simulates health-care data streaming into a model and detects measurements that fall outside a specified range. A Lua snippet in the project subscribes to windows, stores values, and reads a JSON file. Lua connectors are also used to generate and inject data. Note: Use this example with SAS Event Stream Processing 2023.04 and later. |
| onnx_object_detection | Use an ONNX model to detect objects in an incoming video stream. This example demonstrates how you can use an ONNX model to detect objects in an incoming video stream. Post-processing by a Python program converts scoring output to a more usable format. Note: Use this example with SAS Event Stream Processing 2024.04 and later. |
| onnx_pose_estimation | Use an ONNX model to detect keypoints of a person in an incoming video stream. This example uses pose estimation, which is a computer vision technique for recognizing and categorizing the positions of a human body. This example builds on the onnx_object_detection example. Note: Use this example with SAS Event Stream Processing 2024.09 and later. |
| sailing | Visualize data obtained from a set of boats. This example identifies two geographical areas of interest. One area is an exclusion zone that the boats identified are not permitted to enter. The other is an area where a speed restriction has been applied. |
| trades | Identify large trades in stock market transactions. This project identifies large securities transactions and the traders who were involved in those trades. |
| transpose_long | Transpose data from an aircraft, in long mode. This project conceptualizes an event as a row that consists of multiple columns. You can use a Transpose window to interchange an event's rows as columns, and columns as rows. You will process information about the pitch, yaw, roll, and velocity of an aircraft in flight. |
| transpose_wide | Transpose data from an aircraft, in wide mode. This project conceptualizes an event as a row that consists of multiple columns. You can use a Transpose window to interchange an event's rows as columns, and columns as rows. You will process information about the pitch, yaw, roll, and velocity of an aircraft in flight. |
The Analytics directory contains projects that show how to use various algorithms that are packaged with SAS Event Stream Processing.
| Example | Description |
|---|---|
| analytics_cepstrum | Use the Cepstrum Transform (Cepstrum) algorithm. A cepstrum results from taking the inverse Fourier transform of the logarithm of the estimated spectrum of a signal. This application is often used in speech analysis, specifically with voice detection. |
| analytics_changedetection | Use the Change Detection (ChangeDetection) algorithm. With change detection, a stream of measures is monitored and an alert is raised when values deviate from what is expected. |
| analytics_correlation | Use the streaming Pearson's correlation algorithm. Pearson's correlation coefficient is the most common measure of how data correlate with one another. It shows the linear relationship between two sets of data. |
| analytics_dbscan | Use DBSCAN clustering. It applies an insert-only input stream that consists of the following: 1. An ID that acts as the streams key. 2. An x coordinate of data. 3. A y coordinate of data. |
| analytics_distributionfitting | Use the Distribution Fitting algorithm. The algorithm supports fitting a Weibull, Gamma, or Normal distribution to a variable in the incoming data stream. |
| analytics_fitstatistics | Use the FitStat (fit statistics) algorithm. The goodness-of-fit of a statistical model describes how well a model fits a set of data. The measure summarizes the difference between observed values and predicted values of the model under consideration. |
| analytics_kmeans | Use k-means clustering. K-means clustering is often used in data mining. The algorithm assigns data points to their nearest cluster centroid. Each cluster centroid is then recomputed based on the average of the cluster's data points. In k-means clustering, an input event is augmented with a cluster number. This indicates the cluster that the observation falls into. |
| analytics_lagmonitoring | Use the Lag Monitoring (LagMonitor) algorithm. The Lag Monitoring algorithm computes the cross-correlation between a target time series and one or more additional time series. Results contain the selected lags and computed cross-correlation values that correspond to minimum, maximum, and maximum absolute value cross-correlations for each of the variables. |
| analytics_linearregression | Use online linear regression on streaming data. This project has an insert-only input stream that consists of the following: 1. An ID that acts as the streams key. 2. A y coordinate of data. 3. 784 x coordinates. |
| analytics_logisticregression | Use online logistic regression on streaming data. This project has an insert-only input data stream that consists of the following: 1. An ID that acts as the stream's key. 2. A y coordinate of data. 3. 784 x coordinates. |
| analytics_modelReader_digitalFilter | Score data by using SODFIL.sasast. This project shows how to use a Model Reader window to facilitate the scoring of an input data set using a pre-trained analytic store model. The analytic store model is SODFIL.sasast, which is a Digital Filter analytic store of Butterworth filter type. The analytic store was created using CAS. |
| analytics_modelReader_RPCA | Score data by using RPCA.sasast. This project shows how to use a Model Reader window to facilitate the scoring of an input data set using a pre-trained analytic store model. The analytic store model is RPCA.sasast, which is a Robust Principal Components Analysis (RPCA) analytic store. The analytic store was created using CAS. |
| analytics_movingrelativerange | Use the moving relative range (MRR) algorithm. The moving relative range provides a measure of volatility for a nonstationary time series, where the mean and variance of the series changes over time. |
| analytics_ROC | Use the receiver operating curve characteristic (ROC) information algorithm. ROC information shows the diagnostic ability of a classifier system as you vary its discrimination threshold. |
| analytics_segmentedcorrelation | Use the segmented correlation algorithm. Segmented correlation is similar to autocorrelation. It specifies the correlation between the elements of a series and others from the same series that are separated from them by a specified interval. |
| analytics_STFT | Use the Short-Time Fourier Transform (STFT) algorithm. A Fourier transform decomposes a function of time into its underlying frequencies. The amplitude, offset, and rotation speed of every underlying cycle is returned by the function. STFT is commonly used to monitor the time-varying frequency content of a signal. It can be used in digital filters to detect anomalies in continuous streams of data. |
| analytics_streaminghistogram | Use the streaming histogram algorithm. The streaming histogram algorithm processes a stream of numerical data and puts it in bins to generate boundaries in order to create a histogram that fits the data. |
| analytics_subspacetracking | Use the Subspace Tracking (SST) algorithm. When a data set contains a sequence of nx1 vectors: x(t), subspace tracking estimates the covariance matrix for each vector x(t) and then computes the first p principal eigenvectors of the covariance matrix. For each iteration at time t, the covariance matrix C(t) is obtained. SST can be applied to industrial data to detect outliers. Results can be used to identify potential errors before they occur. |
| analytics_summary | Use the streaming summary statistics algorithm. The algorithm is used in a Calculate window, which calculates univariate summary statistics for input variable data that it receives from a Source window. |
| analytics_supportvectormachine | Use a support vector machine. Support vector machines are supervised learning models with associated algorithms. They apply classification and regression analysis on incoming data. You supply training examples and mark them as belonging to a category. A support vector machine builds a model that assigns new examples to that category. |
| analytics_texttokenization | Use a tokenization algorithm. This project shows how to use the tokenization algorithm to perform text tokenization on streaming input. |
| analytics_textvectorization | Use the TextVectorization algorithm. Vectorizing text creates maps from words or n-grams to vector space. A vector space is an algebraic model to represent text documents as vectors of identifiers (for example, index terms). |
| analytics_TFIDF | Use the Term Frequency - Inverse Document Frequency (TFIDF) algorithm. TFIDF can be used as a weighing factor in text mining or general information searches. |
| analytics_tSNE | Use the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm. The t-SNE algorithm is a machine learning algorithm for dimensionality reduction that is used to visualize high-dimensional data sets. |
The following examples in the Advanced directory contain practical use cases.
- activitytracker
- geofence
- onnx_object_detection
- onnx_pose_estimation
- sailing
- trades
The following examples demonstrate how you can use Lua code in projects.
- lua_compute
- lua_connector
- lua_module
- lua_parse
- lua_snippet
- lua_state
- lua_pattern
- trades
- sailing
The following examples demonstrate how you can use Python code in projects.
- python_compute
- python_connector
- python_module
- python_pattern
- python_snippet
- geofence
The following examples use computer vision techniques:
- onnx_object_detection
- onnx_pose_estimation
We welcome your contributions! Please read CONTRIBUTING.md for details on how to submit contributions to this project.
This project is licensed under the Apache 2.0 License.