Disclaimer: TIPSY is currently heavily work in progress!
TIPSY is a benchmark suite to evaluate and compare the performance of programmable data plane technologies and network-function virtualization frameworks over a set of standard scenarios rooted in telecommunications practice. Apart from simple L2 and L3 pipelines, currently there is a rather complex BNG (Broadband Network Gateway) and 5G MGW (Mobile Gateway) pipeline defined and implemented in TIPSY, with further pipelines and implementations to follow soon.
The aim of TIPSY is to provide the networking community a set of standardized telco-oriented scenarios on top of which different programmable data plane technologies can be fairly and comprehensibly evaluated. The target audience is network operators who want to test new data-plane equipment, network engineers evaluating the scalability of a programmable switch in terms of increasingly complex configurations, or researchers who want to compare a new data plane algorithm with existing and established technology.
TIPSY comprises 6 elemental parts, currently existing at varying levels of maturity:
- a set of telco pipelines with working implementations (currently OpenFlow and BESS are supported, contributions are welcome),
- a test suite to validate the implementations (ongoing),
- a configuration system that allows to fine-tune general parameters of the pipelines (supported),
- a tunable trace generator to produce deterministic traffic traces for repeatable experiments and reproducible results (supported),
- a distributed measurement infrastructure to feed the traffic traces to the system-under-test, controller code to drive dynamic benchmarks, and an evaluation framework that visualizes the results (rudimentary),
- an evaluation and visualization framework to generate production-quality reports from the benchmark results (planned).
- A collection of pipelines that model real telco use cases: the intention for TIPSY is to serve as the “de facto” data-plane benchmarking suite for practitioners and researchers.
- A measurement support framework: TIPSY will automatically generate detailed configs, working pipelines, and complete traffic traces to start testing right away.
- A tool to perform repeatable experiments: TIPSY devotes huge emphasis to support reproducibility; it automatically generates deterministic traffic traces, supports multiple traffic-dropping policies to never lose packets, etc.
- A tool to present benchmark results in a comprehensible form, like diagrams, charts, and tables, and a flexible way to configure and fine-tune these visualizations (TODO).
- A standalone automatic measurement system: TIPSY will never be able to setup the measurement completely unattended and there are certain steps, like connecting the SUT and the tester, that will need to be done manually anyway; the intention is to minimize manual intervention as much as possible.
- An exhaustive collection of telco pipelines (contributions are welcome) or canonical implementations of the ones supported (of course, there is more that one way to do it).
- An Internet scale performance measurement framework: the TIPSY reference configuration is a single device-under-test connected back-to-back to a traffic generator/measurement device; benchmarking multi-hop setups or complex network topologies are out of scope for TIPSY.
You can find a comprehensive list of alternative data plane benchmark suites and lots of background on data plane testing in the below paper.
Matthias Holdorf, How-To Compare Performance of Data Plane Devices, Proceedings of the Seminars Future Internet (FI) and Innovative Internet Technologies and Mobile Communications (IITM), 2016.
The general TIPSY setup will contain a System-Under-Test (SUT) that runs the data plane to be evaluated, a Tester that feeds the SUT with a traffic trace and measures performance, and a Controller running on the SUT that sets up the static pipeline and, optionally, exercises the performance of the SUT under updates. Drawing from telco practice, TIPSY distinguishes between the uplink direction (user-to-network direction) and the downlink direction (network-to-user direction); the downlink port is the attachment port for the access network (users) and the uplink port connects to the public Internet. In pratice, however, the uplink and downlink ports of the SUT are connected back-to-back to the same Tester device. A management access is also required between the two to let the Tester configure the SUT via Secure Shell.
+----------------+ +---------------------+ | | | | | | | | | uplink_port|<----------->|downlink_port | | | | | | | | | | *Tester* | | *System Under Test* | | | | (SUT) | | | | + | | | | *Controller* | | | | | | | | | | downlink_port|<----------->|uplink_port | | | | | | | | | | |<---mgmt---->| | +----------------+ +---------------------+
Apart from static pipeline configurations, aimed for evaluating the SUT in steady state, TIPSY also contains additional scenarios to benchmark the SUT in dynamic workloads, i.e., when the control plane updates the data plane program at various (and configurable) intensities. The implementations that come with TIPSY pipelines contain code to feed the updates, via the Controller module, into the SUT.
The below table provides an overview of TIPSY pipelines along with the features each one uses and the current availability of an implementation in the TIPSY code.
| Encap/Decap | ParseField | SetField | RateLimit | Firewall | NAT | Backends | |
|---|---|---|---|---|---|---|---|
| Port forward (PORTfwd) | - | - | - | - | - | - | ovs, bess, erfs, lagopus, ofdpa, t4p4s, vpp |
| L2 forward (L2fwd) | - | L2 | - | - | - | - | ovs, bess, erfs, lagopus, ofdpa, t4p4s |
| L3 forward (L3fwd) | - | L2/L3 | L2/L3 | - | - | - | ovs, bess, erfs, lagopus, ofdpa, t4p4s, vpp |
| Encap/Decap | VXLAN | - | L2/L3 | - | - | - | |
| RateLimit | - | - | - | x | - | - | |
| Firewall | - | L2/L3/L4 | - | - | x | - | ovs, bess, erfs, lagopus |
| NAT | - | L2/L3/L4 | L2/L3/L4 | - | - | x | ovs |
| Data Center GW (DCGW) | VXLAN | L2/L3 | L2/L3 | - | - | x | |
| Mobile GW (MGW) | GTP | L2/L3/L4 | L2/L3 | x | - | - | ovs, bess, erfs, lagopus |
| Broadband Network GW (BNG) | GRE | L2/L3/L4 | L2/L3 | x | x | x | ovs, bess, erfs, lagopus |
TIPSY depends on external software components. To run TIPSY, it is necessary to install the following software:
On SUT:
- sudo,
- ssh,
- screen.
On Tester:
- make,
- ssh,
- sudo,
- python-jsonschema,
- matplotlib,
- pdflatex,
- scapy.
TIPSY does not require explicit installation but the tipsy executable
must always be available and executable; the easiest setup is to add the
TIPSY main directory to the PATH.
git clone https://github.com/hsnlab/tipsy
cd tipsy
export PATH=$PWD:$PATHApply the following patch to timestamping.lua of MoonGen, otherwise this program cannot set the correct packet sizes, and might not be able to read back timestamps from received packets.
diff --git a/lua/timestamping.lua b/lua/timestamping.lua
index bd6c81f..ee68513 100644
--- a/lua/timestamping.lua
+++ b/lua/timestamping.lua
@@ -87,7 +87,7 @@ function timestamper:measureLatency(pktSize, packetModifier, maxWait)
-- change timestamped UDP port as each packet may be on a different port
self.rxQueue:enableTimestamps(buf:getUdpPacket().udp:getDstPort())
end
- buf:getUdpPtpPacket():setLength(pktSize)
+ -- buf:getUdpPtpPacket():setLength(pktSize)
self.txBufs:offloadUdpChecksums()
if self.rxQueue.dev.reconfigureUdpTimestampFilter and not skipReconfigure then
-- i40e driver fdir filters are brokenInput to TIPSY is a high-level description of the intended measurements and the measuring environment. The environment configuration involves the paramaters of the Tester (traffic generator, runtime of the measurement, etc.) and the SUT (pipeline implementation, custom setup/teardown scripts, etc.).
Descriptions of the measurements are the name and other general parameters (number of users, number of Internet routes, etc.) of the pipeline to be coded into the SUT. The high-level configurations are later transformed into a set of configurations/data-plane programs that can be loaded into the SUT and a set of traffic traces for each config.
A detailed TIPSY configuration guide can be found here.
For the pipeline specific parameters, see the pipeline’s
documentation. Below is a sample configuration snippet that defines a
benchmark on the Mobile Gateway (MGW) (mgw) pipeline,
with pipeline-specific settings user (number of users) and bst (number
of base stations) as set in the pipeline section.
{
"benchmark":
[
{
"id": "my_benchmark",
"scale": "joint",
"pipeline": {
"name": "mgw",
"user": [1,2],
"bst": [5,10]
},
"traffic": {
"pkt-num": 10000,
"pkt-size": [64, 128],
"dir": ["uplink"]
},
"sut": {
"type": "bess"
},
"tester": {
"type": "moongen",
"test-time": 30
}
}
],The id parameter sets a name for the benchmark and scale describes the
way the individual benchmark instances in the scalability benchmark are to
be executed. TIPSY allows to easily request and perform scalability tests
by repeating the benchmark multiple times, each time setting one or all
parameters as controlled by the scale setting:
none: do not perform scalability tests (default),outer: take the outer product of all settings specified for the benchmark and generate a separate test case for all,joint: scale the parameters jointly.
In the above example scale is set to joint, which tells TIPSY to scale
the parameters specified as lists in the config (user and bst)
jointly, that is, take the first setting in the list for each parameter,
then the second, etc., and generate a test for each such tuple. In the
above example, this will result in two tests to be run, one when user is
set to 1 and bst is set to 5 (the first elements of the lists), and one
when user is set to 2 and bst is 10 (the second elements of the lists).
If scale is set to outer, then a separate test will be run for each
combination of the multi-valued settings (user and bst), i.e., we get 4
tests, first setting the (user, bst) tuple to (1,5), then to (1,10),
then to (2,5), and finally to (2,10).
Setting scale to none ignores parameter lists and generates a single
benchmark for the first scalar for each argument.
The traffic section sets the test traffic parameters. TIPSY calculates
the outer product of the scaled pipeline configurations and the traffic
configurations in order to measure each scenario with all types of test
traffic.
The sut and the tester section contains parameters of the SUT and
the Tester respectively.
In continuation of the example, the following snippet presents the
default section of the TIPSY configuration. It can contain the same
parameters as the benchmarks. The purpose of the default section is
to simplify the definition of recurring parameters of benchmarks.
Further details are in the detailed TIPSY configuration guide.
"default":
{
"id": "my_benchmark",
"scale": "joint",
"pipeline": {
"name": "mgw",
"user": [1,2],
"bst": [5,10]
},
"traffic": {
"pkt-num": 10000,
"pkt-size": [64, 128],
"dir": ["uplink"]
},
"sut": {
"type": "bess"
},
"tester": {
"type": "moongen",
"test-time": 30
}
}
}For generating an empty configuration with the default setting for each configurable parameter, use:
tipsy init <pipeline>You may then start to edit the resultant JSON configuration accordingly.
The first step to run a benchmark is to create a root directory that will
contain all files (configurations, traffic traces, data plane configs,
results, etc.) associated with the benchmark and to write the main JSON
configuration. Then, to actually run the benchmark, simply issue the
necessary TIPSY commands executing the benchmark driver tipsy in the
benchmark root directory with different command line arguments.
mkdir my_benchmark
cd my_benchmark
tipsy init <pipeline_name>
<edit TIPSY benchmark configuration JSON file>
tipsy config
make
tipsy cleanThe basic workflow is described in more details here.
You can make the output a bit more readable by installing
module/openflow/color_log.py, read the header of
module/openflow/color_log.py for how to do that.
TIPSY is a free software and licensed under GPLv3+.