# The MooBench Observability Overhead Micro-Benchmark The MooBench micro-benchmarks can be used to quantify the performance overhead caused by observability framework components and different observability frameworks. Observability is achieved through its three pillars: * Logs, i.e., timestamped information about system events, * Metrics, i.e., numerical measurements of system behaviour, and * Traces, i.e., representations of request, transaction or operation executions. MooBench can measure the overhead that is created by obtaining any of these three pillars of observability from program execution. Continuous measurement results are available here: * Kiel University Server (Intel Xeon CPU E5620 @ 2.40 GHz, Debian 12): https://kieker-monitoring.net/performance-benchmarks/ * GH Actions Runner (Ubuntu 26.04 -- see `.github/workflows/benchmark*.yaml` for current version): https://kieker-monitoring.github.io/moobench-data/dev/bench/ Currently (fully) supported observability frameworks are: * Elastic APM with Java (https://github.com/elastic/apm-agent-java) * inspectIT with Java (https://inspectit.rocks/) * Kieker with Java (http://kieker-monitoring.net) * Kieker with Python (http://kieker-monitoring.net) * k-perf with Kotlin * OpenTelemetry with Java (https://opentelemetry.io/) * OpenTelemetry with Python (https://opentelemetry.io/) * PinPoint with Java (https://github.com/pinpoint-apm/pinpoint) * Scouter with Java (https://github.com/scouter-project/scouter) * Skywalking with Java (https://skywalking.apache.org/) For all combinations of supported observability frameworks $FRAMEWORK and languages $LANGUAGE, the folder `frameworks` contains a folder $FRAMEWORK-$LANGUAGE. ## Approach MooBench measures the overhead of gathering observability data by executing an example workload using different configurations, including *no instrumentation* (and hence no data gathering) at all, full distributed tracing and data serialization via *binary writer*. The example workload consists of `$RECURSION_DEPTH` recursive calls of a function to itself. For example, the following graph shows the execution of MooBench in the *no instrumentation* configuration: ```mermaid graph TD; BenchmarkingThreadNano.run-->MonitoredClassSimple.monitoredMethod; MonitoredClassSimple.monitoredMethod-->MonitoredClassSimple.monitoredMethod; MonitoredClassSimple.monitoredMethod-->id["Busy Wait"] ``` The *binary writer* configuration on the other hand includes the probe code, that is injected by the observability tool before and after the operation. For the Kieker monitoring framework, the probe inserts records into the `WriterController.writerQueue`, and these are then processed for finally writing binary data to the hard disk. ```mermaid flowchart TD; instrumentedCall["`probeBeforeOperation() MonitoredClassSimple.monitoredMethod probeAfterOperation`"] BenchmarkingThreadNano.run-->instrumentedCall; instrumentedCall-->instrumentedCall; instrumentedCall-->id["Busy Wait"]; subgraph Kieker direction LR WriterController.writerQueue-->FileWriter.writeMonitoringRecord; FileWriter.writeMonitoringRecord-->BinaryLogStreamHandler.serialize; end instrumentedCall-->Kieker; ``` ## Benchmark Execution By one execution of the MooBench, we can measure the overhead of one observability framework. ### Prerequisites To use MooBench, please make sure the following tools are installed: - A linux system, capable of running bash 5.x (or newer), including `curl`, `wget`, `awk`, `nc` (netcat), and `gnuplot` (`sudo dnf install gawk curl wget nmap-ncat gnuplot`) - A recent R installation (Rocky Linux: `sudo dnf install epel-release; sudo dnf config-manager --set-enabled crb; sudo dnf install R`) - For Elastic APM, Pinpoint, and Skywalking: Docker 28.0.0 or newer (for containers that manage observability data persistence) - For Java agents (Kieker, OpenTelemetry, Elastic APM, Pinpoint, Scouter, inspectIT, Skywalking) * A recent JDK installation (OpenJDK 17 or newer); `$JAVA_HOME` needs to be set - For Python agents (Kieker, OpenTelemetry) * Python 3.11 or newer - For Kotlin: * A recent JDK installation (OpenJDK 21 or newer) * Node.js ### Benchmark Execution Compile the application and install it in the repository root directory. This can be done automatically be calling `./setup.sh` (for Kotlin configurations: `./setup.sh withKotlin`). Afterwards, you can switch to the framework folder (`frameworks` and then `$AGENT-$TECHNOLOGY`, e.g., OpenTelemetry as observability agent in Java) and run `./measure.sh`. For example, a simple benchmark execution is: ``` ./setup.sh cd frameworks/OpenTelemetry-java/ ./measure.sh ``` After each run, the main results are displayed in stdout, for example: ``` 1 2 3 4 5 6 mean " 0.2300" " 7.8581" " NA" " 10.5748" " 11.0599" " NA" sd " 0.1356" "115.0105" " NA" " 3.2623" " 3.4707" " NA" ci95% " 0.0038" " 3.1879" " NA" " 0.0904" " 0.0962" " NA" md25% " 0.1900" " 4.2380" " NA" " 8.0750" " 8.7310" " NA" md50% " 0.1910" " 4.5580" " NA" " 11.2710" " 11.3110" " NA" md75% " 0.2800" " 8.0450" " NA" " 11.5620" " 12.5040" " NA" max " 6.5920" "8135.7560" " NA" " 89.2280" " 99.5470" " NA" min " 0.1800" " 4.0970" " NA" " 7.5140" " 7.0230" " NA" ``` For each configuration (1-6), this gives us statistics of the execution. For example, here we see that configuration 4 (Writing OpenTelemetry traces to Zipkin) has a mean execution time of 10.5748 microseconds and configuration 0 (baseline, no instrumentation at all) has a mean execution time of 0.2300 microseconds. All experiments are started with the provided "External Controller" scripts. The following scripts are available for every supported framework ($FRAMEWORK) and language ($LANGUAGE): * In `frameworks/$FRAMEWORK-$LANGUAGE/measure.sh` a script is provided for regular execution (with default parameters) * In `frameworks/measureScaling.sh` a script is provided for execution with increased workload. This can be called with the mode, e.g., `./measureScaling.sh Kieker-java DEPTH` for call tree depth (`DEPTH`, alternative: `THREADS`). Each scripts will start different factorial experiments (started `$NUM_OF_LOOPS` times for repeatability), which will be: - baseline execution - execution with instrumentation but without processing or serialization - execution with serialization to hard disc (currently not available for inspectIT) - execution with serialization to tcp receiver, which might be a simple receiver (Kieker), or Zipkin and Prometheus (OpenTelemetry and inspectIT) All scripts have been tested on Ubuntu, Rocky 9.6 and Raspbian. The execution may be parameterized by the following environment variables: * `SLEEP_TIME` between executions (default 30 seconds) * `NUM_OF_LOOPS` number of repetitions (default 10) * `THREADS` concurrent benchmarking threads (default 1) * `RECURSION_DEPTH` recursion up to this depth (default 10) * `TOTAL_NUM_OF_CALLS` count of root method invocations per SuT start (default 2,000,000 calls) * `METHOD_TIME` the time per monitored call (default 0 ns or 500 us) * `DROP_OUTLIERS` drop the smallest and largest 0.5% of each run's measurements in the statistics (default false) * `TIME_UNIT` divisor for the reported times: 1 = ns, 1000 = µs, 1000000 = ms (default 1000) * `REPORT_METHOD_TIME` true: report the time per method call (recursion chain time divided by `RECURSION_DEPTH`); false: report the time per recursion chain (default false) If they are unset, the values are set via `init.sh` in the repository root. ## Formatting All shell files should be formatted using `shfmt -w -i 2 -sr -kp`. ## Data Analysis Each benchmark execution calls an R script providing mean, standard deviation and confidence intervals for the benchmark variants. If you got data from a run with exponential growing call tree depth, unzip them first (`for file in *.zip; do unzip $file; done`), copy all `results-$framework` folder to a common folder and run `./getExponential.sh` in analysis. This will create a graph for each framework and an overview graph for external processing of the traces (zipkin for OpenTelemetry and inspectIT, TCP for Kieker). In the folder /bin/r are some R scripts provided to generate graphs to visualize the results. In the top the files, one can configure the required paths and the configuration used to analyze the data.