{ "cells": [ { "metadata": { "collapsed": true }, "cell_type": "markdown", "source": [ "# Compare Benchmark Runs\n", "This notebook demonstrates how you can analyze the differences between two benchmark runs of the same benchmark and find the tests that differ the most, which probably means that they require further analysis to figure out why they changed.\n", "\n", "Several projects exist in the `examples` folder, but this notebook assumes we are working on the\n", "JVM part of the `kotlin-multiplatform` project. But the same approach can be used for the other projects.\n", "\n", "First, you need to run the benchmark twice. This can be done by running these commands from the root of the project:\n", "\n", "```shell\n", "> ./gradlew :examples:kotlin-multiplatform:jvmBenchmark\n", "> ./gradlew :examples:kotlin-multiplatform:jvmBenchmark\n", "```\n", "\n", "Once it is completed, run this notebook, and it will automatically find the latest result." ] }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_3_jupyter", "Line_4_jupyter", "Line_5_jupyter", "Line_6_jupyter", "Line_7_jupyter", "Line_8_jupyter", "Line_9_jupyter", "Line_10_jupyter", "Line_11_jupyter", "Line_12_jupyter", "Line_13_jupyter", "Line_14_jupyter", "Line_15_jupyter", "Line_16_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:09.608819Z", "start_time": "2025-11-12T23:01:04.678470Z" } }, "cell_type": "code", "source": "%use serialization, dataframe, kandy", "outputs": [], "execution_count": 1 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_17_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:10.065548Z", "start_time": "2025-11-12T23:01:09.615279Z" } }, "cell_type": "code", "source": [ "// Serialization classes matching the JMH-alike JSON format.\n", "// We define these classes manually so we can keep `params` as a JsonObject, as it means we can handle them\n", "// in a generic manner. If you benchmark have fixed params, using `\"\".deserializeThis()` is\n", "// faster and easier.\n", "\n", "@Serializable\n", "public data class Benchmark(\n", " public val benchmark: String,\n", " public val mode: String,\n", " public val warmupIterations: Int,\n", " public val warmupTime: String,\n", " public val measurementIterations: Int,\n", " public val measurementTime: String,\n", " public val primaryMetric: PrimaryMetric,\n", " public val secondaryMetrics: Map,\n", " public val params: JsonObject? = null\n", ")\n", "\n", "@Serializable\n", "public data class PrimaryMetric(\n", " public val score: Double,\n", " public val scoreError: Double,\n", " public val scoreConfidence: List,\n", " public val scorePercentiles: Map,\n", " public val scoreUnit: String,\n", " public val rawData: List>,\n", ")" ], "outputs": [], "execution_count": 2 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_18_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:10.621667Z", "start_time": "2025-11-12T23:01:10.097804Z" } }, "cell_type": "code", "source": [ "import java.nio.file.Files\n", "import java.nio.file.attribute.BasicFileAttributes\n", "import kotlin.io.path.exists\n", "import kotlin.io.path.forEachDirectoryEntry\n", "import kotlin.io.path.isDirectory\n", "import kotlin.io.path.listDirectoryEntries\n", "import kotlin.io.path.readText\n", "\n", "// Find latest result file, based on the their timestamp.\n", "val runsDir = notebook.workingDir.resolve(\"kotlin-multiplatform/build/reports/benchmarks/main\")\n", "val outputFiles = runsDir.listDirectoryEntries()\n", " .filter { it.isDirectory() }\n", " .sortedByDescending { dir -> Files.readAttributes(dir, BasicFileAttributes::class.java).creationTime() }\n", " .subList(0, 2)\n", " .map { it.resolve(\"jvm.json\") }" ], "outputs": [], "execution_count": 3 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_19_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:10.770928Z", "start_time": "2025-11-12T23:01:10.626280Z" } }, "cell_type": "code", "source": [ "// Convert to typed JSON\n", "val json = Json { ignoreUnknownKeys = true }\n", "val newRun = json.decodeFromString>(outputFiles.first().readText())\n", "val oldRun = json.decodeFromString>(outputFiles.last().readText())" ], "outputs": [], "execution_count": 4 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_20_jupyter", "Line_21_jupyter", "Line_22_jupyter", "Line_23_jupyter", "Line_24_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:11.249299Z", "start_time": "2025-11-12T23:01:10.775339Z" } }, "cell_type": "code", "source": [ "// Convert to DataFrames for easier processing. As there is not \"id\" keys for the benchmark, we invent one by just\n", "// assigning the test row index as their \"primary key\". We could attempt to use the benchmark name and param values,\n", "// but that is complicated by how paramers are represented in the JSON file. So, since we assume that the two files\n", "// are equal using row index should be safe.\n", "val oldDf = oldRun.toDataFrame().addId(\"rowIndex\")\n", "val newDf = newRun.toDataFrame().addId(\"rowIndex\")" ], "outputs": [], "execution_count": 5 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_25_jupyter", "Line_26_jupyter", "Line_27_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:11.560635Z", "start_time": "2025-11-12T23:01:11.256258Z" } }, "cell_type": "code", "source": [ "val combinedData = oldDf.innerJoin(newDf) { rowIndex }\n", "// Un-commont this to see the intermediate dataframe:\n", "// combinedData" ], "outputs": [], "execution_count": 6 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_28_jupyter", "Line_29_jupyter", "Line_30_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:11.982028Z", "start_time": "2025-11-12T23:01:11.565945Z" } }, "cell_type": "code", "source": [ "import kotlinx.serialization.json.encodeToJsonElement\n", "\n", "// Reduce the combined data into the exact format we need\n", "val resultData = combinedData.mapToFrame {\n", " \"name\" from { it.benchmark }\n", " \"params\" from {\n", " it.params?.entries.orEmpty()\n", " .sortedBy { it.key }\n", " .joinToString(\",\") { entry -> \"${entry.key}=${entry.value.jsonPrimitive.content}\" }\n", " }\n", " \"mode\" from { it.mode } // \"avgt\" or \"thrpt\"\n", " \"unit\" from { it.primaryMetric.scoreUnit }\n", " \"runOld\" {\n", " \"score\" from { it.primaryMetric.score }\n", " \"range\" from { it.primaryMetric.scoreConfidence[0]..it.primaryMetric.scoreConfidence[1] }\n", " }\n", " \"runNew\" {\n", " \"score\" from { it.primaryMetric1.score }\n", " \"range\" from { it.primaryMetric1.scoreConfidence[0]..it.primaryMetric1.scoreConfidence[1] }\n", " }\n", "}\n", "// Un-commont this to see the intermediate dataframe:\n", "// resultData" ], "outputs": [], "execution_count": 7 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_31_jupyter", "Line_32_jupyter", "Line_33_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:12.181560Z", "start_time": "2025-11-12T23:01:11.996744Z" } }, "cell_type": "code", "source": [ "// Flatten the data so it is easier to plot\n", "val mergedData = resultData.unfold { runOld and runNew }.flatten()\n", "// Un-commont this to see the intermediate dataframe:\n", "// mergedData" ], "outputs": [], "execution_count": 8 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_34_jupyter", "Line_35_jupyter", "Line_36_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:12.492447Z", "start_time": "2025-11-12T23:01:12.187988Z" } }, "cell_type": "code", "source": [ "// Before plotting the data, we calculate the change between the two runs. This is saved\n", "// in \"scoreDiff\". This is done slightly different depending on the test mode:\n", "//\n", "// - \"avgt\": For the average time we use \"oldScore - newScore\", so improvements in the\n", "// benchmark result in positive numbers.\n", "// - \"thrpt\": For throughput, we use \"newScore - oldScore\", so improvements here also\n", "// result in positive numbers.\n", "//\n", "// We also normalize this value as a percentage change from `scoreOld`. This is saved in\n", "// \"scoreDiffPercentage\".\n", "val plotData = mergedData\n", " .add(\"diffScore\") {\n", " when (mode) {\n", " \"avgt\" -> score - score1\n", " \"thrpt\" -> score1 - score\n", " else -> error(\"Unknown mode: $mode\")\n", " }\n", " }\n", " .add(\"diffScorePercentage\") {\n", " (get(\"diffScore\") as Double) * 100.0 / score\n", " }\n", " .add(\"testLabel\") {\n", " if (params.isNullOrBlank()) {\n", " name\n", " } else {\n", " \"$name\\n[$params]\"\n", " }\n", " }\n", " .add(\"barColor\") {\n", " val value = get(\"diffScorePercentage\") as Double\n", " if (value < 0.0) \"neg\" else \"pos\"\n", " }\n", "plotData" ], "outputs": [ { "data": { "text/html": [ " \n", " \n", " \n", " \n", " \n", " \n", "
nameparamsmodeunitscorerangescore1range1diffScorediffScorePercentagetestLabelbarColor
test.InheritedBenchmark.baseBenchmarkthrptops/s999715.573058974954.2809652109..1024476.86515133131140852.9572291106659.7386652725..1175046.1757937148141137.38417114.117754test.InheritedBenchmark.baseBenchmarkpos
test.InheritedBenchmark.inheritedBenc...thrptops/s132519854.0027531.2936737342079675E8..1.3567233458470...146792560.8958071.4482756334251586E8..1.4875755844909...14272706.89305410.770240test.InheritedBenchmark.inheritedBenc...pos
test.ParamBenchmark.mathBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms196902.036134191654.7423267418..202149.32994077116217133.491804215027.16316066374..219239.8204476190120231.45567010.274884test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms198982.020454194497.3197739599..203466.72113419612216593.775142212594.8732753121..220592.67700943317611.7546888.850928test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms76487.50862874595.04831374026..78379.9689426085486523.58851782315.21514352418..90731.9618897417210036.07988813.121201test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms77218.53689575599.16372724227..78837.9100630139385944.38196181177.07416021802..90711.689761014968725.84506511.300195test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.otherBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms2312642.7736442091779.438724716..2533506.10856339242616589.1987902579570.7359826625..2653607.6615965427303946.42514613.142818test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms2295922.9388222089935.2366487498..2501910.64099451342629054.4416712524752.102688715..2733356.780654085333131.50285014.509699test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms2295223.3640072115768.594299648..2474678.13371464332646402.7370522616649.4130033795..2676156.0611009924351179.37304515.300444test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms2295128.5932112082445.0002591067..2507812.1861624582652047.7421422627344.681389795..2676750.802894536356919.14893115.551161test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.textContentCheckdata=1,text=a "string" with quotes,va...thrptops/ms139497.981811135894.93943472698..143101.02418653402154381.766970153348.64196544446..155414.8919749425514883.78516010.669534test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=1,text=a "string" with quotes,va...thrptops/ms139472.551215135249.9703875965..143695.13204299458153578.254496150716.46891485332..156440.0400775279614105.70328110.113605test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=2,text=a "string" with quotes,va...thrptops/ms137951.951688133789.3696336104..142114.53374258292153262.850584150745.58896373163..155780.1122049946315310.89889611.098719test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=2,text=a "string" with quotes,va...thrptops/ms135829.985631131190.7048701939..140469.26639132493153672.584595150946.56541126606..156398.6037785117517842.59896413.135979test.ParamBenchmark.textContentCheck\n", "...pos
test.nested.CommonBenchmark.mathBench...thrptops/ms133932.193305129878.2239827778..137986.16262644873148921.638469148674.35770053475..149168.9192370649514989.44516411.191816test.nested.CommonBenchmark.mathBench...pos
test.CommonBenchmark.longBenchmarkavgtms/op0.0010209.902780506238168E-4..0.0010504284290...0.0008548.428241181867443E-4..8.6492699881309...0.00016616.315691test.CommonBenchmark.longBenchmarkpos
test.CommonBenchmark.longBlackholeBen...avgtms/op0.0000262.5596365181913836E-5..2.697949020758...0.0000232.284205842928573E-5..2.3608924259914...0.00000311.649592test.CommonBenchmark.longBlackholeBen...pos
test.CommonBenchmark.mathBenchmarkavgtms/op0.0000087.456424871291536E-6..7.7734997336766...0.0000076.685553120802533E-6..6.8730115882504...0.00000110.974184test.CommonBenchmark.mathBenchmarkpos
test.JvmTestBenchmark.cosBenchmarkavgtns/op3.9713383.8358525869703075..4.10682434140826753.4958493.48368627561432..3.50801161595222370.47549011.973029test.JvmTestBenchmark.cosBenchmarkpos
test.JvmTestBenchmark.sqrtBenchmarkavgtns/op0.6065980.5901377228601924..0.623059022507190.5411620.5348433328074729..0.54747986807174540.06543710.787496test.JvmTestBenchmark.sqrtBenchmarkpos
\n", " \n", " \n", " " ], "application/kotlindataframe+json": "{\"$version\":\"2.1.1\",\"metadata\":{\"columns\":[\"name\",\"params\",\"mode\",\"unit\",\"score\",\"range\",\"score1\",\"range1\",\"diffScore\",\"diffScorePercentage\",\"testLabel\",\"barColor\"],\"types\":[{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.ranges.ClosedFloatingPointRange\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.ranges.ClosedFloatingPointRange\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"}],\"nrow\":20,\"ncol\":12},\"kotlin_dataframe\":[{\"name\":\"test.InheritedBenchmark.baseBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/s\",\"score\":999715.5730582711,\"range\":\"974954.2809652109..1024476.8651513313\",\"score1\":1140852.9572294937,\"range1\":\"1106659.7386652725..1175046.1757937148\",\"diffScore\":141137.38417122257,\"diffScorePercentage\":14.117753886684326,\"testLabel\":\"test.InheritedBenchmark.baseBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.InheritedBenchmark.inheritedBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/s\",\"score\":1.3251985400275281E8,\"range\":\"1.2936737342079675E8..1.3567233458470887E8\",\"score1\":1.4679256089580712E8,\"range1\":\"1.4482756334251586E8..1.4875755844909838E8\",\"diffScore\":1.4272706893054307E7,\"diffScorePercentage\":10.770240429602211,\"testLabel\":\"test.InheritedBenchmark.inheritedBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":196902.03613375648,\"range\":\"191654.7423267418..202149.32994077116\",\"score1\":217133.49180414138,\"range1\":\"215027.16316066374..219239.82044761901\",\"diffScore\":20231.455670384894,\"diffScorePercentage\":10.274883930931813,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":198982.020454078,\"range\":\"194497.3197739599..203466.72113419612\",\"score1\":216593.77514237256,\"range1\":\"212594.8732753121..220592.677009433\",\"diffScore\":17611.754688294546,\"diffScorePercentage\":8.85092766075268,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":76487.5086281744,\"range\":\"74595.04831374026..78379.96894260854\",\"score1\":86523.58851663295,\"range1\":\"82315.21514352418..90731.96188974172\",\"diffScore\":10036.079888458553,\"diffScorePercentage\":13.121201184949738,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":77218.5368951281,\"range\":\"75599.16372724227..78837.91006301393\",\"score1\":85944.38196061649,\"range1\":\"81177.07416021802..90711.68976101496\",\"diffScore\":8725.845065488393,\"diffScorePercentage\":11.300194767143958,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2312642.7736440543,\"range\":\"2091779.438724716..2533506.1085633924\",\"score1\":2616589.1987896026,\"range1\":\"2579570.7359826625..2653607.6615965427\",\"diffScore\":303946.4251455483,\"diffScorePercentage\":13.142817758516887,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295922.9388216315,\"range\":\"2089935.2366487498..2501910.6409945134\",\"score1\":2629054.4416714,\"range1\":\"2524752.102688715..2733356.780654085\",\"diffScore\":333131.5028497684,\"diffScorePercentage\":14.509698788964847,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295223.3640071456,\"range\":\"2115768.594299648..2474678.1337146433\",\"score1\":2646402.737052186,\"range1\":\"2616649.4130033795..2676156.0611009924\",\"diffScore\":351179.3730450403,\"diffScorePercentage\":15.300444329388892,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295128.5932107824,\"range\":\"2082445.0002591067..2507812.186162458\",\"score1\":2652047.7421421655,\"range1\":\"2627344.681389795..2676750.802894536\",\"diffScore\":356919.14893138316,\"diffScorePercentage\":15.551161272060545,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":139497.9818106305,\"range\":\"135894.93943472698..143101.02418653402\",\"score1\":154381.7669701935,\"range1\":\"153348.64196544446..155414.89197494255\",\"diffScore\":14883.785159563005,\"diffScorePercentage\":10.669534402130527,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":139472.55121529553,\"range\":\"135249.9703875965..143695.13204299458\",\"score1\":153578.25449619064,\"range1\":\"150716.46891485332..156440.04007752796\",\"diffScore\":14105.703280895104,\"diffScorePercentage\":10.113605263533874,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":137951.95168809665,\"range\":\"133789.3696336104..142114.53374258292\",\"score1\":153262.85058436313,\"range1\":\"150745.58896373163..155780.11220499463\",\"diffScore\":15310.898896266473,\"diffScorePercentage\":11.098718582020316,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":135829.98563075942,\"range\":\"131190.7048701939..140469.26639132493\",\"score1\":153672.5845948889,\"range1\":\"150946.56541126606..156398.60377851175\",\"diffScore\":17842.598964129487,\"diffScorePercentage\":13.13597942403738,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.nested.CommonBenchmark.mathBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":133932.19330461326,\"range\":\"129878.2239827778..137986.16262644873\",\"score1\":148921.63846879985,\"range1\":\"148674.35770053475..149168.91923706495\",\"diffScore\":14989.445164186589,\"diffScorePercentage\":11.191816391817635,\"testLabel\":\"test.nested.CommonBenchmark.mathBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.longBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":0.0010203532398568873,\"range\":\"9.902780506238168E-4..0.0010504284290899577\",\"score1\":8.538755584999174E-4,\"range1\":\"8.428241181867443E-4..8.649269988130904E-4\",\"diffScore\":1.6647768135696992E-4,\"diffScorePercentage\":16.31569096407434,\"testLabel\":\"test.CommonBenchmark.longBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.longBlackholeBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":2.6287927694751805E-5,\"range\":\"2.5596365181913836E-5..2.6979490207589775E-5\",\"score1\":2.3225491344599873E-5,\"range1\":\"2.284205842928573E-5..2.3608924259914014E-5\",\"diffScore\":3.0624363501519328E-6,\"diffScorePercentage\":11.649592108256316,\"testLabel\":\"test.CommonBenchmark.longBlackholeBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.mathBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":7.614962302484078E-6,\"range\":\"7.456424871291536E-6..7.77349973367662E-6\",\"score1\":6.779282354526504E-6,\"range1\":\"6.685553120802533E-6..6.873011588250474E-6\",\"diffScore\":8.356799479575747E-7,\"diffScorePercentage\":10.974183650061766,\"testLabel\":\"test.CommonBenchmark.mathBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.JvmTestBenchmark.cosBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ns/op\",\"score\":3.9713384641892873,\"range\":\"3.8358525869703075..4.1068243414082675\",\"score1\":3.4958489457832718,\"range1\":\"3.48368627561432..3.5080116159522237\",\"diffScore\":0.4754895184060155,\"diffScorePercentage\":11.973029312249324,\"testLabel\":\"test.JvmTestBenchmark.cosBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.JvmTestBenchmark.sqrtBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ns/op\",\"score\":0.6065983726836912,\"range\":\"0.5901377228601924..0.62305902250719\",\"score1\":0.5411616004396091,\"range1\":\"0.5348433328074729..0.5474798680717454\",\"diffScore\":0.06543677224408206,\"diffScorePercentage\":10.787495514466844,\"testLabel\":\"test.JvmTestBenchmark.sqrtBenchmark\",\"barColor\":\"pos\"}]}" }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "execution_count": 9 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_38_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:13.096488Z", "start_time": "2025-11-12T23:01:12.552457Z" } }, "cell_type": "code", "source": [ "import org.jetbrains.kotlinx.kandy.util.color.Color\n", "import org.jetbrains.letsPlot.core.spec.plotson.fill\n", "import org.jetbrains.letsPlot.label.ggtitle\n", "import org.jetbrains.letsPlot.scale.guideLegend\n", "import org.jetbrains.letsPlot.scale.guides\n", "\n", "// Now we can plot this data. First we create a basic plot just showing the difference in percent between all scores.\n", "plotData.sortBy { diffScorePercentage }.plot {\n", " barsH {\n", " x(diffScorePercentage) {\n", " axis.name = \"Diff %\"\n", " }\n", " y(testLabel) {\n", " axis.name = \"\"\n", " }\n", " fillColor(barColor) {\n", " scale = categorical(\"neg\" to Color.RED, \"pos\" to Color.GREEN)\n", " legend.type = LegendType.None\n", " }\n", " tooltips {\n", " line(diffScorePercentage, format = \".2f\")\n", " }\n", " }\n", " layout {\n", " size = 800 to ((40 * plotData.size().nrow) + 100)\n", " style {\n", " global {\n", " title {\n", " margin(10.0, 0.0)\n", " }\n", " }\n", " }\n", " }\n", "}" ], "outputs": [ { "data": { "text/html": [ " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 10\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 15\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.InheritedBenchmark.inheritedBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.JvmTestBenchmark.sqrtBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.mathBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.nested.CommonBenchmark.mathBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.longBlackholeBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.JvmTestBenchmark.cosBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.InheritedBenchmark.baseBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.longBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " Diff %\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " " ], "application/plot+json": { "output_type": "lets_plot_spec", "output": { "mapping": {}, "data": { "testLabel": [ "test.ParamBenchmark.mathBenchmark\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.textContentCheck\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.mathBenchmark\n[data=1,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.textContentCheck\n[data=1,text=a \"string\" with quotes,value=1]", "test.InheritedBenchmark.inheritedBenchmark", "test.JvmTestBenchmark.sqrtBenchmark", "test.CommonBenchmark.mathBenchmark", "test.ParamBenchmark.textContentCheck\n[data=2,text=a \"string\" with quotes,value=1]", "test.nested.CommonBenchmark.mathBenchmark", "test.ParamBenchmark.mathBenchmark\n[data=2,text=a \"string\" with quotes,value=2]", "test.CommonBenchmark.longBlackholeBenchmark", "test.JvmTestBenchmark.cosBenchmark", "test.ParamBenchmark.mathBenchmark\n[data=2,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.textContentCheck\n[data=2,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.otherBenchmark\n[data=1,text=a \"string\" with quotes,value=1]", "test.InheritedBenchmark.baseBenchmark", "test.ParamBenchmark.otherBenchmark\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.otherBenchmark\n[data=2,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.otherBenchmark\n[data=2,text=a \"string\" with quotes,value=2]", "test.CommonBenchmark.longBenchmark" ], "diffScorePercentage": [ 8.85092766075268, 10.113605263533874, 10.274883930931813, 10.669534402130527, 10.770240429602211, 10.787495514466844, 10.974183650061766, 11.098718582020316, 11.191816391817635, 11.300194767143958, 11.649592108256316, 11.973029312249324, 13.121201184949738, 13.13597942403738, 13.142817758516887, 14.117753886684326, 14.509698788964847, 15.300444329388892, 15.551161272060545, 16.31569096407434 ], "barColor": [ "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos" ] }, "ggsize": { "width": 800.0, "height": 900.0 }, "kind": "plot", "scales": [ { "aesthetic": "x", "name": "Diff %", "limits": [ null, null ] }, { "aesthetic": "y", "discrete": true, "name": "" }, { "aesthetic": "fill", "values": [ "#ee6666", "#3ba272" ], "limits": [ "neg", "pos" ], "guide": "none" } ], "layers": [ { "mapping": { "x": "diffScorePercentage", "y": "testLabel", "fill": "barColor" }, "stat": "identity", "orientation": "y", "sampling": "none", "inherit_aes": false, "position": "dodge", "geom": "bar", "tooltips": { "lines": [ "@|@{diffScorePercentage}" ], "formats": [ { "field": "diffScorePercentage", "format": ".2f" } ], "disable_splitting": true } } ], "theme": { "title": { "margin": [ 10.0, 0.0, 10.0, 0.0 ], "blank": false }, "axis_ontop": false, "axis_ontop_y": false, "axis_ontop_x": false }, "data_meta": { "series_annotations": [ { "type": "float", "column": "diffScorePercentage" }, { "type": "str", "column": "testLabel" }, { "type": "str", "column": "barColor" } ] } }, "apply_color_scheme": true, "swing_enabled": true } }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "execution_count": 10 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_39_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:13.354938Z", "start_time": "2025-11-12T23:01:13.243696Z" } }, "cell_type": "code", "source": [ "// Just comparing the score values is a bit simplistic as the benchmark results are actually a range: score +/- error.\n", "// So, instead of plotting all tests, we want to focus only on the benchmarks that looks \"interesting\". This is\n", "// defined as any benchmark that differ so much that the benchmark ranges do not overlap, i.e., we no longer just\n", "// look at only the score but consider the full error range.\n", "//\n", "// We still use the \"score\" to calculate the change in percent, but now on a filtered list\n", "fun kotlin.ranges.ClosedFloatingPointRange.overlaps(other: ClosedFloatingPointRange): Boolean =\n", " this.start <= other.endInclusive && other.start <= this.endInclusive\n", "\n", "val interestingBenchmarks = plotData.filter {\n", " !it.range.overlaps(it.range1)\n", "}\n", "interestingBenchmarks" ], "outputs": [ { "data": { "text/html": [ " \n", " \n", " \n", " \n", " \n", " \n", "
nameparamsmodeunitscorerangescore1range1diffScorediffScorePercentagetestLabelbarColor
test.InheritedBenchmark.baseBenchmarkthrptops/s999715.573058974954.2809652109..1024476.86515133131140852.9572291106659.7386652725..1175046.1757937148141137.38417114.117754test.InheritedBenchmark.baseBenchmarkpos
test.InheritedBenchmark.inheritedBenc...thrptops/s132519854.0027531.2936737342079675E8..1.3567233458470...146792560.8958071.4482756334251586E8..1.4875755844909...14272706.89305410.770240test.InheritedBenchmark.inheritedBenc...pos
test.ParamBenchmark.mathBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms196902.036134191654.7423267418..202149.32994077116217133.491804215027.16316066374..219239.8204476190120231.45567010.274884test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms198982.020454194497.3197739599..203466.72113419612216593.775142212594.8732753121..220592.67700943317611.7546888.850928test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms76487.50862874595.04831374026..78379.9689426085486523.58851782315.21514352418..90731.9618897417210036.07988813.121201test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.mathBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms77218.53689575599.16372724227..78837.9100630139385944.38196181177.07416021802..90711.689761014968725.84506511.300195test.ParamBenchmark.mathBenchmark\n", "[da...pos
test.ParamBenchmark.otherBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms2312642.7736442091779.438724716..2533506.10856339242616589.1987902579570.7359826625..2653607.6615965427303946.42514613.142818test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=1,text=a "string" with quotes,va...thrptops/ms2295922.9388222089935.2366487498..2501910.64099451342629054.4416712524752.102688715..2733356.780654085333131.50285014.509699test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms2295223.3640072115768.594299648..2474678.13371464332646402.7370522616649.4130033795..2676156.0611009924351179.37304515.300444test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.otherBenchmarkdata=2,text=a "string" with quotes,va...thrptops/ms2295128.5932112082445.0002591067..2507812.1861624582652047.7421422627344.681389795..2676750.802894536356919.14893115.551161test.ParamBenchmark.otherBenchmark\n", "[d...pos
test.ParamBenchmark.textContentCheckdata=1,text=a "string" with quotes,va...thrptops/ms139497.981811135894.93943472698..143101.02418653402154381.766970153348.64196544446..155414.8919749425514883.78516010.669534test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=1,text=a "string" with quotes,va...thrptops/ms139472.551215135249.9703875965..143695.13204299458153578.254496150716.46891485332..156440.0400775279614105.70328110.113605test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=2,text=a "string" with quotes,va...thrptops/ms137951.951688133789.3696336104..142114.53374258292153262.850584150745.58896373163..155780.1122049946315310.89889611.098719test.ParamBenchmark.textContentCheck\n", "...pos
test.ParamBenchmark.textContentCheckdata=2,text=a "string" with quotes,va...thrptops/ms135829.985631131190.7048701939..140469.26639132493153672.584595150946.56541126606..156398.6037785117517842.59896413.135979test.ParamBenchmark.textContentCheck\n", "...pos
test.nested.CommonBenchmark.mathBench...thrptops/ms133932.193305129878.2239827778..137986.16262644873148921.638469148674.35770053475..149168.9192370649514989.44516411.191816test.nested.CommonBenchmark.mathBench...pos
test.CommonBenchmark.longBenchmarkavgtms/op0.0010209.902780506238168E-4..0.0010504284290...0.0008548.428241181867443E-4..8.6492699881309...0.00016616.315691test.CommonBenchmark.longBenchmarkpos
test.CommonBenchmark.longBlackholeBen...avgtms/op0.0000262.5596365181913836E-5..2.697949020758...0.0000232.284205842928573E-5..2.3608924259914...0.00000311.649592test.CommonBenchmark.longBlackholeBen...pos
test.CommonBenchmark.mathBenchmarkavgtms/op0.0000087.456424871291536E-6..7.7734997336766...0.0000076.685553120802533E-6..6.8730115882504...0.00000110.974184test.CommonBenchmark.mathBenchmarkpos
test.JvmTestBenchmark.cosBenchmarkavgtns/op3.9713383.8358525869703075..4.10682434140826753.4958493.48368627561432..3.50801161595222370.47549011.973029test.JvmTestBenchmark.cosBenchmarkpos
test.JvmTestBenchmark.sqrtBenchmarkavgtns/op0.6065980.5901377228601924..0.623059022507190.5411620.5348433328074729..0.54747986807174540.06543710.787496test.JvmTestBenchmark.sqrtBenchmarkpos
\n", " \n", " \n", " " ], "application/kotlindataframe+json": "{\"$version\":\"2.1.1\",\"metadata\":{\"columns\":[\"name\",\"params\",\"mode\",\"unit\",\"score\",\"range\",\"score1\",\"range1\",\"diffScore\",\"diffScorePercentage\",\"testLabel\",\"barColor\"],\"types\":[{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.ranges.ClosedFloatingPointRange\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.ranges.ClosedFloatingPointRange\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.Double\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"},{\"kind\":\"ValueColumn\",\"type\":\"kotlin.String\"}],\"nrow\":20,\"ncol\":12},\"kotlin_dataframe\":[{\"name\":\"test.InheritedBenchmark.baseBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/s\",\"score\":999715.5730582711,\"range\":\"974954.2809652109..1024476.8651513313\",\"score1\":1140852.9572294937,\"range1\":\"1106659.7386652725..1175046.1757937148\",\"diffScore\":141137.38417122257,\"diffScorePercentage\":14.117753886684326,\"testLabel\":\"test.InheritedBenchmark.baseBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.InheritedBenchmark.inheritedBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/s\",\"score\":1.3251985400275281E8,\"range\":\"1.2936737342079675E8..1.3567233458470887E8\",\"score1\":1.4679256089580712E8,\"range1\":\"1.4482756334251586E8..1.4875755844909838E8\",\"diffScore\":1.4272706893054307E7,\"diffScorePercentage\":10.770240429602211,\"testLabel\":\"test.InheritedBenchmark.inheritedBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":196902.03613375648,\"range\":\"191654.7423267418..202149.32994077116\",\"score1\":217133.49180414138,\"range1\":\"215027.16316066374..219239.82044761901\",\"diffScore\":20231.455670384894,\"diffScorePercentage\":10.274883930931813,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":198982.020454078,\"range\":\"194497.3197739599..203466.72113419612\",\"score1\":216593.77514237256,\"range1\":\"212594.8732753121..220592.677009433\",\"diffScore\":17611.754688294546,\"diffScorePercentage\":8.85092766075268,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":76487.5086281744,\"range\":\"74595.04831374026..78379.96894260854\",\"score1\":86523.58851663295,\"range1\":\"82315.21514352418..90731.96188974172\",\"diffScore\":10036.079888458553,\"diffScorePercentage\":13.121201184949738,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.mathBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":77218.5368951281,\"range\":\"75599.16372724227..78837.91006301393\",\"score1\":85944.38196061649,\"range1\":\"81177.07416021802..90711.68976101496\",\"diffScore\":8725.845065488393,\"diffScorePercentage\":11.300194767143958,\"testLabel\":\"test.ParamBenchmark.mathBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2312642.7736440543,\"range\":\"2091779.438724716..2533506.1085633924\",\"score1\":2616589.1987896026,\"range1\":\"2579570.7359826625..2653607.6615965427\",\"diffScore\":303946.4251455483,\"diffScorePercentage\":13.142817758516887,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295922.9388216315,\"range\":\"2089935.2366487498..2501910.6409945134\",\"score1\":2629054.4416714,\"range1\":\"2524752.102688715..2733356.780654085\",\"diffScore\":333131.5028497684,\"diffScorePercentage\":14.509698788964847,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295223.3640071456,\"range\":\"2115768.594299648..2474678.1337146433\",\"score1\":2646402.737052186,\"range1\":\"2616649.4130033795..2676156.0611009924\",\"diffScore\":351179.3730450403,\"diffScorePercentage\":15.300444329388892,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.otherBenchmark\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":2295128.5932107824,\"range\":\"2082445.0002591067..2507812.186162458\",\"score1\":2652047.7421421655,\"range1\":\"2627344.681389795..2676750.802894536\",\"diffScore\":356919.14893138316,\"diffScorePercentage\":15.551161272060545,\"testLabel\":\"test.ParamBenchmark.otherBenchmark\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":139497.9818106305,\"range\":\"135894.93943472698..143101.02418653402\",\"score1\":154381.7669701935,\"range1\":\"153348.64196544446..155414.89197494255\",\"diffScore\":14883.785159563005,\"diffScorePercentage\":10.669534402130527,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=1,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=1,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":139472.55121529553,\"range\":\"135249.9703875965..143695.13204299458\",\"score1\":153578.25449619064,\"range1\":\"150716.46891485332..156440.04007752796\",\"diffScore\":14105.703280895104,\"diffScorePercentage\":10.113605263533874,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=1,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=1\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":137951.95168809665,\"range\":\"133789.3696336104..142114.53374258292\",\"score1\":153262.85058436313,\"range1\":\"150745.58896373163..155780.11220499463\",\"diffScore\":15310.898896266473,\"diffScorePercentage\":11.098718582020316,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=2,text=a \\\"string\\\" with quotes,value=1]\",\"barColor\":\"pos\"},{\"name\":\"test.ParamBenchmark.textContentCheck\",\"params\":\"data=2,text=a \\\"string\\\" with quotes,value=2\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":135829.98563075942,\"range\":\"131190.7048701939..140469.26639132493\",\"score1\":153672.5845948889,\"range1\":\"150946.56541126606..156398.60377851175\",\"diffScore\":17842.598964129487,\"diffScorePercentage\":13.13597942403738,\"testLabel\":\"test.ParamBenchmark.textContentCheck\\n[data=2,text=a \\\"string\\\" with quotes,value=2]\",\"barColor\":\"pos\"},{\"name\":\"test.nested.CommonBenchmark.mathBenchmark\",\"params\":\"\",\"mode\":\"thrpt\",\"unit\":\"ops/ms\",\"score\":133932.19330461326,\"range\":\"129878.2239827778..137986.16262644873\",\"score1\":148921.63846879985,\"range1\":\"148674.35770053475..149168.91923706495\",\"diffScore\":14989.445164186589,\"diffScorePercentage\":11.191816391817635,\"testLabel\":\"test.nested.CommonBenchmark.mathBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.longBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":0.0010203532398568873,\"range\":\"9.902780506238168E-4..0.0010504284290899577\",\"score1\":8.538755584999174E-4,\"range1\":\"8.428241181867443E-4..8.649269988130904E-4\",\"diffScore\":1.6647768135696992E-4,\"diffScorePercentage\":16.31569096407434,\"testLabel\":\"test.CommonBenchmark.longBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.longBlackholeBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":2.6287927694751805E-5,\"range\":\"2.5596365181913836E-5..2.6979490207589775E-5\",\"score1\":2.3225491344599873E-5,\"range1\":\"2.284205842928573E-5..2.3608924259914014E-5\",\"diffScore\":3.0624363501519328E-6,\"diffScorePercentage\":11.649592108256316,\"testLabel\":\"test.CommonBenchmark.longBlackholeBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.CommonBenchmark.mathBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ms/op\",\"score\":7.614962302484078E-6,\"range\":\"7.456424871291536E-6..7.77349973367662E-6\",\"score1\":6.779282354526504E-6,\"range1\":\"6.685553120802533E-6..6.873011588250474E-6\",\"diffScore\":8.356799479575747E-7,\"diffScorePercentage\":10.974183650061766,\"testLabel\":\"test.CommonBenchmark.mathBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.JvmTestBenchmark.cosBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ns/op\",\"score\":3.9713384641892873,\"range\":\"3.8358525869703075..4.1068243414082675\",\"score1\":3.4958489457832718,\"range1\":\"3.48368627561432..3.5080116159522237\",\"diffScore\":0.4754895184060155,\"diffScorePercentage\":11.973029312249324,\"testLabel\":\"test.JvmTestBenchmark.cosBenchmark\",\"barColor\":\"pos\"},{\"name\":\"test.JvmTestBenchmark.sqrtBenchmark\",\"params\":\"\",\"mode\":\"avgt\",\"unit\":\"ns/op\",\"score\":0.6065983726836912,\"range\":\"0.5901377228601924..0.62305902250719\",\"score1\":0.5411616004396091,\"range1\":\"0.5348433328074729..0.5474798680717454\",\"diffScore\":0.06543677224408206,\"diffScorePercentage\":10.787495514466844,\"testLabel\":\"test.JvmTestBenchmark.sqrtBenchmark\",\"barColor\":\"pos\"}]}" }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "execution_count": 11 }, { "metadata": { "executionRelatedData": { "compiledClasses": [ "Line_41_jupyter" ] }, "ExecuteTime": { "end_time": "2025-11-12T23:01:13.586643Z", "start_time": "2025-11-12T23:01:13.376316Z" } }, "cell_type": "code", "source": [ "// Now lets plot the interesting benchmarks, similar to before.\n", "interestingBenchmarks.sortBy { diffScorePercentage }.plot {\n", " barsH {\n", " x(diffScorePercentage) {\n", " axis.name = \"Diff %\"\n", " }\n", " y(testLabel) {\n", " axis.name = \"\"\n", " }\n", " fillColor(barColor) {\n", " scale = categorical(\"neg\" to Color.RED, \"pos\" to Color.GREEN)\n", " legend.type = LegendType.None\n", " }\n", " tooltips {\n", " line(diffScorePercentage, format = \".2f\")\n", " }\n", " }\n", " layout {\n", " size = 800 to ((40 * interestingBenchmarks.size().nrow) + 100)\n", " style {\n", " global {\n", " title {\n", " margin(10.0, 0.0)\n", " }\n", " }\n", " }\n", " }\n", "}" ], "outputs": [ { "data": { "text/html": [ " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 10\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 15\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.InheritedBenchmark.inheritedBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.JvmTestBenchmark.sqrtBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.mathBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.nested.CommonBenchmark.mathBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.longBlackholeBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.JvmTestBenchmark.cosBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.mathBenchmark [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.textContentCheck [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=1,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.InheritedBenchmark.baseBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=1,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=2,text=a "string" with quotes,value=1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.ParamBenchmark.otherBenchmark [data=2,text=a "string" with quotes,value=2]\n", " \n", " \n", " \n", " \n", " \n", " \n", " test.CommonBenchmark.longBenchmark\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " Diff %\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " " ], "application/plot+json": { "output_type": "lets_plot_spec", "output": { "mapping": {}, "data": { "testLabel": [ "test.ParamBenchmark.mathBenchmark\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.textContentCheck\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.mathBenchmark\n[data=1,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.textContentCheck\n[data=1,text=a \"string\" with quotes,value=1]", "test.InheritedBenchmark.inheritedBenchmark", "test.JvmTestBenchmark.sqrtBenchmark", "test.CommonBenchmark.mathBenchmark", "test.ParamBenchmark.textContentCheck\n[data=2,text=a \"string\" with quotes,value=1]", "test.nested.CommonBenchmark.mathBenchmark", "test.ParamBenchmark.mathBenchmark\n[data=2,text=a \"string\" with quotes,value=2]", "test.CommonBenchmark.longBlackholeBenchmark", "test.JvmTestBenchmark.cosBenchmark", "test.ParamBenchmark.mathBenchmark\n[data=2,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.textContentCheck\n[data=2,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.otherBenchmark\n[data=1,text=a \"string\" with quotes,value=1]", "test.InheritedBenchmark.baseBenchmark", "test.ParamBenchmark.otherBenchmark\n[data=1,text=a \"string\" with quotes,value=2]", "test.ParamBenchmark.otherBenchmark\n[data=2,text=a \"string\" with quotes,value=1]", "test.ParamBenchmark.otherBenchmark\n[data=2,text=a \"string\" with quotes,value=2]", "test.CommonBenchmark.longBenchmark" ], "diffScorePercentage": [ 8.85092766075268, 10.113605263533874, 10.274883930931813, 10.669534402130527, 10.770240429602211, 10.787495514466844, 10.974183650061766, 11.098718582020316, 11.191816391817635, 11.300194767143958, 11.649592108256316, 11.973029312249324, 13.121201184949738, 13.13597942403738, 13.142817758516887, 14.117753886684326, 14.509698788964847, 15.300444329388892, 15.551161272060545, 16.31569096407434 ], "barColor": [ "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos", "pos" ] }, "ggsize": { "width": 800.0, "height": 900.0 }, "kind": "plot", "scales": [ { "aesthetic": "x", "name": "Diff %", "limits": [ null, null ] }, { "aesthetic": "y", "discrete": true, "name": "" }, { "aesthetic": "fill", "values": [ "#ee6666", "#3ba272" ], "limits": [ "neg", "pos" ], "guide": "none" } ], "layers": [ { "mapping": { "x": "diffScorePercentage", "y": "testLabel", "fill": "barColor" }, "stat": "identity", "orientation": "y", "sampling": "none", "inherit_aes": false, "position": "dodge", "geom": "bar", "tooltips": { "lines": [ "@|@{diffScorePercentage}" ], "formats": [ { "field": "diffScorePercentage", "format": ".2f" } ], "disable_splitting": true } } ], "theme": { "title": { "margin": [ 10.0, 0.0, 10.0, 0.0 ], "blank": false }, "axis_ontop": false, "axis_ontop_y": false, "axis_ontop_x": false }, "data_meta": { "series_annotations": [ { "type": "float", "column": "diffScorePercentage" }, { "type": "str", "column": "testLabel" }, { "type": "str", "column": "barColor" } ] } }, "apply_color_scheme": true, "swing_enabled": true } }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "execution_count": 12 } ], "metadata": { "kernelspec": { "display_name": "Kotlin", "language": "kotlin", "name": "kotlin" }, "language_info": { "name": "kotlin", "version": "2.2.20", "mimetype": "text/x-kotlin", "file_extension": ".kt", "pygments_lexer": "kotlin", "codemirror_mode": "text/x-kotlin", "nbconvert_exporter": "" } }, "nbformat": 4, "nbformat_minor": 0 }