# @fluid-tools/benchmark This package contains benchmarking tools to profile runtime and memory usage. It provides the tools you need to profile your own code (described below) and various utilities for customizing and configuring benchmarks, as well as authoring alternative reporters and profiling functions (e.g. to support test frameworks other than mocha). ## General use This package exports `benchmarkIt` which is used instead of mocha's `it()` to define benchmark tests. Tests written with `benchmarkIt` double as correctness tests: without `--perfMode` they behave like any other mocha `it()` test. To run them as benchmark tests, invoke `mocha` as you normally would for your package but pass some additional arguments, like this: ```console --v8-expose-gc --perfMode --fgrep @Benchmark --reporter @fluid-tools/benchmark/dist/mocha/Reporter.js ``` ### `--perfMode` (required) Enables performance-testing mode instead of correctness-test mode. In performance-testing mode, many iterations are run and measured for each test. In correctness-test mode, only one iteration is run and no measurement takes place. Can also be set using the `FLUID_TEST_PERF_MODE` environment variable. If using [Mocha's Parallel Mode](https://mochajs.org/features/parallel-mode/), then this must be set using the environment variable. ### `--v8-expose-gc` (required) This is necessary so the package can perform explicit garbage collection between tests to help reduce cross-test contamination and get accurate results for memory tests. ### `--fgrep @Benchmark` All tests created with the tools in this package are tagged with `@Benchmark`, so use `--fgrep @Benchmark` to run only benchmark tests. Each test is also tagged based on its `type` option: `@Measurement` (the default), `@Perspective`, or `@Diagnostic`. Use a more specific filter like `--fgrep @Measurement` to run only tests of a particular type. Note that Mocha only supports one `fgrep` or `grep` argument at a time, so if you have other filtering, it may need to be merged into a single more complex `--grep` option. ### `--reporter ` Lets you specify the path to a custom reporter to output the tests' results. This package includes `dist/mocha/Reporter.js`. If you don't specify one, the default mocha reporter will take over and you won't see benchmark information. ### `--reporterOptions reportFile=` If you use the reporter from this package, you can set the output file path with this. All benchmark results are combined into a single JSON file using a hierarchical `ReportArray` structure. If omitted, no file is written (results are still printed to the console). ### `--parentProcess` If you pass this **optional** flag when, each performance test is run in a forked child process. The forked process runs only that test and propagates the results back to the parent. This can have significant overhead (the child process reruns mocha test discovery which may incur significant startup cost, in addition to the overhead of forking Node.js), but reduces the influence of previous tests on the state of the JIT and heap. Test this thoroughly in your scenario to make sure the tradeoffs are worthwhile. Using `--parentProcess` is currently unsupported with [Mocha's Parallel Mode](https://mochajs.org/features/parallel-mode/). ## Profiling durations To profile runtime durations, use `benchmarkIt` together with `benchmarkDuration`: ```typescript benchmarkIt({ title: "My sync test", ...benchmarkDuration({ benchmarkFn: () => { // synchronous code to benchmark }, }), }); benchmarkIt({ title: "My async test", ...benchmarkDuration({ benchmarkFnAsync: async () => { // asynchronous code to benchmark }, }), }); ``` `benchmarkDuration` accepts a `DurationBenchmark`, which must have exactly one of: - `benchmarkFn` — a synchronous function to benchmark. - `benchmarkFnAsync` — an asynchronous function to benchmark. - `benchmarkFnCustom` — an async function that manages the batching loop directly via a `BenchmarkTimer` argument, for cases where you need full control over timing. Use `state.timeBatch(fn)` for the common case, or `state.timer.now()` / `state.recordBatch()` when you need to exclude setup/teardown from the measured time. It also accepts optional `BenchmarkTimingOptions` to tune `maxBenchmarkDurationSeconds`, `minBatchCount`, and `minBatchDurationSeconds`. See the documentation for `DurationBenchmark` and `BenchmarkTimingOptions` for more details. > **NOTE**: Be aware of pitfalls when benchmarking impure functions. > The test execution strategy assumes each iteration of `benchmarkFn` is an independent event. > For full control over per-batch setup and teardown, use `benchmarkFnCustom`. > See `DurationBenchmarkCustom` for details. ## Profiling custom measurements To report fully custom measurements, call `benchmarkIt` directly and provide a `run` function that returns a `CollectedData`: ```typescript benchmarkIt({ title: "My custom measurement", run: (): CollectedData => [ { // The first element is the primary measurement (all fields required). name: "My metric", value: 42, units: "things/op", type: ValueType.SmallerIsBetter, significance: "Primary", }, // Additional measurements are optional. { name: "Sample count", value: 100, units: "count", }, ], }); ``` `CollectedData` is a tuple `[PrimaryMeasurement, ...Measurement[]]`. The first element is the primary measurement (used for regression detection) and requires all fields: `name`, `value`, `units`, `type`, and `significance: "Primary"`. Additional measurements are optional; their `units`, `type`, and `significance` fields are also optional. `collectDurationData` and `collectMemoryUseData` can be called directly within the `run` function, which is more flexible than `benchmarkDuration` or `benchmarkMemoryUse` when you need to add custom measurements or run setup/teardown outside the timed region: ```typescript benchmarkIt({ title: "My custom duration measurement", run: async (): Promise => { // Optional setup can run here const data = await collectDurationData({ benchmarkFn: () => { // code to benchmark }, }); // Extra measurements can be added: return [...data, { name: "Extra metric", value: 1 }]; }, }); ``` ## Profiling memory usage To profile memory usage, use `benchmarkIt` together with `benchmarkMemoryUse`: ```typescript benchmarkIt({ title: "My memory test", ...benchmarkMemoryUse({ benchmarkFn: async (state) => { // If your test requires one-time setup, do it here: const holder: { value: unknown } = { value: undefined }; while (state.continue()) { // Release the previous allocation, then do any per-iteration setup. holder.value = undefined; // Collect a baseline "before" heap measurement. await state.beforeAllocation(); // Allocate the memory you want to measure. holder.value = createSomething(); // Collect an "after allocation" heap measurement. await state.whileAllocated(); // To help confirm you are measuring the allocation you expect, // you can optionally free it here then call afterDeallocation: // holder.value = undefined; // await state.afterDeallocation(); } }, }), }); ``` This measures the difference in the retained portion of the heap from `beforeAllocation` to `whileAllocated`. This does not include memory that was used during the operation but released before `whileAllocated` was called. The argument to `benchmarkMemoryUse` must implement `MemoryUseBenchmark`, which requires a `benchmarkFn` property. That function receives a `MemoryUseCallbacks` object and must loop until `state.continue()` returns false, following this pattern each iteration: 1. Release references to any memory allocated in the previous iteration (so GC can reclaim it). 2. Set up anything needed to do the allocation under test that should not be included in the measurement. 3. `state.beforeAllocation()` — GC runs and a baseline "before" heap measurement is taken. 4. Do the operation whose memory allocation you want to measure. 5. `state.whileAllocated()` — GC runs and an "after allocation" heap measurement is taken. 6. _(Optional)_ Free memory, then call `state.afterDeallocation()` — if called, GC runs and an "after deallocation" heap measurement is taken as well. The benchmark reports the mean heap difference between the "while allocated" and "before" readings across iterations. Memory must not accumulate across iterations (i.e., what you allocate in step 4 must be fully releasable). For more details, see the documentation for `MemoryUseBenchmark` and `MemoryUseCallbacks`. ## Contribution Guidelines You can [contribute](https://github.com/microsoft/FluidFramework/blob/main/CONTRIBUTING.md) to Fluid Framework in these ways: - Answer questions in [GitHub Discussions](https://github.com/microsoft/FluidFramework/discussions). - [Submit bug reports](https://github.com/microsoft/FluidFramework/issues) and help verify fixes. - Review [source code changes](https://github.com/microsoft/FluidFramework/pulls). - [Contribute bug fixes](https://github.com/microsoft/FluidFramework/blob/main/CONTRIBUTING.md). For detailed instructions, read the [repo documentation](https://github.com/microsoft/FluidFramework/blob/main/docs/content/Home.md). This project follows the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information, read the [Code of Conduct frequently asked questions](https://opensource.microsoft.com/codeofconduct/faq/). For questions or comments, contact . This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft’s [Trademark & Brand Guidelines](https://www.microsoft.com/trademarks). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. ## Help Read the [Fluid Framework documentation](https://fluidframework.com/docs/) for information about Fluid Framework concepts and APIs. To request information that the documentation does not contain, [create an issue](https://github.com/microsoft/FluidFramework/blob/main/docs/content/Contributing/Submitting-Bugs-and-Feature-Requests.md). ## Trademark This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services. Use of these trademarks or logos must follow Microsoft's [Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.