# Comet Tuning Guide Comet provides some tuning options to help you get the best performance from your queries. Every deployment needs to configure how much memory Comet can use, so start with memory tuning. The guide is split into the following pages: - [Memory Tuning](tuning/memory.md): configuring Comet's off-heap memory pool and the executor memory overhead, choosing a memory pool, batch size, and limiting spill disk usage. - [Shuffle Tuning](tuning/shuffle.md): enabling Comet shuffle, the native and columnar shuffle implementations, limiting the native shuffle writer's memory, and shuffle compression. - [Remote Shuffle with Celeborn](tuning/celeborn.md): bounding frame sizes and memory admission for native remote shuffle. - [Parquet Scan Tuning](parquet.md#tuning): row filter pushdown and late materialization, and sizing splits so that row groups spread evenly across scan tasks. - [Iceberg Scan Tuning](iceberg.md#tuning): reading data files in parallel with `spark.comet.scan.icebergNative.dataFileConcurrencyLimit`. - [Operator Tuning](tuning/operators.md): joins, adaptive partial aggregation, and sorting on floating-point values. - [Expression Tuning](tuning/expressions.md): lambdas and higher-order functions such as `filter`, and the built-in array functions that replace them. - [Reducing Row/Columnar Conversion Overhead](tuning/transitions.md): stages in which many operators fall back to Spark, and the experimental direct columnar-to-row converter. ## Nested and Wide Data Rows with many columns, long strings, or nested columns such as arrays of structs make every batch larger, so queries over such data spill more. For these workloads: - Choose the codec for spill files from native sorts, aggregations, and joins, which Comet compresses with `lz4` by default. See [Compressing Spill Files](tuning/memory.md#compressing-spill-files). - Lower the batch size if wide batches cause spilling or out-of-memory errors. See [Batch Size](tuning/memory.md#batch-size). - Check which shuffles can run natively when partitioning keys are nested. See [Native Shuffle](tuning/shuffle.md#native-shuffle). - Watch for row/columnar conversions, which are expensive for such schemas. See [Wide or Deeply Nested Schemas](tuning/transitions.md#wide-or-deeply-nested-schemas). ## Configuring Tokio Runtime Comet uses a global tokio runtime per executor process. By default it starts one worker thread per core that the executor runs tasks on, and allows up to 512 blocking threads, which is tokio's default. Comet takes the number of cores from the first of these that applies: - the thread count of a `local`, `local[N]` or `local[*]` master, in local mode - `spark.executor.cores`, when it is set - the cores per worker `C` of a `local-cluster[N, C, M]` master - the number of processors available to the executor's JVM on a standalone (`spark://`) cluster, since a standalone executor without `spark.executor.cores` takes every core its worker offers, and a worker offers all of its machine's cores unless it is started with a different number (`SPARK_WORKER_CORES`) - one on YARN and Kubernetes, which is their default for `spark.executor.cores` On any other cluster manager, such as Mesos, Comet cannot tell how many cores the executor has when `spark.executor.cores` is not set. It starts one worker thread for each processor available to the executor's JVM, and logs a warning unless `COMET_WORKER_THREADS` is set. Too few worker threads cost more than a few idle ones: native plans that read no input from the JVM, such as a native Parquet scan feeding a sort, run entirely on the worker threads, so tasks share them when an executor runs more tasks at once than it has worker threads. The worker and blocking thread counts can be overridden using the environment variables `COMET_WORKER_THREADS` and `COMET_MAX_BLOCKING_THREADS`. ## Metrics Overhead The SQL metrics described in [Metrics](metrics.md) are always collected. Setting `spark.comet.metrics.enabled=true` additionally publishes plan-coverage counters (`operators.native`, `operators.spark`, `queries.planned`, `transitions`, and `acceleration.ratio`) through Spark's metrics system under the `comet` source. It is disabled by default because it walks every executed plan on the driver after each query, and the counters are only useful with an external sink (for example Prometheus) configured. This setting must be applied before the `SparkSession` is created. ## Explain Plan For an explanation of Comet plan output, the configs that control it, and how fallback to Spark works, see [Understanding Comet Plans](understanding-comet-plans.md).