:toc:
:toclevels: 4

:sectnums:

= MLPerf Training Rules
Jul 24, 2026

== Overview
This document describes how to implement the MLPerf® Training Suite using an ML framework and how to use that implementation to measure the performance of an ML software framework or hardware.

There are seperate rules for the submission, review, and publication process for all MLPerf benchmarks https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here].

The MLPerf name and logo are trademarks of the MLCommons® Association ("MLCommons"). In order to refer to a result using the MLPerf name, the result must conform to the letter and spirit of the rules specified in this document. MLCommons reserves the right to solely determine if a use of its name or logos is acceptable.

=== Definitions (read this section carefully)
The following definitions are used throughout this document:

_Performance_ always refers to execution speed.

_Quality_ always refers to a model’s ability to produce “correct” outputs.

A _system_ consists of a defined set of hardware resources such as processors, memories, disks, and interconnect. It also includes specific versions of all software such as operating system, compilers, libraries, and drivers that significantly influences the running time of a benchmark, excluding the ML framework.

A _framework_ is a specific version of a software library or set of related libraries, possibly with associated offline compiler, for training ML models using a system. Examples include specific versions of Caffe2, MXNet, PaddlePaddle, pyTorch, or TensorFlow.

A _benchmark_ is an abstract problem that can be solved using ML by training a model based on a specific dataset or simulation environment to a target quality level.

A _suite_ is a specific set of benchmarks.

A _division_ is a set of rules for implementing benchmarks from a suite to produce a class of comparable results.

A _reference implementation_ is a specific implementation of a benchmark provided by the MLPerf organization.

A _benchmark implementation_ is an implementation of a benchmark in a particular framework by a user under the rules of a specific division.

A _submission implementation set_ is a set of benchmark implementations for one or more benchmarks from a suite under the rules of a specific division using the same framework.

A _run_ is a complete execution of an implementation on a system, training a model from initialization to the quality target.

A _run result_ is the wallclock time required for a run.

A _reference result_ is the result provided by the MLPerf organization for each reference implementation

A _benchmark result_ is the mean of a benchmark-specific number of run results, dropping the highest and lowest results. The result is then normalized to the reference result for that benchmark. Normalization is of the form (reference result / benchmark result) such that a better benchmark result produces a higher number.

A _submission result set_ is a one benchmark result for each benchmark implementation in a  submission implementation set.

A _submission_ is a submission implementation set and a corresponding submission result set.

A _custom summary result_ is the weighted geometric mean of an arbitrary set of results from a specific submission. MLPerf endorses this methodology for computing custom summary results but does not endorse any official summary result.

_latest version available_ is the last MLPerf submission suite that a benchmark was part of.

== General rules
The following rules apply to all benchmark implementations.

=== Strive to be fair
Benchmarking should be conducted to measure the framework and system performance as fairly as possible. Ethics and reputation matter.

=== System and framework must be consistent
The same system and framework must be used for a submission result set. Note that the reference implementations do not all use the same framework.

=== Benchmark detection is not allowed
The framework and system should not detect and behave differently for benchmarks.

=== Pre-training is not allowed
Unless part of the definition of a benchmark, the implementation should not encode any information about the content of the dataset or a successful model’s state in any form. High-level statistical information about the dataset, such as distribution of sizes, may be used.

For DeepSeekv3, manipulation of metadata which consists of the number of documents in the dataset and the size of each document is allowed as long as the data tokens are not accessed.

For benchmarks which are defined as starting from a fixed set of weights, such as a checkpoint or backbone, the implementation should start from the weights provided in the benchmark reference definition, or if that is not posssible, provide  information and code sufficient for reproducing how those starting weights were obtained.

== Benchmarks
The benchmark suite consists of the benchmarks shown in the following table.

|===
|Area|Problem |Dataset |Latest version available

|Vision |Text to Image |CC12M |v6.1
|Language | Small LLM MoE pretraining |c4/en/3.0.1 |v6.1
| |Small LLM pretraining |c4/en/3.0.1 |v6.1
| |LLM finetuning |SCROLLS GovReport |v6.1
| |LLM MoE pretraining |c4/en/3.0.1 |v6.1
| |LLM post training |R2E-Gym-Subset (easy split) |v6.1
|Commerce |Recommendation |Yambda-5B |v6.1

|===

### Benchmark Lifecycle

Developing high-quality benchmarks requires significant effort, computational resources, and commitment. Therefore, each benchmark is expected to remain part of the benchmark suite for a minimum of two years or four submission rounds, whichever comes first.

A benchmark may be considered for early retirement due to reasons such as, but not limited to, low industry adoption. Early retirement requests will be reviewed by the Training working group, followed by a formal vote to determine the benchmark's status.

### Benchmark reference implementation

MLCommons provides a reference implementation of each benchmark, which includes the following elements:

Code that implements the model in a framework.

A plain text “README.md” file that describes:

* Problem
** Dataset/Environment
** Publication/Attribution
** Data preprocessing
** Training and test data separation
** Training data order
** Test data order
** Simulation environment (RL models only)
** Steps necessary for reproducing the initial set of weights, if an initial set of non-standard weights is used. 
** Publication/Attribution
** List of layers
** Weight and bias initialization
** Loss function
** Optimizer
* Quality
** Quality metric
** Quality target
** Evaluation frequency (training items between quality evaluations)
** Evaluation thoroughness (test items per quality evaluation)
* Directions
** Steps to configure machine
** Steps to download and verify data
** Steps to run and time

A “download_dataset” script that downloads the dataset.

A “verify_dataset” script that verifies the dataset against the checksum.

A “run_and_time” script that executes the benchmark and reports the wall-clock time.

== Divisions
There are two divisions of the benchmark suite, the Closed division and the Open division.

=== Closed Division
The Closed division requires using the same preprocessing, model, training method, and quality target as the reference implementation.

The closed division models and quality targets are:

|===
|Area |Problem |Model |Target |Latest version available

|Vision |Text to image |FLUX.1 |0.586 Eval loss |v6.1
|Language | Small LLM MoE pretraining | GPT_OSS_20B | 3.34 log perplexity |v6.1
|  |Small LLM pretraining |Llama31_8b |3.3 log perplexity |v6.1
| |LLM finetuning |Llama2_70B_LoRA |0.925 Eval loss |v6.1
| |LLM MoE pretraining |deepseekv3_671b |3.6 log perplexity |v6.1
| |LLM post training |Qwen3.5-397B-A17B |0.69 pass@4 |v6.1
|Commerce |Recommendation |dlrm_v4_hstu |0.75 AUC |v6.1
|===

Closed division benchmarks must be referred to using the benchmark name plus the term Closed, e.g. “for the Recommendation Closed benchmark, the system achieved a result of 7.2.”

=== Open Division
The Open division allows using arbitrary training data, preprocessing, model, and/or training method. However, the Open division still requires using supervised or reinforcement machine learning in which a model is iteratively improved based on training data, simulation, or self-play.

Open division benchmarks must be referred to using the benchmark name plus the term Open, e.g. “for the Recommendation Open benchmark, the system achieved a result of 7.2.”

== Basics

=== Random numbers
CLOSED: Random numbers must be generated using stock random number generators.

Random number generators must be seeded from the following sources:

* Clock
* System source of randomness, e.g. /dev/random or /dev/urandom
* Another random number generator initialized with an allowed seed

Random number generators may be initialized repeatedly in multiple processes or threads. For a single run, the same seed may be shared across multiple processes or threads.

From v4.1 onwards, the seeds should be logged and they need to satisfy the following requirements:

* The only way to log seeds is through https://github.com/mlcommons/logging/tree/master/mlperf_logging/mllog[`mllog`]. Any seed logged via any other method is discarded.
* All seeds must be valid integer (convertible via https://docs.python.org/3/library/functions.html#int[`int()`]).
* We expect all runs to log at least one seed.
* If one run logs one seed on a certain line in a certain source file, no other run can log the same seed on the same line in the same source file. What files are considered as source files are defined https://github.com/mlcommons/logging/blob/master/mlperf_logging/package_checker/seed_checker.py#L7[here].

Unsatisfying any of the above requirements will result in seed checker failures reported by the https://github.com/mlcommons/logging/tree/master/mlperf_logging/package_checker[package checker].

If any run logs more than one seed, a warning is raised by the package checker. This is a reminder to submitters to rethink their design because using multiple seeds per run should not be necessary.

OPEN: Any random number generation may be used. The seed is not expected to be logged.

=== Numerical formats
CLOSED: The numerical formats fp64, fp32, tf32, fp16, fp8, mxfp6, nvfp4, mxfp4, bfloat16, Graphcore FLOAT 16.16, int8, uint8, int4, and uint4 are pre-approved for use. Additional formats require explicit approval. Any numerically safe techniques needed to compensate for different precision may be used, provided they leave the real-valued mathematical function of the model, i.e. the function the model would compute at infinite precision, unchanged. 

Some examples of numerically safe techniques that are allowed include but are not limited to:  

* Scaling, such as per-tensor or per-block 
* Orthogonal pre/post transformations, such as https://en.wikipedia.org/wiki/Hadamard_transform[`Hadamard Transforms`] around a linear operator   
* Rounding, such as https://royalsocietypublishing.org/doi/epdf/10.1098/rsos.211631[`Stochastic rounding`] 

Some examples of techniques that are _not_ allowed under this rule: 

* Normalization or gradient clipping different from the reference model 
* Increasing the number of trainable parameters in the model 

Techniques that are not numerically safe require explicit approval from the working group.

Reference Convergence Points must be obtained using FP32 precision, FP32 emulation with explanation of the methodology for emulation, or BF16 precision.

OPEN: Any format and scaling may be used.

=== Lowest precision, parallelism, and run-config logging
From v6.1 onwards, every result log in both the Closed and Open divisions must report lowest numerical precision, parallelism, micro-batch size, and the config filename through https://github.com/mlcommons/logging/tree/master/mlperf_logging/mllog[`mllog`].

The following keys must appear exactly once in each result log:

* `lowest_numerical_precision_in_linear`: lowest numerical precision used in linear / GEMM compute
* `lowest_numerical_precision_in_attn`: lowest numerical precision used in attention compute
* `lowest_numerical_precision_in_comm`: lowest numerical precision used in communication, for example parameter gather or all-gather
* `tensor_parallelism`
* `pipeline_parallelism`
* `context_parallelism`
* `expert_parallelism`
* `micro_batch_size`
* `config_filename`: the configuration file used for the run, named so that it can be matched to the submitted config

Precision values must be one of the pre-approved numerical formats listed in <<Numerical formats>> unless the working group has approved an additional format. Parallelism and `micro_batch_size` values must be positive integers. Unused parallelism dimensions must be logged as `1`. `config_filename` must be a non-empty string.

=== Epoch numbering
Epochs should always be numbered from 1.

=== Result rounding
Public results should be rounded normally.

== Data Set

=== Data State at Start of Run
CLOSED: Each reference implementation includes a script to download the input dataset and script to verify the dataset using a checksum. The data must then be preprocessed in a manner consistent with the reference implementation, excepting any transformations that must be done for each run (e.g. random transformations). The data may also be reformatted for the target system provided that the reformatting does not introduce new information or introduce duplicate copies of data.

OPEN: Any public dataset may be used for training the model, however the evaluation data must be drawn from the benchmark dataset in a manner consistent with the reference.

You must flush the cache or restart the system prior to benchmarking.	Data can start on any durable storage system such as local disks and cloud storage systems. This explicitly excludes RAM.

=== Preprocessing During the Run
Only preprocessing that must be done for each run (e.g. random transformations) must be timed.

CLOSED: The same preprocessing steps as the reference implementation must be used.

OPEN: Any preprocessing steps are allowed for training data. However, each datum must be preprocessed individually in a manner that is not influenced by any other data. The evaluation data must be preprocessed in a manner consistent with reference.

=== Data Representation

CLOSED: Images must have the same size as in the reference implementation. Mathematically equivalent padding of images is allowed.

CLOSED: For benchmarks with sequence inputs, you may choose a length N and either truncate all examples to length N or throw out all examples which exceed length N. This must be done uniformly for all examples. This may only be done on the training set and not the evaluation set.

CLOSED: Two ways to represent the Mask R-CNN mask are permitted. One is a polygon and the other is a scalable bitmask.

OPEN: The closed division data representations restrictions only apply at the start of the run. Data may be represented in an arbitrary fashion during the run.

=== Other Uses of Data

Input encoding data, such as language vocabulary, or the set of possible labels may used during pre-processing or execution without counting as "touching the training data" for timing purposes. Same applies to processing metadata like the number of documents, or document sizes in a dataset.

=== Training and Test Sets
CLOSED: If applicable, the dataset must be separated into training and test sets in the same manner as the reference implementation.

OPEN: If applicable, the test dataset must be extracted in the same manner as the reference implementation. The training data set may not contain data that appears in the test set.

=== Training Data Order
CLOSED: the training and test data must be traversed in the same conceptual order as the reference implementation. For instance, the data might be traversed sequentially or randomly with uniform distribution. Batch size, shard size, and the random number generator will affect order.

Where data pipelines randomly order data, arbitrary sharding, batching, and packing are allowed provided that (1) the data is still overall randomly ordered and not ordered to improve convergence and (2) each datum still appears exactly once. Modifications to data order and/or batching must be presented to the SWG group in advance of the submission deadline for approval if they could affect the ability to borrow hyperparameters and/or approximately follow the learning rate schedule defined by the RCPs.

In the case of dlrm_v4_hstu benchmark, the training data is not shuffled. The dataset is carved by wall clock into a Global Temporal Split, and within the training portion the timeline is sliced into fixed-duration windows (`streaming_window_seconds = 86400`) which are traversed in strict forward chronological order: the loop trains window `T`, then evaluates window `T+1`, then advances. Reordering windows, or any shuffling that moves samples across window boundaries, is prohibited, since doing so would leak future interactions into the training history. Within a window, anchors are consumed in dataset order; the reference exposes an optional partial intra-window shuffle (`STREAMING_SHUFFLE_FRACTION`) which must be left at its default of `0.0` for closed division submissions.

OPEN: The training data may be traversed in any order. The test data must be traversed in the same order as the reference implementation.

== RL Environment
CLOSED: The implementation must use the same RL algorithm and simulator or game as the reference implementation, with the same parameters.

OPEN: The implementation may use a different RL algorithm but must use the same simulator or game with the same parameters. If the reference implementation generates all data online, the Open division implementation must also generate all data online.

It is allowed and encouraged to parallelize and otherwise optimize (e.g. by implementing in a compiled language) the RL environment provided that the semantics are preserved.

== Model
CLOSED: The benchmark implementation must use the same model as the reference implementation, as defined by the remainder of this section.

OPEN: The benchmark implementation may use a different model.

=== Graph Definition

CLOSED: Each of the current frameworks has a graph that describes the operations performed during the forward propagation of training. The frameworks automatically infer and execute the corresponding back-propagation computations from this graph. Benchmark implementations must use the same graph as the reference implementation.

=== Weight and Bias Initialization
CLOSED: Weights and biases must be initialized using the same constant or random value distribution as the reference implementation, unless a pre-trained set of weights, such as a checkpoint or backbone, is used by the reference.

OPEN: Weights and biases must be initialized using a consistent constant or random value distribution.

=== Graph Execution
CLOSED: Frameworks are free to optimize the non-weight parts of the computation graph provided that the changes are mathematically equivalent. So optimizations and graph / code transformations of the flavor of dead code elimination, common subexpression elimination, loop-invariant code motion, and recomputation of node state are entirely allowed.

OPEN: Frameworks are free to alter the graph.

== Training Loop

=== Hyperparameters
CLOSED:

By default, the hyperparameters must be the same as the reference.

Hyperparameters include the optimizer used and values like the regularization norms and weight decays.

The implementation of the optimizer must match the optimizer specified in the Appendex: Allowed Optimizer.  The Appendex lists which optimizers in the popular deep learning frameworks are compliant by default.  If a submission uses an alternate implementation, the submitter must describe the optimizer's equation and demonstrate equivalence with the approved optimizers on that list.

The following table lists the tunable hyperparameters for each allowed model,optimizer combination. The value of each tunable hyperparameter must meet the listed constraint.

The MLPerf verifier scripts checks all hyperparameters except those with names marked with asterisks. If a hyperparameter is marked with one asterisk, it must be checked manually. If a hyperparameter is marked with two asterisks, it is also not logged and it must be checked manually in the code.  If the verifier and the constraints in this table differ, the verifier (specifically, the version on the date of submission unless otherwise decided by the review committee) is the source of truth.

|===
 |Model |Optimizer |Name |Constraint |Definition |Reference Code |Latest version available

 |gpt_oss_20b |adamw |global_batch_size |unconstrained |batch size in sequences |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L53[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_adamw_beta_1 |0.9 |AdamW beta1 |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L66[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_adamw_beta_2 |0.95 |AdamW beta2 |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L67[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_adamw_epsilon |1e-5 |AdamW epsilon |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L68[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |link:https://github.com/mlcommons/training/tree/master/gpt-oss-20b/primus[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_adamw_weight_decay |0.1 |weight decay |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L63[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_learning_rate_warmup_steps |unconstrained |steps taken for linear warmup. |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L60[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_learning_rate_decay_steps |1_200_000 - opt_learning_rate_warmup_steps |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps). |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L61[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_base_learning_rate |unconstrained |base learning rate (after warmup); end LR = opt_base_learning_rate * 0.1 per opt_end_learning_rate |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L58[reference code] |v6.1
 |gpt_oss_20b |adamw |opt_end_learning_rate |opt_base_learning_rate * 0.1 |learning rate at the last step of decay period |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L59[reference code] |v6.1
 |gpt_oss_20b |adamw |dropout |0.0 |Disable all dropouts during training. |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L75[reference code] |v6.1
 |gpt_oss_20b |adamw |sequence_length |8192 |sequence length |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L55[reference code] |v6.1
 |gpt_oss_20b |adamw |gradient_accumulation_steps |unconstrained |Number of fwd/bwd steps between optimizer step. |link:https://github.com/mlcommons/training/tree/master/gpt-oss-20b/primus[reference code] |v6.1
 |gpt_oss_20b |adamw |max_steps |1200000 | maximum number of steps |link:https://github.com/mlcommons/training/blob/4e3737bd2423bcfa05a394e5abc01d95c9e1e436/gpt-oss-20b/primus/conf/gpt_oss_20B-pretrain.yaml#L52[reference code] |v6.1
 |llama31_8b |adamw |global_batch_size |unconstrained |batch size in sequences |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L304[reference code] |v6.1
 |llama31_8b |adamw |opt_adamw_beta_1 |0.9 |AdamW beta1 |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L154[reference code] |v6.1
 |llama31_8b |adamw |opt_adamw_beta_2 |0.95 |AdamW beta2 |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L154[reference code] |v6.1
 |llama31_8b |adamw |opt_adamw_epsilon |1e-5 |AdamW epsilon |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L154[reference code] |v6.1
 |llama31_8b |adamw |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L154[reference code] |v6.1
 |llama31_8b |adamw |opt_adamw_weight_decay |0.1 |weight decay |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L154[reference code] |v6.1
 |llama31_8b |adamw |opt_learning_rate_warmup_steps |unconstrained |steps taken for linear warmup. |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L156[reference code] |v6.1
 |llama31_8b |adamw |opt_learning_rate_decay_steps |1_200_000 - opt_learning_rate_warmup_steps |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps). |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L421[reference code] |v6.1
 |llama31_8b |adamw |opt_base_learning_rate |unconstrained |refer to next table in section "Llama 3.1 learning rates" |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L306[reference code] |v6.1
 |llama31_8b |adamw |opt_end_learning_rate |opt_base_learning_rate * 0.1 |learning rate at the last step of decay period |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L157[reference code] |v6.1
 |llama31_8b |adamw |dropout |0.0 |Disable all dropouts during training. |link:https://github.com/mlcommons/training/blob/master/small_llm_pretraining/nemo[reference code] |v6.1
 |llama31_8b |adamw |sequence_length |8192 |sequence length |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L357[reference code] |v6.1
 |llama31_8b |adamw |gradient_accumulation_steps |unconstrained |Number of fwd/bwd steps between optimizer step. |link:https://github.com/mlcommons/training/blob/master/small_llm_pretraining/nemo[reference code] |v6.1
 |llama31_8b |adamw |max_steps |1200000 | maximum number of steps |link:https://github.com/mlcommons/training/blob/46bbcb7c1881ce7b6eca917b19621c375dcc3240/small_llm_pretraining/nemo/pretrain_llama31.py#L310[reference code] |v6.1
 |llama2_70b_lora |adamw |global_batch_size |unconstrained |batch size in sequences |See PR (From NV and Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |opt_gradient_clip_norm |fixed to referance (0.3) | Gradients are clipped above this norm threshold. |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |lora_dropout |0.1 |fixed to reference (0.1). |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |sequence_length |8196 |the sequence length - fixed to reference |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |lora_alpha |fixed to referance (32) | scaling factor for the LoRA weight matrices |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |opt_weight_decay |fixed to referance (0.0001) |weight decay |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |gradient_accumulation_steps |unconstrained |Numer of fwd/bwd steps between optimizer step. |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |opt_learning_rate_warmup_ratio | unconstrained |ratio of steps out of training for linear warmup during initial checkpoint generation. This only affects the learning rate curve in the benchmarking region. |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |opt_learning_rate_training_steps | unconstrained |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps]. |See PR (From Habana, TODO Link) |v6.1
 |llama2_70b_lora |adamw |opt_base_learning_rate |unconstrained | base leraning rate |See PR (From Habana, TODO Link) |v6.1
 |flux.1 |adamw |global_batch_size |unconstrained |The global batch size for training |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/config_manager.py#L198[reference_code] |v6.1
 |flux.1 |adamw |opt_adamw_beta_1 |0.9 |coefficients used for computing running averages of gradient and its square |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/components/optimizer.py#L281[reference_code] |v6.1
 |flux.1 |adamw |opt_adamw_beta_2 |0.95 |coefficients used for computing running averages of gradient and its square |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/components/optimizer.py#L281[reference_code] |v6.1
 |flux.1 |adamw |opt_adamw_epsilon |1e-08 |term added to the denominator to improve numerical stability |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/config_manager.py#L135[reference_code] |v6.1
 |flux.1 |adamw |opt_adamw_weight_decay |0.1 |weight decay coefficient |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/components/optimizer.py#L282[reference_code] |v6.1
 |flux.1 |adamw |opt_base_learning_rate |unconstrained |base learning rate, this should be the learning rate after warm up |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/config_manager.py#L132[reference_code] |v6.1
 |flux.1 |adamw |opt_learning_rate_warmup_steps |unconstrained |number of steps for learning rate to warm up |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/config_manager.py#L157[reference_code] |v6.1
 |flux.1 |adamw |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |link: https://github.com/pytorch/torchtitan/blob/3d7ea4b6b7e5cd2decf11a1c37581d1cd3678fe0/torchtitan/config_manager.py#L204[reference_code] |v6.1
 |deepseekv3_671b |adamw |global_batch_size |>=15360 |batch size in sequences |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/run_deepseek_v3_671b.sh#L45[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_adamw_beta_1 |0.9 |AdamW beta1 |https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/cf15864f8e3ccd5da25282144c2d95cc208d2e2a/src/megatron/bridge/recipes/utils/optimizer_utils.py#L112[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_adamw_beta_2 |0.95 |AdamW beta2 |https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/cf15864f8e3ccd5da25282144c2d95cc208d2e2a/src/megatron/bridge/recipes/utils/optimizer_utils.py#L113[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_adamw_epsilon |1e-8 |AdamW epsilon |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/pretrain_deepseek_v3_671b.py#L176[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/cf15864f8e3ccd5da25282144c2d95cc208d2e2a/src/megatron/bridge/recipes/utils/optimizer_utils.py#L118[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_adamw_weight_decay |0.1 |weight decay |https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/cf15864f8e3ccd5da25282144c2d95cc208d2e2a/src/megatron/bridge/recipes/utils/optimizer_utils.py#L115[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_learning_rate_warmup_steps |4 |steps taken for linear warmup. |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/run_deepseek_v3_671b.sh#L62[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_learning_rate_decay_steps |12000 - opt_learning_rate_warmup_steps |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps]. |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/pretrain_deepseek_v3_671b.py#L82[reference_code] |v6.1
 |deepseekv3_671b |adamw |opt_base_learning_rate |0.000024 * sqrt(global_batch_size / 16384) |base learning rate |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/run_deepseek_v3_671b.sh#L63[reference_code] |v6.1
 |deepseekv3_671b |adamw |sequence_length |4096 |sequence length |https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/v0.3.0/src/megatron/bridge/models/deepseek/deepseek_provider.py#L62[reference_code] |v6.1
 |deepseekv3_671b |adamw |moe_routed_bias_update_rate |1e-3 |The expert bias is updated based on the number of assigned tokens to each expert
    in a global batch, where the bias is increased for the experts with less assigned tokens
    and decreased for the experts with more assigned tokens. |https://github.com/NVIDIA/Megatron-LM/blob/b0cc2706ddc60d2aefd5fff346445b5c013036a8/megatron/core/transformer/transformer_config.py#L639-L643[reference_code] |v6.1
 |deepseekv3_671b |adamw |gradient_accumulation_steps |unconstrained |Number of fwd/bwd steps between optimizer step. |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/pretrain_deepseek_v3_671b.py#L94[reference_code] |v6.1
 |deepseekv3_671b |adamw |max_steps |unconstrained |maximum number of steps |https://github.com/mlcommons/training/blob/daff3992f09b00326d404eaedf575727ff078170/llm_moe_pretraining/nemo/run_deepseek_v3_671b.sh#L61[reference_code] |v6.1
 |qwen35_397b_grpo |adamw |global_batch_size |must equal num_prompts_per_step * num_generations_per_prompt |global batch size in problems (prompts) per training step |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |num_prompts_per_step |unconstrained positive integer; must satisfy global_batch_size = num_prompts_per_step * num_generations_per_prompt |number of prompts (problems) sampled per training step |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |num_generations_per_prompt |16 |GRPO group size; number of independent trajectories generated per prompt |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |loss_fn.truncated_importance_sampling_ratio |1.002 |upper bound for truncated importance sampling correction |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml#L69[reference code] |v6.1
 |qwen35_397b_grpo |adamw |loss_fn.truncated_importance_sampling_ratio_min |0.999 |lower bound for truncated importance sampling correction |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml#L70[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_adamw_beta_1 |0.9 |AdamW beta1 |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_adamw_beta_2 |0.999 |AdamW beta2 |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_adamw_epsilon |1e-8 |AdamW epsilon |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_adamw_weight_decay |0.0 |weight decay |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_name |`adamw` |optimizer name; fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_gradient_clip_norm |0.125 * sqrt(256 / global_batch_size) |gradients are clipped above this norm threshold |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_base_learning_rate |1.0e-6 * sqrt(global_batch_size / 256) |base learning rate |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_learning_rate_warmup_steps |0 |no linear warmup |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |opt_learning_rate_decay_schedule |constant |learning rate schedule; no decay |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |init_checkpoint_step |0 |first step after loading the pretrained Qwen3.5-397B-A17B checkpoint |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |max_steps |unconstrained |maximum number of training steps |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |max_sequence_length |65536 |maximum sequence length (context + response); also caps max response length |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |eval_samples |256 (TBD pending legal review) |number of held-out R2E-Gym validation tasks; fixed; pass@4 uses `num_val_generations_per_prompt = 4`, so the resulting number of validation rollouts is 1024 (`256 × 4`) |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |agent_max_turns |30 |maximum agent turns per rollout (reference config key `env.nemo_gym.swe_agents_*.responses_api_agents.swe_agents.agent_max_turns`); fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |generation_training_rollout_temperature |1.0 |temperature used for training-time rollouts; fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |generation_training_rollout_top_p |1.0 |top_p used for training-time rollouts; fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |generation_training_rollout_top_k |`null` (disabled) |top_k used for training-time rollouts; the reference sets `policy.generation.top_k: null`, so top_k sampling is disabled |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |generation_validation_rollout_temperature |0.1 |temperature used for validation-time rollouts (pass@4 requires stochastic sampling); fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |generation_validation_rollout_top_p |0.95 |top_p used for validation-time rollouts; fixed |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |data.shuffle |false |the training-data shuffle flag must be off, fixing the global traversal order of training problems to ensure a stable solve rate and convergence curve; within a training step, prompts may still finish generation in any order as their trajectories complete |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |grpo.use_dynamic_sampling |false |dynamic sampling is the reference switch for oversampling (drawing more than num_generations_per_prompt rollouts and discarding unfinished or errored generations); it must be disabled so that submitters use exactly num_generations_per_prompt generations per prompt |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |grpo.async_grpo.max_trajectory_age_steps |tunable positive integer, *not borrowable* |staleness knob: number of policy updates between the time a rollout is generated and the time its trajectory is used for the policy update; each submitter must choose their own value and it cannot be borrowed via Hyperparameter Borrowing |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |env.nemo_gym.swe_agents_*.responses_api_agents.swe_agents.agent_cls |`CodeActAgent` |the OpenHands agent class driving the sandbox must be `CodeActAgent`; the agent control loop is part of the fixed sandbox contract |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |env.nemo_gym.swe_agents_*.responses_api_agents.swe_agents.agent_framework_commit |fixed to reference (`0d766ad06b2be64a42e6f0175b9ebcc4a06599d9`) |the OpenHands framework commit baked into the sandbox image must match the reference, so that the tool specification available to the model is identical across submissions |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |env.nemo_gym.swe_agents_*.responses_api_agents.swe_agents.agent_prompt_overrides |fixed to reference `user_prompt.j2` and `system_prompt.j2` |the injected system/user prompt templates are part of the model context that becomes training data and are treated as equivalent to the dataset; sandbox may run on CPUs of any architecture and does not need to be colocated with training/generation, but any dedicated CPU capacity used for the sandbox counts against the overall submission scale |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |env.nemo_gym.swe_agents_*.responses_api_agents.swe_agents.diversify_tool_names |false |tool-name diversification is part of the fixed sandbox contract and must be off |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |partial_rollouts |disallowed (no reference config key) |saving straggler rollouts into a buffer and restarting them in a later batch is not allowed; there is no reference config knob for this, and it may be reconsidered in future rounds once sandbox environments support persistent partial trajectory state |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |grpo.async_grpo.in_flight_weight_updates |allowed (`true` or `false`) |whether policy weight updates on generation workers occur without stalling ongoing rollouts; permitted to be enabled by submitters (reference RCPs use `false`) |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |lora |disallowed (no reference config key) |LoRA-based GRPO is not allowed because it changes the model architecture and reduces the active parameter count; submitters must perform full-parameter GRPO |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |qwen35_397b_grpo |adamw |policy.megatron_cfg.draft.enabled |false |the Megatron draft-model speculative-decoding path must be disabled; speculative decoding requires model changes (draft model or additional prediction head) that the reference implementation does not support |link:https://github.com/NVIDIA-NeMo/RL/blob/3fca04c9b313d313302923a5bb6b0c8dc0340ed6/qwen_35/configs/grpo_qwen35_397b_swe_openhands_async.yaml[reference code] |v6.1
 |dlrm_v4_hstu |adam |global_batch_size |unconstrained |global batch size; the reference per-rank `BATCH_SIZE` multiplied by the world size |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_base_learning_rate |[0.5 * 1e-6 * (global_batch_size / 8192), 1.5 * 1e-6 * (global_batch_size / 8192)] |dense target (post-warmup) learning rate; the warmup ramps to this value and then holds it, there is no decay. The reference convergence points are generated at the midpoint of this range, 1e-6 * (global_batch_size / 8192) |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_name |`Adam` |dense optimizer name; fixed |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_adam_beta_1 |0.95 |Adam beta1 |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_adam_beta_2 |0.999 |Adam beta2 |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_adam_epsilon |1e-8 |Adam epsilon |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_weight_decay |0.0 |weight decay |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_gradient_clip_norm |1.0 |gradients are clipped above this norm threshold; clipping is applied to the **dense** parameters only, the fused sparse optimizer is unaffected |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_sparse_name |`RowWiseAdagrad` |sparse (embedding table) optimizer name; fixed, fused FBGEMM TBE |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_sparse_base_learning_rate |must equal opt_base_learning_rate |sparse embedding learning rate; the reference ramps both channels in lockstep, so this tracks whichever dense learning rate the submitter chooses |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_sparse_epsilon |1e-8 |RowWiseAdagrad epsilon |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_sparse_weight_decay |0.0 |RowWiseAdagrad weight decay |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_learning_rate_warmup_steps |24000 |steps taken for the linear warmup ramp |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |opt_learning_rate_warmup_start_lr |0.0 |learning rate at global step 0; the warmup always begins from zero |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |max_sequence_length |4096 |maximum HSTU sequence length |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |hstu_attn_num_layers |3 |number of HSTU attention layers; part of the model definition and may not be changed in the closed division |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |history_length |4086 |user interaction history length; part of the model definition and may not be changed in the closed division |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |train_samples |2290835423 |number of training samples in one epoch; fixed by the dataset |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |eval_samples |2058204 |size of the held-out future evaluation window; every evaluation scores the entire window, subsampling is not allowed |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
 |dlrm_v4_hstu |adam |eval_every_data_pct |0.001 |evaluation cadence as a fraction of the training stream; fixed. The mutually exclusive per-window cadence (`eval_every_n_windows`) must be disabled |link:https://github.com/mlcommons/training/blob/master/recommendation/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin[reference code] |v6.1
|===
For the tunable hyperparameters used in the past for deprecated benchmarks see <<Deprecated benchmarks hyperparameters>>

OPEN: Hyperparameters and optimizer may be freely changed. 


==== DeepSeek-V3-671B hyperparameter constraints

Since training large language models is very expensive, the task force aims to limit hyperparameter searches. Thus the allowed range of batch sizes and corresponding learning rates are fixed as follows.

|===
 |global_batch_size |opt_base_learning_rate |opt_learning_rate_warmup_steps

 |15360 |0.000023238 |4
 |16384 |0.000024 |4
 |18432 |0.000025456 |4
|===

* GBS<15360 - not allowed
* GBS within range but not listed: opt_base_learning_rate = 0.000024 * sqrt(GBS / 16384), rounded to the **9th** decimal place.

If a new learning rate is needed for any GBS point, request new RCPs from the task force or normalize the score if permissible.

==== qwen35_397b_grpo hyperparameter constraints

Since GRPO post training of a 397B-parameter mixture-of-experts model is very expensive, the task force aims to limit hyperparameter searches. Closed division submitters must use the reference GRPO loss function documented at link:https://docs.nvidia.com/nemo/rl/0.2.1/guides/grpo.html#loss[NeMo RL GRPO loss]. The base learning rate and gradient clipping norm are constrained by the global batch size (GBS), and the reference set of GBS points is fixed as follows.

|===
 |global_batch_size |opt_base_learning_rate |opt_gradient_clip_norm |num_prompts_per_step |num_generations_per_prompt

 |256 |1.0e-6 |0.125 |16 |16
 |512 |1.4142135624e-6 |0.08838834765 |32 |16
 |1024 |2.0e-6 |0.0625 |64 |16
|===

* GBS within range but not listed: opt_base_learning_rate = 1.0e-6 * sqrt(GBS / 256), rounded to the **10th** decimal place; opt_gradient_clip_norm = 0.125 * sqrt(256 / GBS), rounded to the **11th** decimal place.
* GBS<256 or GBS>1024 - Apply above formula as long as RCP checker passes. If there are issues, reach out to the task force for new RCPs to be generated. 
* GBS must always satisfy global_batch_size = num_prompts_per_step * num_generations_per_prompt with num_generations_per_prompt fixed to 16.

If a new learning rate or clip norm is needed for any GBS point, request new RCPs from the task force or normalize the score if permissible.

Refer to the `qwen35_397b_grpo` compliance checker configuration at link:https://github.com/mlcommons/logging/blob/master/mlperf_logging/compliance_checker/training_6.1.0/closed_qwen35_397b_grpo.yaml[closed_qwen35_397b_grpo.yaml] and the reference convergence points at link:https://github.com/mlcommons/logging/blob/master/mlperf_logging/rcp_checker/training_6.1.0/rcps_qwen35_397b_grpo.json[rcps_qwen35_397b_grpo.json].

=== Hyperparameter Borrowing

Submitters are expected to use their best efforts to submit with optimal hyperparameters for their system.  The intent of Hyperparameter Borrowing is to allow a submitter to update their submission to reflect what they would have submitted had they known about more optimal hyperparameters before submitting, without knowing any other info (ie the performance of other submissions).

During the review period as described in the Submission Rules, a submitter may replace the hyperparameters, once per benchmark entry, in their implementation of a benchmark with hyperparameters from another submitter's implementation of the same benchmark. By default, they may change batch size (local batch size, global batch size, batchnorm span), but must replace all other hyperparameters as a group.

With evidence that the resulting model, using the same batch size as the other submitter's implementation, converges worse in terms of epochs required, the submitter may make a minimum number of additional hyperparameter changes for the purpose of improving convergence and achieving comparable, but not better, convergence in epochs compared to the other submitter's implementation, but preserving any difference in convergence that may exist due to precision choices. In this situation, the other submitter's implementation is considered the reference, and the new submitter must match the convergence behavior of the other submitter in a similar way as we compare any submission to the reference.

A resubmission of a benchmark with borrowed hyperparameters must use the same software (with the exceptions listed in the Software Adoption section of this document), system and system configuration (accelerators, NICs etc) as the original submission.  The largest scale submission for a benchmark from a given system may be resubmitted with borrowed hyperparameters using a change of scale on that system, but only if the new scale is either larger, or enables the resubmission to achieve a faster run result.  In addition, the new scale must not be larger than the largest scale used in an original submission of at least one of the benchmarks on that system in this round.

Since the hyperparameters are fixed for qwen35_397b_grpo and Deepseekv3, hyperparameter borrowing is not allowed for qwen35_397b_grpo and Deepseekv3.

=== Loss function
CLOSED: The same loss function used in the reference implementation must be used.

OPEN: Any loss function may be used. Do not confuse the loss function with target quality measure.

=== Quality measure
Each run must reach a target quality level on the reference implementation quality measure. By default, the time to evaluate the quality is included in the wallclock time. However, if the reference implementation generates timestamped checkpoints and evaluates the quality after the clock has been stopped, then an implementation may either perform evaluation on-the-clock or generate timestamped checkpoints, evaluate them after the clock has been stopped, and update the clock stopped time to the timestamp of the first passing checkpoint. The checkpoint timestamp may be any time after the last weight value included in the checkpoint is updated.

CLOSED: The same quality measure as the reference implementation must be used. The quality measure must be evaluated at the same frequency (in terms of number of training items between test sets) and at least as thoroughly (in terms of number of tests per set) as in the reference implementation. Where applicable, the required evaluation point may be rounded up to the nearest batch size. Typically, a test consists of comparing the output of one forward pass through the network with the desired output from the test set.

|===
|Area |Problem |Model|Evaluation frequency |Latest version available

|Vision |Text to image |Flux.1 |Every 262144 training samples. CEIL(262144 / global_batch_size) * global_batch_size if 262144 is not divisible by GBS |v6.1
|Language | Small LLM MoE pretraining | GPT_OSS_20B| Every 12288 sequences; `CEIL(12288 / global_batch_size) * global_batch_size` steps if 12288 is not divisible by GBS. |v6.1
|        |Small LLM pretraining |Llama31_8b| Every 12288 sequences; `CEIL(12288 / global_batch_size) * global_batch_size` steps if 12288 is not divisible by GBS. |v6.1
|        |LLM finetuning |Llama2_70B_LoRA| Every 384 sequences; `CEIL(384 / global_batch_size) * global_batch_size` steps if 384 is not divisible by GBS. Skipping first `FLOOR(0.125*global_batch_size+2)` evaluations |v6.1
|        |LLM MoE pretraining |deepseekv3_671b| First evaluation at `GBS * FLOOR(42 + 24576 / GBS)` samples; every step thereafter. |v6.1
|        |LLM post training |qwen35_397b_grpo| First evaluation at `global_batch_size * CEIL(2.5 + 3840 / global_batch_size)` samples; every training step (i.e., every `global_batch_size` samples) thereafter. |v6.1
|Commerce|Recommendation |dlrm_v4_hstu|Every `eval_interval_samples = CEIL(0.001 * 2290835423 / global_batch_size) * global_batch_size` training samples, i.e. 2293760 samples at each of GBS 8192, 16384 and 32768. First evaluation at `CEIL(FLOOR((4480 * global_batch_size + 135331840) / 3) / eval_interval_samples) * eval_interval_samples` samples |v6.1
|===
For the quality measure used for deprecated benchmarks see <<Deprecated benchmarks quality measure>>

OPEN: An arbitrary stopping criteria may be used, including but not limited to the closed quality measure, a different quality measure, the number of epochs, or a fixed time. However, the reported results must include the geometric mean of the final quality as measured by the closed quality measure.

Check points can be created at the discretion of submitter. No check points are required to be produced or retained.

=== Equivalence exceptions
The CLOSED division allows limited exemptions to mathematical equivalence between implementations for pragmatic purposes, including:

* Different methods can be used to add color jitter as long as the methods are of a similar distribution and magnitude to the reference.

* If data set size is not evenly divisible by batch size, one of several techniques may be used. The last batch in an epoch may be composed of the remaining samples in the epoch, may be padded, or may be a mixed batch composed of samples from the end of one epoch and the start of the next. If the mixed batch technique is used, quality for the ending epoch must be evaluated after the mixed batch. If the padding technique is used, the first batch may be padded instead of the last batch.

* Values introduced for padding purposes may be reflected in batch norm computations.

* Adam optimizer implementations may use the very small value epsilon to maintain mathematical stability in slightly different ways, provided that methods are reviewed and approved in advance. One such method involves squaring the value of epsilon and moving epsilon inside the square root in the parameter update equation.

* Distributed batch normalization is allowed.

Additional exemptions need to be explicitly requested and approved in advance. In general, exemptions may be approved for techniques that are common industry practice, introduce small differences that would be difficult to engineer around relative to their significance, and do not substantially decrease the required computation. Over time, MLPerf should seek to help the industry converge on standards and remove exemptions.

The OPEN division does not restrict mathematical equivalence.

== Results Resubmission

Submitters may resubmit logs from a previous round in a future round if those logs represent their best result and they wish to showcase it. Log resubmission is permitted only if the system.json matches exactly between both rounds and the resubmitted logs pass the package_checker with the future round's mlperf_logging tag.

Submitters must notify the review committee of their intent to resubmit logs by creating an issue in the submission repository before the first review meeting. The review committee will verify the validity of the resubmission before approval. Late notifications may result in rejection of the resubmission request.

[#section-run-results]
== Run Results
A run result consists of a wall-clock timing measurement for a contiguous period that includes model initialization in excess of a maximum initialization time, any data preprocessing required to be on the clock, using the dataset to train the model, and quality evaluation unless specified otherwise for the benchmark.

Prior to starting the clock, a system may use a maximum model initialization time of 30 minutes for _Closed_ division and 4 hours for _Open_ division. Model initialization time begins when the system first begins to construct or execute the model. This maximum initialization time is intended to ensure that model initialization is not disproportionate on large systems intended to run much larger models, and may be adjusted in the future with sufficient evidence.

The clock must start before any part of the system touches the dataset or when the maximum model initialization time is exceeded. The clock may be stopped as soon as any part of the system determines target accuracy has been reached. The clock may not be paused during the run.

=== Result files
A valid MLPerf result requires that result log files (result_*.txt) include the full framework logs and all MLLOG lines. Submitters may redact internal paths and environment variables, or truncate the logs if the file size exceeds 100 MiB, in accordance with GitHub's file size limit. Because framework specific logging is not standardized, submission automation cannot enforce this requirement. Thus during the review process, any submitter can be asked by a peer reviewer or the review committee to provide the complete untruncated result log file if it was not already included. Failure to provide the untruncated and unmodified log upon request may result in removal of the result at the discretion of the review committee.

== Benchmark Results
Each benchmark result is based on a set of run results. The number of results for each benchmark is based on a combination of the variance of the benchmark result, the cost of each run, and the likelihood of convergence.

|===
|Area|Problem |Minimum Number of Runs |Latest version available

|Vision|Text to Image |10 |v6.1
|Language | Small LLM MoE pretraining | 10 | v6.1
| |Small LLM pretraining |10 |v6.1
| |LLM MoE pretraining |3 |v6.1
| |LLM finetuning |10 |v6.1
| |LLM post training |3* |v6.1
|Commerce |Recommendation |10 |v6.1
|===
For the minimum number of runs that were required for deprecated benchmarks see <<Deprecated benchmarks results>>

Each benchmark result is computed by dropping the fastest and slowest runs, then taking the mean of the remaining times. For this purpose, a single non-converging run may be treated as the slowest run and dropped. A benchmark result is invalid if there is more than one non-converging run.


Each benchmark result should be normalized by dividing the reference result for the corresponding reference implementation by the benchmark result. This normalization produces higher numbers for better results, which better aligns with human intuition.

*LLM post training benchmark sees very high evaluation variance because eval is non-deterministic and GRPO training is very sensitive to changes. To mitigate the impact of this on the benchmark and keep submission costs in check, the LLM post training benchmark does late eval to hide variance and keep minimum number of submission runs low.

=== Scoring

An MLPerf submission score is intended to represent the median expected result across a large number of runs.

To reduce statistical variance and the potential to cherry pick results, each benchmark submission is composed of a set of N independent runs, with N chosen based on the observed variation of the benchmark, as described in the table above.

Running multiple iterations of N independent runs with the goal of validating that the submission is close to a median result is encouraged but not required.  Running multiple iterations of N runs to try find the lowest one is against the spirit of MLPerf and is prohibited – see Section 2.1, “Strive to be fair”.  Results that appear to be too far away from a median result may be rejected.

As a more computationally efficient method of validating that a submission is close to the median result, it is also allowed to run M>N independent runs as a group and to designate N consecutive runs from the group as the runs to be used for scoring, provided that the submitter chooses the N consecutive runs that are closest to the median result.  For the purposes of calculating the median, sets of N consecutive runs that would create an invalid benchmark result should be included in the median calculation as "infinite" scores.  If the median set would be an invalid benchmark result, the entire result is invalid.  Submitting the full run set (vs just the N runs used for scoring) as a reference is optional, but may be required in the future.  For purposes of this scoring, "consecutive" is defined as an objective and deterministic method, such as submission timestamps.  Submitters are not allowed to pick different orderings to improve their score.  Runs may go in parallel on the submitter's compute resources, as long as there is a way to objectively and deterministically sort the runs, for example by timestamp.

An example could be for a benchmark with N=5 runs, a submitter could ahead of time pick M=10, launch 10 runs on their compute resources, sort the 10 runs by their launch time stamp, then take a sliding window of 5 consecutive runs over those 10 runs.  That sliding window would create 6 possible sets of 5 runs.  Each of those 6 sets would be olympically scored, and the set with the median runtime would be submitted as that submitter's score.  Any failed runs within those 10 runs would count as infinity time and need to be included in the olympic scoring (could be thrown away as the slowest score).  It is recommended that a submitter keep the logs for all M runs, because the review committee may ask for the submitter to share the M logs during the review period.

Cherry-picking or unfairly advantaged submissions cannot always be conclusively proven. If such behavior is suspected during the review period, the review chairs may take corrective action. Suspicion may arise from inconsistencies in log files, significant deviations from other results obtained on equivalent hardware/software, or any other equivalent methodology. Corrective actions may include, but are not limited to, requesting a new set of runs using the exact same software and hyperparameters as the original submission, or requiring withdrawal of any result deemed to be in violation. The above guidelines provide a general framework because each situation is unique. The review chairs have the authority to determine the appropriate course of action depending on the specific issue.

=== Score scaling

The score of an MLPerf submission may be scaled if the training committee decides so during the review period. This scaling may be, but not limited to failing to meet the reference convergence limits imposed by the Reference Convergence Points (see following section). To facilitate the automatic generation of the scaled score the scaling factor must be provided in a json file under the name scaling.json in the directory whose scores are going to be scaled.

== Reference Convergence Points (RCPs)

Reference Convergence Points are used to ensure that the convergence of the submission does not deviate from the convergence of the reference. We are interested in avoiding cases where the submission convergence is faster than the reference. Reference implementation convergence sets a lower bound on epoch convergence that a valid submission should not beat. From a statistical standpoint if the submission mean epochs to converge is significantly lower than the reference mean epochs to converge, then submission convergence points belong to a different population than the reference convergence points, and thus the submission should not be accepted. Compliance to reference convergence points is validated as follows

* Reference implementations provide at least 2N epoch convergence numbers, where N is the number of submission runs needed for each benchmark. Since convergence is affected by batch size (larger batch size means slower convergence), reference implementations provide convergence data for a few different batch sizes.
* After a set of Reference Convergence Points is gathered, we find the minimal set of these points that are needed for the fastest possible convergence. For example, if the RCP for batch size 128 is at 10 epochs, the RCP for batch size 256 is at 20 epochs, and the RCP for batch size 512 is also at 20 epochs, then we prune the RCP at the 256 batch size. Based on the assumption that convergence increases with batch size, we expect to be able to converge faster than 20 epochs at batch size 256. In practice we prune ALL RCP points that have slower convergence than the linear interpolation at the same batch size of any two surrounding points. Eventually we end up with a pruned set of RCPs which defines the fastest possible convergence of the reference code as a function of batch size.
* A potential submitter can request generation of new RCPs by suggesting a better set of hparams to the WG or generate new RCPs by running the reference themselves. A request for a new RCP run should be backed by at least one run on either the submitter’s code or the reference code proving faster convergence. A request to generate RCPs should be made in the Training WG meeting at least 8 weeks before submission deadline and the reference owner (or a volunteer appointed by WG) should provide the RCP at least 4 weeks before submission deadline. Subject to WG's approval, requester's set of convergence points (2N runs) may act as temporary RCPs for that round if the RCP request is not met by a timely response.
* For DeepSeekv3 and qwen35_397b_grpo, a request to generate RCPs should be made in the Training WG meeting at least 9 weeks before submission deadline and the reference owner (NVIDIA) should provide RCPs (2N runs each) at least 5 weeks before submission deadline so that all submitters have enough time to train with the new hparams. The RCP requests should be handled in FCFS order and if there are more than 5 RCP requests, the WG should decide if the requester's set of convergence points (2N runs) can be used as temporary RCPs.
* Using the mean and standard deviation of the reference convergence we apply a 1-sided independent two-sample Student's t-test with unequal sample sizes, similar variances with p-value=0.05 (explained link:https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes,_similar_variances_(1/2_%3C_sX1/sX2_%3C_2)[here]) to find the maximum acceptable speedup for submission convergence.
* At submission time, the submission is matched to an RCP based on the submission batch size.
** If there is an RCP for that batch size then mean epochs to converge of the submission is extracted from submission logs. If this does not violate the maximum acceptable speedup condition when compared to the reference then the submission is accepted, otherwise it may be rejected.
** If there is no RCP for that batch size but there are RCPs for smaller and larger batch sizes an interpolated RCP is created, and the mean epochs to converge is compared against the interpolated RCP just like in the previous case
** If the submission batch size is larger than the batch size of any RCP the submitter must provide the missing RCPs by running the reference implementation with their batch size.
** If the submission batch size is smaller that the batch size of any RCP AND the convergence test against the RCP with the minimum batch size fails, then again the submitter must provide the missing RCPs by running the reference implementation with their batch size.
** Accepted submissions with mean epochs lower than RCP mean (faster) but within the acceptable speedup range are normalized to (potentially interpolated) RCP mean epochs for fairness. New normalized score = Submission-olympic-score * (RCP-mean / olympic-submission-epochs)

Please refer to the related Appendix for examples that shed light to the RCP process.


=== RCP Rules and Guidelines

Submitters are encouraged to run the RCP checker script prior to their submission to make sure they do not violate RCP limits.

If a submission fails the RCP test, such as S2 in the Appendix, they have the option to submit with the --rcp-bypass parameter. To submit with the --rcp-bypass when submitting using the submission UI, create a file called package_checker_params in the root directory / system directory / benchmark results directory of your submission and put the flags you want to use there. This will allow the submission to upload, but the submitter MUST notify the results chair, and prepare for the audit process described in the next section where at review time the submitter should be able to justify why their submission is valid while it failed the RCP test.

If a submission is missing the RCP for the batch size they are submitting, such as S4 and S6 in the Appendix they must provide the missing convergence points by making a PR in the logger. All missing RCPs are due 7 days before the submission deadline. (Exception is Deepseekv3 and qwen35_397b_grpo: where RCPs are due 5 weeks before the submission deadline) RCPs are added by making a pull request into the RCP library in the logging repository.

During hyperparameter borrowing, borrowers can use hyperparameters from submissions that passed or failed the RCP test. If their submission fails to pass the RCP test they can have it upload by using --rcp-bypass and then prepare for the audit decribed in the next section.

To extract submission convergence points, logs should report epochs as follows.
|===
| Benchmark | Epoch reporting | Latest version available
| GPT_OSS_20B | Training samples starting from 0 (integer) | v6.1
| Llama31_8b | Training sample starting from 0 (integer) | v6.1
| Llama2_70B_LoRA | Training sample (integer) | v6.1
| FLUX.1 | Training sample (integer) | v6.1
| deepseekv3_671b | Training samples starting from 0 (integer) | v6.1
| qwen35_397b_grpo | Training samples starting from 0 (integer) | v6.1
| dlrm_v4_hstu | Training samples starting from 0 (integer) | v6.1
|===
Refer to epoch reporting for deprecated benchmarks at <<Epoch reporting for deprecated benchmarks>>

=== Handling RCP Failures

In order to reduce the burden on the submitter as well as the Submitter’s Working Group (SWG) during the review period, submitters shall ensure compliance with RCP tests ahead of the submission deadline. Submissions that need new RCPs are required to supply those RCPs 7 days before the submission deadline, as specified in the Training Rules document. While providing new RCPs, a submitter must also include reference run logs for the SWG and reference owner to review.

Submissions with failing RCP tests are rejected by default until the SWG approves the submission. Submitters shall notify the SWG in advance of a potential RCP failure, so they can prefetch requests for additional data and minimize churn during the review period.  A submitter requesting approval for a submission with failing RCP test shall provide additional explanatory data to the SWG explaining why the WG should consider the non-compliant submission a fair comparison to compliant submissions. This list will be decided by the WG for each submission individually. 

A non-exhaustive list of potential requests of data is: 

1. Written statement from the submitter explaining the plausible cause of deviation. This should also be supported by data from A/B experiments.
2. Logs showing training loss of the submission vs training loss of the reference. Note that the reference run should be on reference hardware platform using the reference numerical precision.
3. Model summary showing number of trainable_parameters (weights) in the model vs the same. 
4. Debugging via comparing intermediate activations, distributions of initialization weights, and/or compliant randomization on the reference vs the submission.
The SWG may further request additional information, not listed above, at their discretion.

A submitter requesting approval for their RCP failing submission during the review period shall provide requested information in a timely manner. All evidence supporting the appeal is due at the time of submission.

The SWG must come to majority consensus to approve a submission that fails the RCP test.  If the SWG cannot come to majority consensus to approve a submission, then potential alternatives are:

1. Normalize submission run epochs to reference epochs to pass RCP test irrespective of accuracy achieved
2. Submission is withdrawn due to non-compliance

If a submission fails the RCP checker, but there is evidence from this submission round or previous submission rounds that other comparable submissions using the same or similar hyperparameters and software stack have passed the RCP checker, then the failing submission will be rejected. The reasoning is that, under these conditions, normal convergence behavior should allow the submission to pass the RCP checker; a failure indicates the result is anomalously fast and thus cannot be accepted for fairness. To avoid rejection, submitters must ensure their results pass the RCP checker either by rerunning the submission or by adjusting hyperparameters (such as lowering the learning rate) to slow convergence and bring it closer in line with reference behavior.

== Appendix: Benchmark Specific Rules [[benchmark_specific_rules]]

* Node Classification
** Timed region: Graph and feature loading, training, evaluation are all timed. Graph-partitioning for multi-node runs is not timed.
** Node features are in fp32 in the dataset, but lower precisions are allowed. Feature precision can be converted offline. 
** Any sparse format may be used for storing the graph. Offline conversion is allowed. 
** Graph partitioning algorithm and locality:
*** Any any general non-data-aware partitioning algorithm that is reproducible, either using a fixed seed or a deterministic algorithm
*** We require that each graph node’s feature can only be read from disk on one exclusive training node. Other training nodes that need this graph node’s feature should fetch it over the network
** Caching: Graph caching is allowed, but feature caching is not allowed. 
** Sampler: Submitters are not expected to exactly match reference sampler implementation due to known framework differences, but must meet RCP criteria.

* Flux.1
** 10 runs per submission
** Evaluation must be run every 262,144 samples. CEIL(262144 / global_batch_size) if 262144 is not divisible by GBS.

* Bert

** Clip-normalization order: The 1.0 and 1.1 exception that benchmarks may implement clip-normalization either before or after accelerator all-reduce has been extended indefinitely to future rounds.

** --rcp-bert-train-samples log compliance parameter: For all benchmarks other than Bert, convergence for RCP purposes is reported in the last eval_accuracy line of the log file. For Bert, submitters are allowed to add an extra log line with key set to train_samples and value the number of samples to converge. If that is the case, the package compliance checker should be run with the --rcp-bert-train-samples command line parameter.

== Appendix: Examples of Compliant Optimizers

Analysis to support this can be found in the document "MLPerf Optimizer Review" in the MLPerf Training document area.
TODO: locate the document and provide working link

|===
| Benchmark | Algorithm | Framework | Optimizer Implementations

| Image classification | LARS                     | PyTorch	| [No compliant implementation]
|      |                          |	TensorFlow | MLPERF_LARSOptimizer
|      |                          | MxNet | SGDwFASTLARS
| Image classification | SGD with Momentum        | PyTorch	| apex.optimizers.FusedSGD
|      |                          |	PyTorch | torch.optim.SGD
|      |                          |	TensorFlow | tf.train.MomentumOptimizer
|      |                          | MxNet | [No compliant implementation]
| Object detection (heavy weight)	| SGD with Momentum	  | PyTorch	| apex.optimizers.FusedSGD
|      |                          | PyTorch	| torch.optim.SGD
|      |                          | TensorFlow | tf.train.MomentumOptimizer
| Object detection (light weight)  | ADAM	      | PyTorch	| torch.optim.Adam
|      |                          | TensorFlow | tf.keras.optimizers.Adam
| NLP | LAMB             	      | PyTorch	| apex.optimizers.FusedLAMB
|      |              	          | TensorFlow	| tf.optimizers.LAMB
| Large Language Model | Adam             	      | PyTorch	| apex.optimizers.FusedAdam
|      |                         | PaxML	| praxis.optimizers.Adam
| Speech recognition | LAMB             	      | PyTorch	| apex.optimizers.FusedLAMB
|      |              	          | TensorFlow	| tf.optimizers.LAMB
| Recommendation | Adagrad             	        | PyTorch	| torch.optim.Adagrad (dense layers) + torchrec.optim.Adagrad (embeddings)
| Image segmentation (medical) | SGD with Momentum      | PyTorch	    | torch.optim.SGD
|      |              	          | TensorFlow	| tf.train.MomentumOptimizer
|      |              	          | MXNet	    | mx.optimizer.NAG
|===

== Appendix: v0.7 Specific Rules

For benchmarks which are defined as starting from a fixed set of weights, such as a checkpoint or backbone, the implementation should start from the weights provided in the benchmark reference definition, or if that is not posssible, provide information and code sufficient for reproducing how those starting weights were obtained. For v0.7, sets of weights used in v0.6 are allowed.

== Appendix: v1.0 Specific Rules

This section contains rules specific to the v1.0 round of MLPerf Training.  These do not apply to future rounds, unless explicitly ratified as rules for those rounds, or unless these rules are promoted to official rules in previous sections of this document.

=== MaskRCNN backbone

For v1.0 only, Mask-RCNN submitters may use the non-reference backbone located https://drive.google.com/drive/folders/1lGU_pP2Pr2k578DslciwE4AlOKm6dKCK?usp=sharing[here] with the understanding that it converges similarly to the reference backbone.  If the non-reference backbone is shown to converge faster than the reference backbone at any scale on any submitted hyperparameter set, all uses of that backbone for any submitter are to be re-run with the reference backbone to have their submission published.  For future rounds, the expectation is that all submitters will use the reference backbone, which will fixed at reference code freeze time.

=== BERT clip-normalization order

For v1.0 only, BERT submissions may implement clip-norm either before or after inter-accelerator all-reduce. For future rounds, the expectation is that submissions must use clip-norm-after-reduce, to be consistent with most commonly used public BERT model repos.

For performance consistency of at scale BERT submissions for v1.0, submitters are disallowed from using clip-norm-after-reduce to enable additional overlap of communication and math. If a submitter plans to use clip-norm-after-reduce for v1.0, they must notify the committee before the submission deadline, and be prepared to show code in their submission proving that they do not do overlap as a result of clip-norm-after-reduce.

Furthermore, for simplicity, the RCPs for this round will use clip-norm-before-reduce.  In theory, this could allow clip-norm-after-reduce submissions that converge faster than they should, but still not faster than clip-norm-before-reduce, but the Training Working Group feels that this is ok risk for v1.0, in interest of simplifying the RCPs for v1.0.

=== Compile time

For v1.0 only, the allowed untimed compile time is increased from 20 minutes to 30 minutes.  This is to enable new submitters to submit who were close to the 20 minute limit.  The 20 minute number was chosen empirically for rounds prior to v1.0.  For v1.1 and beyond, the training working group should make a data driven decision on what compile time is reasonable for real user applications.

== Appendix: v1.1 Specific Rules

This section contains rules specific to the v1.1 round of MLPerf Training.  These do not apply to future rounds, unless explicitly ratified as rules for those rounds, or unless these rules are promoted to official rules in previous sections of this document.

=== BERT clip-normalization order

For v1.0 only, BERT submissions may implement clip-norm either before or after inter-accelerator all-reduce. For future rounds, the expectation is that submissions must use clip-norm-after-reduce, to be consistent with most commonly used public BERT model repos.  This exception from v1.0 was extended to v1.1 because of the tight schedule between rounds.

=== Preview Category

For v1.1, we changed the policy documentation to say that a Preview submission needs to be available at the next submission after 140 days, not 180 days like it was before.  However, this does not apply to Preview submissions from v1.0, which will still follow the 180 day policy.  For v1.1 Preview submissions and beyond, the 140 day rule will apply.  This is not necessarily an "exception," but we are listing it here as a special case for the record.

== Appendix: v3.0 Specific Rules

=== Large Language Model (GPT3)

Allowed model initialization compile time for GPT3 benchmark is increased to 60 minutes for Closed Division owing to the large memory footprint of initial checkpoints.

== Appendix: v5.0 Specific Rules

=== Large Language Model (Llama 3.1 405B)

This section contains rules specific to Llama 3.1 405B of the v5.0 round of MLPerf training. Since we observe that a larger global batch size leads to increased variation when training Llama 3.1 405B to convergence, we introduce the following benchmark-specific rules for Llama 3.1 405B benchmark for MLPerf v5.0: 

For v5.0's Llama 3.1 405B benchmark only, the maximum Global Batch Size (GBS) for Llama 3.1 405B benchmark will be constrained at 9216. 

For v5.0's Llama 3.1 405B benchmark only, if a submission with 6912 &le; GBS &le; 9216 is unable to pass the RCP checker, extra runs must be submitted during the review period to demonstrate that the submitted convergence distribution matches the reference. 


For v5.0's Llama 3.1 405B benchmark only, we allow submissions that converge 1 evaluation (46,080 samples) earlier than generally allowed by the reference. This is because we noticed that for some global batch sizes it is possible to fail the RCP checker even when running with the reference codebase (NeMo Framework) and the reference precision of BF16 due to how the benchmark is designed. Additionally, experiments using MaxText Framework and FP32 precision fail RCPs for a few global batch sizes and there are submitters who want to use MaxText in v5.0. Given that we are 2 weeks away from submission, the Working Group has agreed to allow submissions that converge **1 evaluation (46080 samples) earlier than generally allowed by the reference** to pass the RCP checker via link:https://github.com/mlcommons/logging/pull/417[PR417]. To ensure fairness, the scores will still be normalized to the RCP mean as usual. This exception only applies to v5.0 and the task force plans on debugging further for v5.1

Evaluation: In v5.0, we did few sparse evaluation every `46080` sequences. `CEIL(46080 / global_batch_size) * global_batch_size` if `46080` is not divisible by GBS. This infrequent evaluation schedule meant that if convergence was reached just after an evaluation, training would continue for an additional `46080` sequences, resulting in a 12-16% increase in total training time. To address this, after v5.0, the evaluation frequency was increased to reduce this overhead, while still keeping the overall evaluation cost low by skipping the initial few evaluations.

== Appendix: v6.0 Specific Rules

=== Small LLM MoE pretraining  (GPT-OSS 20B)

According to the hyperparameter rules for GPT-OSS-20B, `opt_end_learning_rate` should be `0.1 * opt_base_learning_rate`. But due to a bug in the reference implementation where the default value (`4e-5`) was set for `opt_end_learning_rate`, all the RCPs for GPT-OSS-20B use `opt_end_learning_rate = 4e-5` instead of `opt_end_learning_rate = opt_base_learning_rate * 0.1`

Since we are close to submissions and do not expect convergence to be drastically impacted by either of the `opt_end_learning_rate` values, just for v6.0 only, we will allow submissions that use either opt_end_learning_rate = 4e-5 or opt_end_learning_rate = opt_base_learning_rate * 0.1 as long as they PASS the current RCPs. 

== Appendix: RCP Examples

The RCP checking process is best illustrated with the following examples:

Benchmark A requires 5 submission runs.
The reference implementation provides (at least) 10 convergence points, let's say [16, 14, 16, 17, 16, 16, 15, 16, 15, 16] for batch size 128.
The top and bottom run are excluded from the mean and standard deviation computation.
So in this case the Mean = 15.75 epochs and Stdev = 0.43. Based on the t-test the maximum allowed speedup for p-value=0.05 is 3.53%. In other words the minimum mean epochs to converge for each submission with batch size 128 is 15.21.

The reference also provides convergence points for batch size 256: [20, 21, 21, 20, 22, 22, 21, 21, 20, 20].
In this case Mean = 20.75, Stdev = 0.66 and based on the t-test the maximum allowed speedup for p-value=0.05 is 4.12%.
In other words the minimum mean epochs to converge for batch-256 is 19.93.

Let's consider now the following submission scenarios:

* Submitter S1 makes a submission for A with batch size 128, and from the logs the epochs to converge are [15, 15, 15, 16, 16]. Excluding the top and bottom runs the mean epochs to converge is 15.33 (> 15.21), so S1 passes the RCP test for benchmark A, batch size 128.
* Submitter S2 makes a submission for A with batch size 256, and from the logs the epochs to converge are [19, 19, 19, 20, 21]. Excluding the top and bottom runs the mean epochs to converge is 19.33 (< 19.93), so S2 fails the RCP test for benchmark A, batch size 256.
* Submitter S3 makes a submission for A with batch size 192, and from the logs the epochs to converge are [17, 18, 18, 18, 20]. There are no RCPs for 192, but there are for larger and lower batch sizes. In this situation we find an interpolation of the mean and standard deviations for the RCPs at batch size 192. Mean = 18.25 and Stdev=0.547. Based on the t-test with p-value=0.05 the maximum allowed speedup is 3.68%. Exclusing the top and botton submission runs, the submission mean epochs to converge is 18, which is more than 18.25 / 1.0368, so the submission is accepted for batch size 192.
* Submitter S4 makes a submission for A with batch size 512. Since there is neither RCP for that batch size, nor RCPs for larger batch sizes, S2 needs to provide convergence points by running the reference with that batch size.
* Submitter S5 makes a submission for A with batch size 64 that meets the (stricter) convergence criteria for the RCP with the smallest batch size (128). In this case the submission is accepted.
* Submitter S6 makes a submission for A with batch size 64 that does not meet the convergence criteria for the RCP with the smallest batch size (128). In this case S1 needs to provide convergence points by running the reference with batch size = 64.

== Appendix: Utilization

MLPerf recommends calculating _utilization_ as `model_tensor_flops / (peak_system_tensor_flops_per_second * runtime_seconds)` where:

    * `model_tensor_flops` means only the tensor (ie matrix multiply or convolution) operations that are required by the model definition.  Vector or pointwise ops in the model such as bias add, normalization etc, are not counted as `model_tensor_flops`.  Furthermore, implementations that use activation recomputation methods should not count any of the operations added by activation recomputation as `model_tensor_flops`.

    * `peak_system_tensor_flops_per_second` means the peak tensor operations of the hardware, counting only tensor math throughput and not additional vector or pointwise math datapaths.

    * `runtime_seconds` means the mean of the runtimes of the runs used to calculate the benchmark result.

Use of `hardware_tensor_flops` (defined as model_tensor_flops plus operations added due to activation recomputation), instead of `model_tensor_flops` is strongly discouraged because those are not useful flops for the model. If `hardware_tensor_flops` are used for calculating utilization, it is recommended to also provide an accompanying calculation with `model_tensor_flops`.

Note _utilization_ is not an official MLPerf metric.

== Appendix: Deprecated benchmarks 

=== Deprecated benchmarks

|===
|Area|Problem |Dataset |Latest version available

|Language |Large language model |c4/en/3.0.1 |v4.1
| |NLP |Wikipedia 2020/01/01 |v5.0
| |LLM pretraining |c4/en/3.0.1 |v6.0
|Vision |Image classification |ImageNet |v4.0
| |Image segmentation (medical) |KiTS19 |v4.0
| |Text to Image |LAION-400M-filtered |v5.0
|Vision |Object detection (heavy weight) |COCO |v3.1
| |Object detection (light weight) |A subset of OpenImages |v5.1
|Language |Speech recognition |LibriSpeech |v3.1
|Commerce |Recommendation |Criteo 1TB Click Logs (multi-hot variant) |v2.1
| |Recommendation |Criteo 3.5TB Click Logs (multi-hot variant) |v6.0
|Graphs |Node classification |IGBH-Full |v5.1

|===
=== Deprecated benchmarks models and quality targets

|===
|Area |Problem |Model |Target |Latest version available

|Language |Large Language Model |GPT3 |2.69 log perplexity |v4.1
| |NLP |BERT |0.720 Mask-LM accuracy |v5.0
| |LLM pretraining |Llama31_405B |5.6 log perplexity |v6.0
|Vision |Image classification |ResNet-50 v1.5 |75.90% classification |v4.0
| |Image segmentation (medical) |U-Net3D |0.908 Mean DICE score |v4.0
| |Text to image |Stable Diffusion v2.0 |FID<=90 and and CLIP>=0.15 |v5.0
|Vision |Object detection (heavy weight) |Mask R-CNN |0.377 Box min AP and 0.339 Mask min AP |v3.1
| |Object detection (light weight) |SSD (RetinaNet) |34.0% mAP |v5.1
|Language | Speech recognition | RNN-T | 0.058 Word Error Rate |v3.1
|Graphs |Node classification |RGAT |72.0 % classification |v5.1
|Commerce |Recommendation |DLRMv2 (DCNv2) |0.80275 AUC |v6.0
|===

=== Deprecated benchmarks hyperparameters

The following table lists the tunable hyperparameters that were allowed for deprecated benchmarks.

|===
 |Model |Optimizer |Name |Constraint |Definition |Reference Code |Latest version available

 |ssd |adam |global_batch_size |arbitrary constant |reference --batch-size |link:https://github.com/mlperf/training/blob/master/single_stage_detector/ssd/train.py#L80[reference code] |v5.1
 |ssd |adam |opt_learning_rate_warmup_epochs |integer >= 0 |number of epochs for learning rate to warm up |link:https://github.com/mlperf/training/blob/master/single_stage_detector/ssd/train.py#L87[reference code] |v5.1
 |ssd |adam |opt_learning_rate_warmup_factor |unconstrained |the constant factor applied at learning rate warm up |link:https://github.com/mlperf/training/blob/master/single_stage_detector/ssd/train.py#L89[reference code] |v5.1
 |ssd |adam |opt_base_learning_rate |unconstrained |base learning rate, this should be the learning rate after warm up and before decay |link:https://github.com/mlperf/training/blob/master/single_stage_detector/ssd/train.py#L84[reference code] |v5.1
 |ssd |adam |opt_weight_decay |0 |L2 weight decay |link:https://github.com/mlperf/training/blob/master/single_stage_detector/ssd/train.py#L171[reference code] |v5.1
 |gnn |adam |global_batch_size |arbitrary constant |global batch size |link:https://github.com/alibaba/graphlearn-for-pytorch/blob/main/examples/igbh/train_rgnn_multi_gpu.py#L293[reference code] |v5.1
 |gnn |adam |opt_base_learning_rate |unconstrained |base learning rate|link:https://github.com/alibaba/graphlearn-for-pytorch/blob/main/examples/igbh/train_rgnn_multi_gpu.py#L296[reference code] |v5.1
 |bert |lamb |global_batch_size |unconstrained |The global batch size for training. |--train_batch_size |v5.0
 |bert |lamb |opt_base_learning_rate |unconstrained |The base learning rate. |--learning_rate |v5.0
 |bert |lamb |opt_epsilon |unconstrained |adam epsilon |link:https://github.com/mlperf/training/blob/fb058e3849c25f6c718434e60906ea3b0cb0f67d/language_model/tensorflow/bert/optimization.py#L75[reference code] |v5.0
 |bert |lamb |opt_learning_rate_training_steps |unconstrained |Step at which your reach the lowest learning late |link:https://github.com/mlperf/training/blob/master/language_model/tensorflow/bert/run_pretraining.py#L64[reference code] |v5.0
 |bert |lamb |opt_learning_rate_warmup_steps |unconstrained |"num_warmup_steps" |link:https://github.com/mlperf/training/blob/master/language_model/tensorflow/bert/optimization.py#L34[reference code] |v5.0
 |bert |lamb |num_warmup_steps |unconstrained |Number of steps for linear warmup. |--num_warmup_steps |v5.0
 |bert |lamb |start_warmup_step |unconstrained |--start_warmup_step |--start_warmup_step |v5.0
 |bert |lamb |opt_lamb_beta_1 |unconstrained |adam beta1 |link:https://github.com/mlperf/training/blob/fb058e3849c25f6c718434e60906ea3b0cb0f67d/language_model/tensorflow/bert/optimization.py#L73[reference code] |v5.0
 |bert |lamb |opt_lamb_beta_2 |unconstrained |adam beta2 |link:https://github.com/mlperf/training/blob/fb058e3849c25f6c718434e60906ea3b0cb0f67d/language_model/tensorflow/bert/optimization.py#L74[reference code] |v5.0
 |bert |lamb |opt_lamb_weight_decay_rate |unconstrained |Weight decay |link:https://github.com/mlperf/training/blob/fb058e3849c25f6c718434e60906ea3b0cb0f67d/language_model/tensorflow/bert/optimization.py#L72[reference code] |v5.0
 |stable diffusion |adamw |global_batch_size |unconstrained |The global batch size for training |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/main.py#L633[reference code] |v5.0
 |stable diffusion |adamw |opt_adamw_beta_1 |0.9 |coefficients used for computing running averages of gradient and its square |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1629[reference code] |v5.0
 |stable diffusion |adamw |opt_adamw_beta_2 |0.999 |coefficients used for computing running averages of gradient and its square |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1630[reference code] |v5.0
 |stable diffusion |adamw |opt_adamw_epsilon |1e-08 |term added to the denominator to improve numerical stability |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1631[reference code] |v5.0
 |stable diffusion |adamw |opt_adamw_weight_decay |0.01 |weight decay coefficient |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1632[reference code] |v5.0
 |stable diffusion |adamw |opt_base_learning_rate |unconstrained |base learning rate, this should be the learning rate after warm up |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1633[reference code] |v5.0
 |stable diffusion |adamw |opt_learning_rate_warmup_steps |unconstrained |number of steps for learning rate to warm up |link:https://github.com/mlcommons/training/blob/master/stable_diffusion/ldm/models/diffusion/ddpm.py#L1639[reference code] |v5.0
 |gpt3 |adam |global_batch_size |unconstrained |batch size in sequences |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_adam_beta_1 |0.9 |adam beta1 |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_adam_beta_2 |0.95 |adam beta2 |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_adam_epsilon |1e-8 |adam epsilon |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |dropout |0.0 |Disable all dropouts during training. |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |sequence_length |2048 |sequence length |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_weight_decay |0.1 |weight decay |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |gradient_accumulation_steps |unconstrained |Numer of fwd/bwd steps between optimizer step. |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_learning_rate_warmup_steps |ceil(265 * 1536 / global_batch_size) |steps taken for linear warmup during initial checkpoint generation. This only affects the learning rate curve in the benchmarking region. |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_learning_rate_decay_steps |ceil(108600 * 1536 / global_batch_size) |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps]. |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_init_checkpoint_step |ceil(4000 * 1536 / batch_size) |first step after loading initial checkpoint |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_base_learning_rate |constrained based on global_batch_size |refer to next table in section "GPT3 learning rates" |See PR (From NV and Google, TODO Link) |v4.1
 |gpt3 |adam |opt_end_learning_rate |10% of opt_base_learning_rate |learning rate at the last step of decay period |See PR (From NV and Google, TODO Link) |v4.1
 |resnet |lars |lars_opt_base_learning_rate |arbitrary constant |Base "plr" in the PR linked. |link:https://github.com/mlperf/training/pull/342/files#[reference code] |v4.0
 |resnet |lars |lars_opt_end_learning_rate$$*$$ |fixed to reference |end learning rate for polynomial decay, implied mathemetically from other HPs |N/A |v4.0
 |resnet |lars |lars_opt_learning_rate_decay_poly_power$$*$$ |fixed to reference |power of polynomial decay, no link needed since not tunable |N/A |v4.0
 |resnet |lars |lars_epsilon$$*$$ |Fixed to reference |epsilon in reference |link:https://github.com/mlperf/training/pull/342/files#diff-b7db7d58acb8134acb65b4d1d60b8e90R49[reference code] |v4.0
 |resnet |lars |lars_opt_learning_rate_warmup_epochs |arbitrary constant |w_epochs in PR |link:https://github.com/mlperf/training/pull/342/files#[reference code] |v4.0
 |resnet |lars |lars_opt_momentum | 0.9 for batch<32k, otherwise arbitrary constant |momentum in reference |link:https://github.com/mlperf/training/pull/342/files#diff-b7db7d58acb8134acb65b4d1d60b8e90R49[reference code] |v4.0
 |resnet |lars |lars_opt_weight_decay |(0.0001 * 2 ^ N) where N is any integer |weight_decay in  reference |link:https://github.com/mlperf/training/pull/342/files#diff-b7db7d58acb8134acb65b4d1d60b8e90R49[reference code] |v4.0
 |resnet |lars |lars_opt_learning_rate_decay_steps |unconstrained |num_epochs in reference |link:https://github.com/mlperf/training/blob/master/image_classification/tensorflow/official/resnet/resnet_run_loop.py[reference code] |v4.0
 |resnet |lars |global_batch_size |unconstrained |global batch size in reference
|link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/image_classification/tensorflow/official/utils/arg_parsers/parsers.py#L158[reference code] |v4.0
 |resnet |lars |label smoothing$$*$$$$*$$ |0 or 0.1 | TODO |TODO |v4.0
 |resnet |lars |truncated norm initialization$$*$$$$*$$ |boolean | TODO |TODO |v4.0
 |resnet |sgd |global_batch_size |arbitrary constant |reference --batch_size |See LARS |v4.0
 |resnet |sgd |sgd_opt_base_learning_rate |0.001 * k where is an integer  |the learning rate |See LARS |v4.0
 |resnet |sgd |sgd_opt_end_learning_rate |10^-4 |end learning rate for polynomial decay, implied mathemetically from other HPs |See LARS |v4.0
 |resnet |sgd |sgd_opt_learning_rate_decay_poly_power |2 |power of polynomial decay, no link needed since not tunable |See LARS |v4.0
 |resnet |sgd |sgd_opt_learning_rate_decay_steps |integer >= 0 |num_epochs in reference |See LARS |v4.0
 |resnet |sgd |sgd_opt_weight_decay |(0.0001 * 2 ^ N) where N is any integer |Weight decay, same as LARS. |See LARS |v4.0
 |resnet |sgd |sgd_opt_momentum |0.9 |Momentum for SGD. |See LARS |v4.0
 |resnet |sgd |model_bn_span |arbitrary constant |number of samples whose statistics a given BN layer uses to normalize a training minibatch (may be just the portion of global_batch_size per device, but also may be aggregated over several devices) |See LARS |v4.0
 |resnet |sgd |opt_learning_rate_warmup_epochs |integer >= 0 |number of epochs needed for learning rate warmup |See LARS |v4.0
 |resnet |sgd |label smoothing$$*$$$$*$$ |0 or 0.1 | TODO |TODO |v4.0
 |resnet |sgd |truncated norm initialization$$*$$$$*$$ |boolean | TODO |TODO |v4.0
 |resnet |lars/sgd |opt_name |"lars" or "sgd" |The optimizer that was used. | |v4.0
 |unet3d |sgd |global_batch_size |unconstrained |global batch size |reference --batch_size |v4.0
 |unet3d |sgd |opt_base_learning_rate |unconstrained |base learning rate |reference --learning_rate |v4.0
 |unet3d |sgd |opt_momentum |unconstrained |SGD momentum |reference --momentum |v4.0
 |unet3d |sgd |opt_learning_rate_warmup_steps |unconstrained |number of epochs needed for learning rate warmup|reference --lr_warmup_epochs |v4.0
 |unet3d |sgd |opt_initial_learning_rate |unconstrained |initial learning rate (for LR warm up) |reference --init_learning_rate |v4.0
 |unet3d |sgd |opt_learning_rate_decay_steps |unconstrained |epochs at which the learning rate decays |reference --lr_decay_epochs |v4.0
 |unet3d |sgd |opt_learning_rate_decay_factor |unconstrained |factor used for learning rate decay |reference --lr_decay_factor |v4.0
 |unet3d |sgd |opt_weight_decay |unconstrained |L2 weight decay |reference --weight_decay |v4.0
 |unet3d |sgd |training_oversampling |fixed to reference |training oversampling |reference --oversampling |v4.0
 |unet3d |sgd |training_input_shape |fixed to reference |training input shape |reference --input_shape |v4.0
 |unet3d |sgd |evaluation_overlap |fixed to reference |evaluation sliding window overlap |reference --overlap |v4.0
 |unet3d |sgd |evaluation_input_shape |fixed to reference |evaluation input shape |reference --val_input_shape |v4.0
 |unet3d |sgd |data_train_samples |fixed to reference |number of training samples | N/A |v4.0
 |unet3d |sgd |data_eval_samples |fixed to reference |number of evaluation samples | N/A |v4.0
 |maskrcnn |sgd |global_batch_size |arbitrary constant |global version of reference SOLVER.IMS_PER_BATCH |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/data/build.py#L112[reference code] |v3.1
 |maskrcnn |sgd |opt_learning_rate_decay_factor$$*$$ |fixed to reference (0.1) |learning rate decay factor |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/solver/build.py#L13[reference code] |v3.1
 |maskrcnn |sgd |opt_learning_rate_decay_steps$$*$$ |(60000, 80000) * (1 + K / 10) * 16 / global_batch_size where K is integer |Steps at which learning rate is decayed |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/solver/build.py#L26[reference code] |v3.1
 |maskrcnn |sgd |opt_base_learning_rate |0.02 * K for any integer K. For global_batch_size < 16, 0.02 / K for any integer K is also allowed |base learning rate, this should be the learning rate after warm up and before decay |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/solver/build.py#L12[reference code] |v3.1
 |maskrcnn |sgd |max_image_size$$*$$ |fixed to reference |Maximum size of the longer side |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/data/transforms/build.py#L8[reference code] |v3.1
 |maskrcnn |sgd |min_image_size$$*$$ |fixed to reference |Maximum size of the shorter side |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/data/transforms/build.py#L7[reference code] |v3.1
 |maskrcnn |sgd |num_image_candidates$$*$$ |1000 or 1000 * batches per chip |tunable number of region proposals for given batch size |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/modeling/rpn/inference.py#L183[reference code] |v3.1
 |maskrcnn |sgd |opt_learning_rate_warmup_factor |unconstrained |the constant factor applied at learning rate warm up |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/solver/build.py#L28[reference code] |v3.1
 |maskrcnn |sgd |opt_learning_rate_warmup_steps |unconstrained |number of steps for learning rate to warm up |link:https://github.com/mlperf/training/blob/00570abf77d351e474d57830014f6a3e501dece1/object_detection/pytorch/maskrcnn_benchmark/solver/build.py#L29[reference code] |v3.1
 |rnnt |lamb |global_batch_size                       |unconstrained |reference --batch_size       |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L270-L271[reference code] |v3.1
 |rnnt |lamb |opt_name                                |"lamb"        |The optimizer that was used. |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L357[reference code] |v3.1
 |rnnt |lamb |opt_base_learning_rate                  |unconstrained |base learning rate, this should be the learning rate after warm up and before decay  |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L358[reference code] |v3.1
 |rnnt |lamb |opt_lamb_epsilon                        |1e-9          |LAMB epsilon |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L359[reference code] |v3.1
 |rnnt |lamb |opt_lamb_learning_rate_decay_poly_power |unconstrained |Exponential decay rate |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L360[reference code] |v3.1
 |rnnt |lamb |opt_lamb_learning_rate_hold_epochs      |unconstrained |Number of epochs when LR schedule keeps the base learning rate value |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L362[reference code] |v3.1
 |rnnt |lamb |opt_learning_rate_warmup_epochs         |unconstrained |Number of epochs when LR linearly increases from 0 to base learning rate |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L361[reference code] |v3.1
 |rnnt |lamb |opt_weight_decay                        |1e-3          |L2 weight decay |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L372[reference code] |v3.1
 |rnnt |lamb |opt_lamb_beta_1                         |unconstrained |LAMB beta 1 |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L363[reference code] |v3.1
 |rnnt |lamb |opt_lamb_beta_2                         |unconstrained |LAMB beta 2 |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L364[reference code] |v3.1
 |rnnt |lamb |opt_gradient_clip_norm                  |1 or inf      |Gradients are clipped above this norm threshold. |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L365[reference code] |v3.1
 |rnnt |lamb |opt_gradient_accumulation_steps         |unconstrained |Numer of fwd/bwd steps between optimizer step. |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L222[reference code] |v3.1
 |rnnt |lamb |opt_learning_rate_alt_decay_func        |True          |whether to use alternative learning rate decay function |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/common/optimizers.py#L20-L49[reference code] |v3.1
 |rnnt |lamb |opt_learning_rate_alt_warmup_func       |True          |whether to use alternative learning rate warmup function |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L367[reference code] |v3.1
 |rnnt |lamb |opt_lamb_learning_rate_min              |1e-5          |LR schedule doesn't set LR values below this threshold |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L368[reference code] |v3.1
 |rnnt |lamb |train_samples                           |unconstrained |Number of training samples after filtering out samples longer than data_train_max_duration |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L337[reference code] |v3.1
 |rnnt |lamb |eval_samples                            |2703          |Number of evaluation samples |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L338[reference code] |v3.1
 |rnnt |lamb |data_train_max_duration                 |unconstrained |Samples longer than this number of seconds are not included to training dataset |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L252-L253[reference code] |v3.1
 |rnnt |lamb |data_train_num_buckets                  |unconstrained |Training dataset is split to this number of buckets |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L293[reference code] |v3.1
 |rnnt |lamb |data_train_speed_perturbation_min       |0.85          |Input audio is resampled to a random rample rate not less than this fraction of original sample rate. |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L256-L257[reference code] |v3.1
 |rnnt |lamb |data_train_speed_perturbation_max       |1.15          |Input audio is resampled to a random rample rate not greater than this fraction of original sample rate. |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L254-L255[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_freq_n                |2             |Number of masks for frequency bands |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L258-L259[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_freq_min              |0             |Minimum number of frequencies in a single mask |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L260-L261[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_freq_max              |20            |Maximum number of frequencies in a single mask |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L262-L263[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_time_n                |10            |Number of masks for time band  |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L264-L265[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_time_min              |0             |Minimum number of masked time steps as a fraction of all steps |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L266-L267[reference code] |v3.1
 |rnnt |lamb |data_spec_augment_time_max              |0.03          |Maximum number of masked time steps as a fraction of all steps |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L268-L269[reference code] |v3.1
 |rnnt |lamb |model_eval_ema_factor                   |unconstrained |Smoothing factor for Exponential Moving Average |See link:https://github.com/mlcommons/training/blob/651e7c47bcbd7f4708d633afa567205a826438f1/rnn_speech_recognition/pytorch/train.py#L395[reference code] |v3.1
 |rnnt |lamb |model_weights_initialization_scale      |unconstrained |After random initialization of weight and bias tensors, all are scaled with this factorAfter random initialization of weight and bias tensors, all are scaled with this factor |See link:https://github.com/mwawrzos/training/blob/2126999a1ffff542064bb3208650a1e673920dcf/rnn_speech_recognition/pytorch/train.py#L342[reference code] |v3.1
 |dlrmv2 |adagrad |global_batch_size |unconstrained |global batch size |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L705-L708[reference code] |v6.0
 |dlrmv2 |adagrad |opt_base_learning_rate |unconstrained |learning rate (for both dense layers and embeddings) |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L230-L235[reference code] |v6.0
 |dlrmv2 |adagrad |opt_adagrad_learning_rate_decay |0.0 |learning rate decay |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L73[reference code] |v6.0
 |dlrmv2 |adagrad |opt_weight_decay |0.0 |weight decay |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L76[reference code] |v6.0
 |dlrmv2 |adagrad |opt_adagrad_initial_accumulator_value |0.0 |adagrad initial accumulator value |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L74[reference code] |v6.0
 |dlrmv2 |adagrad |opt_adagrad_epsilon |1e-8 |adagrad epsilon |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L75[reference code] |v6.0
 |dlrmv2 |adagrad |opt_learning_rate_warmup_steps |0 (disabled) |number to steps from 0 to sgd_opt_base_learning_rate with a linear warmup |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L303-L307[reference code] |v6.0
 |dlrmv2 |adagrad |opt_learning_rate_decay_start_step |0 (disabled) |step at which poly decay is started |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L308-L312[reference code] |v6.0
 |dlrmv2 |adagrad |opt_learning_rate_decay_steps |0 (disabled) |the step at which the end learning rate is reached |link:https://github.com/mlcommons/training/blob/a9056b8e5840d811484ad91f9fe23ed09a3f97cf/recommendation_v2/torchrec_dlrm/dlrm_main.py#L313-L317[reference code] |v6.0
 |llama31_405b |adamw |global_batch_size |unconstrained |batch size in sequences |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L301[reference code] |v6.0
 |llama31_405b |adamw |opt_adamw_beta_1 |0.9 |AdamW beta1 |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
 |llama31_405b |adamw |opt_adamw_beta_2 |0.95 |AdamW beta2 |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
 |llama31_405b |adamw |opt_adamw_epsilon |1e-5 |AdamW epsilon |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
 |llama31_405b |adamw |opt_gradient_clip_norm |1.0 |Gradients are clipped above this norm threshold. |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
 |llama31_405b |adamw |dropout |0.0 |Disable all dropouts during training. |link:https://github.com/mlcommons/training/tree/master/large_language_model_pretraining/nemo[reference code] |v6.0
 |llama31_405b |adamw |sequence_length |8192 |sequence length |link:https://github.com/mlcommons/training/tree/master/large_language_model_pretraining/nemo[reference code] |v6.0
 |llama31_405b |adamw |opt_adamw_weight_decay |0.1 |weight decay |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
 |llama31_405b |adamw |gradient_accumulation_steps |unconstrained |Numer of fwd/bwd steps between optimizer step. |link:https://github.com/mlcommons/training/tree/master/large_language_model_pretraining/nemo[reference code] |v6.0
 |llama31_405b |adamw |opt_learning_rate_warmup_steps |ceil(8000 * 1152 / global_batch_size) |steps taken for linear warmup. |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L157[reference code] |v6.0
 |llama31_405b |adamw |opt_learning_rate_decay_steps |ceil(1_200_000 * 1152 / global_batch_size) - ceil(8000 * 1152 / global_batch_size) |Step when the end of cosine learning rate curve is reached. Learning rate cosine decay is in range (opt_learning_rate_warmup_steps + 1,opt_learning_rate_decay_steps]. |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L397[reference code] |v6.0
 |llama31_405b |adamw |opt_init_checkpoint_step |0 |first step after loading initial checkpoint |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L295[reference code] |v6.0
 |llama31_405b |adamw |opt_base_learning_rate |constrained based on global_batch_size |refer to next table in section "Llama 3.1 405b hyperparameter constraints" |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L153[reference code] |v6.0
 |llama31_405b |adamw |opt_end_learning_rate |8e-7 |learning rate at the last step of decay period |link:https://github.com/mlcommons/training/blob/a70765e9885ec1cba8e69a842ecfbdea750336c4/large_language_model_pretraining/nemo/pretrain_llama31.py#L155[reference code] |v6.0
|===

==== GPT3 hyperparameter constraints

Since training large language models is very expensive, the task force aims to limit hyperparameter searches. Thus the allowed range of batch sizes and corresponding batch sizes are fixed as follows.

|===
 |global_batch_size |opt_base_learning_rate

 |1536 |2.0e-5
 |2048 |2.0e-5
 |3072 |2.0e-5
 |4096 |3.0e-5
 |8192 |3.0e-5
|===

* GBS<1536 or GBS>8192 - new RCP needs to be generated, reach out to the task force
* GBS [1536,3072] - opt_base_learning_rate=2.0e-5
* For (3072,4096) - opt_base_learning_rate=2.0e-5 or opt_base_learning_rate=3.0e-5
* GBS [4096,8192] - opt_base_learning_rate=3.0e-5

If a new learning rate is needed for any GBS point, request new RCPs from the task force or normalize the score if permissible.

==== Llama 3.1 405b hyperparameter constraints

Since training large language models is very expensive, the task force aims to limit hyperparameter searches. Thus the allowed range of batch sizes and corresponding batch sizes are fixed as follows.

|===
 |global_batch_size |opt_base_learning_rate

 |1152 |8e-5
 |2304 |16e-5
 |4608 |32e-5
 |9216 |64e-5
|===

* GBS<1152 or GBS>9216 - new RCP needs to be generated, reach out to the task force
* GBS within range but not listed: opt_base_learning_rate = 8e-5 * (GBS / 1152), rounded to the **8th** decimal place. 

If a new learning rate is needed for any GBS point, request new RCPs from the task force or normalize the score if permissible.

=== Deprecated benchmarks quality measure
|===
|Area |Problem |Model|Evaluation frequency |Latest version available

|Language        |large Language Model |GPT3| Every 24576 sequences. CEIL(24576 / global_batch_size) if 24576 is not divisible by GBS |v4.1
|                |NLP |BERT| eval_interval_samples=FLOOR(0.05*(230.23*GBS+3000000), 25000), skipping 0 |v5.0
|                |LLM pretraining |Llama31_405B| Every 18432 sequences; `CEIL(18432 / global_batch_size) * global_batch_size` steps if 18432 is not divisible by GBS. Skipping first `FLOOR(0.0026*global_batch_size+12)` evaluations |v6.0
|Vision |Image classification |Resnet-50 v1.5|Every 4 epochs with offset 0 or 1 or 2 or 3 |v4.0
|       |Image segmentation (medical) |U-Net3D | Starting at `CEILING(1000*168/samples_per_epoch)` epochs, then every `CEILING(20*168/samples_per_epoch)` epochs. Where `samples_per_epoch` is the number of samples processed in a given epoch assuming that in the case of uneven batches the last batch is padded, e.g. `CEILING(168/global_batch_size) * global_batch_size`. |v4.0
|       |Text to image |Stable Diffusion v2.0 | See <<benchmark_specific_rules>> |v5.0
|Vision |Object detection (heavy weight) |Mask R-CNN|Every 1 epoch |v3.1
|       |Object detection (light weight) |SSD (RetinaNet) |Every 1 epoch |v5.1
|Language|Speech recognition |RNN-T|Every 1 epoch |v3.1
|Graphs |Node classification |RGAT |Evaluate 20 times per epoch |v5.1
|Commerce|Recommendation |DLRMv2 (DCNv2)|Every FLOOR(TOTAL_TRAINING_SAMPLES / (GLOBAL_BATCH_SIZE * NUM_EVAL)) samples, where TOTAL_TRAINING_SAMPLES = 4195197692 and NUM_EVAL = 20 |v6.0
|===

=== Deprecated benchmarks results

|===
|Area|Problem |Minimum Number of Runs |Latest version available

|Language |NLP |10 |v5.0
| |LLM pretraining |3 |v6.0
|Vision |Image classification |5 |v4.0
| |Image segmentation (medical) | 40 |v4.0
| |Stable Diffusion v2.0 | 10 |v5.0
| |Object detection (heavy weight) |5 |v3.1
| |Object detection (light weight) |5 |v5.1
|Language |Speech recognition |10 |v3.1
|Graphs |Node classification |10 |v5.1
|Commerce |Recommendation |10 |v6.0
|===


=== Epoch reporting for deprecated benchmarks


To extract submission convergence points, logs should report epochs as follows.
|===
| Benchmark | Epoch reporting | Latest version available

| BERT | Training sample (integer) | v5.0
| Stable-Diffusion | Training sample (integer) | v5.0
| GPT3 | Training token starting from 0 (integer) | v4.1
| Llama31_405B | Training samples starting from 0 (integer) | v6.0
| RN50 | Epoch | v4.0
| UNET3D | Epoch | v4.0
| Mask-RCNN | Epoch | v3.1
| RNN-T | Epoch | v3.1
| SSD (RetinaNet) | Epoch | v5.1
| RGAT | Training iteration as the fraction of a total number of iterations for one epoch (0.05, 0.1, 0.15, ..., 1.0) | v5.1
| DLRMv2 (DCNv2) | Training iteration as the fraction of a total number of iterations for one epoch (0.05, 0.1, 0.15, ..., 1.0) | v6.0
|===
=== Benchmark specific rules for deprecated benchmarks
* Image Classification

** The model may have 1000 or 1001 classes, where the 1001st is "I don't know"

* GPT3 
** For GPT3, manipulation of metadata which consists of the number of documents in the dataset and the size of each document is allowed as long as the data tokens are not accessed.
** Since the hyperparameters are fixed for GPT3, hyperparameter borrowing is not allowed.
** Exceptions for GPT3 OPEN: the open submissions are allowed to choose a language version that’s not English for the C4 dataset. When doing so, the submitter needs to make it clear that the dataset and convergence measures are different from the close division submissions.
** For GPT3 where there are two reference implementations which have been verified to be equivalent with minimum variance, each reference implementation should provide at least N epoch convergence numbers for each RCP.
** For GPT3, a request to generate RCPs should be made in the Training WG meeting at least 9 weeks before submission deadline and both reference owners (NV and Google) should provide RCPs (N runs each) at least 5 weeks before submission deadline so that all submitters have enough time to train with the new hparams. The RCP requests should be handled in FCFS order and if there are more than 5 RCP requests, the WG should decide if the requester's set of convergence points (2N runs) can be used as temporary RCPs.

* UNET3D
** In the case of UNET3D, due to large variance, 40 runs are required. Out of the 40 runs, the 4 fastest and 4 slowest are dropped. There can be maximum of 4 non-converging runs. A run is classified as non-converged if the target quality metric is not reached within `CEILING(10000*168/samples_in_epoch)` epochs. 

* Stable Diffusion
** 10 runs per submission
** Checkpoint must be collected every 512,000 images. CEIL(512000 / global_batch_size) * global_batch_size if 512000 is not divisible by GBS.
** The collected checkpoints may be evaluated freely (in order, out of order, some checkpoints may be skipped), provided that:
1. FID and CLIP scores must to be submitted for all collected checkpoints (up to the first checkpoint with a passing score) for 1/10 of the runs.
2. FID and CLIP scores must to be submitted for the last two checkpoints (the first checkpoint with a passing score and the one before it) for 9/10 of the runs.
** evaluation is done offline, the time is not counted towards the submission time.
** A passing score is FID<=90 and CLIP>=0.15

* Llama31_405B (LLM pretraining)
** For Llama31_405B, manipulation of metadata which consists of the number of documents in the dataset and the size of each document is allowed as long as the data tokens are not accessed.
** Since the hyperparameters are fixed for Llama31_405B, hyperparameter borrowing is not allowed.
** For Llama31_405B, a request to generate RCPs should be made in the Training WG meeting at least 9 weeks before submission deadline and the reference owner (NVIDIA) should provide RCPs (2N runs each) at least 5 weeks before submission deadline so that all submitters have enough time to train with the new hparams. The RCP requests should be handled in FCFS order and if there are more than 5 RCP requests, the WG should decide if the requester's set of convergence points (2N runs) can be used as temporary RCPs.
** For all v5.0 and v5.1 specific rules that applied to Llama31_405B (maximum GBS constraint at 9216, single-evaluation-earlier RCP allowance, sparse evaluation schedule), see the v5.0 Specific Rules appendix for historical detail.

* DLRMv2 (DCNv2)
** Because DLRMv2 (DCNv2) benchmark is trained for at most one epoch, epoch numbering starts from 0 in this case. More precisely, it stands for the fraction of epoch iterations passed.
** DLRMv2 training dataset is shuffled during preprocessing (with a fixed seed) on a per-sample basis. The resulting order of samples should be then used during training and any other extra dataset shuffling is prohibited.
** In the case of DLRMv2, if data set size is not evenly divisible by batch size, the last partial training batch may be dropped in addition to the general padding/mixed-batch options allowed under the Equivalence exceptions section.