--- title: Search query parameter evaluation framework description: AI Search performs A/B test evaluations that compare result relevancy for alternate sets of search query parameter values. These evaluations determine the models that AI Search uses for machine learning relevancy and for Q&A Genius Results.The search query parameter evaluation framework for AI Search uses tables to store results and metrics from A/B testing of live search traffic. AI Search uses the stored data when determining which relevancy model and Q&A Genius Result answer validation model to publish for a search application.Exclude a search profile from A/B testing evaluations of live AI Search traffic. This procedure prevents AI Search from using A/B testing evaluation results when publishing the search profile's search result relevancy model and its Q&A Genius Result answer validation model. locale: en-US canonical_url: https://www.servicenow.com/docs/r/australia/platform-administration/ai-search/srch-qry-parm-eval-fw-ais.html release: australia product: AI Search classification: ai-search topic_type: concept last_updated: "2026-03-12" reading_time_minutes: 9 breadcrumb: [Administer, AI Search, Search administration, Configure core features, Administer the ServiceNow AI Platform] --- # Search query parameter evaluation framework AI Search performs A/B test evaluations that compare result relevancy for alternate sets of search query parameter values. These evaluations determine the models that AI Search uses for machine learning relevancy and for Q&A Genius Results. The search query parameter evaluation framework enables A/B relevancy testing on live search traffic. AI Search considers results from this live testing along with its offline evaluation of aggregated search signal data when making the following determinations: - Determining which relevancy model to publish for a search profile. For details on generation and publication of relevancy models, see [Machine learning relevancy in AI Search](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/machine-learning-relevancy-ais.md). - Determining which machine reading comprehension \(MRC\) model to use when validating potential Q&A Genius Result answers. For more information on the MRC model used for Q&A Genius Results, see [Q&A Genius Results](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/genius-result-q-a-ais.md). AI Search computes A/B testing evaluation results on a nightly basis. **Note:** The search query parameter evaluation framework doesn't support user configured A/B testing. All A/B testing is automatically performed by AI Search as part of its evaluation of machine learning relevancy models. The search query parameter evaluation framework is part of the Adaptive Telemetry suite of features. **Parent Topic:**[Administering AI Search](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/administer-ais.md) ## Search query parameter evaluation framework tables The search query parameter evaluation framework for AI Search uses tables to store results and metrics from A/B testing of live search traffic. AI Search uses the stored data when determining which relevancy model and Q&A Genius Result answer validation model to publish for a search application. ### A/B Testing Evaluation table The A/B Testing Evaluation \[evaluation\] table stores details for A/B testing evaluations performed on live search traffic. **Note:** The system automatically cleans this table, removing records for inactive evaluations that are more than two years old.
Column Description
Number Autogenerated number to identify the evaluation.
Name Name and description for the evaluation.
Source Name of the table containing records to run the evaluation for.
Testing Scope Scope of the evaluation.- Supported values: - **ALL**: Run the evaluation against all records in the table specified by **Source**. - **SELECTED**: Only run the evaluation against records specified in the **Selected Artifacts** field value. - Default value: **ALL**
Artifact Provider List of artifact filtering classes that provide records for testing.
Selected Artifacts Comma-separated list of record sys\_id values. When **Testing Scope** is set to **SELECTED**, the evaluation only runs against records with sys\_id values specified in this list.
Minimum Signals for Evaluation Minimum number of search signals the evaluation should collect before ending.- Type: integer - Default value: **0** \(no minimum\) - Maximum value: **30000** For details on how this condition interacts with the **Minimum Days for Evaluation** condition, see **End Criteria Operator**.
Minimum Days for Evaluation Minimum number of days the evaluation should be active before ending.- Type: integer - Default value: **0** \(no minimum\) - Maximum value: **180** For details on how this condition interacts with the **Minimum Signals for Evaluation** condition, see **End Criteria Operator**.
End Criteria Operator Logical operator to use for evaluation end conditions when both **Minimum Signals for Evaluation** and **Minimum Days for Evaluation** conditions exist.- Supported values: - **AND**: The evaluation remains active until the **Minimum Signals for Evaluation** and **Minimum Days for Evaluation** conditions are both satisfied. - **OR**: The evaluation remains active until either of the **Minimum Signals for Evaluation** or **Minimum Days for Evaluation** conditions is satisfied. - Default value: **OR**
### A/B Testing Evaluation Execution table The A/B Testing Evaluation Execution \[evaluation\_execution\] table stores details of individual operations executed as part of A/B testing evaluations for live search traffic.
Column Description
Number Autogenerated number to identify the evaluation operation.
Evaluation Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table.
Artifact ID Sys\_id for the record analyzed by this evaluation operation.
State State for the record evaluation execution.- Supported values: - **Queued**: The evaluation operation is queued for execution. - **In Progress**: The evaluation operation is underway. - **Scoring**: The evaluation operation is in the scoring phase. - **Complete**: The evaluation operation completed successfully. - **Errored**: The evaluation operation failed with an error. - **Canceled**: The evaluation operation was canceled by the system. - **Suspended**: The evaluation operation was suspended by the system. - Default value: **Queued**
Start Date Date and time when the evaluation operation started.
End Date Date and time when the evaluation operation ended.
### A/B Testing Evaluation Parameter table The A/B Testing Evaluation Parameter \[evaluation\_parameter\] table stores details of individual search query parameters considered in A/B testing evaluations for live search traffic.
Column Description
Evaluation Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table.
Name Name and description for the evaluation parameter.
Use Artifact's Assigned Parameter - Type: true \| false - Supported values: - **true**: Compare the specified **Parameter Values** to the currently assigned search query parameter value on the provided artifact. - **false**: Don't compare the specified **Parameter Values** to the currently assigned search query parameter value on the provided artifact - Default value: **true**
Parameter Values Traffic Percentage Percentage of search queries to apply the specified **Parameter Values** to during the A/B evaluation.- Type: integer - Default value: **0** - Maximum value: **100**
Parameter Values JSON list of values to compare to the current search query parameter value when **Use Artifact's Assigned Parameter** is true.
Parameter Type Type for the search query parameter. The selected value determines how the **Parameter Values** are used during and after an A/B evaluation. Supported values: - **Search Context Parameters**: Merge and override search profile context parameters with the **Parameter Values** for search query requests. When the evaluation ends, use the search query context parameter value set with the winning score to override or merge the relevant search profile context parameters. - **Search QnA Genius Result Configuration**: Apply the **Parameter Values** to Q&A Genius Result configurations used in search query requests. When the evaluation ends, update the relevant search profile to include the Genius Result configuration parameter value with the winning score. - **Search QnA Model**: Apply the **Parameter Values** to Q&A Genius Result models used in search query requests. When the evaluation ends, update the relevant search profile to include the Q&A Genius Result model parameter value with the winning score. - **Search Relevancy Model**: Apply the **Parameter Values** as search relevancy models to use in search query requests. When the evaluation ends, update the relevant search profile to include the relevancy model parameter value with the winning score. - **Search REST Parameters**: Merge and override search profile query parameters with the **Parameter Values** for search query requests. When the evaluation ends, use the search query parameter value set with the winning score to override or merge the relevant search profile parameters.
Score Calculation Type Form of calculation used to compute search query parameter value scores and find the best value. Supported values: - **Average Click Position**: The best search query parameter value is the one with the lowest average click position score. **Note:** AI Search computes the average click position score by dividing the sum of all selected search result ranks by the number of searches. The highest-ranked search result has rank 1, the next-highest has rank 2, and so on. As an example, if you perform two searches, selecting the first result in one case and the second result in the other, your average click position score is \(1 + 2\) / 2 = 1.5. - **Genius Result Helpfulness**: The best search query parameter value is the one with the highest helpfulness score. This calculation takes into account whether a relevant Genius Result answer appeared in your search results. **Note:** The helpfulness score is a metric that indicates whether Genius Result answers were helpful in the context of your search. - **Helpfulness**: The best search query parameter value is the one with the highest helpfulness score. Unlike **Genius Result Helpfulness**, this calculation doesn't take into account whether a relevant Genius Result answer appeared in your search results.
Signal Provider The provider for search signals needed to score the search query parameter. **Search Event Signal Provider** is the only supported value.
### A/B Testing Evaluation Parameter Result table The A/B Testing Evaluation Parameter Result \[evaluation\_parameter\_result\] table stores calculation results for individual search query parameters considered in A/B testing evaluations for live search traffic. |Column|Description| |------|-----------| |Evaluation Execution|Reference to the record for the evaluation execution on the A/B Testing Evaluation Execution \[evaluation\_execution\] table.| |Parameter Evaluation|Reference to the record for the search query parameter on the A/B Testing Evaluation Parameter \[evaluation\_parameter\] table.| |Best Value|Best value for the search query parameter, as determined by the **Winning Score**.| |Winning Score|Numeric score for the search query parameter, determined using the **Score Calculation Type**.| |Score Metadata|Metadata from the score computation for the search query parameter.| ## Exclude a search profile from the search query parameter evaluation framework Exclude a search profile from A/B testing evaluations of live AI Search traffic. This procedure prevents AI Search from using A/B testing evaluation results when publishing the search profile's search result relevancy model and its Q&A Genius Result answer validation model. ### Before you begin Role required: ais\_admin ### About this task The search query parameter evaluation framework performs A/B testing evaluations of search configuration settings using live search traffic. By default, AI Search evaluates configuration settings for all search profiles. Search administrators can exclude individual search profiles from the search query parameter evaluation framework. Excluding a search profile from the framework prevents AI Search from performing A/B testing evaluations for live search traffic that uses the excluded search profile. **Note:** When you exclude a search profile from evaluations, AI Search no longer considers evaluation results when updating the machine learning relevancy and machine reading comprehension \(MRC\) models for that search profile. As a result, relevancy scoring settings and Q&A Genius Result answer filtering settings for the search profile may be less reflective of your search traffic. For more information on how AI Search uses A/B evaluation testing results when publishing these models, see [Machine learning relevancy in AI Search](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/machine-learning-relevancy-ais.md) and [Q&A Genius Results](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/ai-search/genius-result-q-a-ais.md). ### Procedure 1. Navigate to **All** > **AI Search** > **Search Experience** > **Search Profiles**. 2. Open the record for the search profile that you want to exclude from A/B testing evaluations of live search traffic. 3. If the Search Profile form doesn't already display the **Exclude from evaluation** field, configure the form layout to make the field visible. For details on configuring a form layout to make fields visible, see [Configuring the form layout](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/platform-administration/configure-form-layout.md). 4. Select the **Exclude from evaluation** option. 5. Select **Update**. ### Result AI Search no longer performs A/B testing evaluations for traffic that uses the excluded search profile. Machine learning relevancy no longer updates the relevancy model for the search profile.