// Copyright (c) 2017-2026 The Khronos Group Inc. // // SPDX-License-Identifier: CC-BY-4.0 include::{generated}/meta/XR_EXT_eye_gaze_interaction.adoc[] *Last Modified Date*:: 2020-02-20 *IP Status*:: No known IP claims. *Contributors*:: Denny Rönngren, Tobii + Yin Li, Microsoft + Alex Turner, Microsoft + Paul Pedriana, Oculus + Rémi Arnaud, Varjo + Blake Taylor, Magic Leap + Lachlan Ford, Microsoft + Cass Everitt, Oculus *Overview* This extension provides an basetype:XrPath for getting eye gaze input from an eye tracker to enable eye gaze interactions. The intended use for this extension is to provide: * system properties to inform if eye gaze interaction is supported by the current device. * an basetype:XrPath for real time eye tracking that exposes an accurate and precise eye gaze pose to be used to enable eye gaze interactions. * a structure slink:XrEyeGazeSampleTimeEXT that allows for an application to retrieve more information regarding the eye tracking samples. With these building blocks, an application can: discover if the XR runtime has access to an eye tracker, bind the eye gaze pose to the action system, determine if the eye tracker is actively tracking the users eye gaze, and use the eye gaze pose as an input signal to build eye gaze interactions. ==== Eye tracker An eye tracker is a sensory device that tracks eyes and accurately maps what the user is looking at. The main purpose of this extension is to provide accurate and precise eye gaze for the application. Eye tracking data is sensitive personal information and is closely linked to personal privacy and integrity. It is strongly recommended that applications that store or transfer eye tracking data always ask the user for active and specific acceptance to do so. If a runtime supports a permission system to control application access to the eye tracker, then the runtime must: set the pname:isActive field to ename:XR_FALSE on the supplied slink:XrActionStatePose structure, and must: clear ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT, ename:XR_SPACE_LOCATION_POSITION_VALID_BIT, ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT and ename:XR_SPACE_LOCATION_ORIENTATION_VALID_BIT when locating using the tracked space until the application has been allowed access to the eye tracker. When the application access has been allowed, the runtime may: set pname:isActive on the supplied slink:XrActionStatePose structure to ename:XR_TRUE and may: set ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT, ename:XR_SPACE_LOCATION_POSITION_VALID_BIT ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT and ename:XR_SPACE_LOCATION_ORIENTATION_VALID_BIT when locating using the tracked space. ==== Device enumeration When the eye gaze input extension is enabled, an application may: pass in a slink:XrSystemEyeGazeInteractionPropertiesEXT structure in the slink:XrSystemProperties::pname:next chain structure when calling flink:xrGetSystemProperties to acquire information about the connected eye tracker. The runtime must: populate the slink:XrSystemEyeGazeInteractionPropertiesEXT structure with the relevant information to the slink:XrSystemProperties returned by the flink:xrGetSystemProperties call. [open,refpage='XrSystemEyeGazeInteractionPropertiesEXT',type='structs',desc='Eye gaze interaction system properties'] -- include::{generated}/api/structs/XrSystemEyeGazeInteractionPropertiesEXT.adoc[] .Member Descriptions **** * pname:type is the elink:XrStructureType of this structure. * pname:next is code:NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension. For more details about the structure chain, see the structure being extended (slink:XrSystemProperties). * pname:supportsEyeGazeInteraction the runtime must: set this value to ename:XR_TRUE when eye gaze sufficient for use cases such as aiming or targeting is supported by the current device, otherwise the runtime must: set this to ename:XR_FALSE. **** Regardless of the value of pname:supportsEyeGazeInteraction, flink:xrSuggestInteractionProfileBindings must: accept eye gaze binding paths. However, if pname:supportsEyeGazeInteraction is ename:XR_FALSE, the runtime should: never report an action as active if its only suggested binding is an eye gaze binding path. include::{generated}/validity/structs/XrSystemEyeGazeInteractionPropertiesEXT.adoc[] -- ==== Eye gaze input This extension exposes a new interaction profile path pathname:/interaction_profiles/ext/eye_gaze_interaction that is valid for the user path * pathname:/user/eyes_ext with supported input subpath * subpathname:/input/gaze_ext/pose // This extension does not meet the current style guide requirements: // the path name must be vendor-suffixed in new extensions. :legacy-path: /interaction_profiles/ext/eye_gaze_interaction :properly-suffixed-path: /interaction_profiles/ext/eye_gaze_interaction_ext include::{config}/interaction_profile_missing_suffix_warning.adoc[] The eye gaze pose is natively oriented with +Y up, +X to the right, and -Z forward and not gravity-aligned, similar to the ename:XR_REFERENCE_SPACE_TYPE_VIEW. The eye gaze pose may: originate from a point positioned between the user's eyes. The tracking state of position and orientation (direction) is coupled: at any point in time, both the position and direction of the eye pose is either tracked or untracked. This means that the runtime must: either set both ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT and ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT or clear both ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT and ename:XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT. When the runtime provides a "nominal" (high quality) eye gaze pose, the ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT must: be set if the eye otherwise has a fully-tracked pose relative to the other space. A runtime can: provide a sub-nominal eye gaze pose but must: then clear the ename:XR_SPACE_LOCATION_POSITION_TRACKED_BIT. A nominal eye gaze pose is suitable for use cases such as aiming or targeting, while a sub-nominal eye gaze pose has degraded performance and should: not be relied on for all input scenarios. Applications should: be very careful when using sub-nominal eye gaze pose, since behavior varies considerably for different users and manufacturers, and some manufacturers may: choose to not provide sub-nominal eye gaze pose at all. With current technology, some eye trackers may: need to undergo an explicit calibration routine to provide a nominal accurate and precise eye gaze pose. If the eye tracker is in an uncalibrated state when the first call to flink:xrSyncActions is made with an eye gaze action enabled, then the runtime should: request eye tracker calibration from the user if it has not yet been requested. [open,refpage='XrEyeGazeSampleTimeEXT',type='structs',desc='Eye gaze sample time structure'] -- include::{generated}/api/structs/XrEyeGazeSampleTimeEXT.adoc[] .Member Descriptions **** * pname:type is the elink:XrStructureType of this structure. * pname:next is code:NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension. * pname:time is when in time the eye gaze pose is expressed. **** One particularity for eye trackers compared to most other spatial input is that the runtime may: lack the capability to predict or interpolate eye gaze poses. Runtimes that are unable to predict or interpolate eye gaze poses must: clamp the gaze pose requested in the flink:xrLocateSpace call to the value nearest to pname:time requested in the call. To allow for an application to reason about high accuracy eye tracking, the application can: chain in an slink:XrEyeGazeSampleTimeEXT to the next pointer of the slink:XrSpaceLocation structure passed into the flink:xrLocateSpace call. The runtime must: set pname:time in the slink:XrEyeGazeSampleTimeEXT structure to the clamped, predicted, or interpolated time. The application should: inspect the slink:XrEyeGazeSampleTimeEXT::pname:time field to understand when in time the pose is expressed. The pname:time field may: be in the future if a runtime is able to predict gaze poses. The runtime must: set the pname:time field to 0 if the sample time is not available. include::{generated}/validity/structs/XrEyeGazeSampleTimeEXT.adoc[] -- ==== Sample code The following example code shows how to bind the eye pose to the action system. [source,cpp] ---- extern XrInstance instance; extern XrSession session; extern XrPosef pose_identity; // Create action set XrActionSetCreateInfo actionSetInfo { .type = XR_TYPE_ACTION_SET_CREATE_INFO, .actionSetName = "gameplay", .localizedActionSetName = "Gameplay", .priority = 0, }; XrActionSet gameplayActionSet {XR_NULL_HANDLE}; CHK_XR(xrCreateActionSet(instance, &actionSetInfo, &gameplayActionSet)); // Create user intent action XrActionCreateInfo actionInfo { .type = XR_TYPE_ACTION_CREATE_INFO, .actionName = "user_intent", .actionType = XR_ACTION_TYPE_POSE_INPUT, .localizedActionName = "User Intent", }; XrAction userIntentAction {XR_NULL_HANDLE}; CHK_XR(xrCreateAction(gameplayActionSet, &actionInfo, &userIntentAction)); // Create suggested bindings XrPath eyeGazeInteractionProfilePath {XR_NULL_PATH}; CHK_XR(xrStringToPath(instance, "/interaction_profiles/ext/eye_gaze_interaction", &eyeGazeInteractionProfilePath)); XrPath gazePosePath {XR_NULL_PATH}; CHK_XR(xrStringToPath(instance, "/user/eyes_ext/input/gaze_ext/pose", &gazePosePath)); XrActionSuggestedBinding bindings { .action = userIntentAction, .binding = gazePosePath, }; XrInteractionProfileSuggestedBinding suggestedBindings{ .type = XR_TYPE_INTERACTION_PROFILE_SUGGESTED_BINDING, .interactionProfile = eyeGazeInteractionProfilePath, .countSuggestedBindings = 1, .suggestedBindings = &bindings, }; CHK_XR(xrSuggestInteractionProfileBindings(instance, &suggestedBindings)); XrSessionActionSetsAttachInfo attachInfo{ .type = XR_TYPE_SESSION_ACTION_SETS_ATTACH_INFO, .countActionSets = 1, .actionSets = &gameplayActionSet, }; CHK_XR(xrAttachSessionActionSets(session, &attachInfo)); XrActionSpaceCreateInfo createActionSpaceInfo{ .type = XR_TYPE_ACTION_SPACE_CREATE_INFO, .action = userIntentAction, .poseInActionSpace = pose_identity, }; XrSpace gazeActionSpace {XR_NULL_HANDLE}; CHK_XR(xrCreateActionSpace(session, &createActionSpaceInfo, &gazeActionSpace)); XrReferenceSpaceCreateInfo createReferenceSpaceInfo{ .type = XR_TYPE_REFERENCE_SPACE_CREATE_INFO, .referenceSpaceType = XR_REFERENCE_SPACE_TYPE_LOCAL, .poseInReferenceSpace = pose_identity, }; XrSpace localReferenceSpace {XR_NULL_HANDLE}; CHK_XR(xrCreateReferenceSpace(session, &createReferenceSpaceInfo, &localReferenceSpace)); while(true) { XrActiveActionSet activeActionSet{ .actionSet = gameplayActionSet, .subactionPath = XR_NULL_PATH, }; XrTime time = 0; XrActionsSyncInfo syncInfo{ .type = XR_TYPE_ACTIONS_SYNC_INFO, .countActiveActionSets = 1, .activeActionSets = &activeActionSet, }; CHK_XR(xrSyncActions(session, &syncInfo)); XrActionStatePose actionStatePose{ .type = XR_TYPE_ACTION_STATE_POSE, }; XrActionStateGetInfo getActionStateInfo{ .type = XR_TYPE_ACTION_STATE_GET_INFO, .action = userIntentAction, }; CHK_XR(xrGetActionStatePose(session, &getActionStateInfo, &actionStatePose)); if(actionStatePose.isActive){ XrEyeGazeSampleTimeEXT eyeGazeSampleTime{ .type = XR_TYPE_EYE_GAZE_SAMPLE_TIME_EXT, }; XrSpaceLocation gazeLocation{ .type = XR_TYPE_SPACE_LOCATION, .next = &eyeGazeSampleTime, }; CHK_XR(xrLocateSpace(gazeActionSpace, localReferenceSpace, time, &gazeLocation)); // Do things } } ---- *Version History* * Revision 1, 2020-02-20 (Denny Rönngren) ** Initial version * Revision 2, 2022-05-27 (Bryce Hutchings) ** Remove error-prone `XrEyeGazeSampleTimeEXT` validation requirement