== PktIO Processing Before packets can be manipulated they typically need to be _received_ and after they are manipulated they need to be _transmitted_. The ODP abstraction that captures these operations is the *Packet I/O (PktIO)*. PktIOs are represented by handles of type *odp_pktio_t* and represent a logical I/O interface that is mapped in an implementation-defined manner to an underlying integrated I/O adapter or NIC. PktIO objects are manipulated through various state transitions via `odp_pktio_xxx()` API calls as shown below: .ODP PktIO Finite State Machine image::pktio_fsm.svg[align="center"] PktIOs begin in the *Unallocated* state. From here a call `odp_pktio_open()` is used to create an *odp_pktio_t* handle that is used in all subsequent calls to manipulate the object. This call puts the PktIO into the *Unconfigured* state. To become operational, a PktIO must first be *configured* for Input, Output, or both Input and Output via the `odp_pktin_queue_config()` and/or `odp_pktout_queue_config()` APIs, and then *started* via the `odp_pktio_start()` to make it *Ready*. Following the completion of I/O processing, the `odp_pktio_stop()` API returns the PktIO to the *Configured* state. From here it may be *Reconfigured* via additional `odp_pktin_queue_config()` and/or `odp_pktout_queue_config()` calls, or *Closed* via the `odp_pktio_close()` API to return the PktIO to the *Unallocated* state. === PktIO Allocation PktIO objects begin life by being _opened_ with `odp_pktio_open()` API: [source,c] ----- odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, const odp_pktio_param_t *param) ----- The function has three arguments: a *name*, which is an implementation-defined string that identifies the logical interface to be opened, a *pool* that identifies the ODP pool that storage for received packets should be allocated from, and a *param* structure that specifies I/O options to be associated with this PktIO instance. ODP defines *"loop"* as a reserved name to indicate that this PktIO represents a loopback interface. Loopback interfaces are useful as a means of recycling packets back for reclassification after decryption or decapsulation, as well as for diagnostic or testing purposes. For example, when receiving IPsec traffic, the classifier is able to recognize that the traffic is IPsec, however until the traffic is decrypted it is unable to say what that traffic contains. So following decryption, sending the decrypted packet back to a loopback interface allows the classifier to take a "second look" at the packet and properly classify the decrypted payload. Similar considerations apply to tunneled packets that must first be decapsulated to reveal the true payload. The *pool* specifies the default pool to use for packet allocation if not overridden by the classifier due to a specific or default Class-of-Service (CoS) match on the packet. The *param* struct, in turn, specifies the input and output *modes* of the PktIO. === PktIO Capabilities and PktIn/PktOut Configuration Associated with each PktIO is a set of _capabilities_ that provide information such as the maximum number of input/output queues it supports, its configuration options, and the operations is supports. These are aggregated into `odp_pktio_capability_t` the struct, which is returned by `odp_pktio_capability()` API. ==== PktIn Configuration For PktIOs that will receive packets, the `odp_pktin_config_opt_t` struct controls the RX processing to be performed on received packets. For example, `odp_pktin_config_opt_t` includes options for controlling packet timestamping as well as default packet checksum verification processing. ==== PktIO Parsing Configuration For RX processing, packets may also be parsed automatically as part of receipt as controlled by the `odp_pktio_parser_config_t` struct. ==== PktOut Configuration For PktIOs that will transmit packets, the `odp_pktout_config_opt_t` struct controls the TX processing to be performed on transmitted packets. For example, `odp_pktout_config_opt_t` includes options for controlling checksum insertion for transmitted packets. === PktIO Input and Output Modes PktIO objects support four different Input and Output modes, that may be specified independently at *open* time. .PktIO Input Modes * `ODP_PKTIN_MODE_DIRECT` * `ODP_PKTIN_MODE_QUEUE` * `ODP_PKTIN_MODE_SCHED` * `ODP_PKTIN_MODE_DISABLED` .PktIO Output Modes * `ODP_PKTOUT_MODE_DIRECT` * `ODP_PKTOUT_MODE_QUEUE` * `ODP_PKTOUT_MODE_TM` * `ODP_PKTOUT_MODE_DISABLED` The DISABLED modes indicate that either input or output is prohibited on this PktIO. Attempts to receive packets on a PktIO whose `in_mode` is DISABLED return no packets while packets sent to a PktIO whose `out_mode` is DISABLED are discarded. ==== Direct I/O Modes DIRECT I/O is the default mode for PktIO objects. It is designed to support poll-based packet processing, which is often found in legacy applications being ported to ODP, and can also be a preferred mode for some types of packet processing. By supporting poll-based I/O processing, ODP provides maximum flexibility to the data plane application writer. ===== Direct RX Processing The processing of DIRECT input is shown below: .PktIO DIRECT Mode Receive Processing image::pktin_direct_recv.svg[align="center"] In DIRECT mode, received packets are stored in one or more special PktIO queues of type *odp_pktin_queue_t* and are retrieved by threads calling the `odp_pktin_recv()` API. Once opened, setting up a DIRECT mode PktIO is performed by the `odp_pktin_queue_config()` API, whose purpose is to specify the number of PktIn queues to be created and to set their attributes. It is important to note that while `odp_pktio_queue_config()` creates a requested number of RX queues that are associated with the PktIO and accepts optimization advice as to how the application intends to use them, _i.e._, whether the queues need to be safe for concurrent use by multiple threads (OP_MT) or only one thread at a time (OP_MT_UNSAFE), these queues are *not* associated with any specific thread. Applications use a discipline appropriate to their design, which may involve restricting PktIn queue use to separate threads, but that is an aspect of the application design. ODP simply provides a set of tools here, but it is the application that determines how those tools are used. ===== Hash Processing Another feature of DIRECT mode input is the provision of a *hash* function used to distribute incoming packets among the PktIO's PktIn queues. If the `hash_enable` field of the *odp_pktin_queue_param_t* is `true`, then the `hash_proto` field is used to specify which field(s) of incoming packets should be used as input to an implementation-defined packet distribution hash function. Note that the hash function used in PktIO poll mode operation is intended to provide simple packet distribution among multiple PktIn queues associated with the PktIO. It does not have the sophistication of the *ODP Classifier*, however it also does not incur the setup requirements of pattern matching rules, making it a simpler choice for less sophisticated applications. Note that ODP does not specify how the hash is to be performed. That is left to each implementation. The hash only specifies which input packet fields are of interest to the application and should be considered by the hash function in deciding how to distribute packets among PktIn queues. The only expectation is that packets that have the same hash values should all be mapped to the same PktIn queue. ===== PktIn Queues A *PktIn Queue* is a special type of queue that is used internally by PktIOs operating in DIRECT mode. Applications cannot perform enqueues to these queues, however they may obtain references to them via the `odp_pktin_queue()` API. Once configured, prior to receiving packets the PktIO must be placed into the *Ready* state via a call to `odp_pktio_start()`. Once started, the PktIn queue handles are used as arguments to `odp_pktin_recv()` to receive packets from the PktIO. Note that it is the caller's responsibility to ensure that PktIn queues are used correctly. For example, it is an error for multiple threads to attempt to perform concurrent receive processing on the same PktIn queue if that queue has been marked MT_UNSAFE. Performance MAY be improved if the application observes the discipline of associating each PktIn queue with a single RX thread (in which case the PktIn queue can be marked MT_UNSAFE), however this is up to the application to determine how best to structure itself. ===== Direct TX Processing A PktIO operating in DIRECT mode performs TX processing as shown here: .PktIO DIRECT Mode Transmit Processing image::pktout_direct_send.svg[align="center"] Direct TX processing operates similarly to Direct RX processing. Following open, the `odp_pktout_queue_config()` API is used to create and configure one or more *PktOut queues* of type *odp_pktout_queue_t* to be used for packet transmission by this PktIO. As with PktIn queues, the handles for these created PktOut queues may be retrieved by the `odp_pktout_queue()` API. Once the PktIO has been configured for output and started via `odp_pktio_start()`, packets may be transmitted to the PktIO by calling `odp_pktout_send()`: [source,c] ----- int odp_pktout_send(odp_pktout_queue_t queue, const odp_packet_t packets[], int num) ----- Note that the first argument to this call specifies the PktOut queue that the packet is to be added to rather than the PktIO itself. This permits multiple threads (presumably operating on different cores) a more efficient means of separating I/O processing destined for the same interface. ==== Queued I/O Modes To provide additional flexibility when operating in poll mode, PktIOs may also be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode is that QUEUE mode uses standard ODP event queues to service packets. ===== Queue RX Processing The processing for QUEUE input processing is shown below: .PktIO QUEUE Mode Receive Processing image::pktin_queue_recv.svg[align="center"] In QUEUE mode, received packets are stored in one or more standard ODP queues. The difference is that these queues are not created directly by the application. Instead, they are created in response to an `odp_pktin_queue_config()` call. As with DIRECT mode, the `odp_pktin_queue_param_t` specified to this call indicates whether an input hash should be used and if so which field(s) of the packet should be considered as input to the has function. The main difference between DIRECT and QUEUE RX processing is that because the PktIO uses standard ODP event queues, other parts of the application can use `odp_queue_enq()` API calls to enqueue packets to these queues for "RX" processing in addition to those originating from the PktIO interface itself. To obtain the handles of these input queues, the `odp_pktin_event_queue()` API is used. Similarly, threads receive packets from PktIOs operating in QUEUE mode by making standard `odp_queue_deq()` calls to one of the event queues associated with the PktIO. ===== Queue TX Processing Transmit processing for PktIOs operating in QUEUE mode is shown below: .PktIO QUEUE Mode Transmit Processing image::pktout_queue_send.svg[align="center] For TX processing QUEUE mode behaves similar to DIRECT mode except that output queues are regular ODP event queues that receive packets via `odp_queue_enq()` calls rather than special PktOut queues that use `odp_pktout_send()`. Again, these queues are created via a call to `odp_pktout_queue_config()` following `odp_pktio_open()`. The main reason for selecting QUEUE mode for output is flexibility. If an application is designed to use a _pipeline model_ where packets flow through a series of processing stages via queues, then having the PktIO in QUEUE mode means that the application can always use the same enq APIs to pass packets from one stage to the next, including the final transmit output stage. ==== Scheduled I/O Modes The final PktIO mode supported integrates RX and TX processing with the ODP _event model_. For RX processing this involves the use of the *Scheduler* while for TX processing this involves the use of the *Traffic Manager*. Scheduled RX Processing is further divided based on whether or not the Classifier is used. ===== Scheduled RX Processing When a PktIO is opened with `ODP_PKTIN_MODE_SCHED`, it indicates that the input queues created by a subsequent `odp_pktin_queue_config()` call are to be used as input to the *ODP Scheduler*. .PktIO SCHED Mode Receive Processing image::pktin_sched_recv.svg[align="center'] For basic use, SCHED mode simply associates the PktIO input event queues created by `odp_pktin_queue_config()` with the scheduler. Hashing may still be employed to distribute input packets among multiple input queues. However instead of these being plain queues they are scheduled queues and have associated scheduling attributes like priority, scheduler group, and synchronization mode (parallel, atomic, ordered). SCHED mode thus provides both packet distribution (via the optional hash) as well as scalability via the ODP event model. In its fullest form, PktIOs operating in SCHED mode use the *ODP Classifier* to permit fine-grained flow separation on *Class of Service (CoS)* boundaries. .PktIO SCHED Mode Receive Processing with Classification image::pktin_sched_cls.svg[align="center"] In this mode of operation, the hash function of `odp_pktin_queue_config()` is typically not used. Instead, the event queues created by this call, as well as any additional event queues created via separate `odp_queue_create()` calls are associated with classes of service via `odp_cls_cos_create()` calls. Classification is enabled for the PktIO as a whole by assigning a _default_ CoS via the `odp_pktio_default_cos_set()` API. When operating in SCHED mode, applications do not call PktIn receive functions. Instead the PktIn queues are scanned by the scheduler and, if classification is enabled on the PktIO, inbound packets are classified and put on queues associated with their target class of service which are themelves scheduled to threads. Note that on platforms that support hardware classification and/or scheduling these operations will typically be performed in parallel as packets are arriving, so this description refers to the _logical_ sequence of classification and scheduling, and does not imply that this is a serial process. ===== Scheduled TX Processing Scheduled transmit processing is performed via the *ODP Traffic Manager* and is requested when a PktIO is opened with an `out_mode` of `ODP_PKTOUT_MODE_TM`. For TX processing via the Traffic Manager, applications use the `odp_tm_enq()` API. See the *Traffic Manager* section of this document for more information about Traffic Manager configuration and operation.