================================================================================ README for App QoS December 2020 ================================================================================ CONTENTS ======== - Introduction - Requirements and Installation - Configuration - Usage - REST API - Usage Scenario - How To Generate self-signed SSL certificate - Legal Disclaimer INTRODUCTION ============ Application Quality of Service (App QoS) is a proof-of-concept software created to demonstrate the use of Intel(R) RDT technologies (CAT, MBA) and Intel(R) SST technologies Base Frequency (BF), Core Power (CP) to improve QoS for applications via partitioning system resources. App QoS allows to assign Apps to Pools with different resources. In current version each of the Pools (group of cores) has fixed amount of cache and memory bandwidth allocated. Thanks to Intel(R) SST BF and CP support, App QoS allows the CPU to be deployed with an asymmetric core frequency configuration. Amount of resources allocated and cores frequency configuration could depend on Pool's priority. Intel(R) RDT CAT and MBA configuration is done via "libpqos" library and Intel(R) SST BF and CP configuration via external "Intel pwr" library. App QoS provides simple, local, REST API management interface secured with HTTPS and mTLS. REST API management interface allows user to manage Apps and Pools. An App controlled by App QoS could be a process, a container or a VNF. App QoS is reference code written in Python. It is fully configurable and can be easily modified to suit other use cases and allow remote management. REQUIREMENTS AND INSTALLATION ============================= App QoS requires minimum Linux kernel v4.20 (for Intel(R) RDT L3 CAT and MBA) however, for support of Intel(R) SST BF and CP Linux kernel v5.2 is required. App QoS is a part of Intel(R) RDT Software Package, it is located in "appqos" directory. App QoS requires Python v3.x and depends on the following Python modules and external components: - pqos (libqos Python wrapper) - flask - flask_restful - jsonschema - Intel pwr library (github.com, CommsPowerManagement) NOTE: To install mentioned dependencies in virtualenv please use "make setup" command. Intel(R) RDT CAT and MBA configuration is done via libpqos using MSR or OS/resctrl interfaces (user configurable), for more information please see "Software Compatibility" section of README for Intel(R) RDT Software Package. For hardware requirements please see "Hardware Support" section of mentioned README file. For more information about "Intel pwr" library used to configure Intel(R) SST BF and CP please see https://github.com/intel/CommsPowerManagement . CONFIGURATION ============= App QoS supports Pools of Apps, each with defined cores, cache and memory bandwidth allocation. Each App describes a single app, with one or more PIDs. Cache Allocation, Memory Bandwidth Allocation (via CAT and MBA), Base Frequency, Core Power (via SST BF and CP) and affinity are configured by App QoS on per core basis. NOTE: make sure that cores are isolated via "isolcpu=" kernel param. App QoS config is stored in JSON format in file "./appqos.conf" for which JSON Schema file is available in "./schema" directory. Configuration file is "Read-Only", no runtime changes are saved. If there is no "Default" pool (with "id" equal to 0) defined in config file, App QoS will dynamically create one on start-up. All unassigned cores will be assigned to "Default" pool, MBA will be configured to "no throttling" (100% (default) or ~2^32MBps (MBA CTRL enabled)) and CAT CBM to all cache ways. EXAMPLE CONFIG { "rdt_iface": {"interface": "msr"}, "mba_ctrl": {"enabled": false}, "apps": [{"id": 1, "name": "App1", "pids": [1979, 1980]}, {"id": 2, "cores": [4], "name": "App2", "pids": [1592, 1593, 1594]}, {"id": 3, "cores": [5], "name": "App3", "pids": [1576, 1577, 1578]}], "pools": [{"id": 1, "cores": [1, 2, 3], "name": "HP", "mba": 100, "l3cbm": "0xFF0", "apps": [1], "power_profile": 1}, {"id": 2, "cores": [4, 5, 6], "name": "MP", "mba": 50, "l3cbm": "0xC", "apps": [2, 3]}, {"id": 3, "cores": [7, 8, 9], "name": "LP", "mba": 10, "l3cbm": "0x3", "apps": []}], "power_profiles": [{"id": 1, "min_freq": 2500, "max_freq": 2500, "epp": "performance", "name": "HP"}], "sstbf": {"configured": false}, "power_profiles_expert_mode": false, "power_profiles_verify": true } "rdt_iface" section, Intel(R) RDT configuration. - "interface" - requested libpqos interface ("msr" or "os") "mba_ctrl" section, Intel(R) RDT configuration. - "enabled" - MBA CTRL requested state (requires "os" interface) "apps" section, Apps being managed by App QoS. - "id" - App's ID - "name" - App's name (optional) - "cores" - cores being used by App (optional) - "pids" - list of App's PIDs "pools" section, Pools of Apps. - "id" - Pool's ID - "name" - Pool's name - "apps" - list of Apps being part of the Pool - "l3cbm" - Intel RDT CAT CacheWayBitmask assigned to Pool - MBA related configuration: - "mba" - Intel RDT MBA rate [%] assigned to Pool (default, 1 - 100 [%]) OR - "mba_bw" - Intel RDT MBA rate [MBps] assigned to Pool (1 - 2^32-1 [MBps]) (requires MBA CTRL to be enabled, please see config's "mba_ctrl" section) - "cores" - cores being assigned to Pool - "power_profile" - Power Profile ID to be applied on pool's cores "power_profiles" section, Power Profiles/SST-CP. - "id" - Profile's ID - "name" - Profiles's name - "min_freq" - min. frequency [MHz] - "max_freq" - max. frequency [MHz] - "epp" - Energy Performance Preference "sstbf" section, Intel(R) SST-BF configuration. - "configured" - SST-BF requested state NOTE: "power profiles" are ignored when SST-BF is configured. Global options: - "power_profiles_expert_mode" - make power profiles editable via REST API (Default: False) - "power_profiles_verify" - Admission Control feature for config file content, verifies Power Profiles and Pools configuration (Default: True) USAGE ===== App QoS is a Python script accepting the following, optional, command line options: - -h, --help, show this help message and exit - -c PATH, --config PATH, Configuration file path - --port PORT, REST API port (default: 5000) - -V, --verbose, Verbose mode NOTE: App QoS requires root privileges. By default it will attempt to read from "appqos.conf" file from current folder. Example command line to run App QoS in verbose mode and config file in non-default location: sudo python3 ./appqos -c /tmp/appqos.conf -V NOTE: To run App QoS in virtualenv please use "make run" command. For more information about configuration file please see CONFIGURATION paragraph, for information about runtime configuration please see REST API paragraph. For more info please see USAGE SCENARIO paragraph. REST API ======== The REST API is a local (but can be easily modified to allow remote access), secured interface that allows the user to control App QoS. The REST API allows to: - add, remove or move App between Pools, - add, remove or modify Pools, - configure Intel(R) SST BF and CP technologies - configure Intel(R) RDT interface type and MBA mode NOTE: None of configuration changes made via REST API are saved to configuration file. App QoS REST API uses HTTPS and mutual TLS (mTLS) for authorization and authentication. App QoS certificate and client certificate are signed using CA certificate - please see HOW TO GENERATE APP QOS AND CLIENT CERTIFICATE paragraph for more info. For curl, parameters for client cert, client key and cacert, need to be passed in command-line: $ curl --cert client_appqos.crt --key client_appqos.key --cacert ca.crt REST API URIs ------------- JSON Schema files for REST API commands and responses are available in "./schema" directory. - GET /apps - get all/collection of apps - POST /apps - create new app Example request: {"pool_id": 2, "name": "hello", "cores": [1,2], "pids": [1]} Result: {"id": 5} - GET /apps/{id} - get app for given id Example response: {"id": 6, "cores": [2, 3, 11], "name": "App", "pids": [1748, 1749, 1750], "pool_id": 3 } - PUT /apps/{id} - update app (e.g.:move between pools) for given id Example request: {"pool_id": 2} - DELETE /apps/{id} - delete app for given id - GET /pools - get all/collection of pools - POST /pools - create new pool - GET /pools/{id} - get pool for given id - PUT /pools/{id} - modify pool for given id - DELETE /pools/{id} - delete empty pool for given id - GET /power_profiles - get all/collection of power profiles - POST /power_profiles - create new power profile - GET /power_profiles/{id} - get power profile for given id - PUT /power_profiles/{id} - modify power profile for given id - DELETE /power_profiles/{id} - delete power profile for given id - GET /stats - get stats - GET /caps - get system capabilities Example response: {"capabilities": ["l3cat","mba","sstbf","power"] } - GET /caps/sstbf - get Intel(R) SST-BF details Example response: {"configured": true, "hp_cores": [2,...,88], "std_cores": [0,...,95] } - PUT /caps/sstbf - configure Intel(R) SST-BF Example request: {"configured": true} - GET /caps/rdt_iface - get Intel(R) RDT interface type Example response: {"interface": "msr", "interface_supported": ["msr", "os"] } - PUT /caps/rdt_iface - set Intel(R) RDT interface type Example request: {"interface": "os"} NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system. "Default" pool will be reset to default values (full cache and MEM BW access). - GET /caps/mba - get Intel(R) RDT MBA details Example response: {"mba_enabled": true, "mba_bw_enabled": false } - GET /caps/mba_ctrl - get Intel(R) RDT MBA CTRL state Example response: {"enabled": false, "supported": true } - PUT /caps/mba_ctrl - set Intel(R) RDT MBA CTRL state Example request: {"enabled": true} NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system. "Default" pool will be reset to default values (full cache and MEM BW access). USAGE SCENARIO ============== App QoS, leveraging Intel(R) RDT (CAT, MBA) and SST (BF, CP) Technologies, is to be used to prioritize and protect performance of high priority applications. Please see below for complete step-by-step guide to configure App QoS to leverage Intel(R) RDT (CAT, MBA) and SST-BF Technologies to prioritize and protect performance of high priority application. SETUP Get mTLS certificates for App QoS CA, App QoS and REST client (for more info please see HOW TO GENERATE APP QOS AND CLIENT CERTIFICATE paragraph). Create appqos.conf file (for more info please see Configuration). cat /tmp/appqos.conf { "apps": [ ], "sstbf": { "configured": true }, "pools": [ ], "power_profiles_expert_mode": true, "rdt_iface": { "interface": "msr"}, "mba_ctrl": { "enabled": false } } Empty config file with essential configuration only: - RDT interface and MBA mode configuration - SST-BF configuration - option to enable Power Profile "Expert Mode" to allow us to add power profiles via REST API App QoS command line parameters (for more info please see USAGE paragraph) python3 ./appqos --help usage: appqos [-h] [-c PATH] [--port PORT] [-V] optional arguments: -h, --help show this help message and exit -c PATH, --config PATH Configuration file path --port PORT REST API port -V, --verbose Verbose mode NOTE: REST API port (5000 by default) can be set via "--port" START APP QOS sudo make run INFO Interface MSR, MBA BW: unsupported. INFO Interface OS, MBA BW: supported. INFO Supported RDT interfaces: ['msr', 'os'] INFO RDT initialized with 'msr' interface INFO Supported capabilities: INFO ['l3cat', 'mba', 'sstbf', 'power'] * Serving Flask app "rest.rest_server" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off INFO * Running on https://127.0.0.1:5000/ (Press CTRL+C to quit) INFO Configuring SST-BF INFO SST-BF enabled, configured. INFO SST-BF HP cores: [2, 10, 11, 12, 13, 14, 15, 16, 25, 34, 35, 36, 37, 38, 39, 40] INFO SST-BF STD cores: [0, 1, 3, 4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 41, 42, 43, 44, 45, 46, 47] INFO Power Profiles/EPP enabled, not configured, SST-BF is configured INFO Configuring RDT INFO RDT MBA CTRL disabled INFO Configuration changed, processing new config... VERIFY INITIAL CONFIGURATION List all Pools curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq { "id": 0, "mba": 100, "l3cbm": 2047, "name": "Default", "cores": [ 0, 1, 2, 3, ... 43, 44, 45, 46, 47 ] } ] NOTE: "Default" pool automatically created (id=0). Pool takes all cores (48), 100% of MBA and full CAT CBM (all cache ways). List all Apps curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq { "message": "No apps in config file" } NOTE: No Apps defined. List capabilities curl https://localhost:5000/caps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq { "capabilities": [ "l3cat", "mba", "sstbf", "power" ] } NOTE: Detected capabilities listed, Intel(R) RDT CAT, MBA and Intel(R) SST BF, CP technologies supported. List RDT interface configuration details curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"interface": "msr", "interface_supported": ["msr", "os"] } NOTE: MSR interface is used (as per config file). System supports both MSR and OS interfaces. List MBA CTRL/MBA details curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"enabled": false, "supported": false } NOTE: MBA CTRL not supported for MSR interface. curl https://localhost:5000/caps/mba -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"mba_enabled": true, "mba_bw_enabled": false} NOTE: MBA CTRL is NOT enabled (as per config file) so "mba_bw" is not available, user should use "mba" field to configure "pools" MBA in %. If MBA CTRL would be enabled, user would use "mba_bw" to configure "pools" MBA in MBps. Error 404 will be returned when MBA is not supported. List Intel(R) SST-BF details curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"configured":true "hp_cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "std_cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47]} NOTE: SST-BF enabled (as per config file), list of HP and STD cores provided. Verify frequencies settings sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))' Freq. Freq. Core# Min Max EPP 0 2100 2100 balance_performance 1 2100 2100 balance_performance 2 2800 2800 balance_performance 3 2100 2100 balance_performance ... 45 2100 2100 balance_performance 46 2100 2100 balance_performance 47 2100 2100 balance_performance NOTE: HP cores' frequencies are set to 2800MHz, STD cores' to 2100MHz Unconfigure SST-BF curl https://localhost:5000/caps/sstbf -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"configured": false}' { "message": "SST-BF caps modified" } Verify frequencies settings sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))' Freq. Freq. Core# Min Max EPP 0 2300 2300 balance_performance 1 2300 2300 balance_performance 2 2300 2300 balance_performance 3 2300 2300 balance_performance ... 45 2300 2300 balance_performance 46 2300 2300 balance_performance 47 2300 2300 balance_performance NOTE: HP and STD cores' frequencies are set to 2300MHz CREATE APPS Create multiple processes, list their PIDs and core affinity killall sleep for n in {1..5}; do bash -c "sleep 1d &"; done for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0-47 pid 13892's current affinity list: 0-47 pid 13890's current affinity list: 0-47 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 Create App, specify cores curl https://localhost:5000/apps -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "APP1", "cores": [0,1,2,3], "pids": [13894, 13892]}' { "id": 1 } Verify App's creation (list ALL apps) curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt [ { "name": " APP1", "cores": [ 0, 1, 2, 3 ], "pids": [ 13894, 13892 ], "id": 1, "pool_id": 0 } ] NOTE: App was assigned to Pool #0 as cores 0,1,2,3 are part of that Pool Verify PIDs' core affinity for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0-3 pid 13892's current affinity list: 0-3 pid 13890's current affinity list: 0-47 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 Create App, specify destination Pool ID curl https://localhost:5000/apps -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "APP2", "pool_id": 0, "pids": [13890]}' { "id": 2 } Verify App's creation (get specific App) curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt { "name": "APP2", "pids": [ 13890 ], "id": 2, "pool_id": 0 } NOTE: App was assigned to Pool #0 as requested via "pool_id" param Verify PIDs' core affinity for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0-3 pid 13892's current affinity list: 0-3 pid 13890's current affinity list: 0-47 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 NOTE: App2 PID's core affinity is 0-47 as those are "Default" pool cores. MODIFY "DEFAULT" POOL List SST-BF's STD cores curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq .std_cores -c [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47] Modify "Default" pool to exclude HP cores, assign 50% of MBA and 4 LLC CWs (CBM: 0xF/15) curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "mba": 50, "l3cbm": "0xf"}' "POOL 0 updated" Check new "Default" pool configuration curl https://localhost:5000/pools/0 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"id":0, "mba":50, "l3cbm":15, "name":"Default", "cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "apps":[1,2]} Check how does "Default" pool modification has affected Apps in that pool curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt [ { "name": "APP1", "pids": [ 13894, 13892 ], "id": 1, "pool_id": 0 }, { "name": "APP2", "pids": [ 13890 ], "id": 2, "pool_id": 0 } ] for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 NOTE: As some of the cores that were specified for APP1 are not part of "Default" pool anymore, core affinity for APP1 PIDs' was reset to new "Default" pool cores. CREATE HP POOL Get list of HP cores curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq .hp_cores -c [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40] Create HP Pool, with 7 (isolated) LLC CWs allocated and 100% of MBA curl https://localhost:5000/pools -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "HP", "cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "l3cbm": "0x7F0", "mba": 100}' { "id": 7 } NOTE: Pool with "id=7" was created. Get All Pools to verify creation of new HP Pool(#7) curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c [{"id":0, "mba":50, "l3cbm":15, "name":"Default", "cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "apps":[1,2]}, {"name":"HP", "cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "l3cbm":2032, "mba":100, "id":7}] DEFINE AND APPLY POWER PROFILES Define HP power profile curl https://localhost:5000/power_profiles -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "PP HP", "min_freq": 3000, "max_freq": 3000, "epp": "performance"}' { "id": 0, "message": "New POWER PROFILE 0 added" } Define STD power profile curl https://localhost:5000/power_profiles -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "PP STD", "min_freq": 1500, "max_freq": 1500, "epp": "power"}' { "id": 1, "message": "New POWER PROFILE 1 added" } Get All Power Profiles to verify creation of new Power Profiles curl https://localhost:5000/power_profiles -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq [ { "name": "PP HP", "min_freq": 3000, "max_freq": 3000, "epp": "performance", "id": 0 }, { "name": "PP STD", "min_freq": 1500, "max_freq": 1500, "epp": "power", "id": 1 } ] Apply power profiles curl https://localhost:5000/pools/7 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 0}' { "message": "POOL 7 not updated, Power Profiles configuration would cause CPU to be oversubscribed." } NOTE: App QoS supports Admission Control feature, which protects CPU from being oversubscribed, as a result of that, order in which power profile are applied is important. (Configuration change can be forced by setting "verify" flag to "false" for a request) Lets apply the STD power profile first. curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 1}' { "message": "POOL 0 updated" } curl https://localhost:5000/pools/7 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 0}' { "message": "POOL 7 updated" } Verify power profiles configuration in HW sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))' Freq. Freq. Core# Min Max EPP 0 1500 1500 power 1 1500 1500 power 2 3000 3000 performance 3 1500 1500 power ... 45 1500 1500 power 46 1500 1500 power 47 1500 1500 power NOTE: Cores from HP pool have frequency fixed to 3000MHz and EPP set to "performance", STD pool's cores have frequency fixed to 1500MHz and EPP set to "power". MOVE APP#2 Move App#2 to HP Pool(#7) curl https://localhost:5000/apps/2 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"pool_id": 7}' "APP 2 moved to new pool" Verify new configuration, list all apps curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt [ { "name": "APP1", "pids": [ 13894, 13892 ], "id": 1, "pool_id": 0 }, { "name": "APP2", "pids": [ 13890 ], "id": 2, "pool_id": 7 } ] Verify new configuration, list pools curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c [{"id":0, "mba":50, "l3cbm":15, "name":"Default", "cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "apps":[1]}, {"name":"HP", "cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "l3cbm":2032, "mba":100, "id":7, "apps":[2]}] Verify new configuration, check core affinity for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13890's current affinity list: 2,10-16,25,34-40 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 MODIFY APP#2 Modify App core affinity and name curl https://localhost:5000/apps/2 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"cores": [2,11,13], "name": "APP2 HP"}' "APP 2 moved to new pool" Verify new configuration, get App#2 details curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt { "name": "APP2 HP", "pids": [ 13890 ], "id": 2, "cores": [ 2, 11, 13 ], "pool_id": 7 } Verify new configuration, check core affinity for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13890's current affinity list: 2,11,13 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 REMOVE APP#2 Remove App#2 curl https://localhost:5000/apps/2 -X DELETE --cert client_appqos.crt --key client_appqos.key --cacert ca.crt "APP 2 deleted" Verify new configuration, get App#2 and all Apps details curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt { "message": "APP 2 not found in config" } NOTE: App#2 not found curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt [ { "name": "stress-ng", "pids": [ 13894, 13892 ], "id": 1, "pool_id": 0 } ] NOTE: No App#2 listed Verify new configuration, check core affinity for i in `pidof sleep`; do taskset -p -c $i; done pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47 pid 13888's current affinity list: 0-47 pid 13886's current affinity list: 0-47 NOTE: App#2's PID core affinity set to "Default" pool cores REMOVE HP POOL Remove HP Pool (#7) curl https://localhost:5000/pools/7 -X DELETE --cert client_appqos.crt --key client_appqos.key --cacert ca.crt "POOL 7 deleted" Verify new configuration, get Pool#7 and all pools details curl https://localhost:5000/pools/7 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt { "message": "POOL 7 not found in config" } NOTE: Pool#7 not found curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c [{"id":0, "mba":50, "l3cbm":15, "name":"Default", "cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "apps":[1]}] NOTE: No Pool#7 listed RESET CONFIGURATION Perform configuration reset. curl https://localhost:5000/reset -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt "Reset performed. Configuration reloaded." Verify configuration after reset curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -k { "message": "No apps in config file" } NOTE: No Apps configured curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c [{"id":0, "mba":100, "l3cbm":2047, "name":"Default", "cores":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]}] NOTE: "Default" pool reconfigured, takes all cores (48), 100% of MBA and full CAT CBM (all cache ways). ENABLE RDT OS INTERFACE Verify that RDT OS interface is supported. curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"interface": "msr", "interface_supported": ["msr", "os"] } NOTE: RDT "os" interface is supported but NOT enabled. Enable RDT OS interface. curl https://localhost:5000/caps/rdt_iface -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"interface": "os"}' {"message": "RDT Interface modified"} NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0) (e.g.: all non-default, configured Pools were removed). "Default" pool will be reset to default values (full cache and MEM BW access). List RDT configuration details curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"interface": "os", "interface_supported": ["msr", "os"] } NOTE: RDT OS interface is now enabled. ENABLE MBA CTRL Verify MBA CTRL status. curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"enabled": false, "supported": true } NOTE: MBA CTRL is supported but NOT enabled. Enable RDT MBA CTRL curl https://localhost:5000/caps/mba_ctrl -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"enabled": true}' {"message": "MBA CTRL status changed."} NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0) (e.g.: all non-default, configured Pools were removed). "Default" pool will be reset to default values (full cache and MEM BW access). Verify MBA CTRL status mount | grep resctrl resctrl on /sys/fs/resctrl type resctrl (rw,relatime,mba_MBps) NOTE: resctrl fs mounted with "mba_MBps" option, MBA CTRL is enabled. curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"enabled": true, "supported": true } curl https://localhost:5000/caps/mba -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c {"mba_bw_enabled": true "mba_enabled": false} NOTE: MBA CTRL is now enabled. User can use "mba_bw" to configure pool's MBA allocation. Verify MBA configuration for "Default" Pool curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq [ { "id": 0, "mba_bw": 4294967295, "l3cbm": 2047, "name": "Default", "cores": [ 0, 1, ... 45, 46, 47 ] } ] NOTE: MBA for Pool #0 set to maximum value 4294967295MBps (~2^32MBps). Limit Pool #0 MBA to ~5GBps curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"mba_bw": 5000}' {"message": "POOL 0 updated"} Verify new Pool #0 MBA configuration. curl https://localhost:5000/pools/0 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq { "id": 0, "mba_bw": 5000, "l3cbm": 2047, "name": "Default", "cores": [ 0, 1, ... 45, 46, 47 ] } NOTE: MBA for Pool #0 is set to 5000MBps cat /sys/fs/resctrl/schemata L3:0=7ff;1=7ff MB:0=5000;1=5000 NOTE: Resctrl shows new MBA configuration for Pool #0. APP QOS AND CLIENT CERTIFICATE ============================= App QoS uses mutual TLS - both App QoS and client certificates need to be signed using the same root ca certificate. It is the user responsibility to deliver certificate and key files for App QoS. App QoS expects certain names of CA certificate, App QoS certificate and associated private key - files need to be located in 'appqos/ca' directory: - ca.crt - contains the CA certificate, that will be used to authenticate clients and App QoS during mTLS - appqos.crt - contains App QoS certificate, it needs to be signed using ca.crt - appqos.key - contains private key for the App QoS certificate App QoS REST Client certificate needs to be signed by the same CA as App QoS certificate. In this instruction we assume that client certificate will be named 'client_appqos.crt' and client will have access to 'ca.crt' CA certificate. HOW TO GENERATE TEST APP QOS AND CLIENT CERTIFICATE ================================================== appqos/ca/ directory contains sample script called 'gen_test_certs.sh' - it can be used for generating certifacates for the App QoS and the client (e.g: curl) for testing purposes only $ cd appqos/ca $ ./gen_test_certs.sh This script will create: - ca certificate - appqos certificate - client certificate Legal Disclaimer ================ THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.