# AUTOMATICALLY GENERATED BY `process-doc -yaml`. DO NOT EDIT. # Edit `util/formats/newfmt/types.go`, then in `util/process-doc` run `make install`. # Version 0.19.0-devel MetadataObject: fields: producer: type: 'NonemptyString' doc: > The name of the component that generated the data. Example: "sonar", "slurm-monitor" version: type: 'NonemptyString' doc: > The semver of the producer format: type: 'uint64' doc: > The data format version attrs: type: '[]KVPair' doc: > An array of generator-dependent attribute values token: type: 'string' doc: > An API token to be used with Envelope.Data.Attributes.Cluster. It proves that the producer of the datum was authorized to produce data for that cluster name. EXPERIMENTAL / UNDERSPECIFIED. ErrorObject: fields: time: type: 'Timestamp' doc: > Time when the error was generated detail: type: 'NonemptyString' doc: > A sensible English-language error message describing the error cluster: type: 'Clustername' doc: > Canonical cluster name for node generating the error node: type: 'Hostname' doc: > name of node generating the error KVPair: fields: key: type: 'NonemptyString' doc: > A unique key within the array for the attribute value: type: 'string' doc: > Some attribute value SysinfoEnvelope: fields: meta: type: 'MetadataObject' doc: > Information about the producer and data format data: type: '*SysinfoData' doc: > Node data, for successful probes errors: type: '[]ErrorObject' doc: > Error information, for unsuccessful probes SysinfoData: fields: type: type: 'DataType' doc: > Data tag: The value "sysinfo" attributes: type: 'SysinfoAttributes' doc: > The node data themselves SysinfoAttributes: fields: time: type: 'Timestamp' doc: > Time the current data were obtained cluster: type: 'Clustername' doc: > The canonical cluster name node: type: 'Hostname' doc: > The name of the host as it is known to itself os_name: type: 'NonemptyString' doc: > Operating system name (struct utsname.sysname) os_release: type: 'NonemptyString' doc: > Operating system version (struct utsname.release) architecture: type: 'NonemptyString' doc: > Architecture name (struct utsname.machine) numa_nodes: type: 'uint64' doc: > Number of NUMA nodes, zero if we can't compute it. Frequently smaller than the number of sockets. sockets: type: 'NonzeroUint' doc: > Number of CPU sockets cores_per_socket: type: 'NonzeroUint' doc: > Number of physical cores per socket threads_per_core: type: 'NonzeroUint' doc: > Number of hyperthreads per physical core cpu_model: type: 'string' doc: > Manufacturer's model name memory: type: 'NonzeroUint' doc: > Primary memory in kilobytes topo_svg: type: 'string' doc: > Text output (SVG source text) of e.g. `lstopo`, base64 per RFC4648 topo_text: type: 'string' doc: > Text output (human-readable) of e.g. `hwloc-ls`, base64 per RFC4648 cards: type: '[]SysinfoGpuCard' doc: > Per-card information distances: type: '[][]uint64' doc: > Square matrix of NUMA node-to-node distances, normalized to 10 for self distance, empty if we can't compute it SysinfoGpuCard: fields: index: type: 'uint64' doc: > Node-local card index. See notes in preamble. This isn't marked "omitempty" because an index can be 0 and it's confusing to omit the field in that case uuid: type: 'string' doc: > UUID as reported by card. See notes in preamble address: type: 'string' doc: > An intra-system card address, eg PCI address manufacturer: type: 'string' doc: > A keyword, "NVIDIA", "AMD", "Intel" (others TBD) model: type: 'string' doc: > Card-dependent, this is the manufacturer's model string architecture: type: 'string' doc: > Card-dependent, for NVIDIA this is "Turing", "Volta" etc driver: type: 'string' doc: > Card-dependent, the manufacturer's driver string firmware: type: 'string' doc: > Card-dependent, the manufacturer's firmware string memory: type: 'uint64' doc: > GPU memory in kilobytes power_limit: type: 'uint64' doc: > Power limit in watts max_power_limit: type: 'uint64' doc: > Max power limit in watts min_power_limit: type: 'uint64' doc: > Min power limit in watts max_ce_clock: type: 'uint64' doc: > Max clock of compute element max_memory_clock: type: 'uint64' doc: > Max clock of GPU memory SampleEnvelope: fields: meta: type: 'MetadataObject' doc: > Information about the producer and data format data: type: '*SampleData' doc: > Sample data, for successful probes errors: type: '[]ErrorObject' doc: > Error information, for unsuccessful probes SampleData: fields: type: type: 'DataType' doc: > Data tag: The value "sample" attributes: type: 'SampleAttributes' doc: > The sample data themselves SampleAttributes: fields: time: type: 'Timestamp' doc: > Time the current data were obtained cluster: type: 'Clustername' doc: > The canonical cluster name whence the datum originated node: type: 'Hostname' doc: > The name of the node as it is known to the node itself system: type: 'SampleSystem' doc: > State of the node as a whole jobs: type: '[]SampleJob' doc: > State of jobs on the nodes errors: type: '[]ErrorObject' doc: > Recoverable errors, if any SampleSystem: fields: boot: type: 'Timestamp' doc: > The node's boot time cpus: type: '[]SampleCpu' doc: > The state of individual cores (cpu seconds used since boot), cores sorted in index order. The array is always dense and of length Sockets * CoresPerSocket * ThreadsPerCore (from SysinfoAttributes for the node). gpus: type: '[]SampleGpu' doc: > The state of individual GPU devices, cards sorted in card index order. The array may be sparse (some indices missing) and may in principle change in length and index set from sample to sample. disks: type: '[]SampleDisk' doc: > The state of individual disks used_memory: type: 'uint64' doc: > The amount of primary memory in use in kilobytes load1: type: 'NonnegativeFloat64' doc: > One-minute load average load5: type: 'NonnegativeFloat64' doc: > Five-minute load average load15: type: 'NonnegativeFloat64' doc: > Fifteen-minute load average runnable_entities: type: 'uint64' doc: > Number of currently runnable scheduling entities (processes, threads) existing_entities: type: 'uint64' doc: > Number of currently existing sceduling entities SampleGpu: fields: index: type: 'uint64' doc: > Node-local card index. See notes in preamble uuid: type: 'NonemptyString' doc: > Card UUID. See preamble for notes about UUIDs. failing: type: 'uint64' doc: > If not zero, an error code indicating a card failure state. code=1 is "generic failure". Other codes TBD. fan: type: 'uint64' doc: > Percent of primary fan's max speed, may exceed 100% on some cards in some cases compute_mode: type: 'string' doc: > Current compute mode, completely card-specific if known at all performance_state: type: 'ExtendedUint' doc: > Current performance level, card-specific >= 0, or unset for "unknown". memory: type: 'uint64' doc: > Memory use in Kilobytes ce_util: type: 'uint64' doc: > Percent of computing element capability used memory_util: type: 'uint64' doc: > Percent of memory used temperature: type: 'int64' doc: > Degrees C card temperature at primary sensor (note can be negative) power: type: 'uint64' doc: > Watts current power usage power_limit: type: 'uint64' doc: > Watts current power limit ce_clock: type: 'uint64' doc: > Compute element current clock memory_clock: type: 'uint64' doc: > Memory current clock SampleDisk: fields: name: type: 'string' doc: > Disk's local name. This must never be empty. major: type: 'uint64' doc: > Disk's local major device number. minor: type: 'uint64' doc: > Disk's local minor device number. stats: type: '[]uint64' doc: > Disk stats values in the order present in /proc/diskstats. Documentation: Linux 6.x has Documentation/admin-guide/iostats.rst [1]. Older kernels have Documentation/iostats.txt [2]. NOTE: - The layout of this array changes between kernel versions and may require knowledge of the kernel version to interpret. See the OsRelease field in SysinfoAttributes. - In particular, the length of the array is not constant across kernel versions, though it appears new fields are always appended. - Counters may sometimes wrap around, and they will be reset on reboot. See also the Boot field in SampleSystem. - Field interpretation may also vary between versions. - The field should be considered experimental; when it was introduced, we did not know which data would be interesting. It may be augmented by other fields in the future. [1] https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/iostats.rst. [2] https://elixir.bootlin.com/linux/v4.18.20/source/Documentation/iostats.txt. SampleJob: fields: job: type: 'uint64' doc: > The job ID user: type: 'NonemptyString' doc: > User name on the cluster; `_user_` if not determined but user ID is available, `_user_unknown` otherwise. Notably in the case of container processes (see the "InContainer" flag in SampleProcess), the user may be nonsensical (unknown or wrong uid) or root. epoch: type: 'uint64' doc: > Zero for batch jobs, otherwise is a nonzero value that increases (by some amount) when the system reboots, and never wraps around. You may think of it as a boot counter for the node, but you must not assume that the values observed will be densely packed. See notes. processes: type: '[]SampleProcess' doc: > Processes in the job, all have the same Job ID. SampleProcess: fields: resident_memory: type: 'uint64' doc: > Kilobytes of private resident memory. virtual_memory: type: 'uint64' doc: > Kilobytes of virtual data+stack memory cmd: type: 'string' doc: > The command (not the command line), zombie processes get an extra annotation at the end, a la ps. exe: type: 'string' doc: > The executable path, if available (privileged). pid: type: 'uint64' doc: > Process ID. Normally Pid is non-zero. For rolled-up samples in daemon mode (where a set of similar processes that changes over time is rolled up as one process that has no obvious representative Pid), it is a synthesized non-zero value outside the system's Pid range; see below at the "Rolledup" field and the section "Rolled-up samples" in the postamble. However, the Pid may sometimes be zero: - it is zero for rolled-up samples in one-shot mode - it was zero for rolled-up samples in daemon mode before v0.18 - it may be zero in corner cases where coherent information cannot be collected for a process or the system does not provide a Pid (issue #499 is about documenting this fully) ppid: type: 'uint64' doc: > Parent-process ID. in_container: type: 'bool' doc: > True if deemed a part of a container. Notably, for docker this means it is a child of something whose name starts with "containerd" where that parent is itself a child of init. InContainer is a flag on the process and not on the job since it is possible for a job to fork off a container as part of its work but itself not be running in a container. num_threads: type: 'uint64' doc: > The number of threads in the process, minus 1. We don't count the process's main thread, allowing this fields to be omitted in transmission for most processes. cpu_avg: type: 'NonnegativeFloat64' doc: > The running average CPU percentage over the true lifetime of the process as reported by the operating system. 100.0 corresponds to "one full core's worth of computation". See notes. cpu_util: type: 'NonnegativeFloat64' doc: > The current sampled CPU utilization of the process, 100.0 corresponds to "one full core's worth of computation". See notes. cpu_time: type: 'uint64' doc: > Cumulative CPU time in seconds for the process over its lifetime. See notes. data_read: type: 'uint64' doc: > Kilobytes read with all sorts of read calls (rounded up). data_written: type: 'uint64' doc: > Kilobytes written with all sorts of write calls (rounded up). data_cancelled: type: 'uint64' doc: > Kilobytes written but never flushed to physical media (rounded up). These are data held in RAM but then made obsolete by overwriting or file deletion or similar. rolledup: type: 'uint64' doc: > The number of additional samples for processes that are "the same" that have been rolled into this one. That is, if the value is 1, the record represents the sum of the sample data for two processes. See "Rolled-up samples" in the postamble for more. gpus: type: '[]SampleProcessGpu' doc: > GPU sample data for all cards used by the process. SampleProcessGpu: fields: index: type: 'uint64' doc: > Node-local card index. See notes in preamble uuid: type: 'NonemptyString' doc: > Card UUID. See preamble for notes about UUIDs. gpu_util: type: 'NonnegativeFloat64' doc: > The current GPU percentage utilization for the process on the card. (The "gpu_" prefix here and below is sort of redundant but have been retained since it makes the fields analogous to the "cpu_" fields.) gpu_memory: type: 'uint64' doc: > The current GPU memory used in kilobytes for the process on the card. See notes. gpu_memory_util: type: 'NonnegativeFloat64' doc: > The current GPU memory usage percentage for the process on the card. See notes. JobsEnvelope: fields: meta: type: 'MetadataObject' doc: > Information about the producer and data format data: type: '*JobsData' doc: > Jobs data, for successful probes errors: type: '[]ErrorObject' doc: > Error information, for unsuccessful probes JobsData: fields: type: type: 'DataType' doc: > Data tag: The value "jobs" attributes: type: 'JobsAttributes' doc: > The jobs data themselves JobsAttributes: fields: time: type: 'Timestamp' doc: > Time the current data were obtained cluster: type: 'Clustername' doc: > The canonical cluster name slurm_jobs: type: '[]SlurmJob' doc: > Individual job records. There may be multiple records per job, one per job step. SlurmJob: fields: job_id: type: 'NonzeroUint' doc: > The Slurm Job ID that directly controls the task that the record describes, in an array or het job this is the primitive ID of the subjob. sacct: the part of `JobIDRaw` before the separator (".", "_", "+"). slurm: `JOB_INFO.job_id`. job_step: type: 'string' doc: > The step identifier for the job identified by job_id. For the topmost step/stage of a job this will be the empty string. Other values normally have the syntax of unsigned integers, but may also be the strings "extern" and "batch". This field's default value is the empty string. NOTE: step 0 and step "empty string" are different, in fact thinking of a normal number-like step name as a number may not be very helpful. sacct: the part of `JobIDRaw` after the separator. slurm: via jobs: `Job.STEP_STEP.name`, probably. job_name: type: 'string' doc: > The name of the job. sacct: `JobName`. slurm: `JOB_INFO.name`. job_state: type: 'NonemptyString' doc: > The state of the job described by the record, an all-uppercase word from the set PENDING, RUNNING, CANCELLED, COMPLETED, DEADLINE, FAILED, OUT_OF_MEMORY, TIMEOUT. sacct: `State`, though sometimes there's additional information in the output that we will discard ("CANCELLED by nnn"). slurm: `JOB_STATE.current` probably, though that has multiple values. array_job_id: type: 'uint64' doc: > The overarching ID of an array job, see discussion in the postamble. sacct: the n of a `JobID` of the form `n_m.s` slurm: `JOB_INFO.array_job_id`. array_task_id: type: 'uint64' doc: > if `array_job_id` is not zero, the array element's index. Individual elements of an array job have their own plain job_id; the `array_job_id` identifies these as part of the same array job and the array_task_id identifies their position within the array, see later discussion. sacct: the m of a `JobID` of the form `n_m.s`. slurm: `JOB_INFO.array_task_id`. het_job_id: type: 'uint64' doc: > If not zero, the overarching ID of a heterogenous job. sacct: the n of a `JobID` of the form `n+m.s` slurm: `JOB_INFO.het_job_id` het_job_offset: type: 'uint64' doc: > If `het_job_id` is not zero, the het job element's index. sacct: the m of a `JobID` of the form `n+m.s` slurm: `JOB_INFO.het_job_offset`. user_name: type: 'string' doc: > The name of the user running the job. Important for tracking resources by user. sacct: `User` slurm: `JOB_INFO.user_name` account: type: 'string' doc: > The name of the user's account. Important for tracking resources by account. sacct: `Account` slurm: `JOB_INFO.account` submit_time: type: 'Timestamp' doc: > The time the job was submitted. sacct: `Submit` slurm: `JOB_INFO.submit_time` time_limit: type: 'ExtendedUint' doc: > The time limit in minutes for the job. sacct: `TimelimitRaw` slurm: `JOB_INFO.time_limit` partition: type: 'string' doc: > The name of the partition to use. sacct: `Partition` slurm: `JOB_INFO.partiton` reservation: type: 'string' doc: > The name of the reservation. sacct: `Reservation` slurm: `JOB_INFO.resv_name` nodes: type: '[]HostnameRange' doc: > The nodes allocated to the job or step. sacct: `NodeList` slurm: `JOB_INFO.nodes` priority: type: 'ExtendedUint' doc: > The job priority. sacct: `Priority` slurm: `JOB_INFO.priority` distribution: type: 'string' doc: > Requested layout. sacct: `Layout` slurm: `JOB_INFO.steps[i].task.distribution` requested_resources: type: 'string' doc: > Requested resources. If present, this comes from scontrol's ReqTRES field. See DecodeSlurmTRES() in decode_jobs.go in this directory for encoding details. We are most often interested in GPU resources, as other resources are represented in other ways. GPU resources are subfields tagged as "gres/gpu=*" for "any gpu" and "gres/gpu:model=n" for specific GPU model names. Memory sizes in the TRES may be encoded with a suffix K, M, G, T, or P representing (as far as is known) 2^10, 2^20, 2^30, 2^40, and 2^50 bytes. scontrol: `ReqTRES` Note that it's been observed that scontrol produces the items in this field in a surprising order. Normally resource requests express a priority order, so specific GPUs will appear in the list before a generic request. In the output from scontrol for ReqTRES, this is not always so (yet AllocTRES seems to have the expected order). allocated_resources: type: 'string' doc: > Allocated resources. If present, this comes from sacct's AllocTRES field. See ReqTRES documentation for encoding details. sacct: `AllocTRES` requested_cpus: type: 'uint64' doc: > Number of requested CPUs for the job in total. The documentation is not good - it's an assumption that this is for the job in total. sacct: `ReqCPUS` slurm: `JOB_INFO.cpus` requested_memory_per_node: type: 'uint64' doc: > Amount of requested memory in kilobytes. sacct: `ReqMem` slurm: `JOB_INFO.memory_per_node` requested_node_count: type: 'uint64' doc: > Number of requested nodes. sacct: `ReqNodes` slurm: `JOB_INFO.node_count` start_time: type: 'Timestamp' doc: > Time the job started, if started. sacct: `Start` slurm: `JOB_INFO.start_time` suspend_time: type: 'uint64' doc: > Number of seconds the job was suspended. sacct: `Suspended` slurm: `JOB_INFO.suspend_time` end_time: type: 'Timestamp' doc: > Time the job ended (or was cancelled), if ended. sacct: `End` slurm: `JOB_INFO.end_time` exit_code: type: 'uint64' doc: > Job exit code, if ended. sacct: `ExitCode` slurm: `JOB_INFO.exit_code.return_code` sacct: type: '*SacctData' doc: > Data specific to sacct output SacctData: fields: AllocTRES: type: 'string' doc: > Requested resources. These are the resources allocated to the job/step after the job started running. AveCPU: type: 'uint64' doc: > Average (system + user) CPU time of all tasks in job. AveDiskRead: type: 'uint64' doc: > Average number of kilobytes read by all tasks in job. AveDiskWrite: type: 'uint64' doc: > Average number of kilobytes written by all tasks in job. AveRSS: type: 'uint64' doc: > Average resident set size of all tasks in job, in kilobytes. AveVMSize: type: 'uint64' doc: > Average Virtual Memory size of all tasks in job, in kilobytes. ElapsedRaw: type: 'uint64' doc: > The job's elapsed time in seconds. MaxRSS: type: 'uint64' doc: > Maximum resident set size of all tasks in job, in kilobytes. MaxVMSize: type: 'uint64' doc: > Maximum Virtual Memory size of all tasks in job, in kilobytes. MinCPU: type: 'uint64' doc: > Minimum (system + user) CPU time of all tasks in job. SystemCPU: type: 'uint64' doc: > The amount of system CPU time used by the job or job step. UserCPU: type: 'uint64' doc: > The amount of user CPU time used by the job or job step. ClusterEnvelope: fields: meta: type: 'MetadataObject' doc: > Information about the producer and data format data: type: '*ClusterData' doc: > Node data, for successful probes errors: type: '[]ErrorObject' doc: > Error information, for unsuccessful probes ClusterData: fields: type: type: 'DataType' doc: > Data tag: The value "cluster" attributes: type: 'ClusterAttributes' doc: > The cluster data themselves ClusterAttributes: fields: time: type: 'Timestamp' doc: > Time the current data were obtained cluster: type: 'Clustername' doc: > The canonical cluster name slurm: type: 'bool' doc: > True if at least some nodes are under Slurm management. partitions: type: '[]ClusterPartition' doc: > Descriptions of the partitions on the cluster nodes: type: '[]ClusterNodes' doc: > Descriptions of the managed nodes on the cluster ClusterPartition: fields: name: type: 'NonemptyString' doc: > Partition name nodes: type: '[]HostnameRange' doc: > Nodes in the partition ClusterNodes: fields: names: type: '[]HostnameRange' doc: > Constraint: The array of names may not be empty states: type: '[]string' doc: > The state(s) of the nodes in the range. This is the output of sinfo as for the StateComplete specifier, split into individual states, and the state names are always folded to upper case.