v2a: ETURWG GM40 v2b: ETURWG GM41 v2c: ETURWG GM49 v2d: ETURWG GM55 v3: ETURWG GM83 v3a: ETURWG GM90 v3b: ETURWG GM102 F18 F2F v4.0.0: ETURWG
Uncertainty Representation and Reasoning Evaluation Framework's Ontology
A taxonomy for modeling uncertain information.
Current advances in technology, sensor collection, data storage, and data distribution have afforded more complex, distributed, and operational information fusion systems (IFSs). IFSs notionally consist of low-level (data collection, registration, and association in time and space) and high-level information fusion (user coordination, situational awareness, and mission control), which require a common ontology for effective communication and data processing. This ontology reference model is developed as part of the uncertainty representation and reasoning evaluation framework (URREF). The URREF ontology is intended to provide guidance for defining the actual concepts and criteria that together comprise the comprehensive uncertainty evaluation framework being developed by the Evaluation of Technologies for Uncertainty Representation Working Group (ETURWG) of the International Society of Information Fusion (ISIF).
@inproceedings{costa2012towards, title={Towards unbiased evaluation of uncertainty reasoning: The URREF ontology}, author={Costa, Paulo C.G. and Laskey, Kathryn B. and Blasch, Erik and Jousselme, Anne-Laure}, booktitle={15th International Conference on Information Fusion}, pages={2301--2308}, year={2012}, organization={IEEE} }
Evaluation of Technologies for Uncertainty Representation Working Group
International Society of Information Fusion
CamelCase object properties are deprecated in URREF version 4.1.0, they will be removed in version 5.
TODO: decide whether associations are pascalCase or CamelCase.
Closeness of agreement between an evaluation subject value and the true value of the quantity or quality being evaluated.
Ability of the representational model to allow for different configurations of the model.
Measure of the ability of the system to handle the types of uncertainty assessments (e.g., verbal, quantitative, combined) needed for a given problem, and to distinguish them from one another.
Belief functions are closely related to probabilities. Beliefs in a hypothesis is calculated as the sum of the masses of all sets it encloses. A belief function differs from a Bayesian probability model in that one does not condition on those parts of the evidence for which no probabilities are specified. This ability to explicitly model the degree of ignorance makes the theory very appealing and has been applied in areas such as inconsistency handling in OWL ontologies (Nikolov et al., 2007) and ontology mapping (e.g. Yaghlane and Laamari, 2007).
TODO
Measure of how compatible a given knowledge representation is to data standards, and should be related to the degree of flexibility it has in being coded with various standards.
Taken from https://www.researchgate.net/publication/334836334_How_to_Evaluate_High_Level_Fusion_Algorithms
Computes the amount of information that is present in both the result r of the algorithm and the expected ideal result (i.e. ground truth), with regards to the amount of information contained in the ideal result (ground truth),
Computes the amount of information that is present in both the result r of the algorithm and the expected ideal result (i.e. ground truth), with regards to the amount of information contained in the actual result r,
Moved from "ReasoningCriterion" in version 4.0.0.
Measure of how much of the system’s computational resources are required by a given representational technique to produce its results.
[NOT REVIEWED as of GM55] Ability of an uncertainty representation to support combining uncertainty about many propositions and/or variables, of different types, with different types of uncertainty.
Measure of the ability of the reasoning process to produce the same results when provided with the same data under the same conditions.
Measure of the degree to which an evaluation subject can be believed or accepted as true, real, or honest.
Ability of the uncertainty representation to capture dependency among propositions (e.g., cause and effect, relevance, statistical association).
Encompasses all the different aspects that must be considered when evaluating uncertainty handling in multi-sensor fusion systems
Examples covers: STANAG2511Credibility, STANAG2511Reliability, Shanon entropy...
TODO: what about quantitative versus qualitiative?
Entities of the Evaluation class are actual evaluations. According to Patton, Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a program's activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions. Patton, M.Q. (1987). Qualitative Research Evaluation Methods. Thousand Oaks, CA: Sage Publishers.).
An Evaluation Subject is an item which can be assessed according to the criteria defined in the URREF ontology.
An expression in some logical language that evaluates to a truth-value (formula, axiom, assertion). It is then assumed that information will be presented in the form of sentences. So the uncertainty will be associated with sentences.
Evidence, for the purpose of this ontology is considered to be equivalent to: - Sentence - Information
Measure of the power of a knowledge representation formalism to convey all relevant aspects of a given fusion problem.
In contrast to probabilistic formalisms, which allow for representing and processing degrees of uncertainty about ambiguous pieces of information, fuzzy formalisms allow for representing and processing degrees of truth about vague (or imprecise) pieces of information. It is important to point out that vague statements are truth-functional, that is, the degree of truth of a vague complex statement (which is constructed from elementary vague statements via logical operators) can be calculated from the degrees of truth of its constituents, while uncertain complex statements are generally not a function of the degrees of uncertainty of their constituents (Dubois and Prade, 1994).
[NOT REVIEWED as of GM55] Ability of the system to represent uncertainty about the uncertainty model, including parameters, structure, and/or type of model.
Renammed from "DataCriteron" as of version 4.0.0.
Evaluation Criteria that concern aspects of information and their relationship to source, the object being reported upon, and the objectives of the fusion process.
Renamed from "DataHandlingCriterion" in version 4.0.0.
Measure of the way data is managed by the fusion system.
Measure the ability of a given uncertainty representation technique to convey knowledge.
Measure of the extent to which an evaluation subject reports about a matter of fact in an unbiased manner.
Quotient of the change in a result of evaluation subject and the corresponding change in a value of a quality being observed.
[NOT REVIEWED as of GM55] Ability to represent different kinds of outcomes for uncertain variables, e.g., Boolean, qualitative categorical, ordinal, discrete numerical, continuous.
Moved from "ReasoningCriterion" and renamed from "Performance" in version 4.0.0.
Measure how suitable the representational model is to handle the functional requirements of an information fusion system. Other system architecture factors also affect these metrics.
TODO, decide between : - The extent to which a piece of information covers different values. - The extent to which the domain of the information is constrained. - Previous definition: "Closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar evaluation subjects under specified conditions."
Probability theory provides a mathematically sound representation language and formal calculus for rational degrees of belief, which gives different agents the freedom to have different beliefs about a given hypothesis.
Ability to assess the degree of informational quality of the data
A rough set, first described by a Polish computer scientist Zdzisław I. Pawlak, is a formal approximation of a crisp set (i.e., conventional set) in terms of a pair of sets which give the lower and the upper approximation of the original set. In the standard version of rough set theory (Pawlak 1991), the lower- and upper-approximation sets are crisp sets, but in other variations, the approximating sets may be fuzzy set
Encompasses criteria directly affecting how an information fusion system transforms its input data into knowledge. It can also be called as process or inference criteria, as it deals with how the uncertainty model performs operations with information.
[NOT REVIEWED as of GM55] Ability to represent uncertainty about domains with relational structure, e.g., attribute value uncertainty; reference uncertainty; type uncertainty; existence uncertainty.
Measure the degree to which an evaluation subject has direct bearing on the objectives of the fusion process.
Encompasses criteria related to how uncertainty is characterized, captured and stored in a manner that can be processed by the fusion system.
Measure of the ability to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.
Measure of the evaluation subject's assessment of its own credibility.
Assesses the information system's ability to execute common operations without requiring deep knowledge about its inner details.
Changed name in version 4.0.0: from "Correctness" to "Soundness".
The quality of being based on valid reason or good judgement.
A source is the origin of the information.
Any feature of the source which will affect the quality of the information provided by the source.
Measure of the average (and (possibly peak) rate of conversion of inputs to outputs.
Measure of the ability to produce results within a required timeframe.
Measure of the ability of a fusion system to provide an accurate and unbroken historical record of its inputs and the chain of operations that led to its conclusions.
TODO: placeholder to import trust ontologies
Moved as a subclass of InformationCriterion in version 4.0.0.
Uncertainty derivation refers to the way it can be assessed. That is, how the uncertainty metrics can be derived.
Renamed from "UncertaintyFunction" in version 4.0.0.
TODO: - a probability function - a support over something
This captures the information about the nature of the uncertainty, i.e., whether the uncertainty is inherent in the phenomenon expressed by the sentence, or it is the result of lack of knowledge of the agent.
Uncertainty Type is a concept that focuses on underlying characteristics of the information that make it uncertain.
For the purposes of this ontology, Uncertainty type is equivalent to: - type of imperfection
Measure of the extent to which a source reports what it assesses to be the case.
Renamed from "WeightOfEvidence" in version 4.0.0.
Measure of the degree of impact of an evaluation subject on the result of fusion.