Explanation Faithfulness Evaluation Measures Ontology
type: Ontology
abstract: As artificial intelligence systems, especially large language models, gain popularity there is an increased desire to trust these systems. One common method to increase trust is to provide explanations. These explanations provide information about the system’s workings and the knowledge used in its general reasoning processes or the processes behind a specific decision. However, ensuring that these explanations are faithful, that they accurately represent that reasoning process, is a difficult and ongoing task. Over 60 measures have been proposed in the last 5 years to quantify how faithful an explanation is, but it is difficult to compare them to decide which one is most appropriate for a given use case. This ontology provides a structured representation of faithfulness measures that would make comparisons easier. It documents the inferences that researchers use to evaluate and categorize these measures. Researchers proposing new measures can communicate the benefits of their idea with a shared vocabulary. Ideally this will also serve as the schema for a comprehensive knowledge graph of measures which will power a recommendation system for AI explainability researchers.
creator: Danielle Villa: https://tw.rpi.edu/person/danielle-villa
imports: prov-o-20130430
imports: AnnotationVocabulary
imports: Collections
imports: Identifiers
imports: Documents
version IRI: http://www.semanticweb.org/villad4/ontologies/efemo/1.0

Explanation
type: Class
subclass of: Class
  intersection of: Restriction
    on property: hasExplanandum
    some values from: Event
  intersection of: Restriction
    on property: hasExplanans
    some values from: Event
subclass of: Restriction
  on property: hasCondition
  some values from: Situation
subclass of: Restriction
  on property: has presentation
  some values from: Explanation Modality
comment: Class explanation defined as "when X happens, then, due to a given set of circumstances C, Y will occur because of a given law L". In order to be complete, an explanation needs at least one antecedent event (explanans), a posterior event (a posterior event) and has to happen in a context that relates the two events (the context), governed by a law (theory).

Assumption
type: Class
is defined by: Merriam-Webster: Assumption, Definition 3b - https://www.merriam-webster.com/dictionary/assumption
definition: A fact or statement taken for granted

Axiomatic Evaluation Method
type: Class
subclass of: Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: An evaluation method that treats certain principles as necessary conditions for faithfulness and tests if an explanation(s) satisfies them
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Binary Faithfulness Measure
type: Class
subclass of: Faithfulness Measure
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: A faithfulness measure that determines whether an explanation(s) is faithful or not
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
synonym: Faithfulness Test

Collection of Explanations
type: Class
subclass of: Collection
subclass of: Restriction
  on property: comprises
  some values from: Explanation
is defined by: A collection of explanations to be evaluated for a single faithfulness value

Computational-based Evaluation Method
type: Class
subclass of: Evaluation Method
alternative label: Quantitative Evaluation Method
definition: An evaluation method that does not rely on human assessment
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Corollary
type: Class
subclass of: Assumption
subclass of: Restriction
  on property: Inferred from Assumption
  some values from: Assumption
is defined by: Merriam-Webster: Corollary, Definition 1 - https://www.merriam-webster.com/dictionary/corollary
definition: An assumption inferred immediately from another assumption with little or no additional proof

Evaluation Method
type: Class
subclass of: Restriction
  on property: Evaluates Proxy
  some values from: Faithfulness Proxy
is defined by: Merriam-Webster: Method, Definition 1ba - https://www.merriam-webster.com/dictionary/method
definition: The technique used to determine the faithfulness of the explanation(s)

Faithfulness Measure
type: Class
subclass of: Restriction
  on property: Evaluates Faithfulness of Explanations
  some values from: Collection of Explanations
subclass of: Restriction
  on property: Evaluates Modality
  some values from: Explanation Modality
subclass of: Restriction
  on property: Has Granularity
  some values from: Granularity
subclass of: Restriction
  on property: Has Model Specificity
  some values from: Model Specificty
subclass of: Restriction
  on property: Measures Proxy
  some values from: Faithfulness Proxy
subclass of: Restriction
  on property: Requires Access Level
  some values from: Model Access Level
subclass of: Restriction
  on property: Uses Method
  some values from: Evaluation Method
subclass of: Restriction
  on property: wasAttributedTo
  some values from: Faithfulness Document
subclass of: Restriction
  on property: isIdentifiedBy
  some values from: Faithfulness Measure Identifier
source: Agarwal, C., Tanneru, S. H., & Lakkaraju, H. (2024). Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614.
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
alternative label: Faithfulness Metric
definition: A technique for quantifying whether an explanation accurately represents the reasoning process behind a model's prediction
synonym: Accountability Measure, Descriptive Accuracy Measure, Explanation Transparency measure, Fidelity Measure

Faithfulness Measure Identifier
type: Class
subclass of: Identifier
subclass of: Restriction
  on property: identifies
  some values from: Faithfulness Measure
definition: An identifier that is associated with a faithfulness measure

Faithfulness Proxy
type: Class
subclass of: Restriction
  on property: Makes Assumption
  some values from: Assumption
source: Agarwal, C., Tanneru, S. H., & Lakkaraju, H. (2024). Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614.
is defined by: Oxford Reference: Proxy Variable - https://www.oxfordreference.com/display/10.1093/oi/authority.20110803100351624
definition: A property used instead of faithfulness when faithfulness cannot be measured directly
explanatory note: As faithfulness cannot currently be measured, this specifies the alternate property that is used as a substitute property or necessary condition for faithfulness

Feature-based Modality
type: Class
subclass of: Explanation Modality
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: An explanation modality that claims which parts of the input are more important than others to the model's decision
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
synonym: Feature-Attribution Modality, Heat-Map Modality, Token Importance Modality, Input Saliency Modality, Salience Map Modality, Extractive Rationale Modality

Graded Faithfulness Measure
type: Class
subclass of: Faithfulness Measure
definition: A faithfulness measure that determines the extent and likelihood of an explanation(s) being faithful
adapted from: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Granularity
type: Class
source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
definition: The level of specificity of a faithfulness determination
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
explanatory note: Faithfulness measures may determine a faithfulness score for different levels: a single explanation, an entire dataset of explanations, or all explanations produced by a model
synonym: Scope

Human-based Evaluation Method
type: Class
subclass of: Evaluation Method
alternative label: Qualitative Evaluation Method
definition: An evaluation method that relies on human assessment
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Model Access Level
type: Class
source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: The amount of model internals that must be available for a faithfulness measure to function

Model Specific
type: Class
subclass of: Model Specificty
is defined by: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
definition: The model specificity of targeting particular classes of models
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Model Specificty
type: Class
definition: How particular a faithfulness measure is to the architecture of a model
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
explanatory note: This poperty describes whether or not the faithfulness measure is specific to the model's architecture

Natural Language Modality
type: Class
subclass of: Explanation Modality
is defined by: Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024). Local interpretations for explainable natural language processing: A survey. ACM Computing Surveys, 56(9), 1-36.
definition: An explanation modality that is generated text
abbreviation: NLE
direct source: Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024). Local interpretations for explainable natural language processing: A survey. ACM Computing Surveys, 56(9), 1-36.

Perturbation-based Evaluation Method
type: Class
subclass of: Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: An evaluation method that perturbs parts of the input and observes the change in the output
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
explanatory note: This differs from robustness evaluation in that robustness considers extremely similar inputs and expects that the explanation is similar; but in perturbation-based evaluation, we consider inputs that are not necessarily similar, and our expectation of how the explanation should change depends on which parts of the input are perturbed
usage note: For all instances of this class: makes_assumption Linearity_Assumption

Predictive Power Evaluation Method
type: Class
subclass of: Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
alternative label: Simulation Evaluation Method
definition: An evaluation method that uses the explanation to predict model decisions on unseen examples and considers a higher accuracy as an indicator of higher faithfulness
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Robustness Evaluation Method
type: Class
subclass of: Evaluation Method
subclass of: Stability Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: An evaluation method that measures whether the explanation is stable against subtle changes in the input examples
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
usage note: For all instances of this class: makes_assumption Prediction_Assumption

White-box Evaluation Method
type: Class
subclass of: Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: An evaluation method that relies on known ground-truth explanations, against which a candidate explanation can be compared, where the ground-truth explanations come from either transparent tasks or transparent models
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Access the Classifier Method
type: Class
subclass of: Human-based Evaluation Method
subclass of: Predictive Power Evaluation Method
is defined by: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
definition: A human-based evaluation method designed to assess whether the interpretation provides sufficient information to understand the classifier's logic
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Attention Map Explanation Modality
type: NamedIndividual
type: Feature-based Modality
definition: A feature-based modality that uses the attention mechanism(s) in a model
adapted from: Bibal, A., Cardon, R., Alfter, D., Wilkens, R., Wang, X., François, T., & Watrin, P. (2022, May). Is attention explanation? an introduction to the debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 3889-3900).

Attention Weight Access Level
type: NamedIndividual
type: Model Access Level
Higher Access Level: Inference Access Level
is defined by: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
definition: The model's attention weights or attention mechanism must be available to the faithfulness measure
direct source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.

Completeness Proxy
type: NamedIndividual
type: Faithfulness Proxy
is defined by: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.
definition: A faithfulness proxy that states that the explanation should describe the entire dynamic of the model
direct source: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.

Completeness to Model Evaluation Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: White-box Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Surrogate-Model Method
Makes Assumption: Model Corollary 1
is defined by: Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018, October). Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE.
definition: A faithfulness proxy that states that a surrogate model should closely approximate the original model it explains
direct source: Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018, October). Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE.

Comprehensive Evaluation Method
type: Class
subclass of: Computational-based Evaluation Method
is defined by: Craven, M. W. (1996). Extracting comprehensible models from trained neural networks. The University of Wisconsin-Madison.
definition: A computational-based evaluation method that assess how easily we can inspect and understand a model
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Continuity Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Robustness Evaluation Method
is defined by: Nejadgholi, I., Omidyeganeh, M., Drouin, M. A., & Boisvert, J. (2025). A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations. arXiv preprint arXiv:2507.10585.
definition: A faithfulness proxy that states that a small input change should result in a small explanation change
direct source: Nejadgholi, I., Omidyeganeh, M., Drouin, M. A., & Boisvert, J. (2025). A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations. arXiv preprint arXiv:2507.10585.

Contrastivity Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Robustness Evaluation Method
is defined by: Nejadgholi, I., Omidyeganeh, M., Drouin, M. A., & Boisvert, J. (2025). A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations. arXiv preprint arXiv:2507.10585.
definition: A faithfulness proxy that an explanation should highlight differences from alternative outcomes
direct source: Nejadgholi, I., Omidyeganeh, M., Drouin, M. A., & Boisvert, J. (2025). A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations. arXiv preprint arXiv:2507.10585.

Correctness Evaluation Method
type: Class
subclass of: Computational-based Evaluation Method
is defined by: Thomas, F., & David, V. (2022). Representativity and consistency measures for deep neural network explanations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).
alternative label: Fidelity Evaluation
definition: A computational-based evaluation method that assess the ability of the explanations to reflect the behavior of the prediction model
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Chain of Thought Explanation Modality
type: NamedIndividual
type: Natural Language Modality
is defined by: Agarwal, C., Tanneru, S. H., & Lakkaraju, H. (2024). Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614.
definition: A natural language modality that consists of a sequence of intermediate thoughts or steps that lead to the final decision or response of an LLM
acronym: CoT
direct source: Agarwal, C., Tanneru, S. H., & Lakkaraju, H. (2024). Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614.

Embedding Access Level
type: NamedIndividual
type: Model Access Level
Higher Access Level: Inference Access Level
is defined by: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
definition: The model's embedding space must be available to the faithfulness measure
direct source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.

Erasure Method
type: NamedIndividual
type: Computational-based Evaluation Method
type: Perturbation-based Evaluation Method
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: A compuational-based evaluation method where parts of the input are erased in expectation that the model's decision will or will not change, based on the explanation
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Evaluated by Method
type: ObjectProperty
inverse of: Evaluates Proxy
definition: Specifies an evaluation method that evaluates the faithfulness proxy

Evaluates Faithfulness of Explanations
type: ObjectProperty
domain: Faithfulness Measure
range: Collection of Explanations
definition: Specifies the collection of explanations that a faithfulness measure uses to determine faithfulness

Evaluates Modality
type: ObjectProperty
domain: Faithfulness Measure
range: Explanation Modality
property chain axiom: Evaluates Faithfulness of Explanations
property chain axiom: comprises
property chain axiom: has presentation
definition: Specifies the modality of the explanations that a faithfulness measure can determine the faithfulness of

Evaluates Proxy
type: ObjectProperty
domain: Evaluation Method
range: Faithfulness Proxy
definition: Specifies a faithfulness proxy evaluated by the evaluation method

Faithfulness Document
type: Class
subclass of: Entity
subclass of: Document
definition: A document that introduces a faithfulness measure

Feature Importance Agreement Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Axiomatic Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Computational-based Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that states that input tokens that are important (resp. unimportant) for label prediction should also be important (resp. unimportant) for explanation generation
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Find-Alignment Method
type: Class
subclass of: Human-based Evaluation Method
is defined by: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
definition: A human-based evaluation method designed to evaluate how close the explanation is to human reasoning
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Full Access Level
type: NamedIndividual
type: Model Access Level
Higher Access Level: Attention Weight Access Level
Higher Access Level: Embedding Access Level
Higher Access Level: Gradient Access Level
is defined by: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
definition: The entire model and its internals must be available to the faithfulness measure
direct source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
usage note: This level of access should only be used for faithfulness measures that evaluate self-explainable systems

Global Scope
type: NamedIndividual
type: Granularity
definition: The granularity level of specifying the faithfulness of an entire model or dataset
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
synonym: Model-Understanding Scope

Gradient Access Level
type: NamedIndividual
type: Model Access Level
Higher Access Level: Inference Access Level
is defined by: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.
definition: The model's gradient values must be available to the faithfulness measure
direct source: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.

Gradient-based Explanation Modality
type: NamedIndividual
type: Feature-based Modality
definition: A feature-based modality that uses a backwards pass through the model, using the computed gradient values, to determine the importance of each input token
adapted from: Bastings, J., & Filippova, K. (2020). The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. arXiv preprint arXiv:2010.05607.

Has Granularity
type: ObjectProperty
domain: Faithfulness Measure
range: Granularity
definition: Specifies the level of granularity at which a faithfulness measure operates

Has Model Specificity
type: ObjectProperty
domain: Faithfulness Measure
range: Model Specificty
definition: Specifies how specific the model must be to apply the faithfulness measure

Higher Access Level
type: ObjectProperty
type: TransitiveProperty
inverse of: Lower Access Level
definition: Specifices that this model access level requires more detailed access than another model access level
explanatory note: Partial ordering

Human-Annotation Method
type: NamedIndividual
type: Human-based Evaluation Method
type: White-box Evaluation Method
is defined by: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).
definition: A human-based evaluation method which verifies the validity of the attribution scores by comparing them with the human problem-solving process
direct source: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).

Inference Access Level
type: NamedIndividual
type: Model Access Level
Higher Access Level: No Access Level
definition: The model must be available to run inferences for the faithfulness measure
example: API inference-only access to the model
adapted from: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.

Inferred from Assumption
type: ObjectProperty
type: TransitiveProperty
domain: Corollary
range: Assumption
definition: Specifies the assumption that the corollary naturally follows from
adapted from: Merriam-Webster: Corollary, Definition 1 - https://www.merriam-webster.com/dictionary/corollary

Input Sensitivity Proxy
type: NamedIndividual
type: Faithfulness Proxy
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that states that the explanations should be sensitivie (resp. insensitive) to changes in the input that influence (resp. do not influence) the prediction
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Linearity Assumption
type: NamedIndividual
type: Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The assumption that certain parts of the input are more important to the model reasoning than others; the contributions of different parts of the input are independent from each other
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Linearity Corollary
type: NamedIndividual
type: Corollary
Inferred from Assumption: Linearity Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The corollary to the Linearity Assumption that under certain circumstances, heatmap interpretations can be faithful
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Local Scope
type: NamedIndividual
type: Granularity
definition: The granularity level of specifying the faithfulness of a single explanation
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
synonym: Explanation-Understanding Scope

Logic Trap 1
type: NamedIndividual
type: Assumption
is defined by: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).
definition: The assumption that the decision-making process of neural networks is equal to the decision-making process of humans.
direct source: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).

Lower Access Level
type: ObjectProperty
type: TransitiveProperty
domain: Model Access Level
range: Model Access Level
definition: Specifices that this model access level requires less detailed access than another model access level
explanatory note: Partial ordering

LSTM Specific
type: NamedIndividual
type: Model Specific
definition: The model specificity of targeting LSTM-based models

Makes Assumption
type: ObjectProperty
range: Assumption
definition: Specifices the assumption that is relied upon

Meaningful Perturbation Method
type: NamedIndividual
type: Computational-based Evaluation Method
type: Perturbation-based Evaluation Method
is defined by: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).
definition: A computational-based evaluation method which makes modifications to the input instances, in accordance with the generated attribution, and expects significant differences to model predictions
direct source: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).

Measures Proxy
type: ObjectProperty
domain: Faithfulness Measure
range: Faithfulness Proxy
property chain axiom: Uses Method
property chain axiom: Evaluates Proxy
definition: Specifices the faithfulness proxy that the faithfulness measure evaluates in place of faithfulness

Minimality Proxy
type: NamedIndividual
type: Faithfulness Proxy
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that states that an explanation should include only the smallest number of necessary factors
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Model Agnostic
type: NamedIndividual
type: Model Specificty
is defined by: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
definition: The model specificity of being able to be applied to any black-box model regardless of its internal components
direct source: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Model Assumption
type: NamedIndividual
type: Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The assumption that two models will make the same predictions if and only if they use the same reasoning process
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Model Corollary 1
type: NamedIndividual
type: Corollary
Inferred from Assumption: Model Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The corollary to the Model Assumption that an interpretation system is unfaithful if it results in different interpretations of models that make the same decisions
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Model Corollary 2
type: NamedIndividual
type: Corollary
Inferred from Assumption: Model Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The corollary to the Model Assumption that an interpretation is unfaithful if it results in different decisions than the model it interprets
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Model Sensitivity Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Perturbation-based Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Stability Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that states that the explanations should be sensitivie (resp. insensitive) to changes in the model that influence (resp. do not influence) the prediction
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

No Access Level
type: NamedIndividual
type: Model Access Level
definition: No model access is required for the faithfulness measure
adapted from: Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., & Eger, S. (2024). Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75), 1-49.

Occlusion-based Explanation Modality
type: NamedIndividual
type: Feature-based Modality
is defined by: Bastings, J., & Filippova, K. (2020). The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. arXiv preprint arXiv:2010.05607.
definition: A feature-based modality that computes input saliency by erasing input features and measuring how that affects the model
scope note: This includes many different methods of occlusion, including making and removal
direct source: Bastings, J., & Filippova, K. (2020). The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. arXiv preprint arXiv:2010.05607.
synonym: Leave-one-out Explanation Modality

Polarity Consistency Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Axiomatic Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Computational-based Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Perturbation-based Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that uses the consistency between importance-score--based explanations and the impact polarity on model predictions as a faithfulness score
example: If an explanation method assigns a positive weight to a feature as its contribution to some predicted label, then after removing this feature, the model confidence in the label should be suppressed
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Prediction Assumption
type: NamedIndividual
type: Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The assumption that on similar inputs, the model makes similar decisions if and only if its reasoning is similar
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Prediction Corollary
type: NamedIndividual
type: Corollary
Inferred from Assumption: Prediction Assumption
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: The corollary to the Prediction Assumption that an interpretation system is unfaithful if it provides different interpretations for similar inputs and outputs
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Propogation-based Explanation Modality
type: NamedIndividual
type: Feature-based Modality
definition: A feature-based modality that starts with a forward pass to obtain the relevance and then redistributes that relevance among the inputs of each layer
adapted from: Bastings, J., & Filippova, K. (2020). The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. arXiv preprint arXiv:2010.05607.

Requires Access Level
type: ObjectProperty
domain: Faithfulness Measure
range: Model Access Level
definition: Specifies the model access level that the faithfulness measure requires

Robustness Equivalence Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Axiomatic Evaluation Method
type: Restriction
  on property: Evaluated by Method
  some values from: Computational-based Evaluation Method
is defined by: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
definition: A faithfulness proxy that states that the explanation and the predicted label should be equally robust (or non-robust) under noise
direct source: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.

Robustness Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Robustness Evaluation Method
is defined by: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
definition: A faithfulness proxy that states that an explanation should be invariant to small perturbations in the input
direct source: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.

Self-Consistency Proxy
type: NamedIndividual
type: Faithfulness Proxy
is defined by: Parcalabescu, L., & Frank, A. (2023). On measuring faithfulness of natural language explanations. CoRR.
definition: A faithfulness proxy that states that the model should produce explanations that never contradict each other
direct source: Parcalabescu, L., & Frank, A. (2023). On measuring faithfulness of natural language explanations. CoRR.

Sensitivity to Iterative Masking Evaluation Method
type: NamedIndividual
type: Evaluation Method
description: An evaluation method that iteratively removes salient features and measures model responses
is defined by: Crothers, E., Viktor, H., & Japkowicz, N. (2024, September). Robust Infidelity: When Faithfulness Measures on Masked Language Models Are Misleading. In International Conference on Machine Learning, Optimization, and Data Science (pp. 133-147). Cham: Springer Nature Switzerland.
direct source: Crothers, E., Viktor, H., & Japkowicz, N. (2024, September). Robust Infidelity: When Faithfulness Measures on Masked Language Models Are Misleading. In International Conference on Machine Learning, Optimization, and Data Science (pp. 133-147). Cham: Springer Nature Switzerland.

Similar Measure To
type: ObjectProperty
domain: Faithfulness Measure
range: Faithfulness Measure
definition: Specifies a measure that measures the same proxy, uses the same type of evaluation method, evaluates the same modality, has the same granularity level, the same model specificity, and at least one shared assumption and shared access level

Simulatability Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Access the Classifier Method
is defined by: Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., ... & Gurram, P. (2017, August). Interpretability of deep learning models: A survey of results. In 2017 IEEE smartworld, ubiquitous intelligence & computing, advanced & trusted computed, scalable computing & communications, cloud & big data computing, Internet of people and smart city innovation (smartworld/SCALCOM/UIC/ATC/CBDcom/IOP/SCI) (pp. 1-6). IEEE.
definition: A faithfulness proxy that states that a human should be able to use the input data together with the model to reproduce every calculationg step necessary to make the same prediction as the model
direct source: Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., ... & Gurram, P. (2017, August). Interpretability of deep learning models: A survey of results. In 2017 IEEE smartworld, ubiquitous intelligence & computing, advanced & trusted computed, scalable computing & communications, cloud & big data computing, Internet of people and smart city innovation (smartworld/SCALCOM/UIC/ATC/CBDcom/IOP/SCI) (pp. 1-6). IEEE.

Soundness Proxy
type: NamedIndividual
type: Faithfulness Proxy
type: Restriction
  on property: Evaluated by Method
  some values from: Axiomatic Evaluation Method
is defined by: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.
definition: A faithfulness proxy that states that the explanation should be correct and truthful
direct source: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.

Stability Evaluation Method
type: Class
subclass of: Computational-based Evaluation Method
alternative label: Robustness Evaluation Method
definition: A computational-based evaluation method that assess whether slight variations of an instance that did not change the predicted class substantially changed the explanation
adapted from: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.

Surrogate-Model Method
type: Class
subclass of: Computational-based Evaluation Method
is defined by: Arya, V., Bellamy, R. K., Chen, P. Y., Dhurandhar, A., Hind, M., Hoffman, S. C., ... & Zhang, Y. (2019). One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012.
definition: A computational-based evaluation method that build a second, usually directly interpretable model, that approximates a more complex model
direct source: Arya, V., Bellamy, R. K., Chen, P. Y., Dhurandhar, A., Hind, M., Hoffman, S. C., ... & Zhang, Y. (2019). One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012.

Transformer Specific
type: NamedIndividual
type: Model Specific
definition: The model specificity of targeting transformer-based models

Uses Method
type: ObjectProperty
domain: Faithfulness Measure
range: Evaluation Method
definition: Specifies an evaluation method used by the faithfulness measure

Explanation Modality
type: Class
subclass of: SIO_000612
description: a particular form/media in which an explanation exists is expressed
is defined by: Shruthi Chari

has presentation
type: ObjectProperty
sub property of: SIO_000008
description: a property that captures what form of output an entity a is presented in

If a faithfulness method requires access beyond inference, then it's model specific
type: Imp
is rule enabled: true
body: ClassAtom
  argument1: fm
  class predicate: Faithfulness Measure
body: IndividualPropertyAtom
  argument1: fm
  argument2: al
  property predicate: Requires Access Level
body: IndividualPropertyAtom
  argument1: al
  argument2: Inference Access Level
  property predicate: Higher Access Level
body: IndividualPropertyAtom
  argument1: fm
  argument2: ms
  property predicate: Has Model Specificity
head: ClassAtom
  argument1: ms
  class predicate: Model Specific

If a faithfulness measure uses a find alignment method, then it makes the Logic Trap 1 assumption
type: Imp
comment: Alangari, N., El Bachir Menai, M., Mathkour, H., & Almosallam, I. (2023). Exploring evaluation methods for interpretable machine learning: A survey. Information, 14(8), 469.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fem
  argument2: em
  property predicate: Uses Method
body: ClassAtom
  argument1: em
  class predicate: Find-Alignment Method
head: IndividualPropertyAtom
  argument1: fem
  argument2: Logic Trap 1
  property predicate: Makes Assumption

If a faithfulness measure uses a surrogate model method, then it makes the Model Assumption
type: Imp
comment: Arya, V., Bellamy, R. K., Chen, P. Y., Dhurandhar, A., Hind, M., Hoffman, S. C., ... & Zhang, Y. (2019). One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fem
  argument2: em
  property predicate: Uses Method
body: ClassAtom
  argument1: em
  class predicate: Surrogate-Model Method
head: IndividualPropertyAtom
  argument1: fem
  argument2: Model Assumption
  property predicate: Makes Assumption

If a faithfulness measures uses a surrogate model method, it requires inference-level access and assumes Model Corollary 1
type: Imp
comment: Arya, V., Bellamy, R. K., Chen, P. Y., Dhurandhar, A., Hind, M., Hoffman, S. C., ... & Zhang, Y. (2019). One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: em
  property predicate: Uses Method
body: ClassAtom
  argument1: em
  class predicate: Surrogate-Model Method
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level
head: IndividualPropertyAtom
  argument1: fm
  argument2: Model Corollary 1
  property predicate: Makes Assumption

If a faithfulness measure uses the Sensitivity to Iterative Masking method, it must evaluate feature-based explanations, requires inference-level access, and makes the Linearity assumption
type: Imp
comment: Crothers, E., Viktor, H., & Japkowicz, N. (2024, September). Robust Infidelity: When Faithfulness Measures on Masked Language Models Are Misleading. In International Conference on Machine Learning, Optimization, and Data Science (pp. 133-147). Cham: Springer Nature Switzerland.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Sensitivity to Iterative Masking Evaluation Method
  property predicate: Uses Method
body: IndividualPropertyAtom
  argument1: fem
  argument2: em
  property predicate: Evaluates Modality
head: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
head: IndividualPropertyAtom
  argument1: fem
  argument2: Inference Access Level
  property predicate: Requires Access Level
head: IndividualPropertyAtom
  argument1: fem
  argument2: Linearity Assumption
  property predicate: Makes Assumption

If a faithfulness method uses the Completeness to Model Evaluation proxy, then it has a global scope
type: Imp
comment: Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018, October). Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE.
is rule enabled: true
body: ClassAtom
  argument1: fm
  class predicate: Faithfulness Measure
body: IndividualPropertyAtom
  argument1: fm
  argument2: Completeness to Model Evaluation Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Global Scope
  property predicate: Has Granularity

If a faithfulness measure uses the Erasure method, it must evaluates feature-based explanations and makes the Linearity assumption
type: Imp
comment: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Erasure Method
  property predicate: Uses Method
body: IndividualPropertyAtom
  argument1: fem
  argument2: em
  property predicate: Evaluates Modality
head: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
head: IndividualPropertyAtom
  argument1: fem
  argument2: Linearity Assumption
  property predicate: Makes Assumption

If a faithfulness measure uses the Robustness proxy, it requires inference-level access and makes the Prediction Assumption
type: Imp
comment: Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv preprint arXiv:2004.03685.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Robustness Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level
head: IndividualPropertyAtom
  argument1: fm
  argument2: Prediction Assumption
  property predicate: Makes Assumption

If a faithfulness measure uses the Human-Annotation method, it must evaluate feature-based explanations and makes the Logic Trap 1 assumption
type: Imp
comment: Ju, Y., Zhang, Y., Yang, Z., Jiang, Z., Liu, K., & Zhao, J. (2022, May). Logic traps in evaluating attribution scores. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5911-5922).
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Human-Annotation Method
  property predicate: Uses Method
body: IndividualPropertyAtom
  argument1: fem
  argument2: em
  property predicate: Evaluates Modality
head: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
head: IndividualPropertyAtom
  argument1: fem
  argument2: Logic Trap 1
  property predicate: Makes Assumption

If a faithfulness measure uses a perturbation-based evaluation method, then it makes the Linearity Assumption and requires inference-level access
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: pem
  property predicate: Uses Method
body: ClassAtom
  argument1: pem
  class predicate: Perturbation-based Evaluation Method
head: IndividualPropertyAtom
  argument1: fm
  argument2: Linearity Assumption
  property predicate: Makes Assumption
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level

If a faithfulness measure uses a robustnes evaluation method, then it makes the Prediction Assumption and requires inference-level access
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: rem
  property predicate: Uses Method
body: ClassAtom
  argument1: rem
  class predicate: Robustness Evaluation Method
head: IndividualPropertyAtom
  argument1: fm
  argument2: Prediction Assumption
  property predicate: Makes Assumption
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level

If a faithfulness measure uses the Feature Importance Agreement proxy, it makes the Linearity assumption and must evaluate a feature-based explanation modality.
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Feature Importance Agreement Proxy
  property predicate: Measures Proxy
body: IndividualPropertyAtom
  argument1: fm
  argument2: em
  property predicate: Evaluates Modality
head: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
head: IndividualPropertyAtom
  argument1: fm
  argument2: Linearity Assumption
  property predicate: Makes Assumption

If a faithfulness measure uses the Input Sensitivity proxy, then it require inference-level access
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Input Sensitivity Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level

If a faithfulness measure uses the Polarity Consistency proxy, it makes the Linearity Corollary and must evaluate feature-based explanations
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Polarity Consistency Proxy
  property predicate: Measures Proxy
body: IndividualPropertyAtom
  argument1: fm
  argument2: em
  property predicate: Evaluates Modality
head: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
head: IndividualPropertyAtom
  argument1: fm
  argument2: Linearity Corollary
  property predicate: Makes Assumption

If a faithfulness method uses the Model Sensitivity proxy, then it has a global scope and makes the Model Assumption
type: Imp
comment: Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024). Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2), 657-723.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Model Sensitivity Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Global Scope
  property predicate: Has Granularity
head: IndividualPropertyAtom
  argument1: fm
  argument2: Model Assumption
  property predicate: Makes Assumption

If a faithfulness measure uses the Continuity proxy, it requires inference level acess
type: Imp
comment: Nejadgholi, I., Omidyeganeh, M., Drouin, M. A., & Boisvert, J. (2025). A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanations. arXiv preprint arXiv:2507.10585.
is rule enabled: true
body: IndividualPropertyAtom
  argument1: fm
  argument2: Continuity Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Inference Access Level
  property predicate: Requires Access Level

If a faithfulness measure uses the Completeness proxy, then it has a global scope
type: Imp
comment: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.
is rule enabled: true
body: ClassAtom
  argument1: fm
  class predicate: Faithfulness Measure
body: IndividualPropertyAtom
  argument1: fm
  argument2: Completeness Proxy
  property predicate: Measures Proxy
head: IndividualPropertyAtom
  argument1: fm
  argument2: Global Scope
  property predicate: Has Granularity

If a faithfulness measure uses the Soundness proxy with a feature-based explanation modality, it must use a computational method
type: Imp
comment: Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5), 593.
is rule enabled: true
body: ClassAtom
  argument1: fm
  class predicate: Faithfulness Measure
body: IndividualPropertyAtom
  argument1: fm
  argument2: Soundness Proxy
  property predicate: Measures Proxy
body: IndividualPropertyAtom
  argument1: fm
  argument2: em
  property predicate: Evaluates Modality
body: ClassAtom
  argument1: em
  class predicate: Feature-based Modality
body: IndividualPropertyAtom
  argument1: fm
  argument2: http://www.semanticweb.org/villad4/ontologies/m
  property predicate: Uses Method
head: ClassAtom
  argument1: http://www.semanticweb.org/villad4/ontologies/m
  class predicate: Computational-based Evaluation Method