Mayra Russo
Doc-Bias, an ontology to support the documentation of bias in data
Doc-Bias Ontology
GNU General Public License v3.0
TIB – Leibniz Information Centre for Science and Technology, Hannover, Germany
citation
This annotation property is used for exemplifying possible requirements the content pattern provides a solution for. Requirements are expressed as natural language competency questions.
This annotation property should be assigned with a URI, which points to the possible reference ontology which the annotated pattern was extracted from (i.e. the reference ontology that the ontology elements have been deeply or partially cloned by). The range is not explicit in the definition of the annotation property because it would turn the ontology to OWL Full. E.g. The participation pattern is extracted from the Dolce Ultra Lite ontology, hence the value for this annotation property is http://www.ontologydesignpatterns.org/ont/dul/DUL.owl
This annotation property is useful for content ontology design patterns. Its value is a URI, which refers to another content ontology design pattern which is a component of the annotated one.
This annotation property is used for briefly describing the benefits and/or possible trade-offs when using the CP.
This annotation property is used in order to describe the intent of the content pattern.
This property can be used to annotate a unit test (e.g. in the form of a SPARQL query) to be launched to evaluate an ontology against a requirement-based task.
This annotation property is used for referring a cloned ontology entity to its cloning source.
This annotation property is useful for content ontology design patterns and its elements. Its value is a URI, which refers either to a content ontology design pattern that is specialized by the annotated one, or to an ontology element that is specialized by the annotated one.
This annotation property should be assigned with a URI, which points to the concept schema, page, or anything else from which the annotated pattern was reengineered. If the source does not have any URI e.g., a printed book, this property value should provide information as precise as possible in order to identify the source. This property is alternative to the extractedFrom property because it is used when the pattern come from a concept schema which is not an owl ontology. For example content ontology design patterns, which are reengineered from data model patterns, rdf schemas, etc. should be annotatd with this property. E.g. The basicpersonalfoaf pattern is extracted from the rdf FOAF specification, hence the value for this annotation property is http://xmlns.com/foaf/spec/20071002.rdf
This annotation property can be used to indicate other CPs (if any) that specialize, generalize, inlcude, or are components of the CP. Furthermore, this field may indicate other CPs that are typically used in conjunction with the described one. Important similarities and differences with other patterns can be also described here.
This annotation property is used for describing examples of instantiation of the Content OP. For example, for the part-of Content OP (which represents part-whole relations) a possible scenario is the sentence: "the brain is part of the human body". Scenarios are expressed as natural language sentences.
creator
communicates_with_language
association of gender identity concept with a biological gender
has_skin_color
identifies_with_religion
is_of_ethnic_origin
is_of_the_racial_background
self_identifies_gender_as
A main category of the resource. A resource can have multiple themes.
theme
defines measure of metric which is computed on dataset
computedOn
defines annotations related to quality that Dataset has
hasQualityAnnotation
defines Category that given Dimension belongs to
inCategory
inDemension
defines metric which is measured
isMeasurementOf
achieves
definedOn
defines
executes
hasInput
hasPart
hasQuality
implements
realizes
A relation between an entity and the information content entity that specifies it.
specifiedBy
defines dataset for which a quality policy is applicable for
target
attributing provenance of Bias to a scholarly document.
wasAttributedTo
aligns
associatedTo
associates
composedOf
evaluatesFeature
evaluatesForTask
evaluatesInDataset
evaluatesWithMeasure
hasBiasMeasure
hasDataset
hasDocumentationType
hasModelComponent
hasOutput
isAlignedWith
isAssociatedTo
isPartOf
measures
defines files that are used in preparing the Data for training to be used by Model
preparesData
defines date on which Dataset is published
defines date on which Dataset is modified
qualityValue
age
biasClassification
biasMeasureID
Represents the bias score calculated with bias detection method
biasScore
biasThreshold
datasetName
evaluationID
featureSubGroup
hasContentType
hssMeasureValue
hasValue
defines whether dataset is publicly available
modelName
sub-group in feature assessed is considered to be privileged when in comparison with another sub-group.
privFeature
specificationDescription
defines whether dataset was spontaneously collected from a Social Media Platform or computer-generated
spontaneity
sub-group in feature assessed is considered to be underprivileged when in comparison with another sub-group.
underPrivFeature
valueCount
An economic and sociological combined total measure of a person's work experience and of an individual's or family's economic and social position in relation to others, based on household income, earners' education, and occupation are examined, as well as combined income, whereas for an individual's SES only their own attributes are assessed. However, SES is more commonly used to depict an economic difference in society as a whole.
SocioEconomicStatus
"A person having origins in any of the original peoples of North and South America (including Central America), and who maintains tribal affiliation or community attachment." OMB Directive 15
lacking in sexual attraction and interest
"A person having origins in any of the original peoples of the Far East, Southeast Asia, or the Indian subcontinent including, for example, Cambodia, China, India, Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam." OMB Directive 15, 1997
sexual interest in both biological sex, males and females
"A person having origins in any of the black racial groups of Africa. Terms such as "Haitian" or "Negro" can be used in addition to "Black or African American."" OMB Directive 15, 1997
concept expresses the ethnic identification of the ego where it encompasses ethnic origin, how they identify with their ethnicity and their religious idenity affiliation. For example, individuals who have mutliple ethnic origins ("mixed race"), may identify strongly with one than the other, or individuals from multi-faceted backgrounds (e.g. "Afro Cuban") may self identify as Latino, etc.
representation of various facets of self ethnic idenity - strength of association and what they identify
We are utilizing the Office of Management and Budget (OMB) classifcation for ethnicity. While not comprehensive, it is temporary, but we have added some subclasses to these.
Individual association with an ethnic group, independent of their origin. This mirrors Ethnic Origin classification of FOAF+.
Individual association with a gender group, independent of their biological gender. This mirrors Gender classification of FOAF+.
sexual attraction and interest in the opposite biological sex
"A person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish culture or origin, regardless of race. The term, "Spanish origin," can be used in addition to "Hispanic or Latino."" OMB Directive 15, 1997
sexual attraction and interest in the same biological sex
"do not fit the typical definitions for male or female bodies"
concept classification of various spoken languages, not an exhaustive list but limited to the languages presented in YMAP and ADD Health surveys
"A person having origins in any of the original peoples of Hawaii, Guam, Samoa, or other Pacific Islands." OMB Directive 15, 1997
attraction to many/any genders beyond the binary classification
individual racial background based on 1997 Office of Management and Budget (OMB) classification
concept classification of world religions that individuals may adhere to
degree of adherence to religious belief for an individual based on scales used in surveys from YMAP and ADD Health
A concept that is part of the ethnic identification concept that extends to religious identity and how much they adhere to it
a concept referring to the biological gender an individual is sexually attracted to
Skin color classification based on Fitzpatrick scale
"A person having origins in any of the original peoples of Europe, the Middle East, or North Africa." OMB Directive 15
Person
SpatialThing
Collection
Describes the theme of a dataset.
Theme
Concept Scheme
A curated collection of metadata about resources (e.g., datasets and data services in the context of a data catalog).
Catalog
A collection of data, published or curated by a single source, and available for access or download in one or more representations.
Dataset
Category is a group of quality dimensions in which a common type of information is used as quality indicator
Dimension is a criteria relevant for assessing quality. Each dimension is part of a broader category of Data Quality Assessment
Metric is a standard to measure a quality dimension. An observation(instance of dqv:QualityMeasurement) assigns a value in a given unit to a Metric.
QualityMetric
QualityAnnotation is a quality annotations, including ratings, quality certificates or feedback that can be associated to datasets
Quality Certificate is subclass of Quality Annotation. It associates a dataset to a document that certifies the data quality according to a set of quality assessment rules
Quality Measurement is the evaluation of a given dataset a specific quality metric.
QualityPolicy is a policy or agreement that is chiefly governed by data quality concerns
QualityPolicy
An optimization algorithm in machine learning problems minimizes a loss or cost function by systematically choosing input values from within an allowed set and computing the value of the function.
Algorithm
Data is a data item composed of data examples and it may be of a various level of granularity and complexity. With regard to granularity, it can be a whole dataset (for instance, one main table and possibly other tables), or only a single table, or only a feature (e.g., a column of a table), or only an instance (e.g., row of a table), or a single feature-value pair. With regard to complexity, data examples are characterized by their datatype, which may be arbitrarily complex (e.g., instead of a table it can be an arbitrary graph).
Data
Data Characteristic is a distinguishing quality or property that distinguish one data from another. Such characteristics are often statistical ones (e.g., the number of instances or the number of features of a data set). They may be also informationtheoretic measures (e.g., class entropy of a categorical data set) or geometric measures of data complexity (e.g., the highest discriminatory power of any single feature in the data set).
DataCharacteristic
EvaluationMeasure is a measure to assess the performance of the model generated by the process that realizes the task. Examples are predictive accuracy or f-measure.
EvaluationMeasure
EvaluationProcedure is a technique to evaluate machine learning models. Examples are cross-validation and leave-one-out.
EvaluationSpecification is a specification of an evaluation measure and an evaluation procedure to assess the performance of the process that realizes the task.
EvaluationSpecification
The training process of fake news detection model consists of several experiments that are subclasses of the Process and they are part of a Study(research project). Each Experiment is done for a specific goal decided by the study it is part of.
Feature
Implementation
InformationEntity
Model is a generalization of a set of training data able to predict values for unseen instances. It is an output from an execution of a data mining algorithm implementation. Models have a dual nature. They can be treated as data structures and as such represented, stored and manipulated. On the other hand, they act as functions and are executed, taking as input data examples and giving as output the result of applying the function to a data example. Models can also be divided into global or local ones. A global model has global coverage of a data set, i.e., it generalizes the whole data set. A local model, such as a pattern set, is a set of local hypotheses, i.e. each applies to a limited region of the data set.
Model
ModelEvaluation is a setting of a value of the performance measure specified by the evaluation specification. It connects a measure specification with its value.
ModelEvaluation
This class represents the process followed in a classification problem using a particular machine learning model.
Run is an execution of an implementation on a machine (computer). It is limited in time (has a start and end point), can be successful or failed.
Study is a collection of runs that belong together to do some kind of analysis on its results. This analysis can be general or very specific (e.g. a hypothesis test). Can be linked to files, data, that belong to it.
Task is a formal description of a process that needs to be completed (e.g. based on inputs and outputs). A Task is any piece of work that needs to be addressed in the data mining process. In ML Schema, it is defined based on data.
Task
Represents the creator of the dataset.
Creator
Scholarly document that represents the provenance where the bias measure has been defined.
ScholarlyDocument
Age Bias
Aggregation Bias
Arises when false conclusions are drawn about individuals from observing the entire population.
Algorithmic Bias
Algorithmic bias is when the bias is not present in the input data and is added purely by the algorithm
Alienation
Denial of the relevance of socially meaningful categories.
Application
The use, purpose or appliation of artificial intelligence technology.
Pitoura, E. 2020. Social-minded Measures of Data Quality: Fairness, Diversity, and Lack of Bias. J. Data and Information Quality 12, 3, Article 12 (July 2020), 8 pages.
Bias
A concentration on, or interest in one particular area, or subject.
Bias is an inclination, or prejudice for, or against one person, or group, especially in a way considered to be unfair.
BiasDocumentation
This class represents the stage in the documentation pipeline that pertains to the capture of bias knowledge.
This class represents the Evaluation property.
BiasEvaluation
DataCleaning
DataIntegration
DataTransformation
DataReduction
DatasetDocumentation
This class represents the stage in the documentation pipeline that pertains to the capture of knowledge and metadata pertaining to the dataset.
Dehumanization
The viewing and the treatment of other people as though they lack the mental capacities that are commonly attributed to human beings.
An identity axis (e.g., gender, race, age) for which specific instances (e.g., for gender: male, female, non-binary, etc) are evaluated. Instances of a demographic dimension are typically comparatively evaluated in measures through some proxy, e.g., occupations or identity terms.
On Measures of Biases and Harms in NLP,https://aclanthology.org/2022.findings-aacl.24, Dev et al., Findings 2022
DemographicDimension
Demography Bias
Engaging in discriminatory behavior, holding negative attitudes toward, or otherwise having less favorable reactions toward people based on a demographic factor.
Denigration
A harm that arises when a system applies a label that has a long history of being purposefully used to denigrate and demean people.
Direct Discrimination
Discrimination
Discrimination can be considered as a source for unfairness that is due to human prejudice and stereotyping based on the sensitive attributes, which may happen intentionally or unintentionally.
Disparagement
Disparagement encapsulates any behaviour by a model which reinforces the notion that certain groups are less valuable than others and less deserving of respect or resources.
This class represents the stage in the documentation pipeline.
DocumentationType
Erasure
Erasure or social invisibility refers to a group of people in society that can be excluded or systematically ignored from resource allocation procedures or decision-making processes due to data collection practices or historical record keeping.
Erasure refers to the lack of adequate representation of members of a particular social group, intentional or not.
Ethnicity Bias
When models are partial towards certain ethnicity.
Explainable Discrimination
Extrinsic Bias
Gender bias is the tendency to give preferential treatment to one gender over another.
Identification as male/masculine, female/feminine or something else, and association with a (social) role or set of behavioral and cultural traits, clothing, etc; a category to which a person belongs on this basis. Gender is the result of a complex combination of gender role, gender expression, gender identity, and gender modality. http://purl.obolibrary.org/obo/NCIT_C17357
Gender Bias Measure
Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023. Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. In AAAI/ACM Conference on AI, Ethics, and Society (AIES ’23). August 8–10, 2023, Montréal,QC, Canada.
Harm
Adverse lived experiences resulting from an AI system’s deployment and operation in the world.
HatespeechDetection
Predominantly, studied in the context of social network structure and the output of ranking algorithms.
Homophily
The tendency to connect to similar users or people as oneself.
Indirect Discrimination
Inductive Bias
Intrinsic Bias
Learning Bias
This class represents the stage in the documentation pipeline that pertains to the capture of knowledge and metadata pertaining to the learning and training process.
LearningDocumentation
Linking Bias Measure
Longitudinal Data Fallacy
Researchers analyzing temporal data must use longitudinal analysis to track cohorts over time to learn their behavior. Instead, temporal data is often modeled using cross-sectional analysis, which combines diverse cohorts at a single time point. The heterogeneous cohorts can bias cross-sectional analysis, leading to different conclusions than longitudinal analysis.
Automatically translate spoken language and textual content.
MachineTranslation
MeasuresBiasIn
Modifiable Areal Unit Problem
A statistical bias in geospatial analysis, which arises when modeling data at different levels of spatial aggregation. This bias results in different trends learned when data is aggregated at different spatial scales.
NamedEntityRecognitionSystem
Nationality Bias
Engaging in discriminatory behavior, holding negative attitudes toward, or otherwise having less favorable reactions toward people based on their nationality.
Nationality Bias Measure
Omitted Variable Bias
Occurs when one or more important variables are left out of the model.
Popularity Bias Measure
Pre-processing class defines the method by which Pre-procesing is done. Typically it stores the files used for data pre-processing. There are many ways to do pre-processing like data reduction, data cleaning, data integration and data transformation which are the subclasses of Pre-processing class.
DataPre-processing
Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Proceedings of the International Conference on Advances in Neural Information Processing Systems.
Proxy Discrimination
A variable V in a causal graph exhibits potential proxy discrimination, if there exists a directed path from A to V that is blocked by a proxy variable and V itself is not a proxy.
PublicParticipation
When athe implementation of a systems results in diminishing the ability of a particular population to participate in public discourse.
QualityOfService
Quality of service harms result from instances when a model fails to perform equitably for different groups.
Race Bias
Racism
When models are biased against certain races.
Ranking Bias
The idea that top-ranked results are the most relevant and important will result in attraction of more clicks than others.
Recommender system's are widely used in e-commerce platforms, i.e., Amazon, Netflix, to generate product or content recommendations.
RecommenderSystem
A recommendation system is developed with the objective of predicting the preference of a system's user for a particular item.
Reporting Bias
Arises from how we choose, utilize, and measure particular features
Representation
Those harms which arise when a system represents some social groups in a less favorable light than others, demeans them, or fails to recognize their existence altogether.
Representation Bias Measure
Self-SelectionBias
When the sampling method only reaches a portion of the population or the population of interest has changed or is distinct from the population used during model training.
Sex bias
Sexism
Prejudice or discrimination based on one's sex or gender.
Sex is, according to medical databases, classification of studied subjects as male or female according to their reproductive organs and functions assigned by sets of chromosomes. Sex is related to reproductive organs anatomy and physiology. http://purl.obolibrary.org/obo/NCIT_C28421
Simpsons Paradox Measure
SocialInvisibility
NLP tasks break down human text and voice data to support the ability of computers to make sense of what it is ingesting.
SocialNLPLModel
Socio-Demographic Bias Measure
Statistical Discrimination
Stereotyping
Stereotypes are overgeneralized beliefs about the personal attributes of an individual as determined by their demographic group membership.
Stigmatization
When systems e.g., NLP systems, treat some language varieties or practices as less grammatical, more offensive, or less appropriate for public consumption in comparison to others. This reproduces dominant language ideolegies.
Systemic Discrimination
UnderlyingDistributionSkew
The underlying distribution that data is collected from may lack an equal ratio or sufficient representation for all of its subpopulations. In such cases, the underlying distribution is inherently skewed, and there are no discriminatory motives behind it.
Unexplainable Discrimination
Unresolved Discrimination
A variable V in a causal graph exhibits unresolved discrimination if there exists a directed path from A to V that is not blocked by a resolving variable, and V itself is non-resolving
A type of selection bias that occurs when systems/platforms get their training data from their most active users, rather than those less active (or inactive).
Arises when the distribution over prediction outputs is skewed in comparison to the prior distribution of the prediction target.
A cognitive bias, the influence of a particular reference point or anchor on people’s decisions. Often more fully referred to as anchoring-and-adjustment, or anchoring-and-adjusting: after an anchor is set, people adjust insufficiently from that anchor point to arrive at a final answer. Decision makers are biased towards an initially presented value.
When users rely on automation as a heuristic replacement for their own information seeking and processing. A form of individual bias but often discussed as a group bias, or the larger effects on natural language processing models.
When humans over-rely on automated systems or have their skills attenuated by such over-reliance (e.g., spelling and autocorrect or spellcheckers).
A mental shortcut whereby people tend to overweight what comes easily or quickly to mind, meaning that what is easier to recall—e.g., more “available”—receives greater emphasis in judgement and decision-making.
Systematic distortions in user behavior across platforms or contexts, or across users represented in different datasets.
A broad term referring generally to a systematic pattern of deviation from rational judgement and decision-making. A large variety of cognitive biases have been identified over many decades of research in judgement and decision-making, some of which are adaptive mental shortcuts known as heuristics.
A systematic tendency which causes differences between results and facts. The bias exists in numbers of the process of data analysis, including the source of the data, the estimator chosen, and the ways the data was analyzed.
Use of a system outside the planned domain of application, and a common cause of performance gaps between laboratory settings and the real world.
A cognitive bias where people tend to prefer information that aligns with, or confirms, their existing beliefs. People can exhibit confirmation bias in the search for, interpretation of, and recall of information. In the famous Wason selection task experiments, participants repeatedly showed a preference for confirmation over falsification. They were tasked with identifying an underlying rule that applied to number triples they were shown, and they overwhelmingly tested triples that confirmed rather than falsified their hypothesized rule.
Arises when an algorithm or platform provides users with a new venue within which to express their biases, and may occur from either side, or party, in a digital interaction..
Arises from structural, lexical, semantic, and syntactic differences in the contents generated by users.
A statistical bias in which testing huge numbers of hypotheses of a dataset may appear to yield statistical significance even when the results are statistically nonsignificant.
Arises from the addition of synthetic or redundant data samples to a dataset.
Arises when systems are used as decision aids for humans, since the human intermediary may act on predictions in ways that are typically not modeled in the system. However, it is still individuals using the deployed system.
Systematic differences between groups in how outcomes are determined and may cause an over- or underestimation of the size of the effect.
The tendency of people with low ability in a given area or task to overestimate their self-assessed ability. Typically measured by comparing self-assessment with objective performance, often called subjective ability and objective ability, respectively.
Occurs when an inference is made about an individual based on their membership within a group.
Emergent bias is the result of the use and reliance on algorithms across new or unanticipated contexts.
The effect of variables' uncertainties (or errors, more specifically random errors) on the uncertainty of a function based on them.
Arises when the testing or external benchmark populations do not equally represent the various parts of the user population or from the use of performance metrics that are not appropriate for the way in which the model will be used.
When specific groups of user populations are excluded from testing and subsequent analyses.
Effects that may occur when an algorithm learns from user behavior and feeds that behavior back into the model.
Arises when biased results are reported in order to support or satisfy the funding agency or financial supporter of the research study, but it can also be the individual researcher.
A pattern of favoring members of one's in-group over out-group members. This can be expressed in evaluation of others, in allocation of resources, and in many other ways.
A psychological phenomenon that occurs when people in a group tend to make non-optimal decisions based on their desire to conform to the group, or fear of dissenting with the group. In groupthink, individuals often refrain from expressing their personal disagreement with the group, hesitating to voice opinions that do not align with the group.
Referring to the long-standing biases encoded in society over time. Related to, but distinct from, biases in historical description, or the interpretation, analysis, and explanation of history. A common example of historical bias is the tendency to view the larger world from a Western or European view.
A bias wherein individuals perceive benign or ambiguous behaviors as hostile.
Systematic errors in human thought based on a limited number of heuristic principles and predicting values to simpler judgmental operations.
When users rely on automation as a heuristic replacement for their own information seeking and processing.
An unconscious belief, attitude, feeling, association, or stereotype that can affect the way in which humans process information, make decisions, and take actions.
Individual bias is a persistent point of view or limited list of such points of view that one applies ("parent", "academic", "professional", or etc.).
Arises when applications that are built with machine Learning are used to generate inputs for other machine Learning algorithms. If the output is biased in any way, this bias may be inherited by systems using the output as input to learn other models.
In contrast to biases exhibited at the level of individual persons, institutional bias refers to a tendency exhibited at the level of entire institutions, where practices or norms result in the favoring or disadvantaging of certain social groups. Common examples include institutional racism and institutional sexism.
A form of information processing bias that can occur when users interpret algorithmic outputs according to their internalized biases and views.
Arises when network attributes obtained from user connections, activities, or interactions differ and misrepresent the true behavior of the users.
When automation leads to humans being unaware of their situation such that, when control of a system is given back to them in a situation where humans and machines cooperate, they are unprepared to assume their duties. This can be a loss of awareness over what automation is and isn’t taking care of.
Arises when features and labels are proxies for desired quantities, potentially leaving out important factors or introducing group or input-dependent noise that leads to differential performance.
When modal interfaces confuse human operators, who misunderstand which mode the system is using, taking actions which are correct for a different mode but incorrect for their current situation. This is the cause of many deadly accidents, but also a source of confusion in everyday life.
The bias introduced while using the data to select a single seemingly “best” model from a large set of models employing many predictor variables. Model selection bias also occurs when an explanatory variable has a weak relationship with the response variable.
A form of selection bias that occurs when items that are more popular are more exposed and less popular items are under-represented.
When collaborative filtering recommenders typically emphasize popular items (those with more ratings) over other “long-tail”, less popular ones that may only be popular among small groups of users.
A form of selection bias that occurs when items that are more popular are more exposed and less popular items are under-represented. Systematic distortions in demographics or other user characteristics between a population of users represented in a dataset or on a platform and some target population.
Population bias arises when statistics, demographics, representatives, and user characteristics are different in the user population of the platform from the original target population
Biases arising from how information is presented on the Web, via a user interface, due to rating or ranking of output, or through users’ own self-selected, biased interaction.
Judgement modulated by affect, which is influenced by the level of efficacy and efficiency in information processing; in cognitive sciences, processing bias is often referred to as an aesthetic judgement.
The idea that top-ranked results are the most relevant and important and will result in more clicks than other results.
Refers to differences in perspective, memory and recall, interpretation, and reporting on the same event from multiple persons or witnesses.
Arises due to non-random sampling of subgroups, causing trends estimated for one population to not be generalizable to data collected from a new population.
Bias introduced by the selection of individuals, groups, or data for analysis in such a way that proper randomization is not achieved, thereby failing to ensure that the sample obtained is representative of the population intended to be analyzed.
When the sampling method only reaches a portion of the population or the population of interest has changed or is distinct from the population used during model training. https://doi.org/10.1145/3588433
Decision-makers’ inclination to selectively adopt algorithmic advice when it matches their pre-existing beliefs and stereotypes.
Can be positive or negative, and take a number of different forms, but is typically characterized as being for or against groups or individuals based on social identities, demographic factors, or immutable physical characteristics. Societal or social biases are often stereotypes. Common examples of societal or social biases are based on concepts like race, ethnicity, gender, sexual orientation, socioeconomic status, education, and more. Societal bias is often recognized and discussed in the context of NLP (Natural Language Processing) models.
A bias whereby people tend to search only where it is easiest to look.
A human tendency where people opt to continue with an endeavor or behavior due to previously spent or invested resources, such as money, time, and effort, regardless of whether costs outweigh benefits. For example, in AI, the sunk cost fallacy could lead development teams and organizations to feel that because they have already invested so much time and money into a particular AI application, they must pursue it to market rather than deciding to end the effort, even in the face of significant technical debt and/or ethical debt.
Tendency for people to focus on the items, observations, or people that “survive” or make it past a selection process, while overlooking those that did not.
Biases that result from procedures and practices of particular institutions that operate in ways which result in certain social groups being advantaged or favored and others being disadvantaged or devalued.
Bias that arises from differences in populations and behaviors over time.
Arises when predictive algorithms favor groups that are better represented in the training data, since there will be less uncertainty associated with those predictions.
An information-processing bias, the tendency to inappropriately analyze ambiguous stimuli, scenarios and events.
Arises when a user imposes their own self-selected biases and behavior during interaction with data, output, results, etc.
A statistical phenomenon where the marginal association between two categorical variables is qualitatively different from the partial association between the same two variables after controlling for one or more other variables. For example, the statistical association or correlation that has been detected between two variables for an entire population disappears or reverses when the population is divided into subgroups.
Model Components are the components of predictive model in use. Different types of predictive model have different components. For example, a Bayesian Network Model will have components like Bayesian Network Nodes, a Rule Set Model will have Rule Set as components and Neural Network Model will have Neural Layer, Neural Inputs/Outputs as modelling components.
ModelComponent
Output is the output of classification model.
They define p2c to be: sum_of s all values S, dec (s) = theta w(s) / sum_of s all values S w(s) RDc not close to zero means that there is a bias in decision value due to group membership (causal discrimination) or to covariates that have not been accounted for in the analysis (omitted variable bias).
Causal Risk Difference
RDc = p1 − p2c
Data Coverage Measure
To have enough similar entries (sub-categories) for a particular variable (categoy) in a dataset.
cov(q,D) = |G_D(q)|, threshold value k, qu is covered iff cov (q, D) > k.
A preprocessing method for the discovery of the problematic documents in the training corpus that have biases in them and tried to debias the system by perturbing or removing these documents efficiently from the training corpus.
Differential Bias Embeddings
Given a word embedding, this method reveals how perturbing the training corpus would affect the resulting embedding bias.
∆pB = B(w) − B( ˜w)
Direct Bias Embeddings
A method to debias word embeddings by proposing a method that respects the embeddings for gender-specific words but debiases embeddings for gender-neutral words.
1 / |N | sum_of w2N |cos( ~>w, g)|c
ELMo Bias Embeddings
A new intrinsic analysis of how ELMo represents gender in biased ways.
Error Type-1 Unweighted
This is a type of error that measures names that are tagged as non-PERSON, or not tagged at all. In other words, any name not tagged as a PERSON is considered to be an error
∑ n∈Nf I(ntype ≠ PERSON ) / |Nf |
Where f reqf (·) returns the frequency of a name in the female census data in a particular year. Similarly, f reqm(·) will yield the frequency of a name in the male census data. Type-1 errors can be sub-divided into Type-2 and Type-3 errors and serve as a super-set for the following types.
Error Type-1 Weighted
This type of error is similar to Error Type-1 Unweighted; however, we considered how frequent the mistaken name is based on census data while calculating the error.
∑ n∈Nf f reqf (ntype ≠ PERSON ) / ∑ n∈Nf f reqf (n)
where ∅ indicates the name is not tagged.
Error Type-2 Unweighted
This is a type of error in which only names that are tagged, but whose tags are non-PERSON are considered to be errors. This measure is similar to precision, where the model is only punished for incorrect retrieval. This error rate reports the percentage of names that are tagged as non-PERSON entities among all the names in a certain year.
∑ n∈Nf I(ntype /∈ {∅, PERSON }) , |Nf |
Error Type-2 Weighted
This type of error is similar to Error Type-2 Unweighted; however, here we considered how frequent the mistaken name was while calculating the error.
∑ n∈Nf f reqf (ntype /∈ {∅, PERSON }), ∑ n∈Nf f reqf (n)
Error Type-3 Unweighted
This is a type of error in which only names that are not tagged are considered to be errors. We do not consider names that are wrongfully tagged to non-PERSON as an error, but only names that are not tagged are considered erroneous. This error rate reports the percentage of names that are not tagged at all among all the names in a certain year.
∑ n∈Nf I(ntype = ∅), |Nf |
Error Type-3 Weighted
This type of error is similar to Error Type-3 Unweighted; however, we considered how frequent the mistaken name was while calculating the error.
∑ n∈Nf f reqf (ntype = ∅) , ∑n∈Nf f reqf (n)
A metric for measuring gender bias in a generated text from a language model based on recurrent neural networks that is trained on a text corpus along with measuring the bias in the training text itself.
w is any word in the corpus, f is a set of gendered words ("female"), m is a sit of gendered words ("male"). The bias measure uses the mean absolute and the standard deviation.
Gender Bias Score
bias(w) = log ( P(w|f) / P(w|m) )
Gender Bias vSRL
The goal is to identify the correlations that are potentially amplified by a learned model. To achieve this, a bias score is defined for a given output, o, with respect to a demographic variable, g; where c(o, g) is the number of occurrences of o and g in a corpus.
To evaluate the degree of bias amplification, we propose to compare bias scores on the training set, b∗(o, g), with bias scores on an unlabeled evaluation set of images ˜b(o, g) that has been annotated by a predictor. We assume that the evaluation set is identically distributed to the training set. Therefore, if o is positively correlated with g (i.e,b∗(o, g) > 1/‖G‖) and ˜b(o, g) is larger than b∗(o, g), we say bias has been amplified.
b(o, g) = c(o, g) / ∑ g′∈G c(o, g′) '
The criteria followed to determine if there is presence of bias is the higher the Gini coefficient, the more skewed or unequal the in-degree distribution across all nodes.
Gini coefficient of the in-degree distribution
The Gini coefficient allows to demonstrate whether popularity bias is exacerbated by recommendation algorithms regardless of the initial conditions of the network structure, or whether certain types of networks are exempt from this bias. Gini coefficient of the in-degree distribution
Gini(π^in ) = {sum_of_all (2_i − n − 1)π_i^in} / {sum_of_all π_i^in}
In-group links
Measures the fraction of links within groups to determine what type of edges are being recommended more often by the recommendation algorithms.
I_a = {e_aa} / {e_aa + e_ab}
Consider the problem of binary classification in which we are presented with a set of paired training data samples Dt r ain = {(x(i), y(i))}n i=1 consisting of features x ∈ Rm and labels y ∈ Rd. Our goal is to find a functional mapping f : X → Y parameterized by θ which minimizes a certain loss L(θ ) over our entire training dataset.
Learned Latent Structure
A tunable algorithm for mitigating the hidden, and potentially unknown, biases within training data. The algorithm fuses the original learning task with a variational autoencoder to learn the latent structure within the dataset.
θ∗ = arg min_θ 1/ n sum_of Li (θ)
Low-Degree Node Bias
In the context of community detection, when theses methods ignore lowly connected users that have few links—by putting them into insignificant small or singleton communities which are then ignored in the study.
|ci| denotes the size of the community ci, and θ is athreshold value that can be treated as a hyperparameter and will be used in our proposed algorithm. We call node n an insignificantly labeled node if:∃ cinsig such that n ∈ cinsig where cinsig is an insignificant community.
C_insig = {ci| where |ci| < θ}
Nationality Bias in Text Generation https://aclanthology.org/2023.eacl-main.9 Narayanan Venkit et al., EACL 2023
Nationality Group Bias
Demographic group bias is measured by using statistical inferences of different demographic synonym groups d_n and check for parity across all the groups and a standard control group C, using the story generation feature of the GPT-2 language model.
RepresentationRate
An equal base rate is defined as having an equal number of objects for different subgroups in the dataset.
Given a target tuple t with c = c and e = e , all the tuples are ranked according to their distances to t using a distance function defined on an appropriate subset of nonprotected attributes. Then, the tuples that are within top-2K highest in the ranking for one-to-one pairing are selected. The selected tuples with c+ are added into set S+ , and the selected tuples with c are added into set S . The proportion p 1 of tuples with e+ among the tuples in S+, and the proportion p 2 of tuples with e+ among the tuples in S are compared. The dierence between p 1 and p2 , i.e., p 1 p2 reflects the bias due to the membership to the protected group. If p1 p2 is greater than a threshold tau, then t is considered as being discriminated. The value of threshold tau used for discrimination depends on the law.
Situation Testing Causal Inference
A general technique to capture discrimination based on the legally grounded situation testing methodology.
The equations hold if the expected value of Y is a monotonically increasing (or decreasing) function of Xp alone, but conditioned on Xc is a monotonically decreasing (resp. increasing) function of Xp , or is constant.
Y is the dependent variable in the dataset, i.e., an outcome being measured, and as X = {X1, X2, . . . , Xm } the set of m independent variables or features. The goal of the method is to identify pairs of variables (Xp , Xc )—Simpson’s pairs— such that a trend in Y as a function of Xp disappears or reverses when the data is disaggregated by conditioning on Xc .
Uncovering Simpson paradox for trends
A statistical method to systematically uncover instances of Simpson’s paradox in heterogeneous data.
dE[Y |Xp = xp ]/dxp < 0, dE[Y |Xp = xp , Xc = xc ]/dxp ≥ 0
Visibility of the minority group
Change in visibility by computing prediction{f_m} after and before the recommendations to verify whether the minority group is gaining or losing visibility.
delta function_fm = prediction{f_m (after)} − prediction{f_m (before)}
Sentence Encoder Association Test (SEAT)
The method takes as input a set of observations of an outcome, Y , and a set of covariates, or features, X ={X1, X2, ..., Xm} associated with it. The approach has three steps. First, it disaggregates data into more homogenous subgroups based on some covariate Xc. Next, it uses a linear model to capture trends with respect to some other covariate Xj , both within the subgroups and within the aggregate data. Finally, it quantifies how well the models describe the disaggregated data compared to aggregate data to identify the important disaggregations.
Nazanin Alipourfard, Peter G. Fennell, and Kristina Lerman. 2018. Using Simpson’s paradox to discover interesting patterns in behavioral data. In Proceedings of the 12th International AAAI Conference on Web and Social Media.
Simpson’s Disaggregation
A method for automatically uncovering interesting patterns in data based on Simpson's Paradox.
List
Bias measure corresponds to a quantitative metric or indicator that assesses the presence and extent of bias in a particular context.
BiasMeasure
Bias Measure represents the measure defined in some scholarly document to measure bias detected in data in association to a bias.