{
    "Name": "ARAFA",
    "Dialect Subsets": [],
    "HF Link": "https://huggingface.co/datasets/ChristopheKhalil/ARAFA",
    "Link": "https://github.com/chriskhalil/ARAFA",
    "License": "CC BY-NC-SA 4.0",
    "Year": 2025,
    "Language": "ar",
    "Dialect": "Modern Standard Arabic",
    "Source": [
        "wikipedia"
    ],
    "Domain": [
        "history",
        "politics",
        "sports",
        "society",
        "technology",
        "science",
        "art",
        "culture",
        "economics",
        "religion"
    ],
    "Form": "text",
    "Annotation Style": [
        "LLM annotation",
        "human annotation"
    ],
    "Description": "Large-scale Arabic fact-checking dataset generated by LLMs from Wikipedia.",
    "Volume": 181976.0,
    "Unit": "sentences",
    "Ethical Risks": "Low",
    "Provider": [
        "American University of Beirut"
    ],
    "Derived From": [],
    "Paper Title": "ARAFA: An LLM Generated Arabic Fact-Checking Dataset",
    "Paper Link": "https://doi.org/10.21203/rs.3.rs-7335564/v1",
    "Script": "Arab",
    "Tokenized": false,
    "Host": "GitHub",
    "Access": "Upon-Request",
    "Cost": "0",
    "Has Splits": true,
    "Partial": false,
    "Tasks": [
        "fact checking"
    ],
    "Venue Title": "Research Square",
    "Venue Type": "preprint",
    "Venue Name": "Research Square",
    "Authors": [
        "Christophe Khalil",
        "Shady Elbassuoni",
        "Rida Assaf"
    ],
    "Affiliations": [
        "American University of Beirut"
    ],
    "Abstract": "Automatic fact-checking poses a significant challenge in Arabic natural language processing due to the scarcity of datasets and resources. In this manuscript, we introduce ARAFA, a new large-scale dataset for fact-checking in Modern Standard Arabic, constructed through an automated framework leveraging large language models (LLMs). The dataset was constructed through a three-step pipeline: (1) claim generation from Arabic Wikipedia pages with supporting textual evidence, (2) claim mutation to generate challenging counterfactual claims with refuting evidence, and (3) an automatic validation step to validate that the generated claims are either supported or refuted by their accompanying evidence, or if the evidence does not provide enough information to judge the validity of the claims. The resulting dataset comprises 181,976 claim-evidence pairs labeled as supported, refuted, or not enough information. Human evaluation carried out on a test sample from the dataset demonstrated strong inter-annotator agreement (\u03ba = 0.89) using Cohen\u2019s Kappa for supported claims and (\u03ba = 0.94) for refuted claims. Automatic validation based on human-evaluated sample achieved 86% accuracy for supported claims and 88% for refuted ones. To showcase ARAFA\u2019s value as a resource for automatic Arabic fact-checking, four open-source transformer-based models were fine-tuned using ARAFA, with the top-performing model achieving a Macro F1-score of 77% on the test data. In addition to ARAFA being the first large-scale dataset for Arabic fact-checking, our framework presents a scalable approach for developing similar resources for other low-resource languages.",
    "Added By": "Zaid Alyafeai"
}