--- id: ATLAS name: Adversarial Threat Landscape for AI Systems version: 4.2.0 matrices: - id: ATLAS name: ATLAS Machine Learning Threat Matrix tactics: - id: AML.TA0002 name: Reconnaissance object-type: tactic ATT&CK-reference: id: TA0043 url: https://attack.mitre.org/tactics/TA0043/ description: 'The adversary is trying to gather information about the machine learning system they can use to plan future operations. Reconnaissance consists of techniques that involve adversaries actively or passively gathering information that can be used to support targeting. Such information may include details of the victim organizations machine learning capabilities and research efforts. This information can be leveraged by the adversary to aid in other phases of the adversary lifecycle, such as using gathered information to obtain relevant ML artifacts, targeting ML capabilities used by the victim, tailoring attacks to the particular models used by the victim, or to drive and lead further Reconnaissance efforts. ' - id: AML.TA0003 name: Resource Development object-type: tactic ATT&CK-reference: id: TA0042 url: https://attack.mitre.org/tactics/TA0042/ description: 'The adversary is trying to establish resources they can use to support operations. Resource Development consists of techniques that involve adversaries creating, purchasing, or compromising/stealing resources that can be used to support targeting. Such resources include machine learning artifacts, infrastructure, accounts, or capabilities. These resources can be leveraged by the adversary to aid in other phases of the adversary lifecycle, such as ML Attack Staging. ' - id: AML.TA0004 name: Initial Access object-type: tactic ATT&CK-reference: id: TA0001 url: https://attack.mitre.org/tactics/TA0001/ description: 'The adversary is trying to gain access to the machine learning system. The target system could be a network, mobile device, or an edge device such as a sensor platform. The machine learning capabilities used by the system could be local with onboard or cloud-enabled ML capabilities. Initial Access consists of techniques that use various entry vectors to gain their initial foothold within the system. ' - id: AML.TA0000 name: ML Model Access object-type: tactic description: 'The adversary is attempting to gain some level of access to a machine learning model. ML Model Access enables techniques that use various types of access to the machine learning model that can be used by the adversary to gain information, develop attacks, and as a means to input data to the model. The level of access can range from the full knowledge of the internals of the model to access to the physical environment where data is collected for use in the machine learning model. The adversary may use varying levels of model access during the course of their attack, from staging the attack to impacting the target system. Access to an ML model may require access to the system housing the model, the model may be publically accessible via an API, or it may be accessed indirectly via interaction with a product or service that utilizes ML as part of its processes. ' - id: AML.TA0005 name: Execution object-type: tactic ATT&CK-reference: id: TA0002 url: https://attack.mitre.org/tactics/TA0002/ description: 'The adversary is trying to run malicious code embedded in machine learning artifacts or software. Execution consists of techniques that result in adversary-controlled code running on a local or remote system. Techniques that run malicious code are often paired with techniques from all other tactics to achieve broader goals, like exploring a network or stealing data. For example, an adversary might use a remote access tool to run a PowerShell script that does Remote System Discovery. ' - id: AML.TA0006 name: Persistence object-type: tactic ATT&CK-reference: id: TA0003 url: https://attack.mitre.org/tactics/TA0003/ description: 'The adversary is trying to maintain their foothold via machine learning artifacts or software. Persistence consists of techniques that adversaries use to keep access to systems across restarts, changed credentials, and other interruptions that could cut off their access. Techniques used for persistence often involve leaving behind modified ML artifacts such as poisoned training data or backdoored ML models. ' - id: AML.TA0007 name: Defense Evasion object-type: tactic ATT&CK-reference: id: TA0005 url: https://attack.mitre.org/tactics/TA0005/ description: 'The adversary is trying to avoid being detected by machine learning-enabled security software. Defense Evasion consists of techniques that adversaries use to avoid detection throughout their compromise. Techniques used for defense evasion include evading ML-enabled security software such as malware detectors. ' - id: AML.TA0008 name: Discovery object-type: tactic ATT&CK-reference: id: TA0007 url: https://attack.mitre.org/tactics/TA0007/ description: 'The adversary is trying to figure out your machine learning environment. Discovery consists of techniques an adversary may use to gain knowledge about the system and internal network. These techniques help adversaries observe the environment and orient themselves before deciding how to act. They also allow adversaries to explore what they can control and what''s around their entry point in order to discover how it could benefit their current objective. Native operating system tools are often used toward this post-compromise information-gathering objective. ' - id: AML.TA0009 name: Collection object-type: tactic ATT&CK-reference: id: TA0009 url: https://attack.mitre.org/tactics/TA0009/ description: 'The adversary is trying to gather machine learning artifacts and other related information relevant to their goal. Collection consists of techniques adversaries may use to gather information and the sources information is collected from that are relevant to following through on the adversary''s objectives. Frequently, the next goal after collecting data is to steal (exfiltrate) the ML artifacts, or use the collected information to stage future operations. Common target sources include software repositories, container registries, model repositories, and object stores. ' - id: AML.TA0001 name: ML Attack Staging object-type: tactic description: 'The adversary is leveraging their knowledge of and access to the target system to tailor the attack. ML Attack Staging consists of techniques adversaries use to prepare their attack on the target ML model. Techniques can include training proxy models, poisoning the target model, and crafting adversarial data to feed the target model. Some of these techniques can be performed in an offline manor and are thus difficult to mitigate. These techniques are often used to achieve the adversary''s end goal. ' - id: AML.TA0010 name: Exfiltration object-type: tactic ATT&CK-reference: id: TA0010 url: https://attack.mitre.org/tactics/TA0010/ description: 'The adversary is trying to steal machine learning artifacts or other information about the machine learning system. Exfiltration consists of techniques that adversaries may use to steal data from your network. Data may be stolen for it''s valuable intellectual property, or for use in staging future operations. Techniques for getting data out of a target network typically include transferring it over their command and control channel or an alternate channel and may also include putting size limits on the transmission. ' - id: AML.TA0011 name: Impact object-type: tactic ATT&CK-reference: id: TA0040 url: https://attack.mitre.org/tactics/TA0040/ description: 'The adversary is trying to manipulate, interrupt, erode confidence in, or destroy your machine learning systems and data. Impact consists of techniques that adversaries use to disrupt availability or compromise integrity by manipulating business and operational processes. Techniques used for impact can include destroying or tampering with data. In some cases, business processes can look fine, but may have been altered to benefit the adversaries'' goals. These techniques might be used by adversaries to follow through on their end goal or to provide cover for a confidentiality breach. ' techniques: - id: AML.T0000 name: Search for Victim's Publicly Available Research Materials object-type: technique description: 'Adversaries may search publicly available research to learn how and where machine learning is used within a victim organization. The adversary can use this information to identify targets for attack, or to tailor an existing attack to make it more effective. Organizations often use open source model architectures trained on additional proprietary data in production. Knowledge of this underlying architecture allows the adversary to craft more realistic proxy models ([Create Proxy ML Model](/techniques/AML.T0005)). An adversary can search these resources for publications for authors employed at the victim organization. Research materials may exist as academic papers published in [Journals and Conference Proceedings](/techniques/AML.T0000.000), or stored in [Pre-Print Repositories](/techniques/AML.T0000.001), as well as [Technical Blogs](/techniques/AML.T0000.002). ' tactics: - AML.TA0002 - id: AML.T0000.000 name: Journals and Conference Proceedings object-type: technique description: 'Many of the publications accepted at premier machine learning conferences and journals come from commercial labs. Some journals and conferences are open access, others may require paying for access or a membership. These publications will often describe in detail all aspects of a particular approach for reproducibility. This information can be used by adversaries to implement the paper. ' subtechnique-of: AML.T0000 - id: AML.T0000.001 name: Pre-Print Repositories object-type: technique description: 'Pre-Print repositories, such as arXiv, contain the latest academic research papers that haven''t been peer reviewed. They may contain research notes, or technical reports that aren''t typically published in journals or conference proceedings. Pre-print repositories also serve as a central location to share papers that have been accepted to journals. Searching pre-print repositories provide adversaries with a relatively up-to-date view of what researchers in the victim organization are working on. ' subtechnique-of: AML.T0000 - id: AML.T0000.002 name: Technical Blogs object-type: technique description: 'Research labs at academic institutions and Company R&D divisions often have blogs that highlight their use of machine learning and its application to the organizations unique problems. Individual researchers also frequently document their work in blogposts. An adversary may search for posts made by the target victim organization or its employees. In comparison to [Journals and Conference Proceedings](/techniques/AML.T0000.000) and [Pre-Print Repositories](/techniques/AML.T0000.001) this material will often contain more practical aspects of the machine learning system. This could include underlying technologies and frameworks used, and possibly some information about the API access and use case. This will help the adversary better understand how that organization is using machine learning internally and the details of their approach that could aid in tailoring an attack. ' subtechnique-of: AML.T0000 - id: AML.T0001 name: Search for Publicly Available Adversarial Vulnerability Analysis object-type: technique description: 'Much like the [Search for Victim''s Publicly Available Research Materials](/techniques/AML.T0000), there is often ample research available on the vulnerabilities of common models. Once a target has been identified, an adversary will likely try to identify any pre-existing work that has been done for this class of models. This will include not only reading academic papers that may identify the particulars of a successful attack, but also identifying pre-existing implementations of those attacks. The adversary may [Adversarial ML Attack Implementations](/techniques/AML.T0016.000) or [Develop Adversarial ML Attack Capabilities](/techniques/AML.T0017) their own if necessary.' tactics: - AML.TA0002 - id: AML.T0003 name: Search Victim-Owned Websites object-type: technique description: 'Adversaries may search websites owned by the victim for information that can be used during targeting. Victim-owned websites may contain technical details about their ML-enabled products or services. Victim-owned websites may contain a variety of details, including names of departments/divisions, physical locations, and data about key employees such as names, roles, and contact info. These sites may also have details highlighting business operations and relationships. Adversaries may search victim-owned websites to gather actionable information. This information may help adversaries tailor their attacks (e.g. [Develop Adversarial ML Attack Capabilities](/techniques/AML.T0017) or [Manual Modification](/techniques/AML.T0043.003)). Information from these sources may reveal opportunities for other forms of reconnaissance (e.g. [Search for Victim''s Publicly Available Research Materials](/techniques/AML.T0000) or [Search for Publicly Available Adversarial Vulnerability Analysis](/techniques/AML.T0001)) ' tactics: - AML.TA0002 - id: AML.T0004 name: Search Application Repositories object-type: technique description: 'Adversaries may search open application repositories during targeting. Examples of these include Google Play, the iOS App store, the macOS App Store, and the Microsoft Store. Adversaries may craft search queries seeking applications that contain a ML-enabled components. Frequently, the next step is to [Acquire Public ML Artifacts](/techniques/AML.T0002). ' tactics: - AML.TA0002 - id: AML.T0006 name: Active Scanning object-type: technique ATT&CK-reference: id: T1595 url: https://attack.mitre.org/techniques/T1595/ description: 'An adversary may probe or scan the victim system to gather information for targeting. This is distinct from other reconnaissance techniques that do not involve direct interaction with the victim system. ' tactics: - AML.TA0002 - id: AML.T0002 name: Acquire Public ML Artifacts object-type: technique description: 'Adversaries may search public sources, including cloud storage, public-facing services, and software or data repositories, to identify machine learning artifacts. These machine learning artifacts may include the software stack used to train and deploy models, training and testing data, model configurations and parameters. An adversary will be particularly interested in artifacts hosted by or associated with the victim organization as they may represent what that organization uses in a production environment. Adversaries may identify artifact repositories via other resources associated with the victim organization (e.g. [Search Victim-Owned Websites](/techniques/AML.T0003) or [Search for Victim''s Publicly Available Research Materials](/techniques/AML.T0000)). These ML artifacts often provide adversaries with details of the ML task and approach. ML artifacts can aid in an adversary''s ability to [Create Proxy ML Model](/techniques/AML.T0005). If these artifacts include pieces of the actual model in production, they can be used to directly [Craft Adversarial Data](/techniques/AML.T0043). Acquiring some artifacts requires registration (providing user details such email/name), AWS keys, or written requests, and may require the adversary to [Establish Accounts](/techniques/AML.T0021). Artifacts might be hosted on victim-controlled infrastructure, providing the victim with some information on who has accessed that data. ' tactics: - AML.TA0003 - id: AML.T0002.000 name: Datasets object-type: technique description: 'Adversaries may collect public datasets to use in their operations. Datasets used by the victim organization or datasets that are representative of the data used by the victim organization may be valuable to adversaries. Datasets can be stored in cloud storage, or on victim-owned websites. Some datasets require the adversary to [Establish Accounts](/techniques/AML.T0021) for access. Acquired datasets help the adversary advance their operations, stage attacks, and tailor attacks to the victim organization. ' subtechnique-of: AML.T0002 - id: AML.T0002.001 name: Models object-type: technique description: 'Adversaries may acquire public models to use in their operations. Adversaries may seek models used by the victim organization or models that are representative of those used by the victim organization. Representative models may include model architectures, or pre-trained models which define the architecture as well as model parameters from training on a dataset. The adversary may search public sources for common model architecture configuration file formats such as yaml or python configuration files, and common model storage file formats such as ONNX (.onnx), HDF5 (.h5), Pickle (.pkl), PyTorch (.pth), or TensorFlow (.pb, .tflite). Acquired models are useful in advancing the adversary''s operations and are frequently used to tailor attacks to the victim model. ' subtechnique-of: AML.T0002 - id: AML.T0016 name: Obtain Capabilities object-type: technique ATT&CK-reference: id: T1588 url: https://attack.mitre.org/techniques/T1588/ description: 'Adversaries may search for and obtain software capabilities for use in their operations. Capabilities may be specific to ML-based attacks [Adversarial ML Attack Implementations](/techniques/AML.T0016.000) or generic software tools repurposed for malicious intent ([Software Tools](/techniques/AML.T0016.001)). In both instances, an adversary may modify or customize the capability to aid in targeting a particular ML system.' tactics: - AML.TA0003 - id: AML.T0016.000 name: Adversarial ML Attack Implementations object-type: technique description: Adversaries may search for existing open source implementations of machine learning attacks. The research community often publishes their code for reproducibility and to further future research. Libraries intended for research purposes, such as CleverHans, the Adversarial Robustness Toolbox, and FoolBox, can be weaponized by an adversary. Adversaries may also obtain and use tools that were not originally designed for adversarial ML attacks as part of their attack. subtechnique-of: AML.T0016 - id: AML.T0016.001 name: Software Tools object-type: technique ATT&CK-reference: id: T1588.002 url: https://attack.mitre.org/techniques/T1588/002/ description: 'Adversaries may search for and obtain software tools to support their operations. Software designed for legitimate use may be repurposed by an adversary for malicious intent. An adversary may modify or customize software tools to achieve their purpose. Software tools used to support attacks on ML systems are not necessarily ML-based themselves. ' subtechnique-of: AML.T0016 - id: AML.T0017 name: Develop Adversarial ML Attack Capabilities object-type: technique description: Adversaries may develop their own adversarial attacks. They may leverage existing libraries as a starting point ([Adversarial ML Attack Implementations](/techniques/AML.T0016.000)). They may implement ideas described in public research papers or develop custom made attacks for the victim model. tactics: - AML.TA0003 - id: AML.T0008 name: Acquire Infrastructure object-type: technique description: 'Adversaries may buy, lease, or rent infrastructure for use throughout their operation. A wide variety of infrastructure exists for hosting and orchestrating adversary operations. Infrastructure solutions include physical or cloud servers, domains, mobile devices, and third-party web services. Free resources may also be used, but they are typically limited. Use of these infrastructure solutions allows an adversary to stage, launch, and execute an operation. Solutions may help adversary operations blend in with traffic that is seen as normal, such as contact to third-party web services. Depending on the implementation, adversaries may use infrastructure that makes it difficult to physically tie back to them as well as utilize infrastructure that can be rapidly provisioned, modified, and shut down. ' tactics: - AML.TA0003 - id: AML.T0008.000 name: ML Development Workspaces object-type: technique description: 'Developing and staging machine learning attacks often requires expensive compute resources. Adversaries may need access to one or many GPUs in order to develop an attack. They may try to anonymously use free resources such as Google Colaboratory, or cloud resources such as AWS, Azure, or Google Cloud as an efficient way to stand up temporary resources to conduct operations. Multiple workspaces may be used to avoid detection. ' subtechnique-of: AML.T0008 - id: AML.T0008.001 name: Consumer Hardware object-type: technique description: 'Adversaries may acquire consumer hardware to conduct their attacks. Owning the hardware provides the adversary with complete control of the environment. These devices can be hard to trace. ' subtechnique-of: AML.T0008 - id: AML.T0019 name: Publish Poisoned Datasets object-type: technique description: 'Adversaries may [Poison Training Data](/techniques/AML.T0020) and publish it to a public location. The poisoned dataset may be a novel dataset or a poisoned variant of an existing open source dataset. This data may be introduced to a victim system via [ML Supply Chain Compromise](/techniques/AML.T0010). ' tactics: - AML.TA0003 - id: AML.T0010 name: ML Supply Chain Compromise object-type: technique description: 'Adversaries may gain initial access to a system by compromising the unique portions of the ML supply chain. This could include [GPU Hardware](/techniques/AML.T0010.000), [Data](/techniques/AML.T0010.002) and its annotations, parts of the ML [ML Software](/techniques/AML.T0010.001) stack, or the [Model](/techniques/AML.T0010.003) itself. In some instances the attacker will need secondary access to fully carry out an attack using compromised components of the supply chain. ' tactics: - AML.TA0004 - id: AML.T0010.000 name: GPU Hardware object-type: technique description: 'Most machine learning systems require access to certain specialized hardware, typically GPUs. Adversaries can target machine learning systems by specifically targeting the GPU supply chain. ' subtechnique-of: AML.T0010 - id: AML.T0010.001 name: ML Software object-type: technique description: 'Most machine learning systems rely on a limited set of machine learning frameworks. An adversary could get access to a large number of machine learning systems through a comprise of one of their supply chains. Many machine learning projects also rely on other open source implementations of various algorithms. These can also be compromised in a targeted way to get access to specific systems. ' subtechnique-of: AML.T0010 - id: AML.T0010.002 name: Data object-type: technique description: 'Data is a key vector of supply chain compromise for adversaries. Every machine learning project will require some form of data. Many rely on large open source datasets that are publicly available. An adversary could rely on compromising these sources of data. The malicious data could be a result of [Poison Training Data](/techniques/AML.T0020) or include traditional malware. An adversary can also target private datasets in the labeling phase. The creation of private datasets will often require the hiring of outside labeling services. An adversary can poison a dataset by modifying the labels being generated by the labeling service. ' subtechnique-of: AML.T0010 - id: AML.T0010.003 name: Model object-type: technique description: 'Machine learning systems often rely on open sourced models in various ways. Most commonly, the victim organization may be using these models for fine tuning. These models will be downloaded from an external source and then used as the base for the model as it is tuned on a smaller, private dataset. Loading models often requires executing some saved code in the form of a saved model file. These can be compromised with traditional malware, or through some adversarial machine learning techniques. ' subtechnique-of: AML.T0010 - id: AML.T0040 name: ML Model Inference API Access object-type: technique description: 'Adversaries may gain access to a model via legitimate access to the inference API. Inference API access can be a source of information to the adversary ([Discover ML Model Ontology](/techniques/AML.T0013), [Discover ML Model Family](/techniques/AML.T0013)), a means of staging the attack ([Verify Attack](/techniques/AML.T0042), [Craft Adversarial Data](/techniques/AML.T0043)), or for introducing data to the target system for Impact ([Evade ML Model](/techniques/AML.T0015), [Erode ML Model Integrity](/techniques/AML.T0031)). ' tactics: - AML.TA0000 - id: AML.T0047 name: ML-Enabled Product or Service object-type: technique description: 'Adversaries may use a product or service that uses machine learning under the hood to gain access to the underlying machine learning model. This type of indirect model access may reveal details of the ML model or its inferences in logs or metadata. ' tactics: - AML.TA0000 - id: AML.T0041 name: Physical Environment Access object-type: technique description: 'In addition to the attacks that take place purely in the digital domain, adversaries may also exploit the physical environment for their attacks. If the model is interacting with data collected from the real world in some way, the adversary can influence the model through access to wherever the data is being collected. By modifying the data in the collection process, the adversary can perform modified versions of attacks designed for digital access. ' tactics: - AML.TA0000 - id: AML.T0044 name: Full ML Model Access object-type: technique description: 'Adversaries may gain full "white-box" access to a machine learning model. This means the adversary has complete knowledge of the model architecture, its parameters, and class ontology. They may exfiltrate the model to [Craft Adversarial Data](/techniques/AML.T0043) and [Verify Attack](/techniques/AML.T0042) in an offline where it is hard to detect their behavior. ' tactics: - AML.TA0000 - id: AML.T0013 name: Discover ML Model Ontology object-type: technique description: 'Adversaries may discover the ontology of a machine learning model''s output space, for example, the types of objects a model can detect. The adversary may discovery the ontology by repeated queries to the model, forcing it to enumerate its output space. Or the ontology may be discovered in a configuration file or in documentation about the model. The model ontology helps the adversary understand how the model is being used by the victim. It is useful to the adversary in creating targeted attacks. ' tactics: - AML.TA0008 - id: AML.T0014 name: Discover ML Model Family object-type: technique description: 'Adversaries may discover the general family of model. General information about the model may be revealed in documentation, or the adversary may used carefully constructed examples and analyze the model''s responses to categorize it. Knowledge of the model family can help the adversary identify means of attacking the model and help tailor the attack. ' tactics: - AML.TA0008 - id: AML.T0020 name: Poison Training Data object-type: technique description: 'Adversaries may attempt to poison datasets used by a ML model by modifying the underlying data or its labels. This allows the adversary to embed vulnerabilities in ML models trained on the data that may not be easily detectable. Data poisoning attacks may or may not require modifying the labels. The embedded vulnerability is activated at a later time by data samples with an [Insert Backdoor Trigger](/techniques/AML.T0043.004) Poisoned data can be introduced via [ML Supply Chain Compromise](/techniques/AML.T0010) or the data may be poisoned after the adversary gains [Initial Access](/tactics/AML.TA0004) to the system. ' tactics: - AML.TA0003 - AML.TA0006 - id: AML.T0021 name: Establish Accounts object-type: technique ATT&CK-reference: id: T1585 url: https://attack.mitre.org/techniques/T1585/ description: 'Adversaries may create accounts with various services for use in targeting, to gain access to resources needed in [ML Attack Staging](/tactics/AML.TA0001), or for victim impersonation. ' tactics: - AML.TA0003 - id: AML.T0005 name: Create Proxy ML Model object-type: technique description: 'Adversaries may obtain models to serve as proxies for the target model in use at the victim organization. Proxy models are used to simulate complete access to the target model in a fully offline manner. Adversaries may train models from representative datasets, attempt to replicate models from victim inference APIs, or use available pre-trained models. ' tactics: - AML.TA0001 - id: AML.T0005.000 name: Train Proxy via Gathered ML Artifacts object-type: technique description: 'Proxy models may be trained from ML artifacts (such as data, model architectures, and pre-trained models) that are representative of the target model gathered by the adversary. This can be used to develop attacks that require higher levels of access than the adversary has available or as a means to validate pre-existing attacks without interacting with the target model. ' subtechnique-of: AML.T0005 - id: AML.T0005.001 name: Train Proxy via Replication object-type: technique description: 'Adversaries may replicate a private model. By repeatedly querying the victim''s [ML Model Inference API Access](/techniques/AML.T0040), the adversary can collect the target model''s inferences into a dataset. The inferences are used as labels for training a separate model offline that will mimic the behavior and performance of the target model. A replicated model that closely mimic''s the target model is a valuable resource in staging the attack. The adversary can use the replicated model to [Craft Adversarial Data](/techniques/AML.T0043) for various purposes (e.g. [Evade ML Model](/techniques/AML.T0015), [Spamming ML System with Chaff Data](/techniques/AML.T0046)). ' subtechnique-of: AML.T0005 - id: AML.T0005.002 name: Use Pre-Trained Model object-type: technique description: 'Adversaries may use an off-the-shelf pre-trained model as a proxy for the victim model to aid in staging the attack. ' subtechnique-of: AML.T0005 - id: AML.T0007 name: Discover ML Artifacts object-type: technique description: 'Adversaries may search private sources to identify machine learning artifacts that exist on the system and gather information about them. These artifacts can include the software stack used to train and deploy models, training and testing data management systems, container registries, software repositories, and model zoos. This information can be used to identify targets for further collection, exfiltration, or disruption, and to tailor and improve attacks. ' tactics: - AML.TA0008 - id: AML.T0011 name: User Execution object-type: technique ATT&CK-reference: id: T1204 url: https://attack.mitre.org/techniques/T1204/ description: 'An adversary may rely upon specific actions by a user in order to gain execution. Users may inadvertently execute unsafe code introduced via [ML Supply Chain Compromise](/techniques/AML.T0010). Users may be subjected to social engineering to get them to execute malicious code by, for example, opening a malicious document file or link. ' tactics: - AML.TA0005 - id: AML.T0011.000 name: Unsafe ML Artifacts object-type: technique description: 'Adversaries may develop unsafe ML artifacts that when executed have a deleterious effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a [ML Supply Chain Compromise](/techniques/AML.T0010). Serialization of models is a popular technique for model storage, transfer, and loading. However, this format without proper checking presents an opportunity for code execution. ' subtechnique-of: AML.T0011 - id: AML.T0012 name: Valid Accounts object-type: technique ATT&CK-reference: id: T1078 url: https://attack.mitre.org/techniques/T1078/ description: 'Adversaries may obtain and abuse credentials of existing accounts as a means of gaining Initial Access. Credentials may take the form of usernames and passwords of individual user accounts or API keys that provide access to various ML resources and services. Compromised credentials may provide access to additional ML artifacts and allow the adversary to perform [Discover ML Artifacts](/techniques/AML.T0007). Compromised credentials may also grant and adversary increased privileges such as write access to ML artifacts used during development or production. ' tactics: - AML.TA0004 - id: AML.T0015 name: Evade ML Model object-type: technique description: 'Adversaries can [Craft Adversarial Data](/techniques/AML.T0043) that prevent a machine learning model from correctly identifying the contents of the data. This technique can be used to evade a downstream task where machine learning is utilized. The adversary may evade machine learning based virus/malware detection, or network scanning towards the goal of a traditional cyber attack. ' tactics: - AML.TA0004 - AML.TA0007 - AML.TA0011 - id: AML.T0018 name: Backdoor ML Model object-type: technique description: 'Adversaries may introduce a backdoor into a ML model. A backdoored model operates performs as expected under typical conditions, but will produce the adversary''s desired output when a trigger is introduced to the input data. A backdoored model provides the adversary with a persistent artifact on the victim system. The embedded vulnerability is typically activated at a later time by data samples with an [Insert Backdoor Trigger](/techniques/AML.T0043.004) ' tactics: - AML.TA0006 - AML.TA0001 - id: AML.T0018.000 name: Poison ML Model object-type: technique description: 'Adversaries may introduce a backdoor by training the model poisoned data, or by interfering with its training process. The model learns to associate a adversary defined trigger with the adversary''s desired output. ' subtechnique-of: AML.T0018 - id: AML.T0018.001 name: Inject Payload object-type: technique description: 'Adversaries may introduce a backdoor into a model by injecting a payload into the model file. The payload detects the presence of the trigger and bypasses the model, instead producing the adversary''s desired output. ' subtechnique-of: AML.T0018 - id: AML.T0024 name: Exfiltration via ML Inference API object-type: technique description: 'Adversaries may exfiltrate private information via [ML Model Inference API Access](/techniques/AML.T0040). ML Models have been shown leak private information about their training data (e.g. [Infer Training Data Membership](/techniques/AML.T0024.000), [Invert ML Model](/techniques/AML.T0024.001)). The model itself may also be extracted ([Extract ML Model](/techniques/AML.T0024.002)) for the purposes of [ML Intellectual Property Theft](/techniques/AML.T0045). Exfiltration of information relating to private training data raises privacy concerns. Private training data may include personally identifiable information, or other protected data. ' tactics: - AML.TA0010 - id: AML.T0024.000 name: Infer Training Data Membership object-type: technique description: 'Adversaries may infer the membership of a data sample in its training set, which raises privacy concerns. Some strategies make use of a shadow model that could be obtained via [Train Proxy via Replication](/techniques/AML.T0005.001), others use statistics of model prediction scores. This can cause the victim model to leak private information, such as PII of those in the training set or other forms of protected IP. ' subtechnique-of: AML.T0024 - id: AML.T0024.001 name: Invert ML Model object-type: technique description: 'Machine learning models'' training data could be reconstructed by exploiting the confidence scores that are available via an inference API. By querying the inference API strategically, adversaries can back out potentially private information embedded within the training data. This could lead to privacy violations if the attacker can reconstruct the data of sensitive features used in the algorithm. ' subtechnique-of: AML.T0024 - id: AML.T0024.002 name: Extract ML Model object-type: technique description: 'Adversaries may extract a functional copy of a private model. By repeatedly querying the victim''s [ML Model Inference API Access](/techniques/AML.T0040), the adversary can collect the target model''s inferences into a dataset. The inferences are used as labels for training a separate model offline that will mimic the behavior and performance of the target model. Adversaries may extract the model to avoid paying per query in a machine learning as a service setting. Model extraction is used for [ML Intellectual Property Theft](/techniques/AML.T0045). ' subtechnique-of: AML.T0024 - id: AML.T0025 name: Exfiltration via Cyber Means object-type: technique description: 'Adversaries may exfiltrate ML artifacts or other information relevant to their goals via traditional cyber means. See the ATT&CK [Exfiltration](https://attack.mitre.org/tactics/TA0010/) tactic for more information. ' tactics: - AML.TA0010 - id: AML.T0029 name: Denial of ML Service object-type: technique description: 'Adversaries may target machine learning systems with a flood of requests for the purpose of degrading or shutting down the service. Since many machine learning systems require significant amounts of specialized compute, they are often expensive bottlenecks that can become overloaded. Adversaries can intentionally craft inputs that require heavy amounts of useless compute from the machine learning system. ' tactics: - AML.TA0011 - id: AML.T0046 name: Spamming ML System with Chaff Data object-type: technique description: 'Adversaries may spam the machine learning system with chaff data that causes increase in the number of detections. This can cause analysts at the victim organization to waste time reviewing and correcting incorrect inferences. ' tactics: - AML.TA0011 - id: AML.T0031 name: Erode ML Model Integrity object-type: technique description: 'Adversaries may degrade the target model''s performance with adversarial data inputs to erode confidence in the system over time. This can lead to the victim organization wasting time and money both attempting to fix the system and performing the tasks it was meant to automate by hand. ' tactics: - AML.TA0011 - id: AML.T0034 name: Cost Harvesting object-type: technique description: 'Adversaries may target different machine learning services to send useless queries or computationally expensive inputs to increase the cost of running services at the victim organization. Sponge examples are a particular type of adversarial data designed to maximize energy consumption and thus operating cost. ' tactics: - AML.TA0011 - id: AML.T0035 name: ML Artifact Collection object-type: technique description: 'Adversaries may collect ML artifacts for [Exfiltration](/tactics/AML.TA0010) or for use in [ML Attack Staging](/tactics/AML.TA0001). ML artifacts include models and datasets as well as other telemetry data produced when interacting with a model. ' tactics: - AML.TA0009 - id: AML.T0036 name: Data from Information Repositories object-type: technique ATT&CK-reference: id: T1213 url: https://attack.mitre.org/techniques/T1213/ description: 'Adversaries may leverage information repositories to mine valuable information. Information repositories are tools that allow for storage of information, typically to facilitate collaboration or information sharing between users, and can store a wide variety of data that may aid adversaries in further objectives, or direct access to the target information. Information stored in a repository may vary based on the specific instance or environment. Specific common information repositories include Sharepoint, Confluence, and enterprise databases such as SQL Server. ' tactics: - AML.TA0009 - id: AML.T0037 name: Data from Local System object-type: technique ATT&CK-reference: id: T1005 url: https://attack.mitre.org/techniques/T1005/ description: 'Adversaries may search local system sources, such as file systems and configuration files or local databases, to find files of interest and sensitive data prior to Exfiltration. This can include basic fingerprinting information and sensitive data such as ssh keys. ' tactics: - AML.TA0009 - id: AML.T0042 name: Verify Attack object-type: technique description: 'Adversaries can verify the efficacy of their attack via an inference API or access to an offline copy of the target model. This gives the adversary confidence that their approach works and allows them to carry out the attack at a later time of their choosing. The adversary may verify the attack once but use it against many edge devices running copies of the target model. The adversary may verify their attack digitally, then deploy it in the [Physical Environment Access](/techniques/AML.T0041) at a later time. Verifying the attack may be hard to detect since the adversary can use a minimal number of queries or an offline copy of the model. ' tactics: - AML.TA0001 - id: AML.T0043 name: Craft Adversarial Data object-type: technique description: 'Adversarial data are inputs to a machine learning model that have been modified such that they cause the adversary''s desired effect in the target model. Effects can range from misclassification, to missed detections, to maximising energy consumption. Typically, the modification is constrained in magnitude or location so that a human still perceives the data as if it were unmodified, but human perceptibility may not always be a concern depending on the adversary''s intended effect. For example, an adversarial input for an image classification task is an image the machine learning model would misclassify, but a human would still recognize as containing the correct class. Depending on the adversary''s knowledge of and access to the target model, the adversary may use different classes of algorithms to develop the adversarial example such as [White-Box Optimization](/techniques/AML.T0043.000), [Black-Box Optimization](/techniques/AML.T0043.001), [Black-Box Transfer](/techniques/AML.T0043.002), or [Manual Modification](/techniques/AML.T0043.003). The adversary may [Verify Attack](/techniques/AML.T0042) their approach works if they have white-box or inference API access to the model. This allows the adversary to gain confidence their attack is effective "live" environment where their attack may be noticed. They can then use the attack at a later time to accomplish their goals. An adversary may optimize adversarial examples for [Evade ML Model](/techniques/AML.T0015), or to [Erode ML Model Integrity](/techniques/AML.T0031). ' tactics: - AML.TA0001 - id: AML.T0043.000 name: White-Box Optimization object-type: technique description: 'In White-Box Optimization, the adversary has full access to the target model and optimizes the adversarial example directly. Adversarial examples trained in this manor are most effective against the target model. ' subtechnique-of: AML.T0043 - id: AML.T0043.001 name: Black-Box Optimization object-type: technique description: 'In Black-Box attacks, the adversary has black-box (i.e. [ML Model Inference API Access](/techniques/AML.T0040) via API access) access to the target model. With black-box attacks, the adversary may be using an API that the victim is monitoring. These attacks are generally less effective and require more inferences than [White-Box Optimization](/techniques/AML.T0043.000) attacks, but they require much less access. ' subtechnique-of: AML.T0043 - id: AML.T0043.002 name: Black-Box Transfer object-type: technique description: 'In Black-Box Transfer attacks, the adversary uses one or more proxy models (trained via [Create Proxy ML Model](/techniques/AML.T0005) or [Train Proxy via Replication](/techniques/AML.T0005.001)) models they have full access to and are representative of the target model. The adversary uses [White-Box Optimization](/techniques/AML.T0043.000) on the proxy models to generate adversarial examples. If the set of proxy models are close enough to the target model, the adversarial example should generalize from one to another. This means that an attack that works for the proxy models will likely then work for the target model. If the adversary has [ML Model Inference API Access](/techniques/AML.T0040), they may use this [Verify Attack](/techniques/AML.T0042) that the attack is working and incorporate that information into their training process. ' subtechnique-of: AML.T0043 - id: AML.T0043.003 name: Manual Modification object-type: technique description: 'Adversaries may manually modify the input data to craft adversarial data. They may use their knowledge of the target model to modify parts of the data they suspect helps the model in performing its task. The adversary may use trial and error until they are able to verify they have a working adversarial input. ' subtechnique-of: AML.T0043 - id: AML.T0043.004 name: Insert Backdoor Trigger object-type: technique description: 'The adversary may add a perceptual trigger into inference data. The trigger may be imperceptible or non-obvious to humans. This technique is used in conjunction with [Poison ML Model](/techniques/AML.T0018.000) and allows the adversary to produce their desired effect in the target model. ' subtechnique-of: AML.T0043 - id: AML.T0045 name: ML Intellectual Property Theft object-type: technique description: 'Adversaries may exfiltrate ML artifacts to steal intellectual property and cause economic harm to the victim organization. Proprietary training data is costly to collect and annotate and may be a target for [Exfiltration](/tactics/AML.TA0010) and theft. MLaaS providers charge for use of their API. An adversary who has stolen a model via [Exfiltration](/tactics/AML.TA0010) or via [Extract ML Model](/techniques/AML.T0024.002) now has unlimited use of that service without paying the owner of the intellectual property. ' tactics: - AML.TA0011 - id: AML.T0048 name: System Misuse for External Effect object-type: technique description: 'Adversaries may abuse their access to a system to use its resources or capabilities to further their goals by causing effects outside of the victim system. ' tactics: - AML.TA0011 case-studies: - id: AML.CS0000 object-type: case-study name: Evasion of Deep Learning Detector for Malware C&C Traffic summary: 'The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available [paper by Le et al.](https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.' incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0002 technique: AML.T0000.001 description: 'We identified a machine learning based approach to malicious URL detection as a representative approach and potential target from the paper [URLNet: Learning a URL representation with deep learning for malicious URL detection](https://arxiv.org/abs/1802.03162), which was found on arXiv (a pre-print repository).' - tactic: AML.TA0003 technique: AML.T0002.000 description: We acquired a command and control HTTP traffic dataset consisting of approximately 33 million benign and 27 million malicious HTTP packet headers. - tactic: AML.TA0001 technique: AML.T0005 description: 'We trained a model on the HTTP traffic dataset to use as a proxy for the target model. Evaluation showed a true positive rate of ~ 99% and false positive rate of ~ 0.01%, on average. Testing the model with a HTTP packet header from known malware command and control traffic samples was detected as malicious with high confidence (> 99%).' - tactic: AML.TA0001 technique: AML.T0043.003 description: We crafted evasion samples by removing fields from packet header which are typically not used for C&C communication (e.g. cache-control, connection, etc.). - tactic: AML.TA0001 technique: AML.T0042 description: We queried the model with our adversarial examples and adjusted them until the model was evaded. - tactic: AML.TA0007 technique: AML.T0015 description: 'With the crafted samples, we performed online evasion of the ML-based spyware detection model. The crafted packets were identified as benign with > 80% confidence. This evaluation demonstrates that adversaries are able to bypass advanced ML detection techniques, by crafting samples that are misclassified by an ML model.' actor: Palo Alto Networks AI Research Team target: Palo Alto Networks malware detection system case-study-type: exercise references: - title: 'Le, Hung, et al. "URLNet: Learning a URL representation with deep learning for malicious URL detection." arXiv preprint arXiv:1802.03162 (2018).' url: https://arxiv.org/abs/1802.03162 - id: AML.CS0001 name: Botnet Domain Generation Algorithm (DGA) Detection Evasion object-type: case-study case-study-type: exercise actor: Palo Alto Networks AI Research Team target: Palo Alto Networks ML-based DGA detection module summary: 'The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules DGA and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they is deployed to the production environment.' incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0002 technique: AML.T0000 description: 'DGA detection is a widely used technique to detect botnets in academia and industry. The research team searched for research papers related to DGA detection.' - tactic: AML.TA0003 technique: AML.T0002 description: 'The researchers acquired a publicly available CNN-based DGA detection model and tested it against a well-known DGA generated domain name data sets, which includes ~50 million domain names from 64 botnet DGA families. The CNN-based DGA detection model shows more than 70% detection accuracy on 16 (~25%) botnet DGA families.' - tactic: AML.TA0003 technique: AML.T0017 description: The researchers developed a generic mutation technique that requires a minimal number of iterations. - tactic: AML.TA0001 technique: AML.T0043.001 description: The researchers used the mutation technique to generate evasive domain names. - tactic: AML.TA0001 technique: AML.T0042 description: The experiment results show that the detection rate of all 16 botnet DGA families drop to less than 25% after only one string is inserted once to the DGA generated domain names. - tactic: AML.TA0007 technique: AML.T0015 description: The DGA generated domain names mutated with this technique successfully evade the target DGA Detection model, allowing an adversary to continue communication with their [Command and Control](https://attack.mitre.org/tactics/TA0011/) servers. references: - title: Yu, Bin, Jie Pan, Jiaming Hu, Anderson Nascimento, and Martine De Cock. "Character level based detection of DGA domain names." In 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2018. url: http://faculty.washington.edu/mdecock/papers/byu2018a.pdf - title: Degas source code url: https://github.com/matthoffman/degas - id: AML.CS0002 object-type: case-study name: VirusTotal Poisoning summary: McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. Case investigation revealed that many samples of that particular ransomware family were submitted through a popular virus-sharing platform within a short amount of time. Further investigation revealed that based on string similarity the samples were all equivalent, and based on code similarity they were between 98 and 74 percent similar. Interestingly enough, the compile time was the same for all the samples. After more digging, researchers discovered that someone used 'metame' a metamorphic code manipulating tool to manipulate the original file towards mutant variants. The variants would not always be executable, but are still classified as the same ransomware family. incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0003 technique: AML.T0016.000 description: The actor obtained [metame](https://github.com/a0rtega/metame), a simple metamorphic code engine for arbitrary executables. - tactic: AML.TA0001 technique: AML.T0043 description: The actor used a malware sample from a prevalent ransomware family as a start to create "mutant" variants. - tactic: AML.TA0004 technique: AML.T0010.002 description: The actor uploaded "mutant" samples to the platform. - tactic: AML.TA0006 technique: AML.T0020 description: 'Several vendors started to classify the files as the ransomware family even though most of them won''t run. The "mutant" samples poisoned the dataset the ML model(s) use to identify and classify this ransomware family.' actor: Unknown target: VirusTotal reporter: McAfee Advanced Threat Research case-study-type: incident references: [] - id: AML.CS0003 object-type: case-study name: Bypassing Cylance's AI Malware Detection summary: Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file. incident-date: 2019-09-07 incident-date-granularity: DATE procedure: - tactic: AML.TA0002 technique: AML.T0000 description: The researchers read publicly available information about Cylance's AI Malware detector. They gathered this information from various sources such as public talks as well as patent submissions by Cylance. - tactic: AML.TA0000 technique: AML.T0047 description: The researchers used Cylance's AI Malware detector and enabled verbose logging to understand the inner workings of the ML model, particularly around reputation scoring and model ensembling. - tactic: AML.TA0003 technique: AML.T0017 description: 'The researchers used the reputation scoring information to reverse engineer which attributes provided what level of positive or negative reputation. Along the way, they discovered a secondary model which was an override for the first model. Positive assessments from the second model overrode the decision of the core ML model.' - tactic: AML.TA0001 technique: AML.T0043.003 description: Using this knowledge, the researchers fused attributes of known good files with malware to manually create adversarial malware. - tactic: AML.TA0007 technique: AML.T0015 description: Due to the secondary model overriding the primary, the researchers were effectively able to bypass the ML model. actor: Skylight Cyber target: CylancePROTECT, Cylance Smart Antivirus case-study-type: exercise references: - title: Skylight Cyber Blog Post, "Cylance, I Kill You!" url: https://skylightcyber.com/2019/07/18/cylance-i-kill-you/ - title: Statement's from Skylight Cyber CEO url: https://www.security7.net/news/the-new-cylance-vulnerability-what-you-need-to-know - id: AML.CS0004 object-type: case-study name: Camera Hijack Attack on Facial Recognition System summary: 'This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government''s tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fraudulently collect $77 million. ' incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0003 technique: AML.T0008.001 description: The attackers bought customized low-end mobile phones. - tactic: AML.TA0003 technique: AML.T0016.001 description: The attackers obtained customized Android ROMs and a virtual camera application. - tactic: AML.TA0003 technique: AML.T0016.000 description: The attackers obtained software that turns static photos into videos, adding realistic effects such as blinking eyes. - tactic: AML.TA0003 technique: AML.T0021 description: The attackers collected user identity information and high definition face photos from an online black market and used the victim's information to register accounts. - tactic: AML.TA0000 technique: AML.T0047 description: The attackers used the virtual camera app to present the generated video to the ML-based facial recognition service used for user verification. - tactic: AML.TA0004 technique: AML.T0015 description: The attackers successfully evaded the face recognition system. This allowed the attackers to impersonate the victim and verify their their identity in the tax system. - tactic: AML.TA0011 technique: AML.T0048 description: The attackers used their privileged access to the tax system to send invoices to supposed clients and further their fraud scheme. reporter: Ant Group AISEC Team actor: Two individuals target: Shanghai government tax office's facial recognition service case-study-type: incident references: - title: Faces are the next target for fraudsters url: https://www.wsj.com/articles/faces-are-the-next-target-for-fraudsters-11625662828 - id: AML.CS0005 object-type: case-study name: Attack on Machine Translation Service - Google Translate, Bing Translator, and Systran Translate summary: 'Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.' incident-date: 2020-04-30 incident-date-granularity: DATE procedure: - tactic: AML.TA0002 technique: AML.T0000 description: The researchers used published research papers to identify the datasets and model architectures used by the target translation services. - tactic: AML.TA0003 technique: AML.T0002.000 description: The researchers gathered similar datasets that the target translation services used. - tactic: AML.TA0003 technique: AML.T0002.001 description: The researchers gathered similar model architectures that the target translation services used. - tactic: AML.TA0000 technique: AML.T0040 description: They abused a public facing application to query the model and produced machine translated sentence pairs as training data. - tactic: AML.TA0001 technique: AML.T0005.001 description: Using these translated sentence pairs, the researchers trained a model that replicates the behavior of the target model. - tactic: AML.TA0011 technique: AML.T0045 description: By replicating the model with high fidelity, the researchers demonstrated that an adversary could steal a model and violate the victim's intellectual property rights. - tactic: AML.TA0001 technique: AML.T0043.002 description: The replicated models were used to generate adversarial examples that successfully transferred to the black-box translation services. - tactic: AML.TA0011 technique: AML.T0015 description: The adversarial examples were used to evade the machine translation services by a variety of means. This included targeted word flips, vulgar outputs, and dropped sentences. - tactic: AML.TA0011 technique: AML.T0031 description: Adversarial attacks can cause errors that cause reputational damage to the company of the translation service and decrease user trust in AI-powered services. actor: Berkeley Artificial Intelligence Research target: Google Translate, Bing Translator, Systran Translate case-study-type: exercise references: - title: Wallace, Eric, et al. "Imitation Attacks and Defenses for Black-box Machine Translation Systems" EMNLP 2020 url: https://arxiv.org/abs/2004.15015 - title: Project Page, "Imitation Attacks and Defenses for Black-box Machine Translation Systems" url: https://www.ericswallace.com/imitation - title: Google under fire for mistranslating Chinese amid Hong Kong protests url: https://thehill.com/policy/international/asia-pacific/449164-google-under-fire-for-mistranslating-chinese-amid-hong-kong/ - id: AML.CS0006 object-type: case-study name: ClearviewAI Misconfiguration summary: 'Clearview AI makes a facial recognition tool that searches publicly available photos for matches. This tool has been used for investigative purposes by law enforcement agencies and other parties. Clearview AI''s source code repository, though password protected, was misconfigured to allow an arbitrary user to register an account. This allowed an external researcher to gain access to a private code repository that contained Clearview AI production credentials, keys to cloud storage buckets containing 70K video samples, and copies of its applications and Slack tokens. With access to training data, a bad-actor has the ability to cause an arbitrary misclassification in the deployed model. These kinds of attacks illustrate that any attempt to secure ML system should be on top of "traditional" good cybersecurity hygiene such as locking down the system with least privileges, multi-factor authentication and monitoring and auditing.' incident-date: 2020-04-16 incident-date-granularity: MONTH procedure: - tactic: AML.TA0003 technique: AML.T0021 description: A security researcher gained initial access to Clearview AI's private code repository via a misconfigured server setting that allowed an arbitrary user to register a valid account. - tactic: AML.TA0009 technique: AML.T0036 description: 'The private code repository contained credentials which were used to access AWS S3 cloud storage buckets, leading to the discovery of assets for the facial recognition tool, including: - Released desktop and mobile applications - Pre-release applications featuring new capabilities - Slack access tokens - Raw videos and other data' - tactic: AML.TA0003 technique: AML.T0002 description: Adversaries could have downloaded training data and gleaned details about software, models, and capabilities from the source code and decompiled application binaries. - tactic: AML.TA0011 technique: AML.T0031 description: As a result, future application releases could have been compromised, causing degraded or malicious facial recognition capabilities. actor: Researchers at spiderSilk target: Clearview AI facial recognition tool case-study-type: incident references: - title: TechCrunch Article, "Security lapse exposed Clearview AI source code" url: https://techcrunch.com/2020/04/16/clearview-source-code-lapse/ - title: Gizmodo Article, "We Found Clearview AI's Shady Face Recognition App" url: https://gizmodo.com/we-found-clearview-ais-shady-face-recognition-app-1841961772 - title: New York Times Article, "The Secretive Company That Might End Privacy as We Know It" url: https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html - id: AML.CS0007 name: GPT-2 Model Replication object-type: case-study case-study-type: exercise actor: Researchers at Brown University target: OpenAI GPT-2 summary: 'OpenAI built GPT-2, a language model capable of generating high quality text samples. Over concerns that GPT-2 could be used for malicious purposes such as impersonating others, or generating misleading news articles, fake social media content, or spam, OpenAI adopted a tiered release schedule. They initially released a smaller, less powerful version of GPT-2 along with a technical description of the approach, but held back the full trained model. Before the full model was released by OpenAI, researchers at Brown University successfully replicated the model using information released by OpenAI and open source ML artifacts. This demonstrates that a bad actor with sufficient technical skill and compute resources could have replicated GPT-2 and used it for harmful goals before the AI Security community is prepared. ' incident-date: 2019-08-22 incident-date-granularity: DATE procedure: - tactic: AML.TA0002 technique: AML.T0000 description: Using the public documentation about GPT-2, the researchers gathered information about the dataset, model architecture, and training hyper-parameters. - tactic: AML.TA0003 technique: AML.T0002.001 description: The researchers obtained a reference implementation of a similar publicly available model called Grover. - tactic: AML.TA0003 technique: AML.T0002.000 description: The researchers were able to manually recreate the dataset used in the original GPT-2 paper using the gathered documentation. - tactic: AML.TA0003 technique: AML.T0008.000 description: The researchers were able to use TensorFlow Research Cloud via their academic credentials. - tactic: AML.TA0001 technique: AML.T0005.000 description: 'The researchers modified Grover''s objective function to reflect GPT-2''s objective function and then trained on the dataset they curated using used Grover''s initial hyperparameters. The resulting model functionally replicates GPT-2, obtaining similar performance on most datasets. A bad actor who followed the same procedure as the researchers could then use the replicated GPT-2 model for malicious purposes.' references: - title: Wired Article, "OpenAI Said Its Code Was Risky. Two Grads Re-Created It Anyway" url: https://www.wired.com/story/dangerous-ai-open-source/ - title: 'Medium BlogPost, "OpenGPT-2: We Replicated GPT-2 Because You Can Too"' url: https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc - id: AML.CS0008 object-type: case-study name: ProofPoint Evasion summary: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM. incident-date: 2019-09-09 incident-date-granularity: DATE procedure: - tactic: AML.TA0000 technique: AML.T0047 description: The researchers first gathered the scores from the Proofpoint's ML system used in email headers by sending a large number of emails through the system and scraping the model scores exposed in the logs. - tactic: AML.TA0001 technique: AML.T0005.001 description: 'The researchers converted the collected scores into a dataset, which they used to train a functional copy of the ProofPoint model. Basic correlation was used to decide which score variable speaks generally about the security of an email. The "mlxlogscore" was selected in this case due to its relationship with spam, phish, and core mlx and was used as the label. Each "mlxlogscore" was generally between 1 and 999 (higher score = safer sample). Training was performed using an Artificial Neural Network (ANN) and Bag of Words tokenizing. ' - tactic: AML.TA0001 technique: AML.T0043.002 description: 'Next, the ML researchers algorithmically found samples from this "offline" proxy model that helped give desired insight into its behavior and influential variables. Examples of good scoring samples include "calculation", "asset", and "tyson". Examples of bad scoring samples include "software", "99", and "unsub".' - tactic: AML.TA0011 technique: AML.T0015 description: Finally, these insights from the "offline" proxy model allowed the researchers to create malicious emails that received preferable scores from the real ProofPoint email protection system, hence bypassing it. target: ProofPoint Email Protection System actor: Researchers at Silent Break Security case-study-type: exercise references: - title: National Vulnerability Database entry for CVE-2019-20634 url: https://nvd.nist.gov/vuln/detail/CVE-2019-20634 - title: '2019 DerbyCon presentation "42: The answer to life, the universe, and everything offensive security"' url: https://github.com/moohax/Talks/blob/master/slides/DerbyCon19.pdf - title: Proof Pudding (CVE-2019-20634) Implementation on GitHub url: https://github.com/moohax/Proof-Pudding - title: '2019 DerbyCon video presentation "42: The answer to life, the universe, and everything offensive security"' url: https://www.youtube.com/watch?v=CsvkYoxtexQ&ab_channel=AdrianCrenshaw - id: AML.CS0009 object-type: case-study name: Tay Poisoning summary: 'Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay''s machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot''s failure. ' incident-date: 2016-03-23 incident-date-granularity: DATE procedure: - tactic: AML.TA0000 technique: AML.T0047 description: Adversaries were able to interact with Tay via Twitter messages. - tactic: AML.TA0004 technique: AML.T0010.002 description: 'Tay bot used the interactions with its Twitter users as training data to improve its conversations. Adversaries were able to coordinate with the intent of defacing Tay bot by exploiting this feedback loop.' - tactic: AML.TA0006 technique: AML.T0020 description: By repeatedly interacting with Tay using racist and offensive language, they were able to bias Tay's dataset towards that language as well. This was done by adversaries using the "repeat after me" function, a command that forced Tay to repeat anything said to it. - tactic: AML.TA0011 technique: AML.T0031 description: As a result of this coordinated attack, Tay's conversation algorithms began to learn to generate reprehensible material. Tay's internalization of this detestable language caused it to be unpromptedly repeated during interactions with innocent users. reporter: Microsoft target: Microsoft's Tay AI Chatbot actor: 4chan Users case-study-type: incident references: - title: 'AIID - Incident 6: TayBot' url: https://incidentdatabase.ai/cite/6 - title: Microsoft BlogPost, "Learning from Tay's introduction" url: https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/ - title: IEEE Article, "In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation" url: https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation - id: AML.CS0010 name: Microsoft Azure Service Disruption object-type: case-study case-study-type: exercise actor: Microsoft AI Red Team target: Internal Microsoft Azure Service summary: The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples. incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0002 technique: AML.T0000 description: The team first performed reconnaissance to gather information about the target ML model. - tactic: AML.TA0004 technique: AML.T0012 description: The team used a valid account to gain access to the network. - tactic: AML.TA0009 technique: AML.T0035 description: The team found the model file of the target ML model and the necessary training data. - tactic: AML.TA0010 technique: AML.T0025 description: The team exfiltrated the model and data via traditional means. - tactic: AML.TA0001 technique: AML.T0043.000 description: Using the target model and data, the red team crafted evasive adversarial data in an offline manor. - tactic: AML.TA0000 technique: AML.T0040 description: The team used an exposed API to access the target model. - tactic: AML.TA0001 technique: AML.T0042 description: The team submitted the adversarial examples to the API to verify their efficacy on the production system. - tactic: AML.TA0011 technique: AML.T0015 description: The team performed an online evasion attack by replaying the adversarial examples and accomplished their goals. references: [] - id: AML.CS0011 name: Microsoft Edge AI Evasion object-type: case-study summary: 'The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use a automated system to continuously manipulate a target image to cause the ML model to produce misclassifications. ' incident-date: 2020-02-01 incident-date-granularity: MONTH procedure: - tactic: AML.TA0002 technique: AML.T0000 description: 'The team first performed reconnaissance to gather information about the target ML model. ' - tactic: AML.TA0003 technique: AML.T0002 description: 'The team identified and obtained the publicly available base model to use against the target ML model. ' - tactic: AML.TA0000 technique: AML.T0040 description: 'Using the publicly available version of the ML model, the team started sending queries and analyzing the responses (inferences) from the ML model. ' - tactic: AML.TA0001 technique: AML.T0043.001 description: 'The red team created an automated system that continuously manipulated an original target image, that tricked the ML model into producing incorrect inferences, but the perturbations in the image were unnoticeable to the human eye. ' - tactic: AML.TA0011 technique: AML.T0015 description: 'Feeding this perturbed image, the red team was able to evade the ML model by causing misclassifications. ' target: New Microsoft AI Product actor: Azure Red Team case-study-type: exercise references: [] - id: AML.CS0012 name: Face Identification System Evasion via Physical Countermeasures object-type: case-study summary: 'MITRE''s AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional ATT&CK enterprise techniques such as finding Valid account, and Executing code via an API - all interleaved with adversarial ML specific attacks. ' incident-date: 2020-01-01 incident-date-granularity: YEAR procedure: - tactic: AML.TA0002 technique: AML.T0000 description: 'The team first performed reconnaissance to gather information about the target ML model. ' - tactic: AML.TA0004 technique: AML.T0012 description: 'The team gained access to said service via a valid account to gain access and knowledge to api. ' - tactic: AML.TA0000 technique: AML.T0040 description: 'The team accessed the inference API of the target model. ' - tactic: AML.TA0008 technique: AML.T0013 description: 'The team identified the list of identities targeted by the model by querying the target model''s inference API. ' - tactic: AML.TA0003 technique: AML.T0002.000 description: 'The team acquired representative open source data. ' - tactic: AML.TA0001 technique: AML.T0005 description: 'The team developed a proxy model using the open source data. ' - tactic: AML.TA0001 technique: AML.T0043.000 description: 'Using the proxy model, the red team optimized adversarial visual pattern as a physical domain patch-based attack using expectation over transformation. ' - tactic: AML.TA0000 technique: AML.T0041 description: 'The team placed the physical countermeasure from the previous step and placed it in the physical environment to cause issues in the face identification system. ' - tactic: AML.TA0011 technique: AML.T0015 description: 'The team successfully evaded the model using the physical countermeasure and causing targeted misclassifications. ' target: Commercial Face Identification Service actor: MITRE AI Red Team case-study-type: exercise references: [] - id: AML.CS0013 object-type: case-study name: Backdoor Attack on Deep Learning Models in Mobile Apps summary: 'Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.' incident-date: 2021-01-18 incident-date-granularity: DATE procedure: - tactic: AML.TA0002 technique: AML.T0004 description: To identify a list of potential target models, the researchers searched the Google Play store for apps that may contain embedded deep learning models by searching for deep learning related keywords. - tactic: AML.TA0003 technique: AML.T0002.001 description: 'The researchers acquired the apps'' APKs from the Google Play store. They filtered the list of potential target applications by searching the code metadata for keywords related to TensorFlow or TFLite and their model binary formats (.tf and .tflite). The models were extracted from the APKs using Apktool.' - tactic: AML.TA0000 technique: AML.T0044 description: This provided the researchers with full access to the ML model, albeit in compiled, binary form. - tactic: AML.TA0003 technique: AML.T0017 description: 'The researchers developed a novel approach to insert a backdoor into a compiled model that can be activated with a visual trigger. They inject a "neural payload" into the model that consists of a trigger detection network and conditional logic. The trigger detector is trained to detect a visual trigger that will be placed in the real world. The conditional logic allows the researchers to bypass the victim model when the trigger is detected and provide model outputs of their choosing. The only requirements for training a trigger detector are a general dataset from the same modality as the target model (e.g. ImageNet for image classification) and several photos of the desired trigger.' - tactic: AML.TA0006 technique: AML.T0018.001 description: 'The researchers poisoned the victim model by injecting the neural payload into the compiled models by directly modifying the computation graph. The researchers then repackage the poisoned model back into the APK' - tactic: AML.TA0001 technique: AML.T0042 description: To verify the success of the attack, the researchers confirmed the app did not crash with the malicious model in place, and that the trigger detector successfully detects the trigger. - tactic: AML.TA0004 technique: AML.T0010.003 description: In practice, the malicious APK would need to be installed on victim's devices via a supply chain compromise. - tactic: AML.TA0001 technique: AML.T0043.004 description: The trigger is placed in the physical environment, where it is captured by the victim's device camera and processed by the backdoored ML model. - tactic: AML.TA0000 technique: AML.T0041 description: At inference time, only physical environment access is required to trigger the attack. - tactic: AML.TA0011 technique: AML.T0015 description: 'Presenting the visual trigger causes the victim model to be bypassed. The researchers demonstrated this can be used to evade ML models in several safety-critical apps in the Google Play store.' actor: Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, Yunxin Liu target: ML-based Android Apps case-study-type: exercise references: - title: 'DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection' url: https://arxiv.org/abs/2101.06896 - id: AML.CS0014 object-type: case-study name: Confusing Antimalware Neural Networks summary: 'Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users'' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky''s antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.' incident-date: 2021-06-23 incident-date-granularity: DATE procedure: - tactic: AML.TA0002 technique: AML.T0001 description: 'The researchers performed a review of adversarial ML attacks on antimalware products. They discovered that techniques borrowed from attacks on image classifiers have been successfully applied to the antimalware domain. However, it was not clear if these approaches were effective against the ML component of production antimalware solutions.' - tactic: AML.TA0002 technique: AML.T0003 description: Kaspersky's use of ML-based antimalware detectors is publicly documented on their website. In practice, an adversary could use this for targeting. - tactic: AML.TA0000 technique: AML.T0047 description: 'The researches used access to the target ML-based antimalware product throughout this case study. This product scans files on the user''s system, extracts features locally, then sends them to the cloud-based ML malware detector for classification. Therefore, the researchers had only black-box access to the malware detector itself, but could learn valuable information for constructing the attack from the feature extractor.' - tactic: AML.TA0003 technique: AML.T0002.000 description: 'The researchers collected a dataset of malware and clean files. They scanned the dataset with the target ML-based antimalware solution and labeled the samples according the ML detector''s predictions.' - tactic: AML.TA0001 technique: AML.T0005 description: 'A proxy model was trained on the labeled dataset of malware and clean files. The researchers experimented with a variety of model architectures.' - tactic: AML.TA0003 technique: AML.T0017 description: 'By reverse engineering the local feature extractor, the researchers could collect information about the input features, used for the cloud-based ML detector. The model collects PE Header features, section features and section data statistics, and file strings information. A gradient based adversarial algorithm for executable files was developed. The algorithm manipulates file features to avoid detection by the proxy model, while still containing the same malware payload' - tactic: AML.TA0001 technique: AML.T0043.002 description: Using a developed gradient-driven algorithm, malicious adversarial files for the proxy model were constructed from the malware files for black-box transfer to the target model. - tactic: AML.TA0001 technique: AML.T0042 description: The adversarial malware files were tested against the target antimalware solution to verify their efficacy. - tactic: AML.TA0007 technique: AML.T0015 description: 'The researchers demonstrated that for most of the adversarial files, the antimalware model was successfully evaded. In practice, an adversary could deploy their adversarially crafted malware and infect systems while evading detection.' target: Kaspersky's Antimalware ML Model actor: Kaspersky ML Research Team case-study-type: exercise references: - title: Article, "How to confuse antimalware neural networks. Adversarial attacks and protection" url: https://securelist.com/how-to-confuse-antimalware-neural-networks-adversarial-attacks-and-protection/102949/ - id: AML.CS0015 object-type: case-study name: Compromised PyTorch Dependency Chain summary: 'Linux packages for PyTorch''s pre-release version, called Pytorch-nightly, were compromised from December 25 to 30, 2022 by a malicious binary uploaded to the Python Package Index (PyPI) code repository. The malicious binary had the same name as a PyTorch dependency and the PyPI package manager (pip) installed this malicious package instead of the legitimate one. This supply chain attack, also known as "dependency confusion," exposed sensitive information of Linux machines with the affected pip-installed versions of PyTorch-nightly. On December 30, 2022, PyTorch announced the incident and initial steps towards mitigation, including the rename and removal of `torchtriton` dependencies.' incident-date: 2022-12-25 incident-date-granularity: DATE procedure: - tactic: AML.TA0004 technique: AML.T0010.001 description: 'A malicious dependency package named `torchtriton` was uploaded to the PyPI code repository with the same package name as a package shipped with the PyTorch-nightly build. This malicious package contained additional code that uploads sensitive data from the machine. The malicious `torchtriton` package was installed instead of the legitimate one because PyPI is prioritized over other sources. See more details at [this GitHub issue](https://github.com/pypa/pip/issues/8606).' - tactic: AML.TA0009 technique: AML.T0037 description: 'The malicious package surveys the affected system for basic fingerprinting info (such as IP address, username, and current working directory), and steals further sensitive data, including: - nameservers from `/etc/resolv.conf` - hostname from `gethostname()` - current username from `getlogin()` - current working directory name from `getcwd()` - environment variables - `/etc/hosts` - `/etc/passwd` - the first 1000 files in the user''s `$HOME` directory - `$HOME/.gitconfig` - `$HOME/.ssh/*.`' - tactic: AML.TA0010 technique: AML.T0025 description: All gathered information, including file contents, is uploaded via encrypted DNS queries to the domain `*[dot]h4ck[dot]cfd`, using the DNS server `wheezy[dot]io`. reporter: PyTorch actor: Unknown target: PyTorch case-study-type: incident references: - title: PyTorch statement on compromised dependency url: https://pytorch.org/blog/compromised-nightly-dependency/ - title: Analysis by BleepingComputer url: https://www.bleepingcomputer.com/news/security/pytorch-discloses-malicious-dependency-chain-compromise-over-holidays/