AttackTrace

Security field guide

ATT&CK Threat Activity ATLAS Search /

Browse techniques Threat activity Browse ATLAS Search

Loading AttackTrace...

AttackTrace - execution, telemetry, detection, investigation, mitigation.

Technique index

MITRE ATLAS

Adversarial threats to AI systems

A verified, source-linked view of the Adversarial Threat Landscape for Artificial-Intelligence Systems.

Resource Development

AI Model Access

Privilege Escalation

Defense Evasion

Credential Access

Lateral Movement

AI Attack Staging

Command and Control

Search techniques

170 verified ATLAS techniques

Search Open Technical Databases

Adversaries may search for publicly available research and technical documentation to learn how and where AI is used within a victim organization. The adversary can use this information to identify targets for attack, or to tailor an existing attack to make it more effective. Organizations often use open source model a

Journals and Conference Proceedings

Many of the publications accepted at premier artificial intelligence conferences and journals come from commercial labs. Some journals and conferences are open access, others may require paying for access or a membership. These publications will often describe in detail all aspects of a particular approach for reproduc

Pre-Print Repositories

Pre Print repositories, such as arXiv, contain the latest academic research papers that haven't been peer reviewed. They may contain research notes, or technical reports that aren't typically published in journals or conference proceedings. Pre print repositories also serve as a central location to share papers that ha

Reconnaissance

Demonstrated

Technical Blogs

Research labs at academic institutions and company R&D divisions often have blogs that highlight their use of artificial intelligence and its application to the organization's unique problems. Individual researchers also frequently document their work in blogposts. An adversary may search for posts made by the target v

Search Open AI Vulnerability Analysis

Much like the Search Open Technical Databases, there is often ample research available on the vulnerabilities of common AI models. Once a target has been identified, an adversary will likely try to identify any pre existing work that has been done for this class of models. This will include not only reading academic pa

Acquire Public AI Artifacts

Adversaries may search public sources, including cloud storage, public facing services, and software or data repositories, to identify AI artifacts. These AI artifacts may include the software stack used to train and deploy models, training and testing data, model configurations and parameters. An adversary will be par

Resource Development

Datasets

Adversaries may collect public datasets to use in their operations. Datasets used by the victim organization or datasets that are representative of the data used by the victim organization may be valuable to adversaries. Datasets can be stored in cloud storage, or on victim owned websites. Some datasets require the adv

Resource Development

Models

Adversaries may acquire public models to use in their operations. Adversaries may seek models used by the victim organization or models that are representative of those used by the victim organization. Representative models may include model architectures, or pre trained models which define the architecture as well as

Resource Development

AI Agent Configuration

Adversaries may acquire publicly accessible AI agent configuration files to understand agent capabilities, gain unauthorized access to tools and data sources, or identify credentials for further attacks. Configuration files define what tools an agent can use, credentials for external services, system prompts, and behav

Resource Development

Search Victim-Owned Websites

Adversaries may search websites owned by the victim for information that can be used during targeting. Victim owned websites may contain technical details about their AI enabled products or services. Victim owned websites may contain a variety of details, including names of departments/divisions, physical locations, an

Search Application Repositories

Adversaries may search open application repositories during targeting. Examples of these include Google Play, the iOS App store, the macOS App Store, and the Microsoft Store. Adversaries may craft search queries seeking applications that contain AI enabled components. Frequently, the next step is to Acquire Public AI A

Create Proxy AI Model

Adversaries may obtain models to serve as proxies for the target model in use at the victim organization. Proxy models are used to simulate complete access to the target model in a fully offline manner. Adversaries may train models from representative datasets, attempt to replicate models from victim inference APIs, or

AI Attack Staging

Train Proxy via Gathered AI Artifacts

Proxy models may be trained from AI artifacts (such as data, model architectures, and pre trained models) that are representative of the target model gathered by the adversary. This can be used to develop attacks that require higher levels of access than the adversary has available or as a means to validate pre existin

AI Attack Staging

Train Proxy via Replication

Adversaries may replicate a private model. By repeatedly querying the victim's AI Model Inference API Access, the adversary can collect the target model's inferences into a dataset. The inferences are used as labels for training a separate model offline that will mimic the behavior and performance of the target model.

AI Attack Staging

Use Pre-Trained Model

Adversaries may use an off the shelf pre trained model as a proxy for the victim model to aid in staging the attack.

AI Attack Staging

Active Scanning

An adversary may probe or scan the victim system to gather information for targeting. This is distinct from other reconnaissance techniques that do not involve direct interaction with the victim system. Adversaries may scan for open ports on a potential victim's network, which can indicate specific services or tools th

Discover AI Artifacts

Adversaries may search private sources to identify AI learning artifacts that exist on the system and gather information about them. These artifacts can include the software stack used to train and deploy models, training and testing data management systems, container registries, software repositories, and model zoos.

Acquire Infrastructure

Adversaries may buy, lease, or rent infrastructure for use throughout their operation. A wide variety of infrastructure exists for hosting and orchestrating adversary operations. Infrastructure solutions include physical or cloud servers, domains, mobile devices, and third party web services. Free resources may also be

Resource Development

AI Development Workspaces

Developing and staging AI attacks often requires expensive compute resources. Adversaries may need access to one or many GPUs in order to develop an attack. They may try to anonymously use free resources such as Google Colaboratory, or cloud resources such as AWS, Azure, or Google Cloud as an efficient way to stand up

Resource Development

Consumer Hardware

Adversaries may acquire consumer hardware to conduct their attacks. Owning the hardware provides the adversary with complete control of the environment. These devices can be hard to trace.

Resource Development

Domains

Adversaries may acquire domains that can be used during targeting. Domain names are the human readable names used to represent one or more IP addresses. They can be purchased or, in some cases, acquired for free. Adversaries may use acquired domains for a variety of purposes (see ATT&CK). Large AI datasets are often di

Resource Development

Physical Countermeasures

Adversaries may acquire or manufacture physical countermeasures to aid or support their attack. These components may be used to disrupt or degrade the model, such as adversarial patterns printed on stickers or T shirts, disguises, or decoys. They may also be used to disrupt or degrade the sensors used in capturing data

Resource Development

Serverless

Adversaries may purchase and configure serverless cloud infrastructure, such as Cloudflare Workers, AWS Lambda functions, or Google Apps Scripts, that can be used during targeting. By utilizing serverless infrastructure, adversaries can make it more difficult to attribute infrastructure used during operations back to t

Resource Development

AI Service Proxies

Adversaries may utilize commercial proxy services that resell access to AI services such as frontier model APIs. This infrastructure can be used to conduct large scale campaigns to perform Exfiltration via AI Inference API via distillation. Adversaries may also use this infrastructure to Generate Malicious Commands for

Resource Development

AI Supply Chain Compromise

Adversaries may gain initial access to a system by compromising the unique portions of the AI supply chain. This could include Hardware, Data and its annotations, parts of the AI AI Software stack, or the Model itself. In some instances the attacker will need secondary access to fully carry out an attack using compromi

Hardware

Adversaries may target AI systems by disrupting or manipulating the hardware supply chain. AI models often run on specialized hardware such as GPUs, TPUs, or embedded devices, but may also be optimized to operate on CPUs.

AI Software

Adversaries may target software packages that are commonly used in AI enabled systems or are part of the AI DevOps lifecycle. This can include deep learning frameworks used to build AI models (e.g. PyTorch, TensorFlow, Jax), generative AI integration frameworks (e.g. LangChain, LangFlow), inference engines, and AI DevO

Data

Data is a key vector of supply chain compromise for adversaries. Every AI project will require some form of data. Many rely on large open source datasets that are publicly available. An adversary could rely on compromising these sources of data. The malicious data could be a result of Poison Training Data or include tr

Model

AI enabled systems often rely on open sourced models in various ways. Most commonly, the victim organization may be using these models for fine tuning. These models will be downloaded from an external source and then used as the base for the model as it is tuned on a smaller, private dataset. Loading models often requi

Container Registry

An adversary may compromise a victim's container registry by pushing a manipulated container image and overwriting an existing container name and/or tag. Users of the container registry as well as automated CI/CD pipelines may pull the adversary's container image, compromising their AI Supply Chain. This can affect dev

AI Agent Tool

Adversaries may target AI agent tools as a means to compromise a victim's AI supply chain. Tools add capabilities to AI agents, allowing them to interact with other services, connect to data sources, access internet resources, run system tools, and execute code. They are an attractive target for adversaries because com

User Execution

An adversary may rely upon specific actions by a user in order to gain execution. Users may inadvertently execute unsafe code introduced via AI Supply Chain Compromise. Users may be subjected to social engineering to get them to execute malicious code by, for example, opening a malicious document file or link.

Unsafe AI Artifacts

Adversaries may develop unsafe AI artifacts that when executed have a deleterious effect. The adversary can use this technique to establish persistent access to systems. These models may be introduced via a AI Supply Chain Compromise. Serialization of models is a popular technique for model storage, transfer, and loadi

Malicious Package

Adversaries may develop malicious software packages that when imported by a user have a deleterious effect. Malicious packages may behave as expected to the user. They may be introduced via AI Supply Chain Compromise. They may not present as obviously malicious to the user and may appear to be useful for an AI related

Poisoned AI Agent Tool

A victim may invoke a poisoned tool when interacting with their AI agent. A poisoned tool may execute an LLM Prompt Injection or perform AI Agent Tool Invocation. Poisoned AI agent tools may be introduced into the victim's environment via AI Software, or the user may configure their agent to connect to remote tools.

Malicious Link

An adversary may rely upon a user clicking a malicious link in order to gain execution. Users may be subjected to social engineering to get them to click on a link that will lead to code execution. This user action will typically be observed as follow on behavior from Spearphishing Link. Clicking on a link may also lea

Valid Accounts

Adversaries may obtain and abuse credentials of existing accounts as a means of gaining Initial Access. Credentials may take the form of usernames and passwords of individual user accounts or API keys that provide access to various AI resources and services. Compromised credentials may provide access to additional AI a

Initial AccessPrivilege Escalation

Discover AI Model Ontology

Adversaries may discover the ontology of an AI model's output space, for example, the types of objects a model can detect. The adversary may discovery the ontology by repeated queries to the model, forcing it to enumerate its output space. Or the ontology may be discovered in a configuration file or in documentation ab

Discover AI Model Family

Adversaries may discover the general family of model. General information about the model may be revealed in documentation, or the adversary may use carefully constructed examples and analyze the model's responses to categorize it. Knowledge of the model family can help the adversary identify means of attacking the mod

Evade AI Model

Adversaries can Craft Adversarial Data that prevents an AI model from correctly identifying the contents of the data or Generate Deepfakes that fools an AI model expecting authentic data. This technique can be used to evade a downstream task where AI is utilized. The adversary may evade AI based virus/malware detection

Initial AccessDefense EvasionImpact

Obtain Capabilities

Adversaries may search for and obtain software capabilities for use in their operations. Capabilities may be specific to AI based attacks Adversarial AI Attack Implementations or generic software tools repurposed for malicious intent (Software Tools). In both instances, an adversary may modify or customize the capabili

Resource Development

Adversarial AI Attack Implementations

Adversaries may search for existing open source implementations of AI attacks. The research community often publishes their code for reproducibility and to further future research. Libraries intended for research purposes, such as CleverHans, the Adversarial Robustness Toolbox, and FoolBox, can be weaponized by an adve

Resource Development

Software Tools

Adversaries may search for and obtain software tools to support their operations. Software designed for legitimate use may be repurposed by an adversary for malicious intent. An adversary may modify or customize software tools to achieve their purpose. Software tools used to support attacks on AI systems are not necess

Resource Development

Generative AI

Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Con

Resource Development

Develop Capabilities

Adversaries may develop their own capabilities to support operations. This process encompasses identifying requirements, building solutions, and deploying capabilities. Capabilities used to support attacks on AI enabled systems are not necessarily AI based themselves. Examples include setting up websites with adversari

Resource Development

Adversarial AI Attacks

Adversaries may develop their own adversarial attacks. They may leverage existing libraries as a starting point (Adversarial AI Attack Implementations). They may implement ideas described in public research papers or develop custom made attacks for the victim model.

Resource Development

Manipulate AI Model

Adversaries may directly manipulate an AI model to change its behavior or introduce malicious code. Manipulating a model gives the adversary a persistent change in the system. This can include poisoning the model by changing its weights, modifying the model architecture to change its behavior, and embedding malware whi

AI Attack StagingPersistence

Poison AI Model

Adversaries may manipulate an AI model's weights to change it's behavior or performance, resulting in a poisoned model. Adversaries may poison a model by directly manipulating its weights, training the model on poisoned data, further fine tuning the model, or otherwise interfering with its training process. The change

AI Attack StagingPersistence

Modify AI Model Architecture

Adversaries may directly modify an AI model's architecture to re define it's behavior. This can include adding or removing layers as well as adding pre or post processing operations. The effects could include removing the ability to predict certain classes, adding erroneous operations to increase computation costs, or

AI Attack StagingPersistence

Embed Malware

Adversaries may embed malicious code into AI Model files. AI models may be packaged as a combination of instructions and weights. Some formats such as pickle files are unsafe to deserialize because they can contain unsafe calls such as exec. Models with embedded malware may still operate as expected. It may allow them

AI Attack StagingPersistence

Publish Poisoned Datasets

Adversaries may Poison Training Data and publish it to a public location. The poisoned dataset may be a novel dataset or a poisoned variant of an existing open source dataset. This data may be introduced to a victim system via AI Supply Chain Compromise.

Resource Development

Poison Training Data

Adversaries may attempt to poison datasets used by an AI model by modifying the underlying data or its labels. This allows the adversary to embed vulnerabilities in AI models trained on the data that may not be easily detectable. Data poisoning attacks may or may not require modifying the labels. The embedded vulnerabi

Resource DevelopmentPersistence

Establish Accounts

Adversaries may create accounts with various services for use in targeting, to gain access to resources needed in AI Attack Staging, or for victim impersonation.

Resource Development

Exfiltration via AI Inference API

Adversaries may exfiltrate private information via AI Model Inference API Access. AI Models have been shown leak private information about their training data (e.g. Infer Training Data Membership, Invert AI Model). The model itself may also be extracted (Extract AI Model) for the purposes of AI Intellectual Property Th

Infer Training Data Membership

Adversaries may infer the membership of a data sample or global characteristics of the data in its training set, which raises privacy concerns. Some strategies make use of a shadow model that could be obtained via Train Proxy via Replication, others use statistics of model prediction scores. This can cause the victim m

Invert AI Model

AI models' training data could be reconstructed by exploiting the confidence scores that are available via an inference API. By querying the inference API strategically, adversaries can back out potentially private information embedded within the training data. This could lead to privacy violations if the attacker can

Extract AI Model

Adversaries may extract a functional copy of a private model. By repeatedly querying the victim's AI Model Inference API Access, the adversary can collect the target model's inferences into a dataset. The inferences are used as labels for training a separate model offline that will mimic the behavior and performance of

Exfiltration via Cyber Means

Adversaries may exfiltrate AI artifacts or other information relevant to their goals via traditional cyber means. See the ATT&CK Exfiltration tactic for more information.

Denial of AI Service

Adversaries may target AI enabled systems with a flood of requests for the purpose of degrading or shutting down the service. Since many AI systems require significant amounts of specialized compute, they are often expensive bottlenecks that can become overloaded. Adversaries can intentionally craft inputs that require

Erode AI Model Integrity

Adversaries may degrade the target model's performance with adversarial data inputs to erode confidence in the system over time. This can lead to the victim organization wasting time and money both attempting to fix the system and performing the tasks it was meant to automate by hand.

Cost Harvesting

Adversaries may deliberately drive a victim's AI services beyond normal operating capacity with the intent of increasing the cost of services. This may be achieved via high volume, low complexity queries (Excessive Queries) or low volume, high complexity queries (Resource Intensive Queries). In Generative AI or Agentic

Excessive Queries

Adversaries may send an excessive number of otherwise normal or low complexity queries to an AI system with the goal of overwhelming its capacity and increasing operating costs. The attacker can automate high volume request generation, exploiting rate limits, autoscaling policies, and pay per use billing models to driv

Resource-Intensive Queries

Adversaries may craft inputs specifically designed to increase the compute resources required for processing. For generative AI models, adversaries may use long input sequences, requests for extremely long outputs, or prompts that require complex reasoning as strategies for increasing compute costs OWASP Top 10 for LLM

Agentic Resource Consumption

Adversaries may coerce an agentic AI system into performing computationally expensive tool calls that waste resources and consume API budgets. They may utilize LLM Prompt Injection or AI Agent Tool Data Poisoning with directives that push the agent to perform unnecessary API queries, excessive query fan outs, or many d

AI Artifact Collection

Adversaries may collect AI artifacts for Exfiltration or for use in AI Attack Staging. AI artifacts include models and datasets as well as other telemetry data produced when interacting with a model.

Data from Information Repositories

Adversaries may leverage information repositories to mine valuable information. Information repositories are tools that allow for storage of information, typically to facilitate collaboration or information sharing between users, and can store a wide variety of data that may aid adversaries in further objectives, or di

Data from Local System

Adversaries may search local system sources, such as file systems and configuration files or local databases, to find files of interest and sensitive data prior to Exfiltration. This can include basic fingerprinting information and sensitive data such as ssh keys.

AI Model Inference API Access

Adversaries may gain access to a model via legitimate access to the inference API. Inference API access can be a source of information to the adversary (Discover AI Model Ontology, Discover AI Model Family), a means of staging the attack (Verify Attack, Craft Adversarial Data), or for introducing data to the target sys

AI Model Access

Physical Environment Access

In addition to the attacks that take place purely in the digital domain, adversaries may also exploit the physical environment for their attacks. If the model is interacting with data collected from the real world in some way, the adversary can influence the model through access to wherever the data is being collected.

AI Model Access

Verify Attack

Adversaries can verify the efficacy of their attack via an inference API or access to an offline copy of the target model. This gives the adversary confidence that their approach works and allows them to carry out the attack at a later time of their choosing. The adversary may verify the attack once but use it against

AI Attack Staging

Craft Adversarial Data

Adversarial data are inputs to an AI model that have been modified such that they cause the adversary's desired effect in the target model. Effects can range from misclassification, to missed detections, to maximizing energy consumption. Typically, the modification is constrained in magnitude or location so that a huma

AI Attack Staging

White-Box Optimization

In White Box Optimization, the adversary has full access to the target model and optimizes the adversarial example directly. Adversarial examples trained in this manner are most effective against the target model.

AI Attack Staging

Black-Box Optimization

In Black Box attacks, the adversary has black box (i.e. AI Model Inference API Access via API access) access to the target model. With black box attacks, the adversary may be using an API that the victim is monitoring. These attacks are generally less effective and require more inferences than White Box Optimization at

AI Attack Staging

Black-Box Transfer

In Black Box Transfer attacks, the adversary uses one or more proxy models (trained via Create Proxy AI Model or Train Proxy via Replication) they have full access to and are representative of the target model. The adversary uses White Box Optimization on the proxy models to generate adversarial examples. If the set of

AI Attack Staging

Manual Modification

Adversaries may manually modify the input data to craft adversarial data. They may use their knowledge of the target model to modify parts of the data they suspect helps the model in performing its task. The adversary may use trial and error until they are able to verify they have a working adversarial input.

AI Attack Staging

Insert Backdoor Trigger

The adversary may add a perceptual trigger into inference data. The trigger may be imperceptible or non obvious to humans. This technique is used in conjunction with Poison AI Model and allows the adversary to produce their desired effect in the target model.

AI Attack Staging

Full AI Model Access

Adversaries may gain full "white box" access to an AI model. This means the adversary has complete knowledge of the model architecture, its parameters, and class ontology. They may exfiltrate the model to Craft Adversarial Data and Verify Attack in an offline where it is hard to detect their behavior.

AI Model Access

Spamming AI System with Chaff Data

Adversaries may spam the AI system with chaff data that causes increase in the number of detections. This can cause analysts at the victim organization to waste time reviewing and correcting incorrect inferences. Adversaries may also spam AI agents with excessive low severity auditable events or agentic actions that re

AI-Enabled Product or Service

Adversaries may use a product or service that uses artificial intelligence under the hood to gain access to the underlying AI model. This type of indirect model access may reveal details of the AI model or its inferences in logs or metadata.

AI Model Access

External Harms

Adversaries may abuse their access to a victim system and use its resources or capabilities to further their goals by causing harms external to that system. These harms could affect the organization (e.g. Financial Harm, Reputational Harm), its users (e.g. User Harm), or the general public (e.g. Societal Harm).

Financial Harm

Financial harm involves the loss of wealth, property, or other monetary assets due to theft, fraud or forgery, or pressure to provide financial resources to the adversary.

Reputational Harm

Reputational harm involves a degradation of public perception and trust in organizations. Examples of reputation harming incidents include scandals or false impersonations.

Societal Harm

Societal harms might generate harmful outcomes that reach either the general public or specific vulnerable groups such as the exposure of children to vulgar content.

User Harm

User harms may encompass a variety of harm types including financial and reputational that are directed at or felt by individual victims of the attack rather than at the organization level.

AI Intellectual Property Theft

Adversaries may exfiltrate AI artifacts to steal intellectual property and cause economic harm to the victim organization. Proprietary training data is costly to collect and annotate and may be a target for Exfiltration and theft. AIaaS providers charge for use of their API. An adversary who has stolen a model via Exfi

Exploit Public-Facing Application

Adversaries may attempt to take advantage of a weakness in an Internet facing computer or program using software, data, or commands in order to cause unintended or unanticipated behavior. The weakness in the system can be a bug, a glitch, or a design vulnerability. These applications are often websites, but can include

Command and Scripting Interpreter

Adversaries may abuse command and script interpreters to execute commands, scripts, or binaries. These interfaces and languages provide ways of interacting with computer systems and are a common feature across many different platforms. Most systems come with some built in command line interface and scripting capabiliti

LLM Prompt Injection

An adversary may craft malicious prompts as inputs to an LLM that cause the LLM to act in unintended ways. These "prompt injections" are often designed to cause the model to ignore aspects of its original instructions and follow the adversary's instructions instead. Prompt Injections can be an initial access vector to

Direct

An adversary may inject prompts directly as a user of the LLM. This type of injection may be used by the adversary to gain a foothold in the system or to misuse the LLM itself, as for example to generate harmful content.

Indirect

An adversary may inject prompts indirectly via separate data channel ingested by the LLM such as include text or multimedia pulled from databases or websites. These malicious prompts may be hidden or obfuscated from the user. This type of injection may be used by the adversary to gain a foothold in the system or to tar

Triggered

An adversary may trigger a prompt injection via a user action or event that occurs within the victim's environment. Triggered prompt injections often target AI agents, which can be activated by means the adversary identifies during Discovery (See Activation Triggers). These malicious prompts may be hidden or obfuscated

Phishing

Adversaries may send phishing messages to gain access to victim systems. All forms of phishing are electronically delivered social engineering. Phishing can be targeted, known as spearphishing. In spearphishing, a specific individual, company, or industry will be targeted by the adversary. More generally, adversaries c

Initial AccessLateral Movement

Spearphishing via Social Engineering LLM

Adversaries may turn LLMs into targeted social engineers. LLMs are capable of interacting with users via text conversations. They can be instructed by an adversary to seek sensitive information from a user and act as effective social engineers. They can be targeted towards particular personas defined by the adversary.

Initial AccessLateral Movement

Deepfake-Assisted Phishing

Adversaries may use deepfakes (AI generated synthetic images, audio, or video) in phishing campaigns to impersonate trusted individuals, executives, or organizations. These attacks exploit human trust by presenting fraudulent voice or video communications as legitimate, enabling adversaries to manipulate targets into d

Initial AccessLateral Movement

AI Agent Tool Invocation

Adversaries may use their access to an AI agent to invoke tools the agent has access to. LLMs are often connected to other services or resources via tools to increase their capabilities. Tools may include integrations with other applications, access to public or private data sources, and the ability to execute code. Th

ExecutionPrivilege Escalation

LLM Jailbreak

Adversaries may induce a large language model (LLM) to ignore, circumvent, or override its safety/alignment behaviors and/or guardrails to elicit outputs the model is intended to withhold. Once jailbroken, the LLM may be used in unintended ways by the adversary. Jailbreaks may be achieved via adversarial prompting, or

Defense EvasionPrivilege Escalation

Unsecured Credentials

Adversaries may search compromised systems to find and obtain insecurely stored credentials. These credentials can be stored and/or misplaced in many locations on a system, including plaintext files (e.g. bash history), environment variables, operating system, or application specific repositories (e.g. Credentials in R

Credential Access

Extract LLM System Prompt

Adversaries may attempt to extract a large language model's (LLM) system prompt. This can be done via prompt injection to induce the model to reveal its own system prompt or may be extracted from a configuration file. System prompts can be a portion of an AI provider's competitive advantage and are thus valuable intell

LLM Data Leakage

Adversaries may craft prompts that induce the LLM to leak sensitive information. This can include private user data or proprietary information. The leaked information may come from proprietary training data, data sources the LLM is connected to, or information from other users of the LLM.

Publish Poisoned Models

Adversaries may publish a poisoned model to a public location such as a model registry or code repository. The poisoned model may be a novel model or a poisoned variant of an existing open source model. This model may be introduced to a victim system via AI Supply Chain Compromise.

Resource Development

Erode Dataset Integrity

Adversaries may poison or manipulate portions of a dataset to reduce its usefulness, reduce trust, and cause users to waste resources correcting errors.

Publish Hallucinated Entities

Adversaries may create an entity they control, such as a software package, website, or email address to a source hallucinated by an LLM. The hallucinations may take the form of package names commands, URLs, company names, or email addresses that point the victim to the entity controlled by the adversary. When the victi

Resource Development

LLM Prompt Self-Replication

An adversary may use a carefully crafted LLM Prompt Injection designed to cause the LLM to replicate the prompt as part of its output. This allows the prompt to propagate to other LLMs and persist on the system. The self replicating prompt is typically paired with other malicious instructions (ex: LLM Jailbreak, LLM Da

Discover LLM Hallucinations

Adversaries may prompt large language models and identify hallucinated entities. They may request software packages, commands, URLs, organization names, or e mail addresses, and identify hallucinations with no connected real world source. Discovered hallucinations provide the adversary with potential targets to Publish

Discover AI Model Outputs

Adversaries may discover model outputs, such as class scores, whose presence is not required for the system to function and are not intended for use by the end user. Model outputs may be found in logs or may be included in API responses. Model outputs may enable the adversary to identify weaknesses in the model and dev

Gather RAG-Indexed Targets

Adversaries may identify data sources used in retrieval augmented generation (RAG) systems for targeting purposes. By pinpointing these sources, attackers can focus on poisoning or otherwise manipulating the external data repositories the AI relies on. RAG indexed data may be identified in public documentation about th

LLM Prompt Crafting

Adversaries may use their acquired knowledge of the target generative AI system to craft prompts that bypass its defenses and allow malicious instructions to be executed. The adversary may iterate on the prompt to ensure that it works as intended consistently.

Resource Development

Retrieval Content Crafting

Adversaries may write content designed to be retrieved by user queries and influence a user of the system in some way. This abuses the trust the user has in the system. The crafted content can be combined with a prompt injection. It can also stand alone in a separate document or email. The adversary must get the crafte

Resource Development

LLM Trusted Output Components Manipulation

Adversaries may utilize prompts to a large language model (LLM) which manipulate various components of its response in order to make it appear trustworthy to the user. This helps the adversary continue to operate in the victim's environment and evade detection by the users it interacts with. The LLM may be instructed t

Defense Evasion

Citations

Adversaries may manipulate the citations provided in an AI system's response, in order to make it appear trustworthy. Variants include citing a providing the wrong citation, making up a new citation, or providing the right citation but for adversary provided data.

Defense Evasion

LLM Prompt Obfuscation

Adversaries may hide or otherwise obfuscate prompt injections or retrieval content to avoid detection from humans, large language model (LLM) guardrails, or other detection mechanisms. For text inputs, this may include modifying how the instructions are rendered such as small text, text colored the same as the backgrou

Defense Evasion

Discover LLM System Information

The adversary is trying to discover something about the large language model's (LLM) system information. This may be found in a configuration file containing the system instructions or extracted via interactions with the LLM. The desired information may include the full system prompt, special characters that have signi

Special Character Sets

Adversaries may discover delimiters and special characters sets used by the large language model. For example, delimiters used in retrieval augmented generation applications to differentiate between context and user prompts. These can later be exploited to confuse or manipulate the large language model into misbehaving

System Instruction Keywords

Adversaries may discover keywords that have special meaning to the large language model (LLM), such as function names or object names. These can later be exploited to confuse or manipulate the LLM into misbehaving and to make calls to plugins the LLM has access to.

System Prompt

Adversaries may discover a large language model's system instructions provided by the AI system builder to learn about the system's capabilities and circumvent its guardrails.

RAG Poisoning

Adversaries may inject malicious content into data indexed by a retrieval augmented generation (RAG) system to contaminate a future thread through RAG based search results. This may be accomplished by placing manipulated documents in a location the RAG indexes (see Gather RAG Indexed Targets). The content may be target

False RAG Entry Injection

Adversaries may introduce false entries into a victim's retrieval augmented generation (RAG) database. Content designed to be interpreted as a document by the large language model (LLM) used in the RAG system is included in a data source being ingested into the RAG database. When RAG entry including the false document

Defense Evasion

Reverse Shell

Adversaries may utilize a reverse shell to communicate and control the victim system. Typically, a user uses a client to connect to a remote machine which is listening for connections. With a reverse shell, the adversary is listening for incoming connections initiated from the victim system.

Command and Control

Impersonation

Adversaries may impersonate a trusted person or organization in order to persuade and trick a target into performing some action on their behalf. For example, adversaries may communicate with victims (via Phishing, or Spearphishing via Social Engineering LLM) while impersonating a known sender such as an executive, col

Defense Evasion

Masquerading

Adversaries may attempt to manipulate features of their artifacts to make them appear legitimate or benign to users and/or security tools. Masquerading occurs when the name or location of an object, legitimate or malicious, is manipulated or abused for the sake of evading defenses and observation. This may include mani

Defense Evasion

Cloud Service Discovery

Adversaries may attempt to enumerate the cloud services running on a system after gaining access. These methods can differ from platform as a service (PaaS), to infrastructure as a service (IaaS), software as a service (SaaS), or AI as a service (AIaaS). Many services exist throughout the various cloud providers and ca

Corrupt AI Model

An adversary may purposefully corrupt a malicious AI model file so that it cannot be successfully deserialized in order to evade detection by a model scanner. The corrupt model may still successfully execute malicious code before deserialization fails.

Defense Evasion

LLM Response Rendering

An adversary may get a large language model (LLM) to respond with private information that is hidden from the user when the response is rendered by the user's client. The private information is then exfiltrated. This can take the form of rendered images, which automatically make a request to an adversary controlled ser

Drive-by Compromise

Adversaries may gain access to an AI system through a user visiting a website over the normal course of browsing, or an AI agent retrieving information from the web on behalf of a user. Websites can contain an LLM Prompt Injection which, when executed, can change the behavior of the AI model. The same approach may be u

Stage Capabilities

Adversaries may upload, install, or otherwise set up capabilities that can be used during targeting. To support their operations, an adversary may need to take capabilities they developed (Develop Capabilities) or obtained (Obtain Capabilities) and stage them on infrastructure under their control. These capabilities ma

Resource Development

AI Agent Context Poisoning

Adversaries may attempt to manipulate the context used by an AI agent's large language model (LLM) to influence the responses it generates or actions it takes. This allows an adversary to persistently change the behavior of the target agent and further their goals. Context poisoning can be accomplished by prompting the

Memory

Adversaries may manipulate the memory of a large language model (LLM) in order to persist changes to the LLM to future chat sessions. Memory is a common feature in LLMs that allows them to remember information across chat sessions by utilizing a user specific database. Because the memory is controlled via normal conver

Thread

Adversaries may introduce malicious instructions into a chat thread of a large language model (LLM) to cause behavior changes which persist for the remainder of the thread. A chat thread may continue for an extended period over multiple sessions. The malicious instructions may be introduced via Direct or Indirect Promp

Modify AI Agent Configuration

Adversaries may modify the configuration files for AI agents on a system. This allows malicious changes to persist beyond the life of a single agent and affects any agents that share the configuration. Configuration changes may include modifications to the system prompt, tampering with or replacing knowledge sources, m

PersistenceDefense Evasion

RAG Credential Harvesting

Adversaries may attempt to use their access to a large language model (LLM) on the victim's system to collect credentials. Credentials may be stored in internal documents which can inadvertently be ingested into a RAG database, where they can ultimately be retrieved by an AI agent.

Credential Access

Credentials from AI Agent Configuration

Adversaries may access the credentials of other tools or services on a system from the configuration of an AI agent. AI Agents often utilize external tools or services to take actions, such as querying databases, invoking APIs, or interacting with cloud resources. To enable these functions, credentials like API keys, t

Credential Access

Discover AI Agent Configuration

Adversaries may attempt to discover configuration information for AI agents present on the victim's system. Agent configurations can include tools or services they have access to. Adversaries may directly access agent configuring dashboards or configuration files. They may also obtain configuration details by prompting

Embedded Knowledge

Adversaries may attempt to discover the data sources a particular agent can access. The AI agent's configuration may reveal data sources or knowledge. The embedded knowledge may include sensitive or proprietary material such as intellectual property, customer data, internal policies, or even credentials. By mapping wha

Tool Definitions

Adversaries may discover the tools the AI agent has access to. By identifying which tools are available, the adversary can understand what actions may be executed through the agent and what additional resources it can reach. This knowledge may reveal access to external data sources such as OneDrive or SharePoint, or ex

Activation Triggers

Adversaries may discover keywords or other triggers (such as incoming emails, documents being added, incoming message, or other workflows) that activate an agent and may cause it to run additional actions. Understanding these triggers can reveal how the AI agent is activated and controlled. This may also expose additio

Call Chains

Adversaries may extract call chains from AI agent configurations, which can reveal potentially targets for remote code execution (RCE) or other vulnerabilities. Vulnerable call chains often connect user inputs or LLM outputs to an execution sink (e.g. exec, eval, os.popen). The vulnerabilities may be later exploited vi

Data from AI Services

Adversaries may use their access to a victim organization's AI enabled services to collect proprietary or otherwise sensitive information. As organizations adopt generative AI in centralized services for accessing an organization's data, such as with chat agents which can access retrieval augmented generation (RAG) dat

RAG Databases

Adversaries may prompt the AI service to retrieve data from a RAG database. This can include the majority of an organization's internal documents.

AI Agent Tools

Adversaries may prompt the AI service to invoke various tools the agent has access to. Tools may retrieve data from different APIs or services in an organization.

Exfiltration via AI Agent Tool Invocation

AI agent tools capable of performing write operations may be invoked to exfiltrate data to an adversary. Sensitive information can be encoded into the tool's input parameters and transmitted to an adversary controlled location (such as an inbox, document, or server) as part of a seemingly legitimate action. Variants in

Gather Victim Identity Information

Adversaries may gather information about the victim's identity that can be used during targeting. Information about identities may include a variety of details, including personal data (ex: employee names, email addresses, photos, etc.) as well as sensitive details such as credentials or multi factor authentication (MF

Generate Deepfakes

Adversaries may use generative artificial intelligence (GenAI) to create synthetic media (i.e. imagery, video, audio, and text) that appear authentic. These "deepfakes" may mimic a real person or depict fictional personas. Adversaries may use deepfakes for impersonation to conduct Phishing or to evade AI applications s

AI Attack Staging

Process Discovery

Adversaries may attempt to get information about processes running on a system. Once obtained, this information could be used to gain an understanding of common AI related software/applications running on systems within the network. Administrator or otherwise elevated access may provide better process details. Identify

OS Credential Dumping

Adversaries may extract credentials from OS caches, application memory, or other sources on a compromised system. Credentials are often in the form of a hash or clear text, and can include usernames and passwords, application tokens, or other authentication keys. Credentials can be used to perform Lateral Movement to a

Credential Access

Use Alternate Authentication Material

Adversaries may use alternate authentication material, such as password hashes, Kerberos tickets, and application access tokens, in order to move laterally within an environment and bypass normal system access controls. AI services commonly use alternate authentication material as a primary means for users to make quer

Lateral Movement

Application Access Token

Adversaries may use stolen application access tokens to bypass the typical authentication process and access restricted accounts, information, or services on remote systems. These tokens are typically stolen from users or services and used in lieu of login credentials. Application access tokens are used to make authori

Lateral Movement

Manipulate User LLM Chat History

Adversaries may manipulate a user's large language model (LLM) chat history to cover the tracks of their malicious behavior. They may hide persistent changes they have made to the LLM's behavior, or obscure their attempts at discovering private information about the user. To do so, adversaries may delete or edit existi

Defense Evasion

Prompt Infiltration via Public-Facing Application

An adversary may introduce malicious prompts into the victim's system via a public facing application with the intention of it being ingested by an AI at some point in the future and ultimately having a downstream effect. This may occur when a data source is indexed by a retrieval augmented generation (RAG) system, whe

Initial AccessPersistence

Delay Execution of LLM Instructions

Adversaries may include instructions to be followed by the AI system in response to a future event, such as a specific keyword or the next interaction, in order to evade detection or bypass controls placed on the AI system. For example, an adversary may include "If the user submits a new request..." followed by the mal

Defense Evasion

Search Open Websites/Domains

Adversaries may search public websites and/or domains for information about victims that can be used during targeting. Information about victims may be available in various online sites, such as social media, new sites, or domains owned by the victim. Adversaries may find the information they seek to gather via search

Code Repositories

Adversaries may search public code repositories for information about a victim or victim system that can be used during targeting. Victims may store code or artifacts related to their AI systems in repositories on various third party websites such as GitHub, GitLab, SourceForge, and BitBucket. Adversaries may search co

AI Service API

Adversaries may communicate using the API of an AI service on the victim's system. The adversary's commands to the victim system, and often the results, are embedded in the normal traffic of the AI service. An AI service API command and control channel is covert because the adversary's commands blend in with normal com

Command and Control

Virtualization/Sandbox Evasion

Adversaries may employ various means to detect and avoid virtualization and analysis environments. This may include changing behaviors based on the results of checks for the presence of artifacts indicative of a virtual machine environment (VME) or sandbox. If the adversary detects a VME, they may alter their malware t

Defense Evasion

AI Agent Tool Credential Harvesting

Adversaries may attempt to use their access to an AI agent on the victim's system to retrieve data from available agent tools to collect credentials. Agent tools may connect to a wide range of sources that may contain credentials including document stores (e.g. SharePoint, OneDrive or Google Drive), code repositories (

Credential Access

AI Agent Tool Data Poisoning

Adversaries may place malicious content on a victim's system where it can be retrieved by an AI Agent Tool. This may be accomplished by placing documents in a location that will be ingested by a service the AI agent has associated tools for. The content may be targeted such that it would often be retrieved by common qu

AI Agent Clickbait

Adversaries may craft deceptive web content designed to bait Computer Using AI agents or AI web browsers into taking unintended actions, such as clicking buttons, copying code, or navigating to specific web pages. These attacks exploit the agent's interpretation of UI content, visual cues, or prompt like language embed

Data Destruction via AI Agent Tool Invocation

Adversaries may invoke an AI agent's tool capable of performing mutative operations to perform Data Destruction. Adversaries may destroy data and files on specific systems or in large numbers on a network to interrupt availability to systems, services, and network resources.

Generate Malicious Commands

Adversaries may use large language models (LLMs) to dynamically generate malicious commands from natural language. Dynamically generated commands may be harder detect as the attack signature is constantly changing. AI generated commands may also allow adversaries to more rapidly adapt to different environments and adju

AI Attack Staging

Deploy AI Agent

Adversaries may launch AI agents in the victim's environment to execute actions on their behalf. AI agents may have access to a wide range of tools and data sources, as well as permissions to access and interact with other services and systems in the victim's environment. The adversary may leverage these capabilities t

Publish Poisoned AI Agent Tool

Adversaries may create and publish poisoned AI agent tools. Poisoned tools may contain an LLM Prompt Injection, which can lead to a variety of impacts. Tools may be published to open source version control repositories (e.g. GitHub, GitLab), to package registries (e.g. npm), or to repositories specifically designed for

Resource Development

Escape to Host

Adversaries may break out of a container or virtualized environment to gain access to the underlying host. This can allow an adversary access to other containerized or virtualized resources from the host level or to the host itself. In principle, containerized / virtualized resources should provide a clear separation o

Privilege Escalation

Exploitation for Credential Access

Adversaries may exploit software vulnerabilities in an attempt to collect credentials. Exploitation of a software vulnerability occurs when an adversary takes advantage of a programming error in a program, service, or within the operating system software or kernel itself to execute adversary controlled code.

Credential Access

Exploitation for Defense Evasion

Adversaries may exploit a system or application vulnerability to bypass security features. Exploitation of a vulnerability occurs when an adversary takes advantage of a programming error in a program, service, or within the operating system software or kernel itself to execute adversary controlled code. Vulnerabilities

Defense Evasion

AI Agent

Adversaries may abuse AI agents present on the victim's system for command and control. AI agents are often granted access to tools that can execute shell commands, reach out to the internet, and interact with other services in the victim's environment, making them capable C2 agents. The adversary may modify the behavi

Command and Control

AI Supply Chain Rug Pull

Adversaries may publish legitimate AI components or software, gain user adoption, then push an update with a malicious variant, leading to AI Supply Chain Compromise. More scrutiny is often placed on a supply chain dependency when it is first being considered for inclusion in an AI system. Performing a rug pull may all

Defense Evasion

AI Agent Tool Poisoning

Adversaries may achieve persistence by poisoning tools used by AI agents including built in tools or tools available to the agent via Model Context Protocol (MCP) connections. This involves compromising benign tools already integrated into the agent's environment. By altering tool behavior such as modifying parameters

AI Supply Chain Reputation Inflation

AI Supply Chain Reputation Inflation is the process of building or leveraging genuinely credible looking trust signals to increase the perceived legitimacy of AI supply chain components, with the goal of driving adoption of malicious or compromised assets. Adversaries use established developer accounts with a history o

Defense Evasion

Machine Compromise

Adversaries may compromise a machine by exploiting or manipulating AI enabled components on the system. Compromising a victim system allows the adversary to execute arbitrary code, steal credentials, exfiltrate data, and continue to persist on the system. Adversaries may target a Local AI Agent which if compromised gra

Local AI Agent

Adversaries may achieve full system compromise by abusing AI agents running locally on a host, such as computer use agents or AI driven browsers. These agents are designed to autonomously interact with the operating system, applications, and external services, often with broad permissions to execute commands, access fi

AI Artifacts

Adversaries may achieve full system compromise by introducing malicious AI artifacts, such as models or data, that contain embedded malware or other malicious commands. AI artifacts are often stored in model registries or data stores and may affect many systems that pull these resources. Malicious content stored in AI