Generative AI

Overview

Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Content Crafting, or to generate Phishing content.

Adversaries may obtain open source models and serve them locally using frameworks such as Ollama or vLLM. They may host them using cloud infrastructure. Or, they may leverage AI service providers such as HuggingFace.

They may need to jailbreak the model (see LLM Jailbreak) to bypass any restrictions put in place to limit the types of responses it can generate. They may also need to break the terms of service of the model's developer.

Generative AI models may also be "uncensored" meaning they are designed to generate content without any restrictions such as guardrails or content filters. Uncensored GenAI is ripe for abuse by cybercriminals Cybercriminal abuse of large language models Cybercriminals Exploit LLM Models to Enhance Hacking Activities. Models may be fine-tuned to remove alignment and guardrails erichartford or be subjected to targeted manipulations to bypass refusal [2406.11717] Refusal in Language Models Is Mediated by a Single Direction resulting in uncensored variants of the model. Uncensored models may be built for offensive and defensive cybersecurity TAICO | WhiteRabbitNeo: An Uncensored, Open Source AI Model for Red & Blue Team Cybersecurity, which can be abused by an adversary. There are also models that are expressly designed and advertised for malicious use BlackHat AI Tool WormGPT Enhanced with Grok and Mixtral.