The advent of generative AI - particularly Large Language Models (LLMs) - marks a paradigm shift in how we must approach security and privacy. As security professionals, we can leverage AI to enhance and augment our defenses, but we must also recognize the new attack surfaces that AI introduces.
Unlike traditional applications that are designed for a specific function or data set, AI-powered applications - such as chatbots - are more general by nature. Users might enter anything from personal questions to highly sensitive data, increasing the potential for inadvertent data leakage. Although it is possible to build highly specialized AI applications, the inherent flexibility of LLM-based systems often exposes them to more varied and unpredictable input.
Before diving into how these new AI applications can be attacked and defended, let’s examine a headline-making example of where things went wrong: DeepSeek.
DeepSeek: A Cautionary Tale
DeepSeek has been front-and-center in recent news for a variety of reasons, mostly unfavorable. If you haven’t heard of it, here’s the gist: DeepSeek is a Chinese AI company offering both the DeepSeek chat application and several openly available models (e.g., on Hugging Face).
OpenAI has accused DeepSeek of using unethical practices - specifically model distillation - based on OpenAI’s own models. More pertinent to this discussion, DeepSeek has experienced severe security lapses resulting in a potential data leak. An unprotected database exposed millions of chat records and sensitive user data. Furthermore, a prompt-injection vulnerability (“jailbreak”) revealed the application’s system prompt, which can disclose sensitive information such as API keys, internal endpoints, and other confidential data.
Why It Matters
In any application, a database breach is a grave issue. In a popular AI chat application, it’s catastrophic. Chat histories can contain extremely sensitive information—especially in corporate contexts where employees may (often unwittingly) provide confidential data or intellectual property. The DeepSeek case is a stark reminder that when it comes to AI applications, sensitive data is at even greater risk.
There are also geopolitical dimensions—DeepSeek being a Chinese company with servers in China raises additional concerns about government scrutiny. While that may be beyond the scope of this post, the DeepSeek incident is instructive for understanding today’s AI security landscape. Let’s use it as a case study in what not to do and how to protect yourself if you’re developing an AI-based application. After all, nobody wants to be the next DeepSeek.
Using AI Securely
Generative AI transforms the defense side of security as much as the attack surface. It introduces new risks, from data exfiltration and prompt injection to malicious model manipulation. Here are some considerations for building secure AI applications.
Data First
Let’s assume you are developing an AI-powered chat application - call it SeepDeek - that allows users to chat with a model about their medical records. Because this application processes highly sensitive information, you must address:
- Data Sensitivity
- What type of data is being processed?
- How sensitive is it?
- Data Flow
- How does data move through your system?
- When and how does the model get access?
- Access Controls
- Who (or what) is allowed to see which data, and under what conditions?
- How are authentication and authorization enforced?
In this hypothetical example, SeepDeek can query medical records from another system. If the model is responsible for retrieving user data, you must ensure robust mechanisms that prevent it from accessing records of users who are not the requester. One approach is to forward the authenticated user’s credentials to an internal API, rather than letting the model itself handle authentication or authorization.
Treat the Model Like an Untrusted Actor
An LLM can be seen as an AI user or actor within your system. It might exhibit “intelligent” behavior (albeit artificially), but it can also be manipulated—or simply make mistakes. With user input at scale, even small errors can become significant threats.
This means building your surrounding infrastructure with the assumption that the model:
- Could act maliciously if prompted or exploited.
- Will sometimes behave erratically or “stupidly,” requiring guardrails not just in the model itself, but also designed into the application logic to prevent data leakage or unauthorized access.
The Model as Part of Your Supply Chain
LLMs are usually trained on massive datasets scraped from the internet, and are sometimes further fine-tuned for specific tasks. Because you likely won’t be training your own LLM from scratch (a costly and complex endeavor), the model itself becomes part of your software supply chain. As such, you should ask:
- Do I trust the company that produced this model?
- Are there assurances the model was trained securely and ethically?
- Is there a possibility the model is backdoored or biased?
By treating the model as you would any other third-party software component, you ensure that supply chain security considerations—such as verifying authenticity and integrity—are taken into account.
AI Attack Surface
To summarize, here are broad categories of AI-specific threats:
- Prompt Injection & Jailbreaking
Attackers exploit LLM prompts or system instructions to bypass safeguards, potentially exposing sensitive data or hijacking the model’s behavior. - Unprotected Data Pipelines
Sensitive data can leak through logs, backups, or unsecured APIs if not properly encrypted and access-controlled. - Model Manipulation & Poisoning
If attackers gain access to model-training processes or fine-tuning pipelines, they could embed backdoors or bias into the model. - Supply Chain Vulnerabilities
Third-party models, data, or fine-tuning providers may introduce hidden threats if not rigorously vetted. - Infrastructure Misconfigurations
As with any modern application, misconfigured servers, publicly exposed databases, and inadequate network security can lead to massive breaches—DeepSeek being a prime example.
This of course doesn’t fully cover the spectrum of possible security issues. The OWASP AI top 10 does a pretty great job of covering a lot of the potential vulnerabilities that AI systems can create and expose.
Using AI for Security Good
While AI certainly adds new risks, it also offers powerful tools that benefit both offensive (penetration testing) and defensive (blue team) security practices. Here’s how AI can be harnessed as a force for good in cybersecurity:
- Enhanced Vulnerability Discovery
- Automated Reconnaissance: LLM-powered tools can analyze massive data sets—from open-source intelligence to network logs—to highlight anomalies or potential weak points.
- Pattern Recognition: Machine learning algorithms excel at detecting unusual behaviors or coding patterns in applications, helping penetration testers uncover hidden vulnerabilities.
- Faster Exploit Development and Testing
- Code Generation: Tools like ChatGPT can accelerate the creation of proof-of-concept exploits or scripts, allowing penetration testers to focus more on strategy rather than boilerplate code.
- Automated Testing: AI-driven frameworks can simulate various attack vectors more rapidly, providing real-time feedback on an application’s resilience.
- Improved Incident Detection and Response
- Threat Intelligence: Blue teams can leverage AI models to quickly parse threat feeds and correlate events across multiple data sources. This automation can drastically reduce detection time and false positives.
- Adaptive Defenses: By learning from incoming threats, AI-driven security platforms can adapt firewall rules or endpoint policies on the fly, continuously improving defense against evolving tactics.
- Adversarial Testing
- AI vs. AI: Security teams can pit their defensive AI models against adversarial AI scripts designed to bypass security controls, effectively simulating advanced persistent threats.
- Reduced Human Error: Automated checks and balances from AI-driven tools can help identify mistakes that human analysts might overlook.
- Continuous Monitoring
- User Behavior Analytics: AI systems can track user behavior patterns and alert security teams when anomalies occur, helping to detect insider threats or compromised accounts.
- System Health and Anomaly Detection: Machine learning can baseline “normal” system behavior and highlight deviations in real time, crucial for rapid intrusion detection.
By embedding AI into both offensive and defensive operations, security professionals can become far more effective and efficient. However, it’s equally important to remember that attackers also have access to these same tools—underscoring the importance of robust AI security measures.
Frameworks for Securely Leveraging AI
Fortunately, resources exist to guide you in securely building AI-based applications. One notable example is the OWASP Top 10 for LLMs, which outlines the most critical vulnerabilities and best practices for generative AI systems. Additionally, the OWASP Generative AI Security and Governance Checklist provides a comprehensive governance and security framework for LLM applications. Standards like ISO 42001 also aim to standardize AI governance and risk management processes.
Conclusion
Generative AI brings both exciting possibilities and significant security challenges. As the DeepSeek incident shows, failing to address these challenges can have dire consequences—ranging from massive data breaches to global scrutiny.
If you’re developing an AI-based application, treat the model as a potentially untrusted component. Understand how data flows through your system, enforce strict authentication and authorization checks, and follow established security frameworks like the OWASP LLM Top 10. By doing so, you’ll be far better equipped to mitigate AI-specific threats and keep both your organization and users safe.
At the same time, remember that AI isn’t just a source of risk; it’s also a powerful ally for security teams. From faster vulnerability discovery to adaptive defenses, AI-driven solutions can help penetration testers and blue teams stay a step ahead of adversaries—provided they incorporate proper guardrails and governance from the start.
By examining real-world incidents such as DeepSeek and following best practices for AI security, we can harness the power of LLMs responsibly and minimize the risks associated with this transformative technology.
If you are interested in learning how Secure Ideas can help you and your organization tackle the emerging security challenges of AI, reach out to info@secureideas.com or cory@secureideas.com. We offer various security testing services such as penetration testing, red teaming, and advisory services. We also offer both public and bespoke training - see our website for more information: https://www.secureideas.com/training.
Also feel free to follow Cory Sabol on various social media platforms, you can find links to his profiles here:
