Try Kipti
AI security

Prompt Injection: Why It Matters in Education

Prompt injection is a central weakness in AI systems. This article examines risks, current research, and protective measures for education.

Björn Schriewer8 min read

Illustration of prompt injection in AI systems

Anyone building an AI-assisted product today—whether a chatbot, document assistant, or learning platform—has to address prompt injection. It is a central security issue for large language models (LLMs), and the entire industry is working on solutions at an impressive pace.

This article explains what prompt injection is, which forms of attack researchers have identified, how defenses are developing, and what this means in practical terms for AI products used in schools.

What makes prompt injection fundamental?

Large language models such as ChatGPT, Claude, and Gemini are controlled through natural language. That is both their greatest strength and a fundamental challenge: there is no hard boundary between instructions and data. An LLM processes the system prompt, user input, and retrieved documents as one stream of text.

Prompt injection exploits that fact. An attacker writes input intended to make the model ignore its original instructions and follow the injected commands instead.

The underlying principle has been familiar in computer science for decades. SQL injection follows the same logic: data is interpreted as code. Separating the two is more difficult for LLMs because everything is natural language. As with SQL injection, however, effective defense strategies exist.

Direct and indirect prompt injection

Research distinguishes two main categories.

Direct prompt injection

The user supplies a manipulative input directly. A typical pattern is “Ignore all previous instructions and do X instead.” This is what many people understand as jailbreaking: an attempt to bypass the model's safeguards.

This matters, but the scenario is comparatively bounded. The attacker is the user, and the immediate effects remain within that user's session.

Indirect prompt injection

The indirect variant is much more complex and is a central focus of current research. The standard reference is Greshake et al. (2023), the first systematic study of how manipulated external data sources can influence applications that integrate LLMs.

The scenario is this: an LLM retrieves information from the outside world, such as websites, email, or documents. An attacker hides instructions inside that data. The model cannot reliably distinguish harmless document text from an embedded command.

Security researcher Simon Willison coined the term “lethal trifecta” for the combination of access to private data, processing untrusted content, and the ability to make external requests. Removing any one of the three substantially reduces the risk, showing how much architectural choices can achieve.

What does current research say?

Many-shot jailbreaking: A strong example of responsible disclosure

In 2024, Anthropic published an attack method that exploits growing context windows: many-shot jailbreaking. A prompt contains a large number of fictional dialogues in which an AI assistant answers harmful requests. Beyond a certain number of examples, the model follows the pattern (Anthropic, 2024).

What is notable about this publication is Anthropic's proactive disclosure. The company informed other providers in advance while developing countermeasures. A follow-up paper (Anthropic, 2025) describes mitigation strategies that substantially reduced the success rate of such attacks. The approach reflects a pattern increasingly seen across the industry: responsible disclosure instead of concealment.

The largest public analysis to date

A recent study by Dziemian et al. (2026) provides the most extensive empirical analysis to date. In a public red-team competition, 464 participants submitted more than 272,000 attacks against 13 leading models.

The findings are nuanced. Every tested model was vulnerable in principle, but success rates ranged from 0.5 to 8.5 percent. For the most robust models, that is a level that the authors' findings suggest can be managed with additional protection layers (Dziemian et al., 2026). The study is also valuable because it identifies attack patterns that generalize across models and can therefore be targeted by defenses.

Simple attacks and clear priorities

Bombieri et al. (2025) show that even simple rewordings can be surprisingly effective. That sounds alarming, but it also provides a constructive direction: defenses need to address basic input patterns as well as technically sophisticated attacks.

Defense is making measurable progress

The defensive side has advanced substantially in recent months.

Anthropic reports an attack rate near one percent

Anthropic reports reducing the attack rate for its Claude Opus 4.5 browser agent to around 1 percent by combining reinforcement learning, specialized classifiers, and system-level safeguards (Anthropic, 2026). The company emphasizes that the problem is not solved, but the reported result indicates measurable progress.

Google: Layered defenses for Gemini

Google uses a layered defense strategy for Gemini. The layers include model hardening through adversarial training, input classifiers, “Security Thought Reinforcement,” Markdown sanitization, and redaction of suspicious URLs (Google, 2026). Google reports that these measures substantially improved the Gemini 2.5 models' defense against indirect prompt injection.

OpenAI: Instruction hierarchy

OpenAI's instruction hierarchy approach trains models to distinguish trusted from untrusted instructions, alongside automated red teaming and bug-bounty programs (OpenAI, 2025). For its Atlas browser agent, OpenAI also uses reinforcement learning to identify new attack vectors systematically and improve defenses iteratively (OpenAI, 2026).

CaMeL: Defense through architecture

The CaMeL framework from Debenedetti et al. (2025) offers a particularly interesting conceptual approach: instead of teaching the LLM to recognize injections, it removes the model's ability to cause harm through an injection. CaMeL first analyzes the trusted user request and derives the steps the system should perform—the actual program flow. External data retrieved later, including documents, web content, or email, cannot change that flow. Even if a document contains a manipulated instruction, it remains ineffective because the decision about what the system will do has already been made. In tests, CaMeL completed 77 percent of tasks with provable security, compared with 84 percent for an unprotected system (Debenedetti et al., 2025). The approach trades a small amount of flexibility for structural safeguards.

PromptGuard and DefensiveToken: More Defensive Tools

The PromptGuard framework (Scientific Reports, 2026) combines four defensive layers: input filtering, structured prompt formatting, semantic output validation, and adaptive response refinement. The authors report reducing the attack rate by up to 67 percent with an F1 score of 0.91 for detection.

The DefensiveToken approach (ACM AISec Workshop, 2025) takes a different path. A small number of additional tokens whose embeddings are optimized for security make an LLM more robust with minimal loss of functionality. For manually created injections, the reported success rate fell to 0.24 percent.

The overall trend

Controlled studies report that layered defense systems can reduce attack rates from above 73 percent to below 10 percent. OWASP, Google, OpenAI, and Anthropic all advocate defense in depth: not one silver-bullet solution, but several independent layers that may each fail without bringing down the whole system.

Why does this matter especially in education?

AI systems in education may process highly sensitive information: learning assessments, support plans, behavioral observations, and accommodations, all directly connected with minors. That creates strict GDPR obligations and a heightened need for protection.

At the same time, a school platform's practical scenario can be more bounded than that of a general browser agent. A school platform does not necessarily browse the open web, open arbitrary emails, or make uncontrolled API requests. Willison's lethal trifecta shows that deliberate design can substantially reduce the attack surface.

What this means for Kipti

We are building Kipti as an AI-assisted documentation platform for schools. Prompt injection directly informs our architectural decisions, and current research provides increasingly robust patterns to draw from.

Several principles follow from the current research:

Minimize the attack surface: We deliberately limit which external data sources the LLM processes and control the context in which it operates. Not every input needs to pass through a language model; when one does, it should do so within a clearly defined frame.

Separate trust from processing: User input and system-retrieved data are treated differently. The model has clear boundaries on which actions it can trigger, following the structural-separation ideas represented by CaMeL and Google's layered defenses.

Use defense in depth: No single safeguard is sufficient. We use several layers, including input validation, controlled context windows, output filtering, and audit trails. This aligns with the approach advocated by OWASP, Google, and Anthropic.

Provide transparency: Teachers need to understand what AI does and what it draws on. Transparency is educationally important and also supports security: what is visible can be reviewed.

A final organizing thought

Prompt injection is not a bug that can be fixed once and forgotten. It is a fundamental property of how today's language models work. But recent progress shows that the industry is taking the problem seriously and that effective defenses are possible.

Anthropic reports attack rates of around 1 percent. Google builds layered defenses directly into Gemini. Academic frameworks such as CaMeL show that architectural security and functionality do not have to be opposites. The community—from OWASP and leading AI labs to security researchers—is collaborating more transparently than before.

For education, this means that anyone building an AI product for schools has both a responsibility and practical tools for treating security as an architectural principle—not as an optional layer added later, but as a foundation.

Ultimately, this is about children's data and teachers' ability to place appropriate trust in AI tools.

References

← Back to insights

Related insights

Digital transformation

AI in Schools 2026: Better Models, Better Education?

New AI models are growing more capable, but better education only emerges through clear frameworks, pedagogical responsibility, and the right deployment scenarios.

Björn Schriewer7 min read

Digital transformation

AI in Schools: Opportunities and Risks

AI is already part of everyday school life. This article examines opportunities and risks for learning, teachers, and the selection of appropriate systems.

Sina Schriewer9 min read

AI in education

Google "Learn Your Way": An Adaptive Textbook?

Google Labs is testing Learn Your Way, an AI-assisted textbook experiment that personalizes content and translates it into different learning formats. We examine its architecture, educational foundations, and reported outcomes.

Björn Schriewer9 min read

Ready to see it in practice?

Try Kipti with a real school workflow and see where it saves time.

Try KiptiContact us
  • Start directly
  • Test a real workflow
  • Data protection considered