The hype around generative AI is accelerating. Executives are being told that large language models are ready for business-critical workflows. What they’re not being told is that the infrastructure behind most of these tools is built for speed, not scrutiny.
Amazon recently experienced a breach with its AI coding assistant in which a hacker successfully added a prompt to wipe a user’s system and delete cloud resources. This is just one of the latest cases that reveal the gaps in AI security protocols as associated risks continue to evolve.
Enterprises have a lot to learn from the legal industry, which is among the earliest to adopt GenAI in high-stakes work. Their experience has exposed a serious flaw in the way these systems are being deployed.
With increased education and exposure to the risks that come with unsanctioned AI tools, 60% of law firms have implemented formal cybersecurity policies, while only a quarter (25%) of businesses have rolled out a proper AI governance program.
If you work in legal, finance, healthcare, insurance, or any other function where trust and accountability define your business, pay attention. Law is the canary in the coal mine. And it is already showing signs of stress.
Legal Work Is High-Stakes by Design
Legal professionals operate in an environment shaped by confidentiality, regulation, and professional obligation. Outputs must be jurisdiction-specific, citation-supported, and defensible under review. GenAI tools introduced into that workflow must meet the same standard.
Most do not.
Many of the legal tools now on the market are built on general-purpose models, trained on public data, and hosted in public cloud infrastructure. Some offer light legal fine-tuning. Others wrap prompts in a user-friendly interface. But the foundation remains the same. The logic is opaque, the sources are unverifiable, and the systems are largely ungoverned.
In a recent survey, more than 70% of law firms using GenAI reported significant rework due to outputs that failed to meet internal legal standards. At the same time, a Stanford study found that large language models hallucinate in 58% to 82% of legal queries.
These are not bugs. They are design limitations. Most models were built to predict language, not apply legal reasoning. That distinction matters.
Public Infrastructure Carries Private Risk
Many GenAI platforms emphasize security and privacy in their messaging. But security is not the same as control. When legal teams send sensitive client information into an AI tool hosted on public infrastructure, they are relying on someone else’s architecture, someone else’s safeguards, and someone else’s interpretation of risk.
With only 20% of firms reporting full auditability across prompts, citations, and model outputs, 61% of law firms have experienced a near miss or compliance concern tied to the use of external GenAI tools.
When high-trust work runs on low-visibility systems, oversight becomes impossible. And without oversight, organizations lose more than control. They lose defensibility.
A Three-Part Framework for Responsible GenAI Deployment
The solution is not better prompts. It is better architecture. The firms moving beyond experimentation are prioritizing three things.
1. Control
AI systems should operate inside environments that the firm governs. That means private cloud or on-premise deployment, with full visibility into how data is processed and how models behave. If you cannot control where your data goes or how it is used, your firm is not in control.
2. Auditability
Every output must be traceable. Firms need clear logging of prompts, retrieval sources, model versions, and output edits. This is not a preference. It is a requirement for client assurance, professional accountability, and regulatory compliance.
3. Internal Knowledge as the Model
Foundation models are trained on the internet. Your firm’s value is not. The highest-performing GenAI systems retrieve from internal precedent, playbooks, prior research, and approved language. In legal, that means firm-specific memos and clauses. In finance, it may mean internal policy or risk models. In healthcare, clinical protocols. Context is not a nice-to-have. It is the only way GenAI becomes useful.
The Broader Warning for Enterprise Leaders
The temptation to adopt AI tools quickly is understandable. But for any organization operating in a high-trust environment, the infrastructure behind those tools matters more than the interface in front of them.
Legal is the first sector to confront this. It will not be the last.
As GenAI becomes embedded in workflows for drafting, advising, reporting, and decision support, the pressure to explain how each output was produced will only grow. Boards will ask. Clients will ask. Regulators will ask. The only acceptable answer will come from systems designed for accountability from the start.
