Should a Company Be Liable for What Its Chatbot Says?

Generative AI doesn't create a liability vacuum because nobody is responsible. It creates one because the frameworks we use to find the responsible party were built for a world where human decisions produced human-scale outputs — and that world no longer fully exists.

The search for a responsible party

Every liability framework starts in the same place. Tort law asks who breached a duty of care. Criminal law asks who formed the intent and carried out the act. Contract law asks who made the promise and who broke it. The frameworks differ in what kind of decision matters, what standard of care applies, and what remedy follows. But they all assume the same thing: that a responsible decision-maker exists and can be found.

The Florida Attorney General's criminal investigation into OpenAI over the FSU shooting puts that assumption under pressure in a way the legal system has not yet had to resolve. What is publicly known is this: the perpetrator had over 200 messages with ChatGPT in the period before the shooting, and more than 200 of those messages have been entered into evidence. The AG's office has reported that queries included what type of gun to use, the lethality of certain ammunition, what time the FSU student union was busiest, and where on campus a higher population would be encountered. The full content and context of those 200 messages is not yet publicly available. The trial is scheduled to begin in October 2026. Until the complete record is accessible, any analysis of this case — including this one — is working from summaries filtered through advocacy positions on both sides. What the AG characterizes as ChatGPT "advising" the shooter, and what OpenAI characterizes as "providing factual information available on the internet," are both descriptions of the same incomplete record. The full picture may confirm either characterization, or neither.

What the case illustrates — regardless of how the facts ultimately read — is a set of questions the legal system has not yet answered. ChatGPT answered whatever it was asked. The perpetrator made every subsequent decision and carried out every act.

The AG's response was direct: "If this were a person on the other side of the screen, we would be charging them with murder."

The framing is rhetorically powerful. It is also legally imprecise — in ways that matter more than the rhetoric. ChatGPT is not a person. It did not act. It cannot form intent. Criminal accessory liability requires all three. The investigation may ultimately rest on a negligence and products liability theory rather than a criminal one — and even that runs into problems that existing frameworks were not designed to solve.

When an AI agent acts — executing tasks, making decisions, taking steps in the world — the liability question is about the deployment chain. Who controlled the agent? Who deployed it? Who bears responsibility for what it did? That question has a working answer in most frameworks: the company that deployed the agent, subject to allocation across the chain of principals. The accountability gap in agentic AI is real, but its shape is recognizable. Liability exists somewhere in the chain. The dispute is about where.

The FSU case poses a different question entirely. ChatGPT was not acting as an agent. It was responding to queries. Nobody deployed it toward the shooting. Nobody instructed it to assist with planning violence. It received input and produced output — the same thing it does hundreds of millions of times a day for users with overwhelmingly legitimate purposes. The question the Florida AG's investigation forces is whether the company that made the LLM available bears responsibility for what a third party chose to do with its output — when that output was a response to queries, not an autonomous action taken on anyone's behalf.

That is a different liability question from the agentic case. Existing frameworks were not designed to answer it. And the tools we currently use to find the responsible party may not be adequate to the task.

Why the search engine analogy fails — and what follows from that

The instinctive defense in cases like this is that ChatGPT is a tool, and tools don't bear liability for how users use them. A hammer used to commit assault does not implicate the manufacturer. Google search results that lead a user to dangerous information do not implicate Google. On this view, the user's decision to use the tool for a harmful purpose is the decisive act — and the causal chain ends there.

That framing is correct as far as it goes. ChatGPT is not a person. It does not initiate. It does not monitor. Modern LLMs including ChatGPT do have safety fine-tuning — RLHF and related approaches that cause the model to refuse certain categories of queries. But that evaluation is statistical and probabilistic, not deliberative. It does not reliably identify harmful intent framed innocuously. It does not aggregate signals across a conversation to detect escalating risk patterns. Whether that level of evaluation met the standard of reasonable care for the foreseeable category of harm the FSU case involves is precisely the question the liability analysis needs to answer — and it is a different question from whether evaluation is possible in principle, which it demonstrably is.

The question is not whether ChatGPT is a tool. It is whether the search engine analogy — the specific tool comparison OpenAI's defense implicitly relies on — holds. And answering that question requires understanding something that most legal analysis of this case has not addressed: ChatGPT does not operate in a single, uniform way.

This layered reality matters because it determines how far the search engine analogy actually fails — and the answer differs by layer.

At the content layer — where the underlying information came from third-party web sources — the search engine analogy is stronger. The content originated elsewhere. In many jurisdictions, platforms that index and display third-party content without material modification benefit from reduced liability exposure compared to platforms that generate their own content. The counterfactual holds more cleanly: the information was retrievable by anyone with a search engine.

At the presentation layer — where the generative model synthesised and structured the response — the analogy weakens. The output is ChatGPT's own, even when the content came from elsewhere. The friction-removal argument applies: the user received a direct, structured answer rather than a list of links to navigate. The reduced liability protections that many jurisdictions extend to platforms displaying third-party content do not apply where the platform has generated the content itself. The liability analysis for this layer looks more like products liability for a generative system than platform immunity for a search engine.

At the calibration layer — where the response was tailored to the full conversational context — the analogy fails most clearly. No search engine calibrates results to a conversation that has been unfolding for twenty minutes. The output the user received was shaped by everything said before — making it more specifically responsive to whatever purpose the conversation revealed than any standalone search result could be. This layer is where the conversational context argument lives, and it applies regardless of whether the underlying content was retrieved or generated.

In the hybrid case — which is likely the most common mode for queries involving specific real-world facts — all three layers operate simultaneously within a single response. The liability analysis differs by layer, and may require different legal frameworks to be applied to different parts of the same response. That is a level of technical granularity that the current public framing of the FSU case has not come close to addressing — and that any GC advising on AI liability exposure needs to understand before reaching a conclusion.

The factual questions that shape the liability analysis

Before the liability frameworks can be applied, two prior factual questions need to be answered — both of which current public framing of the FSU case largely ignores.

First: were the queries asked within a single conversation, or across separate sessions? Second: were they framed in a way that made harmful intent detectable, or were they phrased so that each appeared to have an innocent purpose?

These are not technical details. Together, they produce at least three materially different scenarios — each with a different liability profile. The binary of contextualised versus isolated that most commentary assumes is a significant oversimplification. Which scenario the FSU case falls into depends on facts — the full conversational record — that are not yet publicly available. The framework matters regardless: it determines which factual questions are legally significant, and what their answers mean for the liability analysis.

The context-clearing mechanism exists for legitimate reasons — privacy, session separation, the ability to start fresh. A user who deliberately fragments queries across sessions to avoid pattern detection has taken an affirmative step to defeat the safeguard. A user who frames harmful queries in innocuous terms has done the same thing within a single session. In both cases, deliberate circumvention shifts moral and legal responsibility substantially toward the user.

Scenario B reveals a deeper structural problem. Any safety system that relies on semantic pattern recognition is inherently vulnerable to adversarial framing. If reasonable care requires detecting harmful intent, but harmful intent can be masked by plausible innocent framing, the reasonable care standard may be imposing a requirement that no system can reliably meet. The only ways to close that gap are to surveil more comprehensively or to refuse more aggressively — both of which create problems examined later in this article.

Scenario A presents the hardest version of all. The queries were not clearly innocent. They were not clearly criminal. They were individually ambiguous, and their combination looks damning only in hindsight, after the harm occurred. In criminal law, that hindsight is the point: the prosecution assembles evidence after the crime to establish guilt beyond reasonable doubt. For an AI system to have intervened before the crime, it would have needed to act on a standard far below probable cause — treating every user who asks a similar combination of questions as a potential suspect. That is not reasonable care. That is suspicion at scale. Whether the FSU case is Scenario A, B, or C — or some combination — depends on the full record. The analytical point stands independently of which scenario applies: the liability argument becomes progressively harder to sustain as the queries become more individually ambiguous and more dispersed across sessions.

Why "find the engineer" is harder than it looks

Set aside for a moment the unresolved factual questions about how the FSU queries were structured and framed. However those questions resolve, the next analytical move — assuming any version of the contextualised argument holds — is to trace liability upstream: to the engineers who built the system, the safety team that designed the guardrails, the executives who made deployment decisions. The instinct to look there is correct. But the black box nature of large language models makes the attribution harder than it appears.

Nobody designed ChatGPT's specific response to this specific sequence of questions. That response emerged from billions of parameters trained on trillions of tokens. No engineer reviewed it, approved it, or could have anticipated it in the specific. The causal chain between an identifiable human decision and a harmful output runs through a system whose internal logic no individual fully controls or understands.

This is not a defense. It is a description of why existing negligence doctrine — which requires identifying a specific breach of duty by a specific decision-maker — strains under the weight of the attribution problem. The doctrine was built for a world where human decisions produced human-scale outputs. A world where a harmful output emerges from billions of parameters is a world the doctrine was not designed to address.

Three bases for liability nonetheless survive:

Systemic design choices. The black box argument covers emergent outputs. It does not cover foundational decisions — what safety fine-tuning to implement, what refusal thresholds to set, whether to build context-aware multi-turn risk detection. Those are human decisions with identifiable decision-makers. The question is whether any of those decisions fell below the standard of reasonable care.

Known risk, inadequate response. If OpenAI identified through red-teaming, user reports, or internal research that the model could assist with planning violence under certain conversational patterns, and chose not to implement adequate countermeasures, that knowledge becomes the liability hook. The black box does not shield an organization from liability for risks it knew about and chose not to address.

The pharmaceutical analogy. Drug companies do not understand every molecular mechanism by which a drug causes harm. The human body is, in many respects, its own black box. But pharmaceutical companies are liable if they failed to test systematically, failed to disclose known risks, or failed to implement available safeguards. An equivalent standard for AI — test systematically, document what you find, implement available safeguards, disclose known risks — imposes obligations that are independent of whether the model's internal logic is understood.

Where the existing frameworks land — and where they break

Applying each of the relevant legal frameworks to the generative AI context reveals where the fit is adequate and where the framework strains:

The surveillance trap — and the warrant problem

The Scenario A analysis reveals a structural problem that goes beyond AI safety design. Consider what it implies.

In criminal law, to obtain a warrant, the prosecution must demonstrate probable cause — a reasonable belief, based on articulable facts, that a crime has been or is being committed. The standard exists precisely to prevent general suspicion from justifying surveillance. A prosecutor cannot obtain a warrant to monitor everyone who asks about ammunition, or everyone who asks about campus schedules, or even everyone who asks about both — because the combination, however suspicious in hindsight, does not rise to probable cause when the overwhelming majority of people asking similar questions have innocent purposes.

If the legal system itself — with all its coercive power and public safety mandate — cannot justify monitoring on the basis of ambiguous combinations of innocent acts, on what basis do we impose on a private AI provider a duty to do exactly that? We would be requiring AI providers to detect and intervene on patterns whose harmful interpretation may be visible only in hindsight — at a threshold of suspicion that the law considers insufficient to justify state surveillance. That is not a standard derived from reasonable care. It is a standard derived from the desire to prevent harm at any cost, applied to a private actor without the procedural safeguards that constrain the same logic when applied by the state.

That is not a coherent liability standard. And its practical consequence is what makes it dangerous.

If courts find that OpenAI's failure to detect the conversational pattern created liability, the rational response for every AI provider is to record and retain everything — across sessions, indefinitely — to enable exactly the pattern detection the liability argument implies they should have implemented. More monitoring. Longer retention. Cross-session tracking. Persistent user profiling.

That outcome is deeply problematic — and not only for privacy advocates.

Cross-session surveillance means every query a user has ever asked is retained, analyzed, and potentially subpoenaed. The privacy expectation users reasonably hold when they clear a conversation becomes illusory. A person researching a sensitive medical condition, exploring a controversial political idea, working through a personal crisis, or conducting legitimate professional research leaves a permanent record that can be accessed, hacked, or compelled in litigation.

This incentive collides directly with privacy law. Modern data protection frameworks broadly require data minimization — collecting only what is necessary for a defined purpose — purpose limitation, and retention limits. A cross-session surveillance architecture designed to detect harmful intent patterns implicates all three principles simultaneously.

The chilling effect compounds the problem. A system that retains and analyzes everything changes what people are willing to ask. The researcher who wants to understand extremist rhetoric, the novelist writing about violence, the abuse survivor trying to understand their experience, the lawyer researching a criminal defense — all will hesitate. The determined bad actor, meanwhile, will simply fragment queries across sessions, use different accounts, or use different tools. The population most deterred by comprehensive AI surveillance is the population of legitimate users. The population least deterred is the one the surveillance is designed to stop.

It is tempting to conclude from this that litigation is the wrong instrument and that ex ante regulatory standards would produce better outcomes. That conclusion does not hold. The surveillance incentive does not arise from the form of legal intervention — litigation or regulation. It arises from the substance of the obligation itself. If any legal framework — court judgment or regulatory standard — determines that service providers have a duty to monitor user interactions for harmful intent, the rational response is to monitor as comprehensively as the law permits. A regulatory standard that imposes a monitoring obligation creates the same surveillance incentive as a court judgment that finds liability for failure to monitor.

The tension is not between litigation and regulation. It is between any monitoring obligation and privacy — two requirements that are structurally incompatible. A legal framework that simultaneously requires AI providers to detect harmful intent and prohibits them from collecting the data needed to do it forces providers to choose which obligation to breach. Until that incompatibility is resolved, every AI provider in this space is navigating irreconcilable legal obligations. That is precisely why data governance has become strategically significant — not as a compliance exercise, but as the only instrument that can navigate both sides of an irreconcilable tension at once. An organization that has determined precisely what it collects, for what defined purpose, retained for how long, under what access controls, and with what audit trail, can account for itself under either framework. One that has not cannot account for itself under either.

What a defensible governance posture looks like

The law will settle eventually — through litigation, regulation, or both. The FSU case may produce a significant ruling on AI provider liability. The EU AI Act is developing obligations for high-risk systems. China's AI companion regulation has already mandated context-aware intervention for a specific category of service. How the monitoring-versus-privacy tension is ultimately resolved will shape what all of these frameworks require. What will not change is the underlying logic: organizations that have built rigorous data governance frameworks before the law settles will be better positioned than those that build them in response to it.

In the interim, companies deploying or building on generative AI face a liability landscape that is genuinely uncertain. That uncertainty is not an argument for paralysis. It is an argument for a governance posture that would be defensible under any of the frameworks currently being tested.

The pharmaceutical analogy is the most useful guide. Drug companies operate under pervasive uncertainty about exactly how their products cause harm. What the regulatory framework requires of them is not perfect knowledge — it is systematic behavior: test, document, implement available safeguards, disclose known risks. A company that can demonstrate it did all four is in a materially better position than one that cannot, regardless of which legal theory eventually prevails.

None of this eliminates liability exposure. What it does is demonstrate that the company took its responsibilities seriously, acted on what it knew, and implemented measures proportionate to the foreseeable risks. In a legal environment where the frameworks are still developing, that demonstration is the most defensible position available.

The question that still has no answer

The FSU case will produce a ruling on OpenAI's liability — or a settlement that avoids one. Either way, it will not resolve the foundational question the case exposes.

The liability gap generative AI creates is not only about who the responsible party is. It is about whether the obligation being implied — detect harmful intent from individually ambiguous signals, at scale, in real time, without treating every user who asks similar questions as a suspect — is one that any system can coherently be required to discharge.

The warrant analogy is instructive. Criminal law requires probable cause before permitting surveillance because the alternative — monitoring based on patterns of innocent behavior — catches too many innocent people to be justified. If that standard applies to the state, with its coercive power and public safety mandate, the basis for imposing a lower threshold on private AI providers is not obvious. And if no lower threshold can be justified, the implied obligation — act on patterns that would not justify a warrant — may be one the law cannot coherently impose.

This does not mean OpenAI bears no responsibility. The British Columbia case — where OpenAI's own systems flagged a user, staff considered alerting law enforcement, and the company decided not to — involves identifiable human decisions with identifiable consequences. That is not the black box problem. That is the known-risk-inadequate-response problem, and it has clear legal implications independent of whether the ambiguous-pattern scenario does.

The deeper question — person or tool, and what duty either status creates — remains open. The law has created new categories before when technology demanded it. Corporate personhood. Product liability for latent defects. Platform immunity for intermediaries displaying third-party content. Each created new problems while solving old ones. The category the law eventually creates for generative AI will do the same.

What is certain is that the category will be created under pressure — through litigation that forces imperfect frameworks to produce answers they were not designed to give. The companies that will be best positioned when that happens are the ones that did not wait for the answer before building a governance posture adequate to the uncertainty.