After the Hype: Model Hallucination, Interface Silos and Who Is Liable

The most valuable thing about this edition may not be how capable the robots looked, but that the industry began publicly admitting what it still cannot do. For buyers, writing a vendor's limits into the contract beats writing its capabilities into a brochure.

Trade shows sell possibility. Procurement exists to constrain impossibility. The most useful material to come out of Security China 2026, in our view, was the industry's own list of unresolved problems.

What the Industry Says It Cannot Yet Do

Post-show analysis from a&s Media names the friction created by ecosystem expansion directly: standards, interfaces and business language are not unified across vendors, and cross-brand agent coordination, data interchange and the division of responsibility remain unavoidable obstacles to deployment. Its closing list of present challenges is model hallucination, insufficient algorithm generalisation, high cost of large-scale deployment, a mismatch in talent structure, and lagging standards.

The same analysis offers a deliberately unglamorous technical judgement: large models have not replaced traditional algorithms. In high-certainty tasks such as perimeter breaches, smoke and flame, or hard-hat detection, well-tuned small specialised models remain the cost-effective choice; large models take on cross-modal understanding, complex event reasoning, natural-language work orders and multi-system orchestration. The two are complementary, not substitutive. It also notes that agent behaviour auditing, shadow-AI discovery, prompt-injection protection, model watermarking and provenance appeared alongside the models themselves, signalling a move into an engineering phase defined by controllable, auditable and attributable effects.

At the industry development forum, a former deputy director of the MIIT cybersecurity industry development centre argued that AI is both a threat amplifier and a capability multiplier. Agent loss of control, data poisoning and deepfakes were named as AI-native risks, alongside a security talent gap; proposed directions included using AI against AI, domain-specific security models, agent-driven security operations and AI supply-chain governance.

The 15th Five-Year Plan states its own three challenges plainly: shortfalls in core technology, mounting pressure on video security, and intensifying involution-style competition.

Four Boundaries, Mapped to Vehicles

First, do not replace a validated small model with a large one. Fatigue, distraction, smoking and lane departure are high-certainty tasks where specialised small models win on compute, latency and predictability. Large models belong in cross-modal attribution, natural-language work orders and multi-system orchestration. Our earlier report argued that false-alarm rate is the real battleground; the addition here is that swapping models does not reduce false alarms and may trade a predictable system for an unpredictable one.

Second, liability is harder to write than accuracy. Vehicle unit, platform, algorithm supplier and cloud provider all sit on the chain. When a misjudgement or missed alert contributes to an incident, who produces the evidence and who carries the responsibility? A draft mandatory national standard on automotive data security requirements already proposes binding vehicle makers, driving-automation providers, component suppliers and cloud service providers into a full-chain responsibility model — but it is still a draft, not an enacted obligation. Contracts should not wait for it.

Third, interface silos are the biggest hidden risk in factory-fit adoption. We argued earlier that joining a command system would become an acceptance criterion. With standards, interfaces and business language still unaligned across vendors, 'can integrate' usually means a bespoke joint-debug exercise for each pairing. Ask for a reproducible cross-brand integration report from a system in routine operation, not a booth demonstration.

Fourth, AI-native risk is becoming a precondition for evidence admissibility. With deepfakes and data poisoning now realistic, video used for accident determination, insurance claims and enforcement needs traceable model versions, auditable behaviour, watermarking and tamper-evident signatures. What we earlier called verifiable originality moves from a nice-to-have to the question of whether the evidence can be used at all.

One more cost constraint: infrastructure analysis at the show stressed that security workloads are 7x24 continuous streaming inference rather than one-off training, so stability, fault isolation and scheduling efficiency matter more than peak TOPS. On a vehicle that becomes power, thermal and automotive-grade reliability. A public technology stack is not the same thing as something that survives inside a truck.

Risk Note

The limitations cited here — hallucination, generalisation gaps, lagging standards, unclear liability — come from industry press analysis and forum expert opinion. They are qualitative consensus, not quantified failure rates, and must not be used to estimate reliability or to characterise any specific vendor. The plan's three challenges are the industry's own self-assessment, not findings about individual companies.

Six Questions Before Accepting an AI System

• Are high-certainty tasks still handled by specialised small models?

• Is liability for misjudgement and missed alerts written into the contract?

• Can every alert be tied to a specific model version and triggering rule?

• Is there a reproducible cross-brand integration report rather than a demo?

• What exactly degrades when the network drops, and which capabilities remain?

• Can model updates be grey-released, rolled back and audited afterwards?

Our Judgment

The show demonstrates what is possible; procurement constrains what is not. Asking a supplier to state its boundaries is the single most practical step in this report series. Combine these six questions with the seven false-alarm metrics from the earlier series and you have an acceptance framework that survives contact with a real fleet.

What to Take Away

• Turn the six questions into an AI acceptance checklist alongside the earlier false-alarm metric set.

• Require both a capability list and a boundary list; the second one determines the contract terms.

• Design model versions, watermarking and behaviour auditing into the evidence chain before an incident forces the question.

Sources: a&s Media's 10 November 2026 post-show analysis, the 2026 China Security Industry Development Forum proceedings, and the CSPIA 15th Five-Year Plan interpretation of 3 November 2026. All limitation statements are qualitative industry consensus, not measured statistics.