09 Mar 2025 AI Engineering 5 min read

Manus and the Autonomy Hype: Demos vs. Reproducible Value

Manus launched in March 2025 with viral demos and state-of-the-art GAIA claims. We compare the self-reported numbers with the first independent tests, explain what the benchmark does not measure, and argue why constrained agents with approval gates ship to production today while general autonomy does not — yet.

Autonomy Goes Viral

On March 6, 2025, Butterfly Effect, a startup of a few dozen people in Wuhan and Beijing, released a preview of Manus and called it the first general AI agent. Within days, invite codes were resold on Xianyu for close to 100,000 yuan (about $13,800), the official Discord passed 138,000 members, and commentators declared a second DeepSeek moment. Fewer than 1% of waitlisted users had access.

The pattern is familiar: a polished four-minute demo, one benchmark chart, and restricted access. Restricted access matters because it inverts the burden of proof — claims circulate faster than anyone can test them. We build agent systems for production, so this post separates what is verified about Manus from what is merely claimed, and explains why we still ship constrained agents with approval gates.

A task arrives — the agent plans its next step. 1/4

What Manus Actually Is

Manus is not a new foundation model. Chief scientist Yichao "Peak" Ji confirmed that it orchestrates existing models — Anthropic's Claude 3.5 Sonnet and fine-tuned variants of Alibaba's open-weight Qwen models — together with browser automation. The product wraps them in a multi-agent loop: plan, browse, execute code in a sandboxed virtual machine, deliver a result file. Users can watch each step in a side panel called Manus's Computer.

This is orchestration engineering, not model research. That is not a criticism; most production agent value comes from orchestration. But it defines the ceiling. Manus can be no more reliable than its underlying models plus its scaffolding, and any team with API access can attempt a replication. OpenManus, an open-source clone from the MetaGPT community, appeared within days of the launch.

The GAIA Numbers

The central claim is state-of-the-art performance on GAIA, a benchmark published by Mialon et al. in November 2023. GAIA contains 466 real-world questions that require web browsing, tool use, and multi-step reasoning, split into three difficulty levels. Human respondents score about 92%; GPT-4 with plugins scored 15% at publication. The figures below are the numbers Manus published at launch.

Three details temper the chart. The scores are self-reported, and as of mid-March 2025 no independent replication exists. The Level 2 margin over OpenAI's Deep Research is a single percentage point, which is within run-to-run variance for agentic systems. And while the company states the runs used its production configuration, per-task traces have not been published.

GAIA level	Manus (self-reported)	OpenAI Deep Research	Previous SOTA
Level 1	86.5%	74.3%	67.9%
Level 2	70.1%	69.1%	67.4%
Level 3	57.7%	47.6%	42.3%

What the Benchmark Does Not Measure

GAIA scores pass@1 on curated questions with verifiable answers. It does not measure what the launch demos show: booking flights, ordering food, screening resumes, building games. Those tasks are open-ended, stateful, and partly irreversible — failure modes GAIA never touches. A benchmark for tool-assisted question answering cannot certify a general agent for actions with side effects. The two problem classes share techniques but not risk profiles.

GAIA also ignores three production axes: cost, latency, and variance. Chinese outlet 36Kr put Manus's per-task cost at roughly $2, about a tenth of Deep Research — a genuinely interesting number. But a task that runs 50 minutes and dies at step 18 of 20 costs more than money. Benchmarks report central tendency; operations live in the tail.

Reproducibility in Practice

Independent tests in the first week diverged from the chart. TechCrunch saw a crash on a food order, broken links on a flight search, and an error 30 minutes into a coding task. Pleias co-founder Alexander Doria reported error messages and endless loops. One widely shared video of "Manus" operating smartphone apps was, per Ji himself, not Manus at all. In one shared comparison, Deep Research finished a task in under 15 minutes; Manus failed after 50.

MIT Technology Review's more favorable hands-on found the same texture: better output than ChatGPT's Deep Research on two of three research tasks, but slower, blocked by captchas and paywalls, and prone to freezes under load. Ji acknowledged a higher failure rate than Deep Research. The honest mid-March summary: impressive medians, unreliable tails, unverified claims.

Why Constrained Agents Ship

Autonomy multiplies error rates. An agent that completes each step with 90% reliability finishes a 20-step task about 12% of the time (0.9^20 ≈ 0.12). General autonomy demands long chains across arbitrary domains — exactly where compounding is worst. Constraining an agent shortens the chain and narrows the domain. That is why every agent that shipped by early 2025 is constrained.

OpenAI's Deep Research (February 2, 2025) is read-only: it browses and writes a report; it does not act. Operator (January 23, 2025) acts, but asks the user to confirm before purchases and logins. Anthropic's computer use, in beta since October 22, 2024, is explicitly labeled experimental and recommended for supervised, low-risk tasks. Coding agents propose diffs; humans merge them.

We apply the same pattern in client work: bounded scope, checkpointed side effects, human approval at every irreversible step, full traces for audit. Approval gates are not a concession to weak models. They are the mechanism that turns a 70%-reliable agent into a 100%-accountable workflow — which is the property enterprises actually pay for.

Outlook From March 2025

What we expect from here. Orchestration scaffolds will commoditize quickly — OpenManus needed days, and model vendors will absorb the loop into their own products. Benchmark claims will keep outrunning reproduction until agent benchmarks report cost, latency, and failure tails alongside pass@1; GAIA-style accuracy alone will not settle arguments. Manus itself may yet become a good product — the team is already testing Claude 3.7 Sonnet — but "general agent" will remain a marketing term through 2025.

Our position is unchanged by the demo cycle: deploy agents where the task is narrow, the steps are checkpointed, and a human owns the irreversible ones. Widen autonomy only as measured step reliability rises, not as demos improve. The teams that ship value this year will not be the ones with the boldest chart. They will be the ones that can show their traces.