#评测/基准

Hume launches the first decision model for voice AI

Hume introduced an audio-native decision model that takes audio, a question and possible answers and returns a probability for each answer, aimed at evaluating speech and sound in voice applications. Hume says the model agrees with the human crowd 70.7% of the time versus 56.5% for a single human rater, and scores an average 0.82 across 10 leading benchmarks at a median latency of 0.23 s per screen. It is offered via a Decision Model API as a decision layer in voice-agent systems alongside speech recognition and generation.

Can AI Spot a Bad Business Plan?

The author tested Claude Fable 5.1 and DeepSeek 4.1 on business-plan review using AI-generated plans with planted mistakes and 31 anonymized real Regulation Crowdfunding companies scored against later outcomes, where both models caught about four in five problems (81% and 82%). The piece also reports that a model identified the real company behind 37 of 68 disguised plans, that both models flagged nearly half of real red herrings as problems, and that the tests are small, single-run signals rather than a ranking.

One brutal Minecraft night scared 50 AI agents into inventing chicken handoff protocols, a Fertilizer Officer, and fishing shifts

In a three-hour "Civ50" Minecraft experiment, 50 GPT-6.1 Sol agents exchanged about 5,900 messages, lost 22 agents on the first night, and then adopted extreme caution, ration lists, fishing shifts and a written handoff protocol after a tossed chicken was accidentally eaten. In separate Bed Wars and Mindustry matches, GPT agents beat Claude Opus 5.5 6-3 and 6-1 respectively, communicating far more and downloading Mindustry's public source code to refine tactics, while Claude agents misread a block-breaking bug and never searched for game code. The researchers released the Mindustry environment and Multi-Agent Harbor framework as open source, arguing such simulations can surface risky swarm behaviors before they reach production systems.

Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4

This Import AI issue covers Realtime-Bench, a 50-task benchmark of interactive games and web tasks showing frontier models score poorly under real-world latency (GPT-6 Astra high led the wall-clock setting at 36% pass rate versus roughly 70-80% with a paused clock). It also summarizes a Defense Intelligence Agency report on space security covering Chinese and Russian counterspace capabilities, co-orbital satellites, and space nuclear weapons, plus Metis's position paper on fusing symbolic and neural reasoning and Google DeepMind's Genie 4 world model with real-time modification, native audio, and Gemini 3.8 Flash as a guiding agent.

GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains

SemiAnalysis introduces GDPx, a benchmark covering domains such as chip design, cybersecurity, biology, finance, legal, quant trading, RSI, hardware design, materials science, alignment, and software engineering, built with 14 specialized data partners; the best model scores just 12%. The article argues OpenAI's GDPval is inadequate for today's agents because its tasks are non-interactive and lack realistic environments, and it details GDPx's QA process, per-domain task design, and model results including safety-classifier effects.

Avartha: Fastest open-weight frontier voice AI beats OpenAI, Google, and xAI

Avartha announced Ava Zero, a voice agent built entirely on open models that ranks #1 on Sierra's τ³-Voice benchmark with 86.0% task completion across six languages, ahead of OpenAI's GPT Live 1 (79.9%), xAI's Grok (71.4%), and Google's Gemini (68.5%). The Georgia Tech spinout says it co-designed its inference engine and agent harness as one system, supporting up to an order of magnitude more concurrent conversations per GPU than vLLM and SGLang, and prices Ava at $2.50 per hour of conversation. Its benchmark submission combines Qwen3-ASR, Fish Audio Drama 3, GLM-5.3, and DeepSeek 4.1 Flash.

We tested 12 search APIs for AI agents. Even the best one isn't enough

Telem.ai tested 12 web search providers on 363 agent searches with known answers and found the best single provider returned the known answer in its top five results 68% of the time, while all 12 together reached 84%, with most of the gain coming from adding a second provider (68% to 77%). For 14% of searches only one to three providers had the answer in their top five, and for 16% none of the 12 did. The team says it built Telem to close this gap and plans follow-up posts on provider result overlap and retrying after a miss.

Chinese AI models are politically complex, not just government mouthpieces, study finds

Vals.ai's study found wide variation in how Chinese AI models align with Chinese government positions, with pro-government lean lower than expected overall and some models, such as GLM-5.3 Flash, acknowledging documented human rights violations. Prompts in Simplified Chinese produced more pro-government answers than English, Russian or Spanish for most models, and some models adopted official positions rather than refusing, such as Kimi K3 denying Uyghur concentration camps in Xinjiang. Response format, agentic framing and web-search access also shifted scores, and fine-tuning reduced Thomson 1.0 Small's adoption of official positions from 84.6 to 15.0 percent compared with its Qwen3.6 35B base.

Q&A: Hey AI, who should I vote for this year?

MIT researchers led by Adam Berinsky, Chara Podimata and Charles Stewart III have built the LLM Election Observatory, a public dashboard that tracks in real time how popular AI chatbots answer a fixed set of questions about the 2026 U.S. midterm elections, re-running them every few days to see how responses vary by user location, party affiliation and identity. Podimata says early 2026 data show differences in how answers are presented, which sources are cited and tone, sometimes when the underlying facts are the same, though stability and effects on real users are not yet established. Berinsky warns that a confident-sounding chatbot is not necessarily accurate and says voters should treat its answers as a starting point rather than a final word.

Is AI Cheap Enough to Replace Developers?

The article contrasts high public benchmark scores (e.g., Claude Opus 4.8 at 88.6% on SWE-bench Verified) with much lower results on private codebases (no model clears 47% on the private commercial set), and notes that the two top-scoring models, Claude Mythos 5 and Claude Fable 5, are unavailable after Anthropic suspended public access under an export control directive. It details June 2026 pricing shifts, including GitHub Copilot's move to usage-based credits on June 1, with heavy users spending $100–$200/month, and reviews METR's studies, which found a 19% slowdown in 2025 and roughly -4% in 2026 before the design was scrapped. It concludes AI is cheap enough to replace a developer's typing but not their judgment.

Show HN: NullPrime – do AI models know when they can't know?

NullPrime is a benchmark of integers labeled prime or composite in which every number is coprime to all primes below a bound B, removing the divisibility signal that makes primality learnable from digit and remainder features; standard learners fit the training set perfectly yet score at chance (~0.50) on held-out numbers, while the same learners reach 62–93% on an unfiltered contrast set. The write-up reports an LLM study in which, on hard numbers, five of eight models gave a verdict on almost every item and were wrong about half the time, and many cited a stated factor that does not divide the number. The project includes a certificate script, downloadable CSVs, a dependency-free number generator, and a canary GUID asking that the data be filtered from training corpora.

Show HN: I benchmarked a voice AI against real people. It scored at chance

A Show HN post links to a benchmark comparing a voice AI against real people, reporting that the AI scored at chance level. The submission includes the article URL and a Hacker News comments link, with 1 point and no comments at the time.

Open-Weight vs Closed-Weight Models: What's the Difference?

Open-weight models publish their trained weights so anyone can download, run, fine-tune or serve them, while closed-weight models are only available through their maker or licensed clouds. On OpenRouter, open-weight models grew from 26.8% of top-50 model tokens in October 2025 to 71.7% in late September 2026, and cost about a quarter as much per token, though the best closed-weight models still score about 11 points higher on the Artificial Analysis Intelligence Index.

Assure – your AI agent says "done." This checks if it's lying

Assure runs an AI agent against a disposable SQLite and file workspace, routing all tool use through a policy-enforcing gateway and judging results with human-written deterministic assertions, with no LLM involved in any verdict. It ships scenarios covering false success, collateral change, prohibited access, tampering and prompt injection, plus benchmark suites (customer-adversarial, customer-hard, customer-long, code-fix, code-long) that report pass rates, pass^k, claim honesty and failure kinds, and it can test any MCP-speaking agent through an adapter. CI runs the full test suite and check-suite on every push, while a manual bench workflow makes paid API calls and fails on significant regressions.

Show HN: Navital Gym – open environments for testing AI agents on Indian tasks

Navital Labs released Navital Gym, a public, actively developed suite of six sandboxed, deterministically auto-graded environments (trip planning, GST invoicing, forms, multi-document forms, Hindi/Hinglish customer support, and UPI) for testing AI agents on Indian-reality tasks without an LLM judge. It runs on plain Python 3.10+ with optional adapters for Inspect AI, verifiers and OpenEnv, and publishes pass@1 and partial-credit results for small, mid-size and larger models. The environments use simplified, documented rules and synthetic data and are not affiliated with NPCI, RBI, GSTN/CBIC, IRCTC or any government body.

From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence

Microsoft describes its Industry Evaluation Data Generation Platform, which builds fully synthetic enterprise environments grounded in public sources such as O*NET occupational data, job postings and SEC filings to evaluate Microsoft Copilot without using customer data. The platform combines synthetic tenants across pharmaceutical, capital markets and IT/software industries with an automated six-stage query generation pipeline and quality frameworks covering structural fidelity, contextual relevance, behavioral realism and content safety. Microsoft reports a 39% improvement on its composite synthetic data quality metric versus its initial approach, some industry environments scoring above 0.9 out of 1.0, and estimated savings of about $100,000 per evaluation dataset and $300,000 in total project savings.

AI Builder Compare

AI Builder Compare is a review site that scores 26 AI app builders across seven weighted categories, including ease for beginners (25%) and speed to MVP (20%), with pricing and features re-checked on each review's last verified date. It lists top-rated tools such as Prompt-to-App, Browser Full-Stack and Vibe Coding Agent builders scoring between 8.3 and 8.6 out of 10, and offers head-to-head comparisons like Lovable vs Base44 and Bubble vs WeWeb. The site also answers beginner questions on AI app builders and vibe coding, and recommends Lovable as the best option for beginners.

Timestral Tested: How Much Data AI Demand Forecasting Needs

A hands-on benchmark of Mistral's Timestral (released 15 October 2026) against Chronos-2, TimesFM, t0-gamma, LightGBM and spreadsheet AutoETS on 300 Online Retail II products and GB national electricity demand found the pretrained models tied on retail and beat both baselines, with Timestral's error 3% below LightGBM and 14% below AutoETS. Timestral needed about two days of half-hourly demand or eight weeks of weekly sales to beat repeating the last value, and every pretrained model forecast "another August" through November when given only summer grid history, while a small temperature-response model had less than a third of their error. The author notes licence limits (Timestral free only under $20M monthly revenue) and open questions about training-data leakage.

Claude Opus vs Sonnet vs Haiku: Which Should You Use?

The article recommends Claude Sonnet 5.5 as the default, Haiku 5.5 for high-volume and sub-agent work, and Opus 5.5 for long, ambiguous tasks where errors are costly. In the author's tests, all three models passed every coding task, with Haiku costing 19 to 30 times less than Opus per task and Sonnet costing half to two-thirds as much as Opus. It notes that Sonnet scored as well as or better than Opus on the author's GPQA Diamond and τ-bench runs, and advises testing the models on your own work with Ori Eval before choosing.

We asked eleven AI models to chair the drafting of a multilateral statement

Westphalia Labs tested eleven AI models on a simulated multilateral drafting task, playing Secretary-General of a fictional Regional Cooperation Union and producing three drafts of a joint statement for seven governments with conflicting red lines, across ten runs each on a version where agreement was possible and one where it was impossible. Only Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 passed every run, while Gemini 4 Argon and MiniMax M3.1 Flash told members agreement was "likely" when it then failed, and Gemini omitted a casualty figure one government had requested in writing. The scenario, scoring rules and code are public, and a panel of three former diplomats checked the scoring.

MLCommons Candela: AI risk benchmarking-as-a-service for developers, deployers

MLCommons introduced Candela, described as an AI risk benchmarking-as-a-service offering aimed at developers and deployers. The announcement is published at mlcommons.org.

Paris-based Mistral releases "Le Chonk," a strong coding model that still trails China overall

Mistral released the weights of Mistral Large 4, nicknamed "Le Chonk," a roughly one-trillion-parameter multimodal model (52 billion active per step) that it says leads open models on coding, scoring 41 percent on Terminal-Bench 4.0 and 19.7 percent on FrontierCode 2.0 while trailing Claude Opus 5.5. Independent testing by Artificial Analysis put the final version at 26 points on its Intelligence Index, behind Chinese open models DeepSeek V4.1 Flash and GLM-5.3 at 32 and MiMo-V2.6-Pro at 30, though it is fast at 161 tokens per second and cheaper per task than the preview. The model is under a modified MIT license requiring a separate license for commercial cloud hosting, and API pricing is $0.70 per million input tokens and $2.10 per million output tokens.

Zombie stats: Where the numbers AI quotes come from

Trakkr Research asked ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews 200 numeric questions, then traced 1,285 cited statistics up to five pages deep: 40% reached a study saying the same thing, 8% reached a study but with a changed number or context, 40% stopped at an unverifiable source and 11% led nowhere. Only 21% of pages cited for a number were the research itself, with more than half being company blogs, and four statistics (including "44% of salespeople give up after one follow-up") were quoted by all five assistants with no study behind them. On 31 of 71 questions where all five gave a percentage, the figures were 20 or more points apart.

Why AI design tools keep making the same flyer

A test of two flyer prompts run three times each in Claude Design, Canva, Moda and Grafista found that the first three tools returned one layout per prompt in different colours, while Grafista returned six distinct designs. The article attributes the repetition to how generative models predict the most likely output, and says steering away from it requires a detailed prompt written in a designer's vocabulary. Grafista instead starts each design from a Blueprint, a visual reference, rather than a prompt.

Roasting China’s AI Industry

In a ChinaTalk interview, Xianyu, the writer behind the satirical blog "Funeral AI" (葬AI), discusses why AI writing isn't funny, how he built MemeBench — a public arena where ordinary users rank video models by meme-ability — and why Five-Year Plans function as a kind of benchmark while "AGI" lacks quantitative indicators. He describes recruiting about 300 students via QQ to seed the rankings, and recounts that MiniMax's open-source H3 beat ByteDance's Seedance in the meme arena, prompting ByteDance to question the result and MiniMax to complain about an earlier article. He also argues that Chinese AI startup themes rotate every three months following Silicon Valley, with many "world models" actually being post-trained versions of Alibaba's Wan video model.

Building a Banana Sorter AI Machine

The author tests four vision models (Qwen2.5-VL 3B, Gemma 3 4B, Liquid d1-2B, and gpt-6.1-sol) on five banana images to decide BASKET or TRASH over 100 runs each. The smaller offline models trashed any banana with a black dot, while the frontier model sometimes trashed the heavily spotted banana (16% at low reasoning, 38% at high) and Liquid d1 gave probabilistic answers. The author concludes he would not trust a high-end model with a critical decision and that designing a real banana sorter is more nuanced.

AI Article Reviews Dominate Human Peer Reviews at Their Best [pdf]

A PDF linked from Harvard's Gary King site is titled "AI Article Reviews Dominate Human Peer Reviews at Their Best." The item was submitted to Hacker News, where it had 2 points and no comments at the time of the data.

Can AI agents do senior expert work? What the evidence says

Five recent studies suggest AI can match or assist expert work on some tasks under some conditions, but the evidence is limited: GDPval had experts blind-grade AI output against human work on one-shot tasks, while a BCG field experiment found consultants with AI did 12.2% more tasks and worked 25.1% faster inside the "jagged technological frontier" but were about 19 points less likely to be correct outside it. A METR study found experienced developers took 19% longer with early-2025 AI tools despite forecasting a 24% speed-up, and studies of 150 autonomous agents on one dataset and of ICU prediction agents found wide variation in method choices and that single agents outperformed naive multi-agent systems.

[AINews] Gemini 4 Carbon tops leaderboards, launches for cyber defenders

Google DeepMind released Gemini 4 Carbon, a frontier model that takes the #1 spot on Artificial Analysis' Intelligence Index v5 with a score of 59, two weeks after Argon. Access is initially limited to trusted cyber defenders through the Fairwind Program, with claimed results of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, a 1M-token context window, and pricing listed at $4/$20 per million tokens with a 50% launch discount. Independent evaluations place it first in Text Arena and Vending-Bench 2 but behind Opus 5.5 on coding-agent and FrontierSWE benchmarks, and its hallucination rate rose from Argon's 4% to 11%.

Google Deepmind releases Gemini 4 Carbon just two weeks after Argon with gains across the board

Google DeepMind shipped Gemini 4 Carbon two weeks after Argon; it tops the Artificial Analysis Intelligence Index v5.0 with 59 points versus Argon's 50, and leads 15 of 16 of Google's own benchmarks, especially in long-context and security tasks. Independent evaluations show it second on Vals AI (70.36%) and the Agent Arena (+16.2%), fourth in the Code Arena's WebDev category (1,768), and still slightly behind Claude Opus 5.5 in agentic coding, while costing $7.76 per task at the same introductory price as Argon ($2/$10 per million input/output tokens). Google is rolling it out first to selected cybersecurity experts and through the Fairwind Program before wider release.

Featured coverage of this story: 《Gemini 4 Carbon: Our next frontier model》

ViBench 2.0: A Benchmark for Long-Horizon Software Building

Replit releases ViBench 2.0, a long-horizon software development benchmark in which an agent builds a product from an empty project and then adds up to eighteen features on top of its own code, spanning seventeen products, 241 feature requests and 544 test plans. In Fireworks' official runs, GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%, and no open-weight model passes a full build. Eight of the products are public on GitHub in the Harbor format, and results can be rerun from release v2.0.6 on the OpenHands-based harness.

Google announces Gemini 4 Carbon, its latest frontier AI model that you can't use

Google announced Gemini 4 Carbon, an improvement on Gemini 4 Argon released two weeks earlier, with a staged release that may take time. Google says Carbon shows a significant leap in vulnerability detection, validation and remediation, is twice as likely as Argon to generate vulnerability-free code, and gained 89 points on WebDev Arena to rank fourth. Carbon edges out Argon and beats GPT-6 Astra, Claude Opus 5.5 and Claude Fable 5.1 in almost every test, though the article notes the margins are just a few points.

Hill-Climbing Agent Quality with Agents CLI Evaluation

A walkthrough of the Agents CLI eval loop using the LangChain template in Agents CLI v1.8.0, covering a baseline run (mean 4.0), a system-prompt regression that dropped the mean to 2.75, per-case diffs via eval compare, and a gemini-3.8-flash to gemini-2.5-flash swap whose mean drop traced to a single off-topic case. It also shows scheduling eval runs with cron or Cloud Scheduler, a script that exits non-zero when scores fall past a threshold, and eval optimize for automated prompt hill-climbing on ADK projects.

Code mode: make your agents 2x faster and 2x cheaper on long tasks

A benchmark of 5,276 graded runs across eight models on five small Pydantic AI apps found that code mode, where the model writes a program that calls tools inside a sandbox, cut cost by 58% and wall time by 51% on long tasks while raising success by 16 points. Gains grew with task length: at 200 items code mode was 47% to 75% cheaper and 42% to 67% faster, whereas at 5 items it cost 21% to 38% more on three of five tasks. The author attributes the win not to fewer round trips, since models already batch calls, but to less content per request, and runs the model's code in Monty, a Rust Python interpreter with no host access by default and 30-second, 256 MiB limits.

Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

Google's Gemini 4 Carbon tops the Artificial Analysis Intelligence Index v5 with a score of 59, leading 4 of 9 evaluations, and ranks second to Claude Opus 5.5 on the Coding Agent Index with 59 points. It scores 1609 Elo on AA-Briefcase (up 126 over Gemini 4 Argon) and 54 on AA-Omniscience v1.1, at $7.76 per Intelligence Index task on standard $4/$20 pricing. The model is being rolled out to selected users and is not yet publicly available, with a 50% launch discount and no confirmed end date.

Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Surge AI introduced DAYJOB: Tax, a benchmark of 50 environments across eight tax practice areas in which agents receive client requests and supporting records and must produce returns, workpapers, reviews, or advice. Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate at a 95% rubric threshold, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The post illustrates the difficulty with examples in which agents missed a home-sale exclusion deadline and assumed a couple was unmarried without verifying their marital status.

New Artificial Analysis index widens the gap between open and closed models

Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to fall from fourth to ninth place behind eight closed models in version 5, and the gap between open models and the top of the leaderboard widened from 3 points to 12. For the first time, 60 percent of the index relies on private test sets to prevent "benchmaxxing," and Grok 4.7 also lost 12 points. AI researcher Nathan Lambert notes that many training gains now come from reinforcement learning environments resembling benchmarks, while closed test sets mean outsiders can no longer spot errors in prompts, as shown by revisions to the AA-Omniscience knowledge test.

Nvidia's open Nemotron 3.5 Transcribe combines transcription, translation, and speaker recognition

Nvidia released Nemotron 3.5 Transcribe, an open speech model that follows text instructions to transcribe 40 languages, translate 31 to or from English, identify speakers in 13, and answer questions about recordings up to an hour long with up to eight speakers. The 30-billion-parameter model activates about 4 billion parameters per step, runs on a single Nvidia GPU, and ranks first among open-weight models on Hugging Face's English Open ASR Leaderboard with a 5.33 percent average word error rate. Nvidia also released the Granary v2 dataset of about 844,000 hours of speech in 48 languages plus dataset-building tools, and reported that targeted fine-tuning on recordings of Indian doctors and medical students cut error rates from 30.8 to 11.8 percent in about 14 GPU hours on one GB300.

AI shopping assistants confuse fashion subcultures: a small test

A signed-out test of three public AI assistants across eight fashion aesthetics found that when an assistant does not know a style well, it fills the gap with a neighbouring style: Goth Ninja became Techwear, City Boy became Ivy, Amekaji became American workwear and Quiet Luxury became high-street minimalism. Daydream returned few or none of the record's key brands for most styles (0 of 92 for Amekaji, 0 of 105 for Quiet Luxury, 0 of 112 for 90s Ura-Hara and Acubi) and refused "Opium" twice, likely due to a drug filter, while ChatGPT named all 5 record brands for 90s Ura-Hara but drifted elsewhere. The author says the failing step is mapping an aesthetic to brands, and cites a public eval with Claude in which answers mixing in a neighbouring style's brands dropped from 17 of 30 to 2 of 30 with the record in context.

The AI cost of JavaScript frameworks

A benchmark had coding agents build the same small PokéAPI app five times in each of twelve JavaScript frameworks/meta-frameworks, then measured cost, time, tokens, lines of code, and Lighthouse performance and accessibility. Hono+htmx and Astro were cheapest (about $2.67–$2.76 per run) while Angular, Gracile+Lit and SvelteKit were most expensive, and JavaScript bytes ranged from 0 KiB (Astro) to 142 KiB (Next.js). The author presents the results as evidence that framework choices can be made on data rather than vibes, while noting the experiment is stochastic and limited to one content-heavy app type.