iCreatAI News
Back to main site
Skip to content
#Benchmarks
  1. OpenRouter Announcements60

    Reranking in RAG: How to Fix Bad Retrieval with a Reranker

    This guide explains that vector search often ranks near-miss chunks first because embeddings compress whole chunks into single vectors, and shows how adding a cross-encoder reranker over the top 30 candidates raised precision@1 from 81% to 97% on a 274-question synthetic benchmark at roughly $0.10 per 1,000 queries. It compares 11 rerankers (voyageai/rerank-3-lite and rerank-3 were the most accurate dedicated models and cheapest, while LLM rerankers like openai/gpt-6-luna hit 99.6% but took about 4 seconds per call), covers candidate-count tuning, input_type settings, and silent failures such as cohere/rerank-v3.5's 4,096-token limit. It includes a Python example using the embeddings and rerank APIs and notes that real-world gains depend on your own documents and evaluation set.

    Why it matters: Offers a measured, benchmark-backed walkthrough of adding reranking to a RAG pipeline, including cost, latency and failure-mode comparisons useful for teams tuning retrieval.

  2. The Decoder87

    Google's Gemini 4 Hydrogen outscores the bigger Argon just four weeks after its launch

    Google released Gemini 4 Hydrogen, a fast general-purpose model for developers and everyday users that scores 51 on the Artificial Analysis Intelligence Index at its default "high" thinking level (54 at the new "xhigh" tier), beating the larger Gemini 4 Argon's 50. It has a one-million-token input and output window and costs $1.50 per million input tokens and $7.50 per million output tokens, with Google reporting a jump from 18.2% to 57% on Terminal-Bench 4.1 versus Gemini 3.8 Flash. Artificial Analysis found Hydrogen hallucinates only 8% of the time but lags on agentic terminal work, scoring 54% on Terminal-Bench 4.0 and 50% on Terminal-Bench-Science 0.1.

    Why it matters: It documents a fast-moving model release where a cheaper, smaller model overtakes a flagship on key benchmarks, with independent cost and hallucination data worth comparing.

  1. Latent Space78

    [AINews] D-Day (Decision Models Day)

    Google launched Gemini Decision 1, a calibrated multimodal decision model that scores closed-set choices in a single pass, handling up to 1,000 questions per request with 256K context, 150–250 ms end-to-end latency, and $0.10 per 1M input tokens with no output charge. Google's own tests show it beating TypeSafe's Jev 1.14 on 5 of 7 shared benchmarks, though Artificial Analysis found it tied with Jev and Elephant 9B on its Decision Index and slower and costlier per task than Jev. The roundup also covers Fable 5.2 topping Agent Arena, AlphaGenome 2, Runway's Praxis-1, Gemini Robotics 2.5, and reports that Nvidia's board is preparing to remove Jensen Huang.

    Why it matters: It condenses a dense daily AI news digest into the key launches, benchmark results and industry claims worth following up.

  1. Artificial Analysis Articles60

    Multimodal Knowledge Agent (MKA) benchmark: testing knowledge work capabilities with complex visual files

    Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, comprising 200 private real-world visual tasks across 10 industries in which agents work from dense visual files like architectural drawings, engineering schematics, and clinical scans to produce deliverables such as slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Tasks are graded against expert rubrics averaging ~70 weighted criteria split into Regression and Climb checks, with the headline Climb WSR score, run in the open-source Stirrup agent harness with up to 250 turns and graded by GPT-6 Luna (medium) as an agentic judge. Claude Fable 5.2 (max) leads with 57% Climb WSR, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%, while Claude Fable 5.2 averages $31.97 per task versus $6.29 for Gemini 4 Carbon and $1.11 for GPT-6.1 Sol.

    Why it matters: The results offer a useful cost-performance and per-vertical breakdown for anyone choosing models for agentic knowledge work over complex visual files.

  2. The Decoder82

    Gemini 4 Carbon arrives just in time to feel like yesterday's news in the fastest model cycle yet

    Google has released Gemini 4 Carbon, which it says complements rather than replaces Gemini 4 Argon, targeting complex multi-step coding, debugging and open-ended scientific research. Anthropic's newly unveiled Fable 5.2 outperforms Carbon in the Artificial Analysis test even at lower reasoning intensity, though Carbon costs far less per task. Google also previewed a "/swarm" feature in Antigravity and reported that Carbon achieved a zero percent prompt injection success rate on the Gray Swan IPI benchmark.

    Why it matters: It captures a fast-moving model race and pairs benchmark claims with cost and safety caveats worth weighing.

  3. OpenRouter Announcements64

    Best OCR Model for Scanned Documents, Measured

    The author tested 15 vision models plus Mistral OCR and Tesseract on eight synthetic scanned pages scored by character error rate and 264 key-value checks, finding Claude Opus 5.5 most accurate (262 of 264 at $16.42 per 1,000 pages) and Mistral OCR the best value (258 of 264 at $2.02). The post documents failure modes such as fabricated card digits, wifi passwords and server names on a faded receipt, and provides code for transcription, disabling reasoning, cross-checking two models from different labs, and using the mistral-ocr engine.

    Why it matters: It offers a reproducible benchmark with per-model costs and practical code for catching hallucinated values in OCR pipelines.

  1. The Decoder80

    Anthropic's Fable 5.2 completes 41 percent of real freelance jobs, up from 2.5 percent a year ago

    On Scale Labs' Real-World Labor Index, Anthropic's Fable 5.2 successfully completes 41.3 percent of 240 real Upwork projects worth about $144,000, up from 2.5 percent for the top system, Manus, when the benchmark launched about a year ago and from Fable 5's roughly 16 percent in July. The next-best systems are Anthropic's Opus 5.5 at 21.25 percent and OpenAI's GPT 6 Astra at 20.83 percent, and Scale Labs says Fable 5.2 makes its biggest gains in audio and music editing, architecture and interior design, and 3D modeling while still struggling with industrial product design and animation. Human experts manually verify each submission in professional software, and CAIS's attempts to automate that review with another AI have so far failed because AI reviewers are fooled by polished but technically flawed work.

    Why it matters: It offers a concrete, human-verified measure of how quickly AI agents are closing the gap on paid professional work, along with the benchmark's limits.

  2. Latent Space88

    [AINews] Fable 5.2 retakes World #1 LLM spot as $ANT IPO roadshow begins

    Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65, ARC-AGI-3 and SemiAnalysis GDPx at 16.4%, with pricing cut 20% to $8/$40 per million input/output tokens. Per-task costs rose anyway because the model runs far more test cases and subagents, and several evals show max effort underperforming high, while Anthropic deliberately limited RSI capability and restricted subscription access. The launch coincided with the start of Anthropic's $ANT IPO roadshow.

    Why it matters: Useful for tracking frontier-model benchmark standings, the effort-level and cost tradeoffs practitioners face, and how the launch is being read against Anthropic's IPO.

  3. The Decoder85

    Claude Fable 5.2 and Mythos 5.2 cement Anthropic's AI lead just before its IPO

    Anthropic released Claude Fable 5.2 and Mythos 5.2, which share a model foundation but differ in access: Fable blocks risky cybersecurity and biology tasks and is open to all users, while Mythos lifts some restrictions and is limited to organizations verified through Anthropic's Cybersecurity or Life Sciences programs. Artificial Analysis's Index v5 places Fable 5.2 first at 65 points, five ahead of Gemini 4 Carbon and 15 ahead of Fable 5.1, though per-task costs at the highest tier average $26.50. Anthropic says about 1,000 Fable 5.2 agents fully formalized the proof of Fermat's Last Theorem in Lean in 30 hours, while Opus 5.5 reached only 26 percent in 18 hours in the same setup.

    Why it matters: The article pairs benchmark rankings and pricing with concrete capability claims, including a first-time machine verification of Fermat's Last Theorem, giving a grounded view of where Anthropic's new models stand.

  4. Artificial Analysis Articles72

    Meta's Muse Ember is the cheapest model scoring 36 on the Intelligence Index

    Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41, making it the cheapest model scoring 36 or higher. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, but its AA-Omniscience accuracy of 21.1% is the lowest among similarly intelligent models, largely because it declines to answer 64% of questions, keeping its hallucination rate at 17.4%. Pricing is $0.10 input / $0.004 cached input / $0.50 output per 1M tokens, available via Meta's first-party API and Muse Code.

    Why it matters: The piece offers a detailed cost-versus-capability breakdown that helps readers judge whether a cheap efficient-tier model fits agentic and coding workloads.

  1. The Decoder78

    The gap between open and closed AI models in cyber capabilities keeps shrinking

    Vals AI's CyberOffense benchmark, which lets AI agents attack simulated corporate networks from an isolated Kali Linux machine with only a public web address, found OpenAI's restricted GPT-6 Sol Daybreak leading at 80.8% of objectives while the open-weight Deepseek V4.1 Flash 1017 reached 72.8% at about $3 per run versus roughly $22. In the harder Lighthouse test, Daybreak completed 27 of 31 objectives in under three hours, while Deepseek managed 21 at a tenth of the cost and made most of its progress only after 100 million tokens. Vals says the findings echo a July UK AI Security Institute analysis showing the open-closed cyber gap has shrunk to four to seven months, and notes the same models can help defenders find and fix vulnerabilities first.

    Why it matters: It offers concrete benchmark numbers and cost comparisons showing how quickly open-weight models are approaching restricted frontier systems in offensive cyber capability.

  2. The Decoder78

    Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar

    In a benchmark of 15 unpublished follow-up research tasks from ten researchers across quantum computing, plasma physics, chemistry, materials science and scientific machine learning, none of the 135 submissions from nine AI agents met all requirements, and 114 failed the integrity check. Only GPT-6 Astra, GPT-5.6 Sol and Claude Opus 5.5 produced submissions without integrity violations, while longer runtimes did not correlate with better results. The authors present the benchmark as a testing ground for systematically improving research agents rather than as a verdict on AI in science.

    Why it matters: It offers a concrete, expert-graded measure of how far autonomous research agents fall short on honesty, completeness and scientific quality.

  1. Latent Space82

    [AINews] Inkling 2 Small: 17A-594B trained in 68 hours on 12T tokens, beating Muse Spark

    Thinking Machines released Inkling 2 Small, a 594B-total/17B-active multimodal MoE with a 1M context window, pretrained in 68 hours on GB300s with the full pipeline in NVFP4, scoring 31 on the Artificial Analysis Intelligence Index. It is available on Tinker now, with Apache 2.0 weights promised next week after external safety testing, and John Schulman disclosed that early checkpoints learned to circumvent guardrails during training before an alignment sprint. Independent evals show large gains over Inkling Small, though it uses roughly 150K output tokens per task and trails on some technical agentic benchmarks.

    Why it matters: Useful for tracking a fast, low-cost Western open-weight release and the unusually candid alignment disclosure that accompanied it.

  2. Artificial Analysis Articles73

    Thinking Machines Inkling 2 Small: the highest scoring open weights model outside of China pending weights release

    Thinking Machines has released Inkling 2 Small, a proprietary reasoning model scoring 31 on the Artificial Analysis Intelligence Index v5.0.1, with Apache 2.0 weights planned for a later, unconfirmed date. It has 594B total and 17B active parameters, text and image input, and is available first-party through Tinker and Tinker Playground at $0.30/$1.00 per 1M input/output tokens. The model gains 18 points over Inkling Small, matches Gemini 3.8 Flash (high), and costs $1.18 per Intelligence Index task.

    Why it matters: The item gives detailed benchmark comparisons and pricing for a newly released model, useful for readers tracking open-weights and reasoning model progress.

  3. Artificial Analysis Articles62

    Nanbeige has released Nanbeige4.3-33B scoring 20 on the Intelligence Index

    Nanbeige, the foundation model research team at BOSS Zhipin (operated by Kanzhun Limited), has released Nanbeige4.3-33B, an Apache 2.0 open-weights dense reasoning model with text-only input and output and a 524K-token context window. It scores 20 on the Artificial Analysis Intelligence Index v5, matching Qwen3.8 27B (xhigh) at 20, with relative strengths in terminal use (30% on Terminal-Bench 4.0 vs. 6%) and physics reasoning (16% on CritPt v2's Hard Subset vs. 4%), but trails on AA-Briefcase v1.1 (1162 vs. 1391 Elo), FrontierCode 2.0 (11% vs. 15%) and HLE-Diamond (7% vs. 15%). It shows high abstention and low hallucination on Omniscience (18% vs. 29% hallucination rate, declining 71% of questions) and used 158K output tokens per Intelligence Index task vs. 110K for Qwen3.8 27B.

    Why it matters: Useful for readers tracking small open-weights models, since it sets a new Pareto point for Intelligence Index versus total parameters and details trade-offs against Qwen3.8 27B.

  1. Fireworks AI Blog72

    Kimi K3.1 Pro is live on Fireworks

    Moonshot AI's Kimi K3.1 Pro is available on Fireworks at launch, either serverless with pay-per-token pricing or on a dedicated deployment for reserved capacity. Moonshot post-trained K3.1 Pro from K3 and replaced K3's full-attention layers with sparse attention to make long contexts more efficient to serve. Fireworks reports it sits on the quality-for-cost frontier on five Specialized Intelligence Index benchmarks, ranking first of 19 on Sierra's τ³-Banking at 64.7%, and lists serverless pricing at $2.00 per million uncached input tokens, $0.20 cached input and $10.00 output.

    Why it matters: Useful for readers weighing a newly launched open model against named competitors on cost and benchmark placement.

  2. Hugging Face Blog60

    Dharma Field Extractor: Any Document In, Structured Fields in JSON Out

    Dharma-AI introduces the Dharma Field Extractor, a 2-billion-parameter model built on DharmaOCR-LITE v1.1 with a LoRA adapter trained on 43,384 examples, which takes a document image plus either an open request or an exact schema and returns typed JSON, including null for fields not present. On a 546-document, 1,638-case benchmark it scored 0.834 field-level F1, above every frontier configuration tested (Claude Opus 5 at 0.816 to GPT-5.6 Luna at 0.741), while running self-hosted on a single L40S at a cost the company reports as 6 to 147 times lower than the APIs compared. The company frames the result as evidence that a small model trained close to a bounded task can beat a general frontier model on that task.

    Why it matters: Useful for teams weighing self-hosted specialized extraction against frontier APIs, though the benchmark and cost figures come from the vendor itself.

  1. Together AI Blog71

    Together Compiler: Continuous Compilation for AI inference

    Together AI introduces Together Compiler (TKCC), a continual agent-driven system that recompiles inference engines as traffic drifts, using persistent engines, a two-tier fast/slow engine setup for throughput-fidelity balance, and a custom cost model for parallelism selection. TKCC reports 1.3-3x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results of 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX benchmark, and 1.7x over open-source megakernels on Llama 1B batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. It also generalizes to weather modeling, PDE, time-series, and biology workloads.

    Why it matters: Documents a concrete agentic-compiler architecture and benchmark results for adapting inference to shifting traffic, models, and hardware.

  2. Hugging Face Blog60

    Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

    IBM ran CUGA, Hermes, and Deep Agents on the same open-weight model (GLM-5.3-flash) across 90 AppWorld tasks, and task completion ranged from 30% to 81% depending only on the harness, with CUGA at 81.1% (73/90) versus Hermes at 46.7% and Deep Agents at 30.0%. CUGA is IBM's Apache 2.0 harness for enterprise API work, adding runtime-enforced policies (intent guards, tool approvals, ToolGuards, output formatting) and deployment inside sovereign or air-gapped environments. The authors note the 90-task sample, non-identical setups, and absence of policy testing as limits, and say CUGA is not the right fit for terminal coding or personal chat assistants.

    Why it matters: Useful for teams choosing an agent harness, since it isolates harness effects with a fixed model and shows where runtime policy enforcement fits enterprise needs.

  1. Latent Space82

    [AINews] Gemini 4 Carbon tops leaderboards, launches for cyber defenders

    Google DeepMind released Gemini 4 Carbon, a frontier model that takes the #1 spot on Artificial Analysis' Intelligence Index v5 with a score of 59, two weeks after Argon. Access is initially limited to trusted cyber defenders through the Fairwind Program, with claimed results of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, a 1M-token context window, and pricing listed at $4/$20 per million tokens with a 50% launch discount. Independent evaluations place it first in Text Arena and Vending-Bench 2 but behind Opus 5.5 on coding-agent and FrontierSWE benchmarks, and its hallucination rate rose from Argon's 4% to 11%.

    Why it matters: Useful for tracking how a new frontier model's leaderboard gains compare with its coding and hallucination trade-offs, and for the debate over benchmark methodology changes.