iCreatAI News
Back to main site
Skip to content
Only First party#Benchmarks
  1. OpenRouter Announcements60

    Reranking in RAG: How to Fix Bad Retrieval with a Reranker

    This guide explains that vector search often ranks near-miss chunks first because embeddings compress whole chunks into single vectors, and shows how adding a cross-encoder reranker over the top 30 candidates raised precision@1 from 81% to 97% on a 274-question synthetic benchmark at roughly $0.10 per 1,000 queries. It compares 11 rerankers (voyageai/rerank-3-lite and rerank-3 were the most accurate dedicated models and cheapest, while LLM rerankers like openai/gpt-6-luna hit 99.6% but took about 4 seconds per call), covers candidate-count tuning, input_type settings, and silent failures such as cohere/rerank-v3.5's 4,096-token limit. It includes a Python example using the embeddings and rerank APIs and notes that real-world gains depend on your own documents and evaluation set.

    Why it matters: Offers a measured, benchmark-backed walkthrough of adding reranking to a RAG pipeline, including cost, latency and failure-mode comparisons useful for teams tuning retrieval.

  1. Artificial Analysis Articles60

    Multimodal Knowledge Agent (MKA) benchmark: testing knowledge work capabilities with complex visual files

    Artificial Analysis partnered with Google DeepMind to launch the Multimodal Knowledge Agent (MKA) benchmark, comprising 200 private real-world visual tasks across 10 industries in which agents work from dense visual files like architectural drawings, engineering schematics, and clinical scans to produce deliverables such as slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Tasks are graded against expert rubrics averaging ~70 weighted criteria split into Regression and Climb checks, with the headline Climb WSR score, run in the open-source Stirrup agent harness with up to 250 turns and graded by GPT-6 Luna (medium) as an agentic judge. Claude Fable 5.2 (max) leads with 57% Climb WSR, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%, while Claude Fable 5.2 averages $31.97 per task versus $6.29 for Gemini 4 Carbon and $1.11 for GPT-6.1 Sol.

    Why it matters: The results offer a useful cost-performance and per-vertical breakdown for anyone choosing models for agentic knowledge work over complex visual files.

  2. OpenRouter Announcements64

    Best OCR Model for Scanned Documents, Measured

    The author tested 15 vision models plus Mistral OCR and Tesseract on eight synthetic scanned pages scored by character error rate and 264 key-value checks, finding Claude Opus 5.5 most accurate (262 of 264 at $16.42 per 1,000 pages) and Mistral OCR the best value (258 of 264 at $2.02). The post documents failure modes such as fabricated card digits, wifi passwords and server names on a faded receipt, and provides code for transcription, disabling reasoning, cross-checking two models from different labs, and using the mistral-ocr engine.

    Why it matters: It offers a reproducible benchmark with per-model costs and practical code for catching hallucinated values in OCR pipelines.

  1. Artificial Analysis Articles72

    Meta's Muse Ember is the cheapest model scoring 36 on the Intelligence Index

    Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41, making it the cheapest model scoring 36 or higher. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, but its AA-Omniscience accuracy of 21.1% is the lowest among similarly intelligent models, largely because it declines to answer 64% of questions, keeping its hallucination rate at 17.4%. Pricing is $0.10 input / $0.004 cached input / $0.50 output per 1M tokens, available via Meta's first-party API and Muse Code.

    Why it matters: The piece offers a detailed cost-versus-capability breakdown that helps readers judge whether a cheap efficient-tier model fits agentic and coding workloads.

  1. Artificial Analysis Articles73

    Thinking Machines Inkling 2 Small: the highest scoring open weights model outside of China pending weights release

    Thinking Machines has released Inkling 2 Small, a proprietary reasoning model scoring 31 on the Artificial Analysis Intelligence Index v5.0.1, with Apache 2.0 weights planned for a later, unconfirmed date. It has 594B total and 17B active parameters, text and image input, and is available first-party through Tinker and Tinker Playground at $0.30/$1.00 per 1M input/output tokens. The model gains 18 points over Inkling Small, matches Gemini 3.8 Flash (high), and costs $1.18 per Intelligence Index task.

    Why it matters: The item gives detailed benchmark comparisons and pricing for a newly released model, useful for readers tracking open-weights and reasoning model progress.

  2. Artificial Analysis Articles62

    Nanbeige has released Nanbeige4.3-33B scoring 20 on the Intelligence Index

    Nanbeige, the foundation model research team at BOSS Zhipin (operated by Kanzhun Limited), has released Nanbeige4.3-33B, an Apache 2.0 open-weights dense reasoning model with text-only input and output and a 524K-token context window. It scores 20 on the Artificial Analysis Intelligence Index v5, matching Qwen3.8 27B (xhigh) at 20, with relative strengths in terminal use (30% on Terminal-Bench 4.0 vs. 6%) and physics reasoning (16% on CritPt v2's Hard Subset vs. 4%), but trails on AA-Briefcase v1.1 (1162 vs. 1391 Elo), FrontierCode 2.0 (11% vs. 15%) and HLE-Diamond (7% vs. 15%). It shows high abstention and low hallucination on Omniscience (18% vs. 29% hallucination rate, declining 71% of questions) and used 158K output tokens per Intelligence Index task vs. 110K for Qwen3.8 27B.

    Why it matters: Useful for readers tracking small open-weights models, since it sets a new Pareto point for Intelligence Index versus total parameters and details trade-offs against Qwen3.8 27B.

  1. Fireworks AI Blog72

    Kimi K3.1 Pro is live on Fireworks

    Moonshot AI's Kimi K3.1 Pro is available on Fireworks at launch, either serverless with pay-per-token pricing or on a dedicated deployment for reserved capacity. Moonshot post-trained K3.1 Pro from K3 and replaced K3's full-attention layers with sparse attention to make long contexts more efficient to serve. Fireworks reports it sits on the quality-for-cost frontier on five Specialized Intelligence Index benchmarks, ranking first of 19 on Sierra's τ³-Banking at 64.7%, and lists serverless pricing at $2.00 per million uncached input tokens, $0.20 cached input and $10.00 output.

    Why it matters: Useful for readers weighing a newly launched open model against named competitors on cost and benchmark placement.

  2. Hugging Face Blog60

    Dharma Field Extractor: Any Document In, Structured Fields in JSON Out

    Dharma-AI introduces the Dharma Field Extractor, a 2-billion-parameter model built on DharmaOCR-LITE v1.1 with a LoRA adapter trained on 43,384 examples, which takes a document image plus either an open request or an exact schema and returns typed JSON, including null for fields not present. On a 546-document, 1,638-case benchmark it scored 0.834 field-level F1, above every frontier configuration tested (Claude Opus 5 at 0.816 to GPT-5.6 Luna at 0.741), while running self-hosted on a single L40S at a cost the company reports as 6 to 147 times lower than the APIs compared. The company frames the result as evidence that a small model trained close to a bounded task can beat a general frontier model on that task.

    Why it matters: Useful for teams weighing self-hosted specialized extraction against frontier APIs, though the benchmark and cost figures come from the vendor itself.

  1. Together AI Blog71

    Together Compiler: Continuous Compilation for AI inference

    Together AI introduces Together Compiler (TKCC), a continual agent-driven system that recompiles inference engines as traffic drifts, using persistent engines, a two-tier fast/slow engine setup for throughput-fidelity balance, and a custom cost model for parallelism selection. TKCC reports 1.3-3x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results of 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX benchmark, and 1.7x over open-source megakernels on Llama 1B batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. It also generalizes to weather modeling, PDE, time-series, and biology workloads.

    Why it matters: Documents a concrete agentic-compiler architecture and benchmark results for adapting inference to shifting traffic, models, and hardware.

  2. Hugging Face Blog60

    Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?

    IBM ran CUGA, Hermes, and Deep Agents on the same open-weight model (GLM-5.3-flash) across 90 AppWorld tasks, and task completion ranged from 30% to 81% depending only on the harness, with CUGA at 81.1% (73/90) versus Hermes at 46.7% and Deep Agents at 30.0%. CUGA is IBM's Apache 2.0 harness for enterprise API work, adding runtime-enforced policies (intent guards, tool approvals, ToolGuards, output formatting) and deployment inside sovereign or air-gapped environments. The authors note the 90-task sample, non-identical setups, and absence of policy testing as limits, and say CUGA is not the right fit for terminal coding or personal chat assistants.

    Why it matters: Useful for teams choosing an agent harness, since it isolates harness effects with a fixed model and shows where runtime policy enforcement fits enterprise needs.

  1. Replit Blog60

    ViBench 2.0: A Benchmark for Long-Horizon Software Building

    Replit releases ViBench 2.0, a long-horizon software development benchmark in which an agent builds a product from an empty project and then adds up to eighteen features on top of its own code, spanning seventeen products, 241 feature requests and 544 test plans. In Fireworks' official runs, GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%, and no open-weight model passes a full build. Eight of the products are public on GitHub in the Harbor format, and results can be rerun from release v2.0.6 on the OpenHands-based harness.

    Why it matters: It offers concrete pass rates, costs and failure modes for frontier models on multi-feature product building, with a public, rerunnable harness.

  2. Pydantic AI Articles74

    Code mode: make your agents 2x faster and 2x cheaper on long tasks

    A benchmark of 5,276 graded runs across eight models on five small Pydantic AI apps found that code mode, where the model writes a program that calls tools inside a sandbox, cut cost by 58% and wall time by 51% on long tasks while raising success by 16 points. Gains grew with task length: at 200 items code mode was 47% to 75% cheaper and 42% to 67% faster, whereas at 5 items it cost 21% to 38% more on three of five tasks. The author attributes the win not to fewer round trips, since models already batch calls, but to less content per request, and runs the model's code in Monty, a Rust Python interpreter with no host access by default and 30-second, 256 MiB limits.

    Why it matters: Offers a detailed, methodology-heavy benchmark with per-model cost, latency and correctness numbers, plus an honest list of where code mode costs more.

  3. Artificial Analysis Articles82

    Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

    Google's Gemini 4 Carbon tops the Artificial Analysis Intelligence Index v5 with a score of 59, leading 4 of 9 evaluations, and ranks second to Claude Opus 5.5 on the Coding Agent Index with 59 points. It scores 1609 Elo on AA-Briefcase (up 126 over Gemini 4 Argon) and 54 on AA-Omniscience v1.1, at $7.76 per Intelligence Index task on standard $4/$20 pricing. The model is being rolled out to selected users and is not yet publicly available, with a 50% launch discount and no confirmed end date.

    Why it matters: The piece gives a detailed benchmark breakdown of an unreleased Google model, useful for tracking how Gemini's coding and agentic performance compares with Claude Opus 5.5 and GPT-6 Astra.

  1. Artificial Analysis Articles60

    Announcing Artificial Analysis Intelligence Index v5

    Artificial Analysis released Intelligence Index v5, its biggest upgrade since v4, adding FrontierCode 2.0 and Terminal-Bench-Science and upgrading HLE-Diamond, AA-Omniscience v1.1 and CritPt v2, with 60% of the Index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55), while the open weights gap widens from 3 points to 12. It also launched the Progress Index v1, which tracks model capability over time on one continuous scale.

    Why it matters: It details a major benchmark revision and new leaderboard results, useful for readers tracking how frontier and open-weight models are measured.

  1. Google Research65

    Putting Earth AI to work: Geospatial agents for our planet's biggest challenges

    Google shares a technical report on its experimental Geospatial Reasoning agent, which connects Gemini to Earth AI foundation models and planet-scale data, and reports evaluations across four benchmarks, including two newly released ones (GeoKTools and GeoExpertQueries). The post describes early partner testing with organizations including WHO AFRO, People in Need, GiveDirectly, the Milken Institute and WPP, and announces expanded early access via Ask Google Earth and an experimental agent-to-agent API.

    Why it matters: It offers a concrete look at how a planning-based geospatial agent is evaluated and applied to real public health, disaster and infrastructure workflows.

  1. GitHub Blog · AI & ML61

    ReviewBench: An open benchmark for AI code review

    GitHub introduces ReviewBench, an open benchmark for AI code review built from analysis of 103.9 million GitHub pull requests, containing 219 public pull requests across 19 languages with a multi-source golden set labeled for severity and category. It reports grounded and augmented precision, recall, and F1 metrics, uses Claude Sonnet 5 as the LLM grader under a published rubric, and had its ground-truth findings independently re-labeled by senior engineers with 96.6% agreement. GitHub says ReviewBench's offline predictions for a Copilot code review multi-model ensemble matched a later A/B test, which showed an 8.0% rise in addressed rate, 13.6% in recall, 61% in comment volume, and an 8.0% drop in cost per review.

    Why it matters: It offers a concrete, reproducible methodology and public dataset for comparing AI code reviewers, plus evidence on how offline scores tracked a production experiment.

You have reached the end