AWS Machine Learning Blog· Khaled Thabit·· 19 days agoAI score40
Fix evaluation bias to unlock better models on Amazon Bedrock
Fix evaluation bias to unlock better models on Amazon Bedrock
AI summary
Property Finder found that 96.8% of its evaluation labels came directly from its incumbent vision model's predictions rather than human verification, biasing comparisons against alternatives. After building expert-annotated ground truth (356 UAE images) and running evolutionary prompt optimization from v0 to v27, it selected Llama 4 Maverick 17B Instruct on Amazon Bedrock, reaching a Micro F1 of 0.80 versus 0.56 for its previous model on the evaluation set, at an estimated 86% lower cost per 1,000 images.
Source: AWS Machine Learning Blog · aws.amazon.com