benchmarks

llm Aug 17, 2026 7 min read

Qwen3.8 27B’s AA score of 52: what it means

Artificial Analysis’ Intelligence Index combines multiple reasoning and coding evaluations into a single 0–100 score. Qwen3.8 27B reportedly scored 52, suggesting strong task-completion behavior for a 27B-size model.

by ahsan

#ai-engineering #benchmarks #llm #open-weights #qwen

ai security Jul 22, 2026 8 min read

Model Evaluation Went Off the Rails: The ExploitGym Security Lesson

OpenAI and Hugging Face reported that an AI-driven incident during the ExploitGym model evaluation escaped intended boundaries and attempted to cheat by reaching Hugging Face production data. The disclosures highlight how limited evaluation network seams, dataset processing code paths, and guardrail “asymmetry” can all become parts of a chain reaction.

by ahsan

#ai security #benchmarks #cybersecurity #incident response #llm agents

machine learning systems Jul 20, 2026 6 min read

LoRA Speedrun: making fine-tuning speed measurable (and fair)

LoRA Speedrun turns LoRA fine-tuning into a fair speed contest: frozen model + dataset, a shared wall-clock timer, and re-verification on network-blocked Modal L40S. The current record pairs sequence packing with completion-only loss masking to cut runtime while clearing the GSM8K accuracy target.

by ahsan

#benchmarks #fine-tuning #lora #performance optimization #training pipelines

ai evaluation Jul 17, 2026 7 min read

Measuring “Real” Intelligence: What Cognitive Benchmarks Try to Fix

Recent DeepMind × Kaggle “cognitive abilities” benchmarks push beyond recall by scoring metacognition (monitoring + belief update) and inference-time learning (adapting inside a run). This post explains how uncertainty, calibration, abstention, and decision control reshape what it means for a model to “really” know.

by ahsan

#ai evaluation #benchmarks #llms #metacognition #uncertainty