Analyses & model case studies

01

Three open models below 36 billion parameters

Qwen 3.8 27B, Qwen 3.6 35B-A3B and Gemma 4 31B IT are our editorial sample: two providers, including dense and MoE architectures. This selection is not a leaderboard. The first question is whether comparable measurements exist for the same task.

For each model and benchmark, we use the base result selected by the table rules. Quantized variants, stale or no longer confirmed evidence, conflicts and contamination warnings are excluded. Within each chart, source, test version, tasks, metric, agent, tools and judge must match. Different thinking settings remain visible; this does not establish equal compute budgets.

Scale 0–100%. Only values within the same group share the disclosed setup. We calculate no advantage between groups.

A higher point estimate is not proof of statistical superiority. Disclosed setups may still contain unreported differences.

GPQA is evidence about scientific problem solving. SciCode covers scientific programming; LiveCodeBench covers competitive programming. A lead on GPQA therefore does not establish which model will handle your code better. Start with your task domain and compare values only within the same group.

Our conclusion: relevant shared evidence is more useful for shortlisting than an average across these three tests. Without a shared measurement, a knowledge gap remains. Total parameter count alone establishes neither memory requirements nor the speed of a quantized local installation.

↑ Back to topics

02

The same model, two SWE-bench results

Claude Opus 5 illustrates why model and benchmark names alone are insufficient. We place the independent Vals run alongside the supplementary provider report. Each result remains tied to its own source and evaluation setup.

The Vals record describes mini-swe-agent and Bash. The provider record describes a different or incompletely disclosed setup. Thinking and repetition details differ too. The score gap therefore cannot be attributed to a single cause; it proves neither a model improvement nor an error by either source.

To compare different models, we would start with the uniform Vals group. We use the provider report as additional evidence for the system tested there. Your own deployment still needs evaluation on your repository, with your agent and a fixed time and token budget.

↑ Back to topics