LLM Benchmarks

Source-backed measurements

Page updated
EN
♡ Support

Local GPU Benchmarks: Open-Weight Language Models on Consumer Hardware

Self-measured full-dataset evaluations of local open-weight LLMs on dedicated GPU hardware (2× RTX 5060 Ti, 32 GB VRAM): Core Master, Total Master, wall-clock duration, and per-benchmark time efficiency (0–100).

While the main table compares official vendor claims and frontier cloud models, this page shows how compact open-weight models (in 4-bit and 8-bit quantization) perform on real local hardware—tested both with and without thinking. Every cell displays the score alongside actual runtime and time efficiency, making it easy to spot models that deliver strong results in 30 minutes versus those that spend hours reasoning in circles.

Independent full-dataset runs on local hardware: Every benchmark here runs across the complete dataset without subsampling on our local GPU server. Continuous 24/7 operation, power, GPUs, and external judge APIs unfortunately cost more and take longer than expected, which is why updates appear somewhat irregularly.

More about benchmark effort & future hardware plans

Full datasets instead of small samples: Running all 12,032 MMLU-Pro items, SciCode, or SWE-bench per model and thinking track takes hours to days under full GPU load.

Ongoing operating costs: Continuous 24/7 power draw, GPU hardware, and external judge API costs (such as Gemini for open-ended and agent evaluations) add up over the year.

Evolving hardware profiles: We currently measure on 2× NVIDIA RTX 5060 Ti (32 GB VRAM). When stronger GPUs become available, models will be re-tested and selectable via the hardware filter above.

Thinking track
Master columns
Sort cells by
Hardware
Cell color
Top 10%Top 25%Mid 50%Lower quarterBottom 10%
≥ 75 Highly efficient (strong score in short time)40–74 Moderate< 40 Inefficient (e.g. 3h reasoning loops for 22%)

Local evaluation round in progress – first results coming soon

Full-dataset local evaluations on our test rig (2× RTX 5060 Ti, 32 GB VRAM) are currently running. Once published, scores, wall-clock durations, and time efficiency ratings will appear here.

2× RTX 5060 Ti (32 GB VRAM) · 2× NVIDIA GeForce RTX 5060 Ti (16 GB GDDR7 × 2, Tensor Parallelism 2) · vLLM 0.29.0