← Subnet 114 · test bench

long01 · g_reasoning

stage-1 fail

compressor variants/base_miner_pre_gitshow_fix.py · md5 c35648cae1c2 · 60/60 runs graded · tasks: django__django-13417, django__django-15930, sympy__sympy-15875

Score
-0.119
90% CI -0.29 … -0.03
Weighted saving
-23.9%
pooled per run vs baseline
Resolved
26/30
baseline 28/30
Stage gates
S1 fail · S2 fail
tasks 3/3 (need 2)
Cost
$1.95
30 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted364K of 294K↑ 23.9%
Fresh input29K of 28K↑ 6.6%
Cached input2.29M of 1.78M↑ 28.6%
Output35K of 29K↑ 19.7%
Steps50.0 of 42.4↑ 17.8%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-1341710/1010/10110K103K+0.097maintain25 / 25344 / 333
django__django-1593010/1010/10398K523K-0.394maintain58 / 70588 / 647
sympy__sympy-158758/106/10373K466K-0.320penalty44 / 55979 / 943

Runs

60 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-13417baseliner1resolved1719K372K6K73K390s$0.015
django__django-13417baseliner10resolved2319K501K7K89K447s$0.017
django__django-13417baseliner2resolved2220K489K8K93K274s$0.019
django__django-13417baseliner3resolved3022K700K8K117K259s$0.021
django__django-13417baseliner4resolved2421K581K8K104K301s$0.020
django__django-13417baseliner5resolved3926K1.40M30K257K372s$0.052
django__django-13417baseliner6resolved1918K382K5K72K307s$0.014
django__django-13417baseliner7resolved1615K285K4K54K530s$0.011
django__django-13417baseliner8resolved1619K334K4K66K319s$0.013
django__django-13417baseliner9resolved4326K1.18M11K178K353s$0.029
django__django-13417g_reasoningr1resolved2217K471K9K91K364s$0.019
django__django-13417g_reasoningr10resolved2118K448K7K85K447s$0.017
django__django-13417g_reasoningr2resolved2419K566K9K103K328s$0.020
django__django-13417g_reasoningr3resolved3121K739K9K121K295s$0.021
django__django-13417g_reasoningr4resolved1919K424K8K85K286s$0.018
django__django-13417g_reasoningr5resolved3727K1.09M14K177K406s$0.031
django__django-13417g_reasoningr6resolved2718K570K7K96K365s$0.017
django__django-13417g_reasoningr7resolved2821K654K8K109K519s$0.019
django__django-13417g_reasoningr8resolved2320K517K7K94K328s$0.018
django__django-13417g_reasoningr9resolved1819K370K5K70K283s$0.014
django__django-15930baseliner1resolved4627K1.63M26K269K1156s$0.049
django__django-15930baseliner10resolved6337K2.91M42K455K538s$0.079
django__django-15930baseliner2resolved6339K3.13M42K478K487s$0.081
django__django-15930baseliner3resolved5334K2.16M31K341K480s$0.060
django__django-15930baseliner4resolved4231K1.58M26K265K830s$0.050
django__django-15930baseliner5resolved5640K2.27M25K341K1008s$0.055
django__django-15930baseliner6resolved6336K3.04M41K464K587s$0.079
django__django-15930baseliner7resolved9952K5.86M55K802K1433s$0.116
django__django-15930baseliner8resolved5030K1.98M30K319K561s$0.057
django__django-15930baseliner9resolved4429K1.48M23K247K1057s$0.046
django__django-15930g_reasoningr1resolved6534K3.20M43K484K688s$0.082
django__django-15930g_reasoningr10resolved6833K3.02M38K449K533s$0.074
django__django-15930g_reasoningr2resolved7541K3.73M41K536K500s$0.084
django__django-15930g_reasoningr3resolved9748K5.55M60K784K688s$0.120
django__django-15930g_reasoningr4resolved7642K4.36M70K688K1334s$0.123
django__django-15930g_reasoningr5resolved7942K4.69M64K703K1277s$0.118
django__django-15930g_reasoningr6resolved7236K3.37M39K488K1029s$0.077
django__django-15930g_reasoningr7resolved5229K2.16M39K360K1253s$0.068
django__django-15930g_reasoningr8resolved5236K2.23M29K346K482s$0.059
django__django-15930g_reasoningr9resolved6235K2.61M31K388K455s$0.063
sympy__sympy-15875baseliner1resolved5834K2.96M51K484K1035s$0.089
sympy__sympy-15875baseliner10resolved3928K1.86M39K333K596s$0.067
sympy__sympy-15875baseliner2resolved5528K3.20M67K548K1678s$0.108
sympy__sympy-15875baseliner3resolved4627K2.19M48K391K793s$0.079
sympy__sympy-15875baseliner4resolved4428K2.16M56K412K1080s$0.088
sympy__sympy-15875baseliner5failed5730K2.92M64K513K812s$0.103
sympy__sympy-15875baseliner6resolved3421K1.12M23K201K721s$0.040
sympy__sympy-15875baseliner7resolved4224K1.97M49K368K881s$0.078
sympy__sympy-15875baseliner8resolved1915K409K9K84K524s$0.018
sympy__sympy-15875baseliner9failed5033K2.30M45K398K2068s$0.078
sympy__sympy-15875g_reasoningr1resolved4425K1.75M40K320K714s$0.066
sympy__sympy-15875g_reasoningr10failed3722K1.18M24K212K588s$0.043
sympy__sympy-15875g_reasoningr2resolved4629K2.12M43K371K1606s$0.073
sympy__sympy-15875g_reasoningr3resolved8338K4.34M61K654K1301s$0.110
sympy__sympy-15875g_reasoningr4resolved5735K3.68M81K647K1027s$0.130
sympy__sympy-15875g_reasoningr5failed6933K3.64M54K560K1221s$0.097
sympy__sympy-15875g_reasoningr6resolved5629K2.97M54K487K650s$0.091
sympy__sympy-15875g_reasoningr7failed5437K2.75M59K490K1040s$0.099
sympy__sympy-15875g_reasoningr8failed3922K1.37M31K251K1146s$0.052
sympy__sympy-15875g_reasoningr9resolved6635K4.03M76K666K1064s$0.126