← Subnet 114 · test bench

short01 · g_reasoning

stage-1 pass

compressor variants/base_miner_pre_gitshow_fix.py · md5 c35648cae1c2 · 60/60 runs graded · tasks: django__django-14089, django__django-14373, django__django-15851, sympy__sympy-16886, sympy__sympy-19637 · other arms: g_toolonly

Score
+0.059
90% CI -0.05 … +0.16
Weighted saving
+8.8%
pooled per run vs baseline
Resolved
20/20
baseline 20/20
Stage gates
S1 pass · S2 fail
tasks 5/5 (need 3)
Cost
$0.17
20 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted41K of 45K↓ 8.8%
Fresh input13K of 14K↓ 7.0%
Cached input201K of 220K↓ 8.5%
Output3K of 3K↓ 12.2%
Steps12.4 of 12.9↓ 3.9%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-140894/44/431K34K-0.135maintain9 / 12159 / 136
django__django-143734/44/448K41K+0.238maintain14 / 13262 / 209
django__django-158514/44/464K55K+0.219maintain16 / 14329 / 325
sympy__sympy-168864/44/434K29K+0.212maintain11 / 10130 / 131
sympy__sympy-196374/44/448K46K+0.053maintain15 / 14203 / 206

Runs

40 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-14089baseliner1resolved618K110K1K32K397s$0.007
django__django-14089baseliner2resolved1112K164K2K34K310s$0.007
django__django-14089baseliner3resolved1312K192K2K36K288s$0.007
django__django-14089baseliner4resolved610K80K91121K234s$0.005
django__django-14089g_reasoningr1resolved812K117K1K27K391s$0.006
django__django-14089g_reasoningr2resolved911K126K94826K299s$0.005
django__django-14089g_reasoningr3resolved1913K295K2K50K291s$0.009
django__django-14089g_reasoningr4resolved1012K150K2K32K236s$0.006
django__django-14373baseliner1resolved1414K249K5K53K427s$0.011
django__django-14373baseliner2resolved1012K158K3K36K284s$0.008
django__django-14373baseliner3resolved1914K325K4K58K291s$0.011
django__django-14373baseliner4resolved1413K234K3K46K340s$0.009
django__django-14373g_reasoningr1resolved913K141K2K33K394s$0.007
django__django-14373g_reasoningr2resolved1512K235K2K42K240s$0.008
django__django-14373g_reasoningr3resolved1113K180K3K42K277s$0.009
django__django-14373g_reasoningr4resolved1613K262K3K48K309s$0.009
django__django-15851baseliner1resolved814K122K2K31K273s$0.007
django__django-15851baseliner2resolved1115K180K2K39K374s$0.008
django__django-15851baseliner3resolved2218K445K7K84K314s$0.017
django__django-15851baseliner4resolved2120K463K12K102K320s$0.023
django__django-15851g_reasoningr1resolved1918K452K14K104K358s$0.024
django__django-15851g_reasoningr2resolved1315K210K2K42K317s$0.008
django__django-15851g_reasoningr3resolved1414K225K2K44K343s$0.009
django__django-15851g_reasoningr4resolved813K115K2K30K222s$0.007
sympy__sympy-16886baseliner1resolved812K116K1K26K334s$0.005
sympy__sympy-16886baseliner2resolved1217K225K2K44K244s$0.008
sympy__sympy-16886baseliner3resolved1011K140K1K28K246s$0.005
sympy__sympy-16886baseliner4resolved1312K199K2K38K232s$0.007
sympy__sympy-16886g_reasoningr1resolved1111K157K1K31K388s$0.006
sympy__sympy-16886g_reasoningr2resolved1312K192K2K37K219s$0.007
sympy__sympy-16886g_reasoningr3resolved911K129K1K28K235s$0.006
sympy__sympy-16886g_reasoningr4resolved711K94K81422K223s$0.005
sympy__sympy-19637baseliner1resolved1414K227K3K45K394s$0.009
sympy__sympy-19637baseliner2resolved1213K196K3K41K242s$0.008
sympy__sympy-19637baseliner3resolved1514K256K3K49K256s$0.009
sympy__sympy-19637baseliner4resolved1914K322K4K58K244s$0.011
sympy__sympy-19637g_reasoningr1resolved1614K272K3K52K383s$0.010
sympy__sympy-19637g_reasoningr2resolved1413K222K2K42K253s$0.008
sympy__sympy-19637g_reasoningr3resolved1314K220K3K46K244s$0.010
sympy__sympy-19637g_reasoningr4resolved1414K236K3K46K234s$0.009