← Subnet 114 · test bench

short01 · g_toolonly

stage-1 pass

compressor variants/toolonly.py · md5 49ec21a384f9 · 60/60 runs graded · tasks: django__django-14089, django__django-14373, django__django-15851, sympy__sympy-16886, sympy__sympy-19637 · other arms: g_reasoning

Score
+0.057
90% CI -0.04 … +0.15
Weighted saving
+5.0%
pooled per run vs baseline
Resolved
20/20
baseline 20/20
Stage gates
S1 pass · S2 fail
tasks 5/5 (need 3)
Cost
$0.17
20 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted43K of 45K↓ 5.0%
Fresh input13K of 14K↓ 4.4%
Cached input210K of 220K↓ 4.4%
Output3K of 3K↓ 7.2%
Steps12.6 of 12.9↓ 2.7%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-140894/44/431K26K+0.266maintain9 / 8159 / 144
django__django-143734/44/448K59K-0.280maintain14 / 18262 / 249
django__django-158514/44/464K60K+0.093maintain16 / 16329 / 267
sympy__sympy-168864/44/434K28K+0.280maintain11 / 9130 / 130
sympy__sympy-196374/44/448K42K+0.206maintain15 / 12203 / 231

Runs

40 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-14089baseliner1resolved618K110K1K32K397s$0.007
django__django-14089baseliner2resolved1112K164K2K34K310s$0.007
django__django-14089baseliner3resolved1312K192K2K36K288s$0.007
django__django-14089baseliner4resolved610K80K91121K234s$0.005
django__django-14089g_toolonlyr1resolved911K126K1K27K374s$0.005
django__django-14089g_toolonlyr2resolved711K98K1K24K273s$0.005
django__django-14089g_toolonlyr3resolved1012K143K1K29K287s$0.006
django__django-14089g_toolonlyr4resolved611K81K95822K233s$0.005
django__django-14373baseliner1resolved1414K249K5K53K427s$0.011
django__django-14373baseliner2resolved1012K158K3K36K284s$0.008
django__django-14373baseliner3resolved1914K325K4K58K291s$0.011
django__django-14373baseliner4resolved1413K234K3K46K340s$0.009
django__django-14373g_toolonlyr1resolved1815K320K5K61K402s$0.012
django__django-14373g_toolonlyr2resolved1212K188K3K39K262s$0.008
django__django-14373g_toolonlyr3resolved1915K345K6K66K295s$0.013
django__django-14373g_toolonlyr4resolved2215K394K5K69K286s$0.013
django__django-15851baseliner1resolved814K122K2K31K273s$0.007
django__django-15851baseliner2resolved1115K180K2K39K374s$0.008
django__django-15851baseliner3resolved2218K445K7K84K314s$0.017
django__django-15851baseliner4resolved2120K463K12K102K320s$0.023
django__django-15851g_toolonlyr1resolved2621K579K10K109K1024s$0.022
django__django-15851g_toolonlyr2resolved1113K163K2K35K280s$0.007
django__django-15851g_toolonlyr3resolved1415K234K3K47K343s$0.009
django__django-15851g_toolonlyr4resolved1217K223K4K50K219s$0.011
sympy__sympy-16886baseliner1resolved812K116K1K26K334s$0.005
sympy__sympy-16886baseliner2resolved1217K225K2K44K244s$0.008
sympy__sympy-16886baseliner3resolved1011K140K1K28K246s$0.005
sympy__sympy-16886baseliner4resolved1312K199K2K38K232s$0.007
sympy__sympy-16886g_toolonlyr1resolved811K109K94224K374s$0.005
sympy__sympy-16886g_toolonlyr2resolved1111K157K1K31K265s$0.006
sympy__sympy-16886g_toolonlyr3resolved912K134K1K30K225s$0.006
sympy__sympy-16886g_toolonlyr4resolved911K127K1K28K220s$0.006
sympy__sympy-19637baseliner1resolved1414K227K3K45K394s$0.009
sympy__sympy-19637baseliner2resolved1213K196K3K41K242s$0.008
sympy__sympy-19637baseliner3resolved1514K256K3K49K256s$0.009
sympy__sympy-19637baseliner4resolved1914K322K4K58K244s$0.011
sympy__sympy-19637g_toolonlyr1resolved1314K221K3K47K422s$0.010
sympy__sympy-19637g_toolonlyr2resolved1213K198K3K42K255s$0.009
sympy__sympy-19637g_toolonlyr3resolved1214K195K2K40K252s$0.008
sympy__sympy-19637g_toolonlyr4resolved1113K173K2K38K215s$0.008