← Subnet 114 · test bench

ablB · grephits

stage-1 pass

compressor variants/ablation/ablate_grephits.py · md5 5635358b62e3 · 27/27 runs graded · tasks: conan-io__conan-20032, django__django-13417, getmoto__moto-10057 · other arms: comments

Score
+0.145
90% CI -0.05 … +0.34
Weighted saving
+9.1%
pooled per run vs baseline
Resolved
6/9
baseline 6/9
Stage gates
S1 pass · S2 fail
tasks 2/3 (need 2)
Cost
$0.24
9 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted154K of 169K↓ 9.1%
Fresh input27K of 29K↓ 4.6%
Cached input940K of 1.04M↓ 9.7%
Output11K of 12K↓ 10.8%
Steps31.9 of 33.7↓ 5.3%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
conan-io__conan-200320/30/3177K193K-0.123none31 / 31482 / 505
django__django-134173/33/387K72K+0.270maintain19 / 21366 / 219
getmoto__moto-100573/33/3242K196K+0.310maintain51 / 43283 / 261

Runs

18 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
conan-io__conan-20032baseliner1failed2831K823K10K143K125s$0.026
conan-io__conan-20032baseliner2failed2933K1.01M21K196K191s$0.041
conan-io__conan-20032baseliner3failed3532K1.20M13K192K192s$0.033
conan-io__conan-20032grephitsr1failed2431K642K9K123K183s$0.024
conan-io__conan-20032grephitsr2failed3739K1.40M19K236K852s$0.043
conan-io__conan-20032grephitsr3failed3335K1.23M20K219K173s$0.042
django__django-13417baseliner1resolved1520K331K6K71K286s$0.015
django__django-13417baseliner2resolved3025K787K10K134K255s$0.024
django__django-13417baseliner3resolved1317K257K5K57K220s$0.012
django__django-13417grephitsr1resolved2018K400K4K71K364s$0.013
django__django-13417grephitsr2resolved1715K312K4K57K224s$0.011
django__django-13417grephitsr3resolved2619K528K6K89K232s$0.016
getmoto__moto-10057baseliner1resolved5232K1.71M14K246K261s$0.037
getmoto__moto-10057baseliner2resolved4635K1.40M13K213K240s$0.034
getmoto__moto-10057baseliner3resolved5535K1.85M16K268K682s$0.041
getmoto__moto-10057grephitsr1resolved3629K1.03M10K161K638s$0.027
getmoto__moto-10057grephitsr2resolved6038K2.08M15K292K321s$0.042
getmoto__moto-10057grephitsr3resolved3424K837K9K134K172s$0.023