← Subnet 114 · test bench

ablA · listing

stage-1 fail

compressor variants/ablation/ablate_listing.py · md5 afdcc12fddfb · 18/18 runs graded · tasks: dask__dask-12430, vega__altair-4019 · other arms: bashother

Score
+0.000
90% CI +0.00 … +0.00
Weighted saving
-27.8%
pooled per run vs baseline
Resolved
0/6
baseline 0/6
Stage gates
S1 fail · S2 fail
tasks 0/2 (need 1)
Cost
$0.47
6 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted419K of 328K↑ 27.8%
Fresh input39K of 30K↑ 26.6%
Cached input2.54M of 1.95M↑ 30.3%
Output42K of 34K↑ 23.5%
Steps66.7 of 54.2↑ 23.1%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
dask__dask-124300/30/3324K410K-0.341none56 / 63638 / 679
vega__altair-40190/30/3332K428K-0.367none53 / 70614 / 603

Runs

12 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
dask__dask-12430baseliner1failed6332K2.32M42K388K1355s$0.073
dask__dask-12430baseliner2failed3822K1.01M27K203K354s$0.045
dask__dask-12430baseliner3failed6632K2.39M36K381K1065s$0.068
dask__dask-12430listingr1failed8539K4.15M58K626K1248s$0.106
dask__dask-12430listingr2failed4726K1.46M33K271K346s$0.056
dask__dask-12430listingr3failed5731K1.91M37K333K367s$0.065
vega__altair-4019baseliner1failed5237K1.84M36K327K1153s$0.065
vega__altair-4019baseliner2failed2821K634K14K127K233s$0.027
vega__altair-4019baseliner3failed7839K3.49M51K541K1323s$0.094
vega__altair-4019listingr1failed5349K1.61M36K318K1010s$0.067
vega__altair-4019listingr2failed8242K3.57M52K555K541s$0.096
vega__altair-4019listingr3failed7645K2.52M38K410K1430s$0.074