← Subnet 114 · test bench

s1_edge10 · edge10_footer

stage-1 fail

compressor variants/exp_edge10_footer.py · md5 4438bcc3df54 · 23/150 runs graded · tasks: django__django-12419, django__django-13821, django__django-14373, sympy__sympy-15875, sympy__sympy-19637 · other arms: edge10

Score
-0.033
90% CI -0.21 … +0.16
Weighted saving
-19.3%
pooled per run vs baseline
Resolved
7/7
baseline 8/8
Stage gates
S1 fail · S2 fail
tasks 4/4 (need 2)
Cost
$0.18
7 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted137K of 176K↓ 22.0%
Fresh input23K of 24K↓ 6.8%
Cached input778K of 987K↓ 21.1%
Output12K of 18K↓ 30.6%
Steps24.9 of 27.0↓ 7.9%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-124192/22/2123K232K-0.915maintain24 / 36480 / 677
django__django-138212/22/2206K191K+0.105maintain28 / 32668 / 508
django__django-143732/22/247K34K+0.436maintain15 / 10198 / 222
sympy__sympy-196371/11/156K48K+0.238maintain18 / 15204 / 183

Runs

19 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-12419baseliner1resolved3327K932K13K160K390s$0.030
django__django-12419baseliner2resolved1619K396K9K86K431s$0.019
django__django-12419edge10_footerr1resolved2427K769K20K163K449s$0.036
django__django-12419edge10_footerr2resolved4931K1.91M26K300K449s$0.052
django__django-13821baseliner1resolved2843K1.05M14K189K448s$0.036
django__django-13821baseliner2resolved2933K1.16M24K222K505s$0.046
django__django-13821edge10_footerr1resolved2230K601K11K124K749s$0.026
django__django-13821edge10_footerr2resolved4332K1.60M22K259K541s$0.046
django__django-14373baseliner1resolved1212K174K2K34K391s$0.006
django__django-14373baseliner2resolved1814K314K5K59K414s$0.012
django__django-14373edge10_footerr1resolved1112K169K3K38K336s$0.008
django__django-14373edge10_footerr2resolved1011K146K2K31K429s$0.006
sympy__sympy-15875baseliner1resolved6232K3.56M72K604K1352s$0.117
sympy__sympy-15875baseliner2no result–––––––
sympy__sympy-15875edge10_footerr1no result–––––––
sympy__sympy-15875edge10_footerr2no result–––––––
sympy__sympy-19637baseliner1resolved1814K306K4K56K346s$0.011
sympy__sympy-19637baseliner2no result–––––––
sympy__sympy-19637edge10_footerr1resolved1514K253K3K48K339s$0.009