← Subnet 114 · test bench

s1_tooluse · project

stage-1 pass

compressor variants/base_miner_tooluse_0926.py · md5 513928d92e24 · 17/120 runs graded · tasks: django__django-12419, django__django-13821, django__django-14373, sympy__sympy-15875, sympy__sympy-19637

Score
-0.032
90% CI -0.12 … +0.09
Weighted saving
+10.4%
pooled per run vs baseline
Resolved
8/8
baseline 9/9
Stage gates
S1 pass · S2 pass
tasks 5/5 (need 3)
Cost
$0.25
8 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted151K of 169K↓ 10.4%
Fresh input22K of 22K↑ 1.2%
Cached input809K of 939K↓ 13.8%
Output16K of 18K↓ 9.1%
Steps24.8 of 26.7↓ 7.2%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-124192/22/2107K261K-1.281maintain20 / 36542 / 715
django__django-138212/22/2267K208K+0.358maintain42 / 30627 / 808
django__django-143732/22/254K50K+0.094maintain17 / 16192 / 194
sympy__sympy-158751/11/1568K145K+1.968maintain56 / 261252 / 672
sympy__sympy-196372/21/148K27K+0.849penalty13 / 8305 / 171

Runs

22 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-12419baseliner1resolved2223K574K10K111K565s$0.023
django__django-12419baseliner2resolved1822K482K11K103K373s$0.023
django__django-12419baseliner3no result–––––––
django__django-12419projectr1resolved4235K1.99M38K348K795s$0.068
django__django-12419projectr2resolved3028K989K16K174K380s$0.033
django__django-12419projectr3no result–––––––
django__django-13821baseliner1resolved4132K1.66M33K296K739s$0.059
django__django-13821baseliner2resolved4331K1.48M20K237K412s$0.042
django__django-13821projectr1resolved3026K1.04M27K212K1303s$0.047
django__django-13821projectr2resolved3130K1.09M22K205K400s$0.042
django__django-14373baseliner1resolved1916K342K3K60K509s$0.011
django__django-14373baseliner2resolved1513K246K3K47K312s$0.009
django__django-14373projectr1resolved1015K150K2K36K499s$0.008
django__django-14373projectr2resolved2115K372K4K65K318s$0.012
sympy__sympy-15875baseliner1resolved5636K3.21M70K568K812s$0.114
sympy__sympy-15875baseliner2no result–––––––
sympy__sympy-15875projectr1resolved2619K734K17K145K1116s$0.031
sympy__sympy-15875projectr2no result–––––––
sympy__sympy-19637baseliner1resolved1314K227K4K48K364s$0.010
sympy__sympy-19637baseliner2resolved1314K226K4K49K–$0.010
sympy__sympy-19637projectr1resolved811K114K1K27K298s$0.006
sympy__sympy-19637projectr2no result–––––––