← Subnet 114 · test bench

s1_orig · project

stage-1 pass

compressor variants/base_miner_pre_gitshow_fix.py · md5 c35648cae1c2 · 50/50 runs graded · tasks: django__django-12419, django__django-13821, django__django-14373, sympy__sympy-15875, sympy__sympy-19637

Score
+0.084
90% CI -0.15 … +0.21
Weighted saving
+21.2%
pooled per run vs baseline
Resolved
22/25
baseline 22/25
Stage gates
S1 pass · S2 pass
tasks 4/5 (need 3)
Cost
$0.82
25 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted163K of 207K↓ 21.2%
Fresh input22K of 24K↓ 7.5%
Cached input896K of 1.05M↓ 15.1%
Output17K of 26K↓ 33.6%
Steps26.2 of 27.1↓ 3.5%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-124195/55/5326K149K+1.131maintain34 / 251294 / 646
django__django-138215/55/5251K209K+0.266maintain32 / 31744 / 708
django__django-143735/55/535K48K-0.460maintain11 / 14200 / 268
sympy__sympy-158752/52/5382K362K+0.074maintain46 / 47965 / 869
sympy__sympy-196375/55/542K48K-0.194maintain13 / 14200 / 225

Runs

50 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-12419baseliner1resolved1826K475K12K111K426s$0.026
django__django-12419baseliner2resolved4731K1.85M31K310K496s$0.058
django__django-12419baseliner3resolved5153K2.68M209K946K1118s$0.282
django__django-12419baseliner4resolved3224K905K16K161K287s$0.031
django__django-12419baseliner5resolved2119K507K11K104K257s$0.022
django__django-12419projectr1resolved2429K759K17K155K452s$0.033
django__django-12419projectr2resolved3826K1.26M24K223K442s$0.044
django__django-12419projectr3resolved1724K492K11K108K285s$0.024
django__django-12419projectr4resolved2725K854K20K168K296s$0.036
django__django-12419projectr5resolved1722K439K9K92K297s$0.020
django__django-13821baseliner1resolved5758K3.43M43K530K1533s$0.089
django__django-13821baseliner2resolved2627K854K19K171K370s$0.037
django__django-13821baseliner3resolved2527K842K20K172K353s$0.037
django__django-13821baseliner4resolved3243K1.42M25K258K394s$0.051
django__django-13821baseliner5resolved2124K589K14K124K322s$0.027
django__django-13821projectr1resolved3742K1.46M22K255K507s$0.048
django__django-13821projectr2resolved3131K1.06M17K188K340s$0.036
django__django-13821projectr3resolved2926K965K22K189K353s$0.040
django__django-13821projectr4resolved2330K821K16K160K270s$0.033
django__django-13821projectr5resolved3330K1.29M31K251K408s$0.054
django__django-14373baseliner1resolved1112K164K2K35K370s$0.007
django__django-14373baseliner2resolved1313K196K2K39K248s$0.008
django__django-14373baseliner3resolved1114K182K2K39K218s$0.008
django__django-14373baseliner4resolved1112K160K2K33K212s$0.006
django__django-14373baseliner5resolved812K116K2K29K221s$0.007
django__django-14373projectr1resolved1814K306K4K56K255s$0.011
django__django-14373projectr2resolved1713K265K3K47K253s$0.009
django__django-14373projectr3resolved1415K267K5K56K246s$0.012
django__django-14373projectr4resolved1115K195K4K48K247s$0.011
django__django-14373projectr5resolved1012K154K2K34K244s$0.007
sympy__sympy-15875baseliner1failed3426K1.32M30K248K622s$0.052
sympy__sympy-15875baseliner2failed5028K2.57M48K430K1927s$0.082
sympy__sympy-15875baseliner3failed3318K1.15M29K221K1347s$0.047
sympy__sympy-15875baseliner4resolved4624K2.10M48K377K555s$0.077
sympy__sympy-15875baseliner5resolved6833K3.83M72K633K1414s$0.119
sympy__sympy-15875projectr1failed3822K1.35M30K246K876s$0.050
sympy__sympy-15875projectr2failed6833K3.78M59K588K892s$0.103
sympy__sympy-15875projectr3failed5124K2.34M49K405K563s$0.080
sympy__sympy-15875projectr4resolved4625K2.09M51K386K1003s$0.081
sympy__sympy-15875projectr5resolved3321K1.04M21K187K387s$0.037
sympy__sympy-19637baseliner1resolved911K129K1K28K190s$0.006
sympy__sympy-19637baseliner2resolved1314K223K4K47K265s$0.010
sympy__sympy-19637baseliner3resolved1213K198K2K40K227s$0.008
sympy__sympy-19637baseliner4resolved1614K275K3K52K209s$0.010
sympy__sympy-19637baseliner5resolved1314K213K2K42K229s$0.008
sympy__sympy-19637projectr1resolved1213K189K2K39K238s$0.008
sympy__sympy-19637projectr2resolved1313K204K2K40K250s$0.008
sympy__sympy-19637projectr3resolved1314K223K4K48K234s$0.010
sympy__sympy-19637projectr4resolved1514K259K4K51K207s$0.010
sympy__sympy-19637projectr5resolved1915K336K4K62K254s$0.012