← Subnet 114 · test bench

s1_fix · project

stage-1 pass

compressor base_miner.py (current project) · md5 db938439ee55 · ★ current project · 50/50 runs graded · tasks: django__django-12419, django__django-13821, django__django-14373, sympy__sympy-15875, sympy__sympy-19637

Score
+0.148
90% CI +0.02 … +0.31
Weighted saving
+13.8%
pooled per run vs baseline
Resolved
23/25
baseline 22/25
Stage gates
S1 pass · S2 pass
tasks 5/5 (need 3)
Cost
$0.82
25 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted167K of 193K↓ 13.8%
Fresh input22K of 23K↓ 4.0%
Cached input931K of 1.10M↓ 15.7%
Output17K of 20K↓ 14.0%
Steps24.8 of 28.4↓ 12.7%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-124195/55/5203K121K+0.746maintain32 / 23617 / 472
django__django-138215/55/5224K194K+0.208maintain34 / 30655 / 702
django__django-143735/55/543K31K+0.478maintain12 / 9261 / 220
sympy__sympy-158752/53/5450K439K+0.037maintain49 / 48999 / 894
sympy__sympy-196375/55/547K49K-0.060maintain14 / 14217 / 228

Runs

50 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-12419baseliner1resolved2125K678K14K136K444s$0.029
django__django-12419baseliner2resolved3024K883K15K156K1017s$0.030
django__django-12419baseliner3resolved2924K852K16K158K414s$0.032
django__django-12419baseliner4resolved4532K1.61M21K257K471s$0.045
django__django-12419baseliner5resolved3638K1.73M32K306K525s$0.060
django__django-12419projectr1resolved2733K889K15K168K460s$0.034
django__django-12419projectr2resolved1921K514K13K112K316s$0.025
django__django-12419projectr3resolved2020K479K8K93K369s$0.019
django__django-12419projectr4resolved2621K605K7K104K319s$0.019
django__django-12419projectr5resolved2527K709K10K128K365s$0.024
django__django-13821baseliner1resolved2825K914K17K168K1459s$0.034
django__django-13821baseliner2resolved4136K1.55M24K262K415s$0.048
django__django-13821baseliner3resolved3331K1.27M24K230K502s$0.046
django__django-13821baseliner4resolved3129K1.13M23K209K1099s$0.042
django__django-13821baseliner5resolved3833K1.45M25K252K1049s$0.048
django__django-13821projectr1resolved2527K751K15K149K474s$0.031
django__django-13821projectr2resolved3233K1.20M23K224K341s$0.045
django__django-13821projectr3resolved3131K1.22M36K261K423s$0.060
django__django-13821projectr4resolved2927K952K20K182K385s$0.038
django__django-13821projectr5resolved3428K957K11K156K352s$0.027
django__django-14373baseliner1resolved1112K178K3K40K397s$0.009
django__django-14373baseliner2resolved1514K255K3K49K208s$0.010
django__django-14373baseliner3resolved1114K186K3K43K265s$0.009
django__django-14373baseliner4resolved711K98K1K25K274s$0.006
django__django-14373baseliner5resolved1813K305K5K58K318s$0.011
django__django-14373projectr1resolved912K136K2K30K264s$0.006
django__django-14373projectr2resolved711K99K2K26K274s$0.006
django__django-14373projectr3resolved1312K211K4K46K258s$0.010
django__django-14373projectr4resolved711K97K1K24K273s$0.005
django__django-14373projectr5resolved812K118K2K29K276s$0.006
sympy__sympy-15875baseliner1resolved2618K695K15K133K1551s$0.028
sympy__sympy-15875baseliner2failed4324K1.95M45K355K828s$0.073
sympy__sympy-15875baseliner3failed7137K4.72M82K754K1957s$0.137
sympy__sympy-15875baseliner4resolved6336K3.90M68K631K818s$0.116
sympy__sympy-15875baseliner5failed4326K2.07M49K379K821s$0.079
sympy__sympy-15875projectr1failed1817K508K18K121K376s$0.030
sympy__sympy-15875projectr2failed7142K4.55M77K728K1003s$0.132
sympy__sympy-15875projectr3resolved2717K551K7K93K420s$0.017
sympy__sympy-15875projectr4resolved8750K6.15M112K999K1633s$0.186
sympy__sympy-15875projectr5resolved3624K1.36M31K253K644s$0.052
sympy__sympy-19637baseliner1resolved1313K208K2K41K243s$0.008
sympy__sympy-19637baseliner2resolved1513K249K3K49K242s$0.010
sympy__sympy-19637baseliner3resolved1314K211K3K44K286s$0.009
sympy__sympy-19637baseliner4resolved1514K253K3K47K249s$0.009
sympy__sympy-19637baseliner5resolved1514K260K4K52K247s$0.011
sympy__sympy-19637projectr1resolved1214K200K2K41K239s$0.008
sympy__sympy-19637projectr2resolved1514K251K3K48K242s$0.009
sympy__sympy-19637projectr3resolved1113K176K2K37K288s$0.008
sympy__sympy-19637projectr4resolved1514K253K4K51K277s$0.011
sympy__sympy-19637projectr5resolved1717K336K5K64K261s$0.013