← Subnet 114 · test bench

long02 · notail

stage-1 fail

compressor unsaved version · md5 610317f70de2 · 18/18 runs graded · tasks: django__django-13417, django__django-15930, sympy__sympy-15875

Score
-0.074
90% CI -0.35 … +0.19
Weighted saving
+6.4%
pooled per run vs baseline
Resolved
7/9
baseline 8/9
Stage gates
S1 fail · S2 fail
tasks 2/3 (need 2)
Cost
$0.57
9 graded runs

Tokens per run vs baseline

bar = arm · tick = baseline · mean per graded run
Weighted358K of 382K↓ 6.4%
Fresh input31K of 30K↑ 3.4%
Cached input2.27M of 2.53M↓ 10.4%
Output33K of 33K↑ 1.0%
Steps51.0 of 49.1↑ 3.8%

Tasks

x = baseline resolves · y = arm resolves · r = log2(W base / W arm)
TaskBase x/nArm y/nW baseW armrZoneSteps b / aOut/step b / a
django__django-134173/33/389K108K-0.274maintain23 / 27304 / 323
django__django-159303/33/3726K440K+0.723maintain83 / 67610 / 576
sympy__sympy-158752/31/3331K525K-0.667penalty42 / 60951 / 883

Runs

18 runs incl. baseline
TaskArmRepResultSteps FreshCachedOutputWeightedTimeCost
django__django-13417baseliner1resolved2318K496K8K91K410s$0.018
django__django-13417baseliner2resolved2319K491K7K91K315s$0.018
django__django-13417baseliner3resolved2319K495K6K86K–$0.015
django__django-13417notailr1resolved3122K776K11K132K436s$0.024
django__django-13417notailr2resolved2619K580K9K104K342s$0.020
django__django-13417notailr3resolved2319K492K6K87K–$0.016
django__django-15930baseliner1resolved7138K3.41M36K488K650s$0.075
django__django-15930baseliner2resolved5732K2.43M33K374K1155s$0.064
django__django-15930baseliner3resolved12059K9.90M89K1.32M–$0.184
django__django-15930notailr1resolved6835K2.99M39K452K650s$0.076
django__django-15930notailr2resolved6536K2.89M44K456K1439s$0.080
django__django-15930notailr3resolved6736K2.79M32K411K–$0.066
sympy__sympy-15875baseliner1resolved4837K2.33M40K391K1340s$0.073
sympy__sympy-15875baseliner2failed4124K1.73M40K317K1202s$0.066
sympy__sympy-15875baseliner3resolved3623K1.49M38K285K572s$0.061
sympy__sympy-15875notailr1failed7542K4.68M69K716K2053s$0.123
sympy__sympy-15875notailr2resolved3828K1.55M33K283K1111s$0.058
sympy__sympy-15875notailr3failed6641K3.66M56K576K–$0.102