Research, 2026–Present
Proactive Feasibility Scheduler
- 2026–present
- research
- python · xgboost
- repo
- seeded · docker · CI
- 1 star
- head b491ee0 · 2026-09-11
- ci: passing
- Joseph Wright of Derby, A Philosopher Lecturing on the Orrery, 1766 — public domain, Wikimedia Commons
An evaluation study of ML-based GPU-cluster job scheduling — and a proven negative result.
- counterexamples
- 0
- dispatch instants
- 45,432
- paired TOST
- p = 2.6×10−16
Problem
The research question is narrow and testable: can machine learning improve GPU-cluster job scheduling via learned wait-time prediction? The intuition behind an ML "Proactive" scheduler is that a model predicting each queued job's wait from live cluster state should order the queue better than classical heuristics.
This study answers that question end to end — and the answer is a proven negative result. The learned scheduler is structurally degenerate: at the dispatch instant, its score cannot distinguish two queued jobs by anything except requested size.
Approach
The pipeline begins with a discrete-event cluster simulator (01_simulation/) and an XGBoost regressor that predicts per-job wait time from 12 cluster-state features (can_fit_now, gpu_fit_ratio, node_availability, queue_pressure, and others). Under run-wise 5-fold cross-validation (GroupKFold on the simulation run) the regressor reaches R2 0.811 ± 0.021 and MAE 4.90 ± 0.45 — corrected down in v3.6 from a random-row split that put rows of the same run on both sides; under a within-run chronological split R2 is 0.725. The degeneracy result does not depend on this number.
Fourteen policies run against the simulator — FCFS, SJF (oracle, estimated), SRPT, EASY and conservative backfill, HRRN, and the ML Proactive scheduler — with Holm-adjusted significance testing. Validation then moves off synthetic workloads onto real supercomputer traces from the Parallel Workloads Archive: LANL CM-5 and SDSC SP2, replayed second-exact, 12 policies across 20 paired 7-day windows at load ≈ 0.70, using the real user runtime estimates recorded in the traces.
The degeneracy audit (04_scheduler/ranking_degeneracy.py) asks one question of the full pipeline: can the learned score ever rank two identically-sized queued jobs differently? Everything is reproducible — bash run_all_experiments.sh, Python 3.11+, seeded runs, Docker, CI.
simulator
discrete-event cluster
xgboost regressor
wait-time, R2 0.811
14-policy bench
Holm-adjusted stats
trace replay
LANL CM-5 · SDSC SP2
degeneracy audit
45,432 instants
negative result
TOST p = 2.6×10−16
Decisions
- Test equivalence, not difference
- A standard Holm-adjusted difference test between the ML scheduler and a size sort returns p = 0.17 — no detectable difference, and the tie stays hidden. Running a paired TOST equivalence test instead makes the tie the finding: sorting by requested size is statistically equivalent to the full XGBoost pipeline, p = 2.6×10−16.
- Prove the degeneracy structurally
- At any single dispatch instant, cluster-state features take the same value for every queued job — 7 of the 12 features vary across the queue in 0.0% of instants. That makes the learned score a function of requested size alone, and the audit confirms it empirically: zero counterexamples in 45,432 dispatch instants (41,786 real + 3,646 synthetic) where two identically-sized queued jobs scored differently — 45,268 instants and zero again on a Linux runner; the count is platform-dependent, the result is not.
- Replay real traces second-exact
- The synthetic benchmark looked favorable: 7.9% ± 9.4% mean wait reduction vs FIFO over 40 paired runs (paired t, p = 2.0×10−6). But utilisation and completions were identical in all 40 runs — a reordering effect, not a throughput gain — and per-job fairness moved the wrong way (Gini 0.526 → 0.794, max wait 57.85 → 122.65 ts, per the repo's honest-claims record) — and on real traces the headline gain proved machine-dependent: −20.4% vs FCFS on SDSC (p = 0.042) but −4.5%, p = 0.48, on LANL.
- Name the honest baseline
- The control implied by this feature set is not FIFO — it is a sort by requested size, which needs no training, no inference, and no monitoring. Judged against that ML-free control, the learned scheduler adds nothing: both land on the same mean wait (145.0) in the SDSC replay.
Evidence
| Dispatch instants audited (reference platform) | 45,432 — zero counterexamples (41,786 real, 3,646 synthetic) |
|---|---|
| Equivalence to size sort (paired TOST) | p = 2.6×10−16 |
| Regressor, run-wise 5-fold CV | R2 0.811 ± 0.021 · MAE 4.90 ± 0.45 |
| Synthetic gain vs FIFO (40 paired runs) | 7.9% ± 9.4% (p = 2.0×10−6) |
| SJF vs ML Proactive, SDSC | 20.2% lower wait (Holm p = 0.009) |
| Zero-shot transfer, both real traces | R2 ≈ 0 |
Outcome
What stands proven, since the v3.4 degeneracy audit: at the dispatch instant, the learned score is a function of requested job size alone. Across 45,432 dispatch instants on the reference platform (41,786 replayed from real traces, 3,646 synthetic; 45,268 and zero again on a Linux runner), two identically-sized queued jobs never scored differently, and paired TOST (p = 2.6×10−16) establishes that a size sort is equivalent to the XGBoost pipeline. The simulated gain does not replicate on real hardware traces — the ML scheduler beats FCFS on SDSC (−20.4%, p = 0.042) but not on LANL (−4.5%, p = 0.48), plain SJF on the users' own estimates beats it by 20.2% on SDSC (Holm p = 0.009) and by 15.3% on LANL, where the gap is not significant (Holm-adjusted t p = 0.54, Wilcoxon p = 0.43), and transfer is weak: zero-shot R2 ≈ 0 on both traces, 0.49 (SDSC) and 0.10 (LANL) after retraining.
The constructive output is three findings. A wait-time feature set can only produce a meaningful ranking if it contains a per-job attribute that is not a function of size given cluster state; the honest baseline for an ML scheduler is the ML-free control its feature set implies, not FIFO; and equivalence tests are required — the difference test (Holm-adjusted p = 0.17) would have hidden the tie. The replay also surfaced side results: EASY backfill costs +74% mean wait under real estimate error versus perfect estimates on LANL (+6.2% on SDSC) — significant under the Wilcoxon rank test (Holm p = 2.1×10−5) but not the t-test (raw p = 0.025, 0.23 after Holm), and real estimate distributions (SDSC median 6.91× over-estimate with 0.1% under; LANL 1.51× with 36.3% under-estimates) do not match the f-model's 3.0× over-only assumption.
Next
Finish the LaTeX manuscript, in progress at phases_22_30/phase_28_manuscript/manuscript.tex.
Per-job attributes that satisfy the non-degeneracy condition — runtime estimates, user history, partition identity — define what any non-degenerate successor feature set must contain.