Frozen question · traceable result artifacts
How much difficulty signal is in problem-statement structure?
After excluding rating, solved counts, tags, points, and other strong signals from the model inputs, this study asks whether 41 fixed statement-structure features improve forward-time rating prediction beyond two problem-index features alone.
Retrospective study
Rerun protocol frozen before test evaluation
Auditable outputs
The narrow claim
In this historical cohort and under the frozen comparison, statement structure contains predictive signal beyond problem-index position. The result does not establish prospective performance.
Locked forward-time result
One comparison, reported with its uncertainty.
The 15.9% reduction is relative only to the predeclared index-only comparator. It is not a claim of improvement over the legacy full-API model or the state of the art.
Index-only comparator
MAE 477.2151 index_rank + index_numberIndex + statement structure
MAE 401.2704 the same 2 features + 41 frozen featuresPrimary contrast
−75.9447 primary MAE − comparator MAERelative reduction
15.9% against the index-only comparatorRMSE
575.4 → 503.3 secondary metricR-squared
0.5005 → 0.6179 secondary metric
Paired contest-cluster bootstrap
95% interval [−86.7339, −65.0783]
10,000 paired resamples across 355 test contests, using the frozen PCG64 seed 42. The interval describes uncertainty in this locked historical cohort; it does not guarantee a future effect.
Research design
Controls were fixed before this rerun's final test evaluation.
The repository records the question, split, exact feature allowlists, selection rule, test policy, uncertainty method, and automatic error-analysis rule.
Freeze the question
Compare two index features against the same two features plus 41 named statement-structure features. Unexpected inputs fail the run instead of entering silently.
Split forward in time
Complete contest start-time buckets are ordered chronologically and allocated 70/10/20. Shared timestamps never cross a boundary.
Select on validation only
Alpha is chosen separately for each Ridge model using validation MAE. Both selected 0.01; test metrics do not choose the model.
Seal before testing
Configuration, split membership, prepared data, validation metrics, feature lists, and their SHA-256 digests are captured in the selection lock.
Test and preserve
The policy allows one final evaluation after selection. Git records one committed locked result with row-level predictions, bootstrap output, Top-10 errors, and a result hash manifest.
Interactive evidence explorer
Inspect what was decided, measured, and preserved.
This replaces the old heuristic difficulty slider. It presents the real locked study and never pretends to run a trained model in the browser.
FROZEN COMPARISON
Does statement structure add signal beyond index?
2 features vs 43 features
The comparator sees index_rank and index_number. The primary model adds 41 fixed statement-structure features.
Rating is the target, not an input. Ratings, solved counts, tags, points, and participation signals are forbidden from both input matrices.
Open the frozen configurationAudit trail
Every headline number has a trail.
The compact committed artifacts are enough to recompute selection and final metrics from the prepared data. The original 718 MB page cache is not committed, so HTML-to-feature extraction is not fully reproducible from this repository alone.
01 · governance
Public erratum
What was wrong in the historical evaluation and how claims changed.
02 · design Frozen protocolThe exact question, split, models, metric, test policy, and non-claims.
03 · before test Selection lockSelected alphas, feature lists, input hashes, and test_evaluated: false.
04 · result Final test metricsMachine-readable MAE, RMSE, R-squared, and the primary contrast.
05 · row level Test predictionsPredictions and errors for all 2,468 locked test problems.
06 · integrity Result hashesSHA-256 manifest for the final committed result files.
Automatic Top-10 error analysis
The largest errors suggest likely failure modes.
The ten examples were selected automatically by absolute error; no case was manually substituted after viewing the test.
ICPC letters are identifiers
Manual post-hoc inspection found that all ten largest errors were overpredictions on ICPC-style problems. A likely explanation is that, unlike typical Codeforces rounds, their letters are identifiers rather than a difficulty order.
Ridge predictions were not clipped
An unusual index and statement-feature combination can produce a prediction outside the observed 800–3500 rating range. Changing this after test would invalidate the locked comparison.
One page was a PDF
Problem 2073/L returned a PDF rather than parsable HTML, so its statement features followed the frozen missing-page policy.
Fixes belong in a new protocol
Contest-type interactions, revised index encoding, or clipping may be useful hypotheses, but none can be retrofitted into the result already opened and reported.
Evidence boundary
Strong traceability, deliberately conservative claims.
The evidence supports
- Statement structure added signal beyond the frozen index-only comparator.
- Candidate alphas were selected using validation MAE only.
- Committed prepared data can recompute selection and final metrics.
- Historical evaluation problems and corrections are publicly documented.
The evidence does not establish
- Prospective, independently preregistered, or strict pre-contest performance.
- Understanding of algorithms, proofs, or intrinsic problem difficulty.
- Improvement over the legacy full-API model, a state-of-the-art method, or another platform.
- The page state at contest time or end-to-end extraction without the local cache.
Historical exploration archive
Earlier results are context, not the current headline.
These experiments motivated the present question, but the old split configuration and test-informed selection issues mean their metrics are retained as retrospective, hypothesis-generating work.
Legacy contest-grouped and forward-time headline metrics. Solved count is powerful but reflects exposure and participation as well as difficulty.
Historical metadata-plus-structure comparisons that motivated the narrower locked statement-only backtest.
Bag-of-words statement text added signal in legacy evaluations; it is not deep semantic understanding and was out of scope for the locked backtest.
Project records