The transformer paper-recommendation advantage is real at the head of the impact distribution and decays to null across the broad literature: a 4-checkpoint, all-26-field convergence study of SPECTER vs TF-IDF

A companion single-subfield study showed that SPECTER (a transformer pre-trained on the scientific-paper citation graph) beats a TF-IDF baseline at held-out citation prediction in High-Energy Physics (+15.4% relative MAP, p = 0.0005) — a large win, measured on the citation-dense top-cited slice of one subfield. The natural question — the one a practitioner faces when reaching for a neural paper-recommender — is whether that advantage generalizes.

We answer it with a checkpointed, resumable, producer/consumer pipeline that pulls an impact-ranked corpus (most-cited papers first) across all 26 OpenAlex top-level fields, builds a complete in-corpus citation graph and PageRank per field, embeds title+abstract on a local AMD GPU (ROCm) at a measured 9.1 docs/s, and runs the identical held-out citation-prediction evaluation — SPECTER vs TF-IDF vs word2vec vs a text-free graph recommender, with bootstrap CIs and a paired test — in every field, then grows the corpus and re-measures. The result is a clean convergence finding. At checkpoint 1 (top ~3k works/field, 78,000 works), SPECTER beats TF-IDF in 16 of 26 fields and the across-field edge is large and nearly significant: combined mean ΔMAP +0.0095 (95% CI [−0.0005, +0.0195], bootstrap p = 0.062). As the impact-ranked corpus broadens — checkpoint 2 (130,000), checkpoint 3 (361,800) — the edge decays monotonically toward null: 12/26 then 11/26 wins; combined ΔMAP −0.0008 (p = 0.80) then −0.0019 (95% CI [−0.0075, +0.0034], p = 0.49). Checkpoint 4 found the impact-ranked corpus had plateaued at 361,800 works and reproduced checkpoint 3 exactly, so the result is converged.

The headline: neural paper-recommendation's edge is concentrated in the head of the impact distribution; across the broad literature it is not a general win. The advantage survives where fine-grained phrase meaning carries relevance (Computer Science, Social Sciences, Neuroscience, Biochemistry) and reverses in physical-science / pharmacology fields (Pharmacology −0.040 p < 0.001; Chemistry −0.025 p < 0.001; Earth & Planetary −0.020 p < 0.001) where exact-term matching wins. Citation concentration (Gini 0.243–0.501) and interdisciplinarity (cross-field reference fraction 0.169–0.558) vary by field but do not predict the split.