ThermoMPNN vs ESM Zero-Shot vs FoldX for Stability ΔΔG

ThermoMPNN, FoldX and an ESM zero-shot scan do not answer the same question. ThermoMPNN (Dieckhaus et al., PNAS 2024) predicts the folding ΔΔG of a point mutation in kcal/mol from a structure, and it is the most accurate of the three on every benchmark in its own comparison — RMSE 0.71 kcal/mol and Pearson 0.75 on the mega-scale test set, against FoldX's 3.87 and 0.40. FoldX (Schymkowitz et al., Nucleic Acids Research 2005) is an empirical force field that also returns kcal/mol, is far less accurate in magnitude, and still ranks mutations respectably — Pearson 0.63 on the Ssym direct set. An ESM zero-shot scan (Meier et al., 2021) returns no kcal/mol at all: it gives a unitless log-likelihood ratio per mutation, scores every position in one pass without a structure, and is graded by rank correlation. The practical division of labour: scan with ESM to pick candidate positions, value the shortlist with ThermoMPNN, and test what survives.

This article is about folding stability, not binding. Folding ΔΔG asks whether the mutant still holds its own fold. Binding ΔΔG asks whether it still holds onto a partner. Those are different free energies, measured by different experiments, and in a real engineering campaign they often move in opposite directions — a core mutation that buys you 1.5 kcal/mol of fold stability can cost you the interface. None of the three methods compared here predicts a binding ΔΔG, and nothing below should be read as if it did.

At a Glance#

All ThermoMPNN and FoldX figures in that table are from the ThermoMPNN paper's own Tables 2 and 3, where the literature methods were run with default weights and inference procedures, and where the FoldX benchmark used the PositionScan protocol (Dieckhaus et al., 2024).

What Each Method Actually Computes#

ThermoMPNN: a stability head on inverse-folding features#

ThermoMPNN is a small network — 2.7M parameters — that takes a wild-type structure and a point mutation and emits a ΔΔG. Its central design choice is transfer learning: rather than train a stability model from scratch on the few thousand curated ΔΔG measurements that existed before 2023, it extracts learned features from ProteinMPNN, a network trained on the far larger PDB-wide task of predicting a sequence from a structure, and trains a lightweight head on top of those frozen features. Its final layer subtracts a predicted ΔG for the wild-type residue from the predicted ΔG of the mutant, which turns out to matter: that subtraction is what gives the model its approximate antisymmetry, and a retrained "single target" variant without it degrades badly on reverse mutations (Dieckhaus et al., 2024).

The second enabling piece is data. ThermoMPNN trains on the mega-scale stability dataset of Tsuboyama et al., Nature 2023, which used cDNA display proteolysis to measure folding stability for up to 900,000 protein domains in a one-week experiment, and from 1.8 million measurements curated roughly 776,000 high-quality folding stabilities covering all single amino acid variants of 331 natural and 148 de novo domains

40–72 amino acids long. That length range is the single most important caveat about every model trained on it, and we come back to it under limitations.

FoldX: an empirical force field, fitted to 1,000 mutations#

FoldX computes a free-energy difference from a set of physically motivated terms — van der Waals, solvation, hydrogen bonding, electrostatics, backbone and side-chain entropy — each weighted by a coefficient fitted to experimental data. The original parameterisation was fitted against more than 1,000 point mutations (Guerois, Nielsen & Serrano, Journal of Molecular Biology 2002), and the web server made the force field broadly usable (Schymkowitz et al., 2005). It has been the default stability tool in protein engineering for two decades, it is interpretable term by term in a way no neural network is, and it is still the method most published mutagenesis campaigns were designed with.

It is also the method the newer benchmarks treat least kindly, and the interesting part is how it fails. On the mega-scale test set FoldX's RMSE is 3.87 kcal/mol — a very large error against assays whose dynamic range is a few kcal/mol — while its Spearman rank correlation is 0.57, essentially tied with Rosetta's 0.56 and well above its own Pearson of 0.40. A method can order mutations usefully while being badly miscalibrated in magnitude, and that is the FoldX signature across these tables.

An ESM zero-shot scan: ranking without a structure, and without units#

A protein language model is trained only to fill in masked residues in sequences. Zero-shot variant effect prediction reads a score straight out of that objective: at the mutated position, take the model's log-likelihood for the mutant residue minus its log-likelihood for the wild-type residue. No stability data, no structure, no fine-tuning (Meier et al., 2021; ESM-2 in Lin et al., Science 2023).

The score is a statement about evolutionary plausibility, not thermodynamics. It correlates with stability because unstable proteins tend not to survive selection, which is a real but indirect coupling. Three consequences follow directly, and all three are easy to forget once the numbers are in a spreadsheet: the output has no units, so it cannot be compared to a measured ΔΔG or thresholded in kcal/mol; the score conflates every reason a residue might be conserved, including catalysis and binding, with stability; and because the model scores every substitution at every position in essentially one pass, its natural output is a whole saturation landscape rather than a verdict on a shortlist.

The Benchmark Numbers#

Two benchmark families matter here and they disagree, which is informative rather than annoying.

The ThermoMPNN paper's own comparison on its mega-scale test split and on a high-quality FireProt subset (Dieckhaus et al., 2024, Table 2 — all methods at default settings):

The older curated benchmarks, Ssym (342 mutations with crystal structures of both wild-type and mutant, so it can be scored in both directions) and S669 (669 mutations from 95 proteins, selected for dissimilarity to common training sets), from the same paper's Table 3:

The ThermoMPNN S669 row comes from a model retrained on a homologue-filtered mega-scale training set, scored on the 420 of 669 variants that survive homology filtering; the FoldX, Rosetta and mCSM rows in this table were taken from prior literature rather than re-run. Read the two tables together and the ordering is stable but the margin is not: ThermoMPNN is far ahead on the mega-scale distribution it was trained on, clearly ahead on Ssym, and only mid-field on S669, where it trails PROSTATA and ACDC-NN. The paper says so plainly.

For the ESM side the right benchmark is a different one, because there is no kcal/mol to score. ProteinGym (Notin et al., NeurIPS 2023) assembles over 250 standardised deep mutational scanning assays and grades zero-shot models by Spearman correlation, split into five assay categories of which one is stability — 66 substitution assays measuring how thermostable a protein is. On that split:

Three readings of that table are worth carrying away. Stability is the category where sequence-only language models do best, not worst — an ESM ensemble's 0.500 on stability is well above its 0.415 average. Models that read structure do better still on stability specifically: ESM-IF1 reaches 0.624 there while sitting mid-table on the average, and ProteinMPNN is the extreme case, nearly as good as an ESM ensemble on stability and last in the table overall. And ProteinMPNN being a strong zero-shot stability ranker is exactly the observation ThermoMPNN was built on.

One caution about that benchmark that applies to this whole comparison: 65 of ProteinGym's stability assays come from the same Tsuboyama mega-scale dataset ThermoMPNN trains on, and those are the 40–72 residue domains. The stability column above and the mega-scale column in the first table are substantially the same proteins, reached by two different routes.

When to Use Each#

Reach for ThermoMPNN when you have a structure and a shortlist. It is the most accurate of the three in kcal/mol, it is fast enough to run a saturation scan over a chain, and it does not need a mutant structure — only the wild-type one. It is the default for "I have twelve candidate mutations and I want them ordered with a magnitude attached", and for scoring the stability of designs that came out of a sequence-design run.

Reach for an ESM scan when you have only a sequence, or want the whole landscape. No structure is required, every position is scored, and the output is a map of where the protein tolerates change at all. This is the right first pass on a target with no structure, and the right way to narrow 2,000 substitutions to 30 before spending structure-based compute. Remember what it is ranking: evolutionary plausibility, which includes function.

Reach for FoldX when you need its term decomposition, or continuity with prior work. It remains interpretable in a way the learned models are not — you can read which energy term drives a prediction — and if a project's existing mutation set was designed with FoldX, re-scoring with FoldX is how you stay comparable. It is an external package you license and run yourself; it is not offered on .bloom*.

Use two of them, in series, when the decision is expensive. Scan with ESM, re-score the top of that list with ThermoMPNN, and look hardest at the mutations the two methods disagree about — a residue the language model hates and the stability model likes is usually functionally conserved rather than structurally critical, which is a finding, not a contradiction.

Limitations#

The training distribution is short domains. The mega-scale dataset covers domains 40–72 residues long. ThermoMPNN's per-protein error tracks that: most proteins under 100 residues keep RMSE below 1.0 kcal/mol, larger proteins are more likely to produce higher errors, and 13 of 17 proteins over 100 residues still came in under 1.5 kcal/mol. That is reassuring rather than conclusive, and on a 400-residue multidomain target you are extrapolating.

Dynamic range is compressed. Models trained on the mega-scale data inherit its measurable range, and the paper names this as a limitation of the dataset for training predictors "not explicitly built to extrapolate beyond the dynamic range of the training set". Expect a very destabilising mutation to be under-predicted rather than reported as catastrophic.

Antisymmetry is approximate, not guaranteed. A correct ΔΔG predictor should give exactly the negative value for the reverse mutation. ThermoMPNN is not constructed to be antisymmetric and its Ssym inverse performance is lower than its direct performance (PCC 0.60 against 0.72); FoldX drops further in the same comparison (0.39 against 0.63), and mCSM collapses to 0.14. If the direction of your question is "which reversion rescues this destabilised variant", every one of these methods is on weaker ground than its headline number suggests.

Stabilising predictions are not promises. ThermoMPNN's positive predictive value for stabilising mutations is 56% (34 of 61) on the FireProt high-quality set and 46% (1,312 of 2,852) on the mega-scale set, and the paper notes that studies with experimental validation generally report lower success rates than benchmark metrics imply — citing a meta-analysis of 13 mutagenesis studies using FoldX with an overall success rate of 29.4%. Plan to test more candidates than you expect to need.

Benchmark homology is a live risk. Several methods in the comparison carry an explicit asterisk for possible inflation from close homologues of FireProt proteins in their training data, and the S669 evaluation had to be run on a homology-filtered subset. Any ΔΔG leaderboard should be read with the question "which of these methods has seen this protein's relatives" held open.

An ESM score cannot be thresholded in kcal/mol. It has no units, it mixes stability with every other reason for conservation, and it will mark a catalytic residue as intolerant whether or not substituting it destabilises anything. Do not convert it, and do not compare it across proteins as if it were an energy.

What the Wet Lab Measures, and Why the Units Differ#

The experimental comparator for folding ΔΔG is a thermodynamic one: measure the free energy of unfolding for wild type and mutant and subtract. Chemical or thermal denaturation monitored by circular dichroism or fluorescence gives ΔG directly; differential scanning fluorimetry gives a melting temperature, and a ΔTm in °C is not a ΔΔG in kcal/mol — converting between them requires the unfolding enthalpy at Tm and is an approximation, not an identity. The mega-scale dataset that anchors the modern benchmarks took a third route entirely, inferring folding stability from proteolytic susceptibility in a cDNA display format, which is what made measuring hundreds of thousands of domains feasible at all (Tsuboyama et al., 2023).

So when you compare a prediction to a measurement, check which quantity the measurement is. A predictor benchmarked against proteolysis-inferred ΔG, graded by you against a ΔTm from a thermal shift assay, is being asked a question it was never scored on. And an assay of binding or activity is answering a different question again — the ProteinGym category split exists precisely because activity, binding, expression, organismal fitness and stability assays rank mutations differently.

How to Run It on .bloom*#

The platform runs ThermoMPNN as a job. FoldX is not offered on .bloom* — it is a separately licensed external package, and it appears in this article only as a comparison target.

thermompnn_predict takes a structure and, optionally, a list of point mutations:

• The structure. Fetch it first through the platform's PDB tooling (see the PDB docs) and hand the result straight to the stability job; or predict one when there is no deposited structure, with ESMFold or by fetching an AlphaFold model. ThermoMPNN was trained on computationally modelled structures and the paper reports similar performance on experimentally resolved ones, so a predicted model is a legitimate input — but read it with the usual caution about low-confidence regions, which is the subject of our comparison of pocket prediction on predicted versus experimental structures.
• The mutations. One entry per point mutation, each written [chain:]<wt><pos><mut> — S31A, or H:S31A for an explicit chain. Two separate mutations are two entries; there is no combined shorthand. Leave the list out entirely and the job runs a saturation scan over the chain instead.
• chain_id selects the chain, default A. model_type is single for single-point mutations or mt for multi-point and epistatic ones. temperature is a sampling parameter between 0.01 and 2.0 — it is not Kelvin, and nothing about it describes a thermal condition.

What comes back: a ΔΔG in kcal/mol per mutation, each classified stabilising, neutral or destabilising, the most stabilising mutation found, and a CSV of the full table. A saturation scan also returns a heatmap of ΔΔG across positions and substitutions. Mutations whose wild-type letter disagrees with the residue actually present in the structure are rejected on a per-mutation error channel rather than silently retargeted — the wild-type reported in a prediction is always the residue the model scored.

esm_scan_predict is the sequence-side counterpart: give it a sequence and it scores substitutions across positions with ESM-2, 650M parameters and masked-marginal scoring by default, returning the landscape rather than a shortlist. Check which checkpoint an option loads before quoting a published Spearman at anyone: the figure in a paper describes the model that paper scored, which is not always the one a given option runs.

Both jobs run from a signed-in session. If you are choosing between the protein tools more broadly, which tool? maps goals to jobs, and our ESMFold versus AlphaFold2 comparison covers the structure step that comes first.

Run the stability prediction on the .bloom* platform.

Key Takeaways#

• ThermoMPNN is the accuracy pick in kcal/mol. RMSE 0.71 and Pearson 0.75 on the mega-scale test set, against FoldX's 3.87 and 0.40; 1.12 and 0.72 on Ssym direct, against 1.56 and 0.63.
• FoldX ranks better than it values. Spearman 0.57 on the mega-scale set against a Pearson of 0.40 — use the ordering, distrust the magnitude. It is not offered on .bloom*.
• An ESM scan has no units. It ranks every substitution at every position without a structure, scoring 0.500 (ESM-1v ensemble) on ProteinGym's 66 stability assays, and it cannot be thresholded in kcal/mol or compared to a measurement.
• If stability is the question, structure helps even zero-shot. ESM-IF1 reaches 0.624 on the same stability assays where a sequence-only ESM ensemble gets 0.500.
• The benchmarks are short domains. 40–72 residue domains dominate the training and much of the evaluation; long multidomain targets are an extrapolation.
• Predicted stabilising mutations are hypotheses. 56% and 46% positive predictive value for ThermoMPNN on two sets, and a 29.4% success rate in a FoldX meta-analysis of 13 mutagenesis studies.
• Folding ΔΔG is not binding ΔΔG. None of these three methods answers a binding question, and the two energies routinely move in opposite directions.

References#

1. Dieckhaus, H., Brocidiacono, M., Randolph, N.Z. & Kuhlman, B. "Transfer learning to leverage larger datasets for improved prediction of protein stability changes." Proceedings of the National Academy of Sciences 121 (2024). doi:10.1073/pnas.2314853121 · code: Kuhlman-Lab/ThermoMPNN
2. Tsuboyama, K., Dauparas, J., Chen, J. et al. "Mega-scale experimental analysis of protein folding stability in biology and design." Nature 620, 434–444 (2023). doi:10.1038/s41586-023-06328-6
3. Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F. & Serrano, L. "The FoldX web server: an online force field." Nucleic Acids Research 33, W382–W388 (2005). doi:10.1093/nar/gki387
4. Guerois, R., Nielsen, J.E. & Serrano, L. "Predicting Changes in the Stability of Proteins and Protein Complexes: A Study of More Than 1000 Mutations." Journal of Molecular Biology 320, 369–387 (2002). doi:10.1016/S0022-2836(02)00442-4
5. Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T. & Rives, A. "Language models enable zero-shot prediction of the effects of mutations on protein function." bioRxiv (2021). doi:10.1101/2021.07.09.450648 · code: facebookresearch/esm
6. Lin, Z., Akin, H., Rao, R. et al. "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science 379, 1123–1130 (2023). doi:10.1126/science.ade2574
7. Notin, P., Kollasch, A., Ritter, D. et al. "ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design." Advances in Neural Information Processing Systems 36, 64331–64379 (2023). doi:10.52202/075280-2810 · code: OATML-Markslab/ProteinGym