Predicted Structure vs X-ray vs Cryo-EM: When Is a Predicted Model Good Enough?

A predicted model is good enough when every residue your question depends on is predicted with very high confidence, and when the answer doesn't depend on anything the model can't represent: a bound ligand, a second conformation, a disordered segment, or side-chain placement finer than about an ångström. For most well-folded domains, that covers fold assignment, domain boundaries, construct design and mapping a mutation onto a structure. It doesn't cover placing a ligand, reading an interface, or making any argument that rests on individual side chains. The most direct measurement of the gap compares AlphaFold predictions directly with crystallographic electron density maps. For residues with pLDDT above 90, the median Cα error was 0.6 Å, and about 10% of those residues were still off by more than 2 Å (Terwilliger et al., Nature Methods 2023). The authors' own summary is the right default: treat a prediction as an exceptionally useful hypothesis, and determine the structure experimentally when the details matter.

This article covers the comparison the two earlier posts leave out: predicted against experimental. ESMFold vs AlphaFold2 compares two predictors with each other. Binding pocket prediction on AlphaFold models vs experimental structures follows one downstream task, pocket finding. This one asks how close a predicted model gets to a deposited X-ray or cryo-EM structure, how to read each method's own confidence signal, and how to decide whether a given model can carry a given question.

At a Glance#

The table puts three different kinds of object side by side. The predicted column is a model with a forecast of its own error. The two experimental columns are models fitted to measured data, and their error bars come from how well the data determine the model. Each section below takes one of those error signals and says what it does and doesn't let you conclude.

How Close Does a Predicted Model Get?#

Most accuracy figures for structure predictors compare a prediction with a

deposited model, and a deposited model is itself an interpretation of data. The cleanest test skips that step. Terwilliger and colleagues compared AlphaFold predictions for 102 crystal structures with electron density maps that were computed without reference to the deposited models, so the maps carry no bias toward the PDB entry (Terwilliger et al., 2023). As a baseline they used 926 pairs of high-resolution crystal structures of the same molecule solved in different space groups. Those pairs show how much two experimental structures differ when only the crystal environment changes.

Source: Terwilliger et al. 2023, Table 1 and text. These are AlphaFold predictions, superimposed on the deposited models.

Four findings from that study set the limits of a predicted model:

• Even the best band is about twice the crystal-to-crystal difference. Very-high-confidence predictions differed from the deposited models by about twice as much as two crystal structures of the same protein differed from each other. Over whole chains, the median Cα RMSD was 1.0 Å for predictions against 0.6 Å for crystal pairs.
• Part of the error is long-range distortion. For residues with pLDDT above 70, the median deviation in inter-atomic distances was about 0.1 Å for atoms 4–8 Å apart and grew to 0.7 Å for atoms 48–52 Å apart. That is a slow warp across the molecule, not noise on each atom. Morphing the predictions with a smooth distortion field brought the median RMSD down from 1.0 Å to 0.4 Å.
• Side chains are where confident models go wrong most often. Among moderate-to-high-confidence residues away from crystal contacts, 20% of side chains differed from the deposited model by at least 1.5 Å RMSD, and 7% were clearly incompatible with the density. For the crystal-form pairs the same figures were 6% and 2%. Functionally important residues were as likely to be wrong as any others.
• Where prediction and deposition disagree, the density sides with the deposited model. The authors conclude that the predictions are not better representations of the crystal than the PDB models. They also note that predictions do not account for ligands, covalent modifications or other environmental factors.

This sample was favourable to the predictor. 86% of its residues had pLDDT above 90. Across the whole human proteome, 36% of residues fall in that band and 42% fall below 70. Median errors on a typical target will be larger than the ones in the table.

What about ESMFold?#

The density-map study above measured AlphaFold, not ESMFold, so carry over the direction of its findings and not the numbers. ESMFold's own evaluation compares its predictions with experimental structures on two held-out sets. The figures below come from the bioRxiv preprint of the ESMFold paper (v3), which we read in full (Lin et al., Science 2023; preprint):

Two things follow. On CAMEO, ESMFold's confident predictions are close to AlphaFold2's. On CASP14, the targets picked for their difficulty, the gap widens to 0.17 TM-score. The authors also report that ESMFold's accuracy tracks how well the language model understands the sequence. On CAMEO, language-model perplexity correlates with TM-score at a Pearson r of −0.55. So a sequence the language model finds surprising is the one to check hardest against experiment.

Reading the Confidence Signals Side by Side#

Each method ships with its own error signal, and they don't measure the same thing. Reading them correctly is most of the work of judging fitness for a task.

pLDDT: a forecast of local agreement#

pLDDT is the predictor's estimate, per residue, of the lDDT-Cα it would score against an experimental structure. lDDT is a superposition-free score built from local distance differences, so it stays meaningful when domains move relative to each other (Mariani et al., Bioinformatics 2013). In the AlphaFold2 paper, pLDDT predicted the measured lDDT-Cα with a Pearson r of 0.76 (Jumper et al., Nature 2021). The usual bands were defined on AlphaFold2 for the human proteome (Tunyasuvunakool et al., Nature 2021):

• Above 90: the high-accuracy cut-off. Here AlphaFold χ1 rotamers were 80% correct on a recent-PDB test set.
• Above 70: a generally correct backbone.
• Below 50: long low-confidence stretches take on a ribbon-like look and "should not be interpreted as structures but rather as a prediction of disorder". pLDDT works as a disorder predictor, with an AUC of 0.897 on the CAID benchmark.

Two caveats. pLDDT is local, so a confident domain tells you nothing about where that domain sits relative to the next one. And these band definitions come from AlphaFold2. The ESMFold paper reports its own thresholds as fractions (pLDDT > 0.7 and > 0.9), not as a separate band scheme.

pTM and PAE: the global and pairwise view#

pTM predicts the TM-score of the whole chain. In the AlphaFold2 paper it tracked the measured full-chain TM-score with a Pearson r of 0.85 (Jumper et al., 2021). Use it to ask whether the overall fold is right.

PAE (predicted aligned error) is the expected position error, in ångströms, at residue x when the predicted and true structures are aligned on residue y. AlphaFold DB caps it at 31.75 Å. Low PAE between two domains means AlphaFold predicts their relative position and orientation reliably. High PAE means you should attach no structural meaning to that arrangement. The matrix is asymmetric, so (x, y) and (y, x) can differ (Varadi et al., Nucleic Acids Res. 2021). PAE answers the question pLDDT can't: whether two well-predicted parts are placed correctly relative to each other.

Resolution, B-factors and fit to density: X-ray#

A crystal structure's resolution is a property of the whole dataset. It sets how much detail the map can show anywhere. It doesn't say that every residue was seen equally well. Wlodawer and colleagues' guide for non-crystallographers is the standard reference for reading these parameters, and its point is in its subtitle: get the best, but not more, from a published structure (Wlodawer et al., FEBS J. 2007).

• B-factors are per-atom displacement parameters. A high B-factor means that atom's density is smeared, by motion, static disorder or partial occupancy, and its position is less certain. The AlphaFold2 authors filtered their own side-chain evaluation to experimental atoms with B-factor below 30 Ų and to structures better than 2.5 Å (Jumper et al., 2021). Even the reference set for grading a predictor leaves out the poorly ordered part of a crystal.
• R-free measures how well the model predicts a small held-out set of reflections, typically 5–10%. An R-free much higher than R suggests over-fitting.
• RSRZ scores each residue's fit to the density. The wwPDB validation report flags residues with RSRZ above 2 as outliers, and it ranks each entry against structures of similar resolution (Gore et al., Structure 2017).

Unmodelled loops are their own signal. In a predicted structure every residue has coordinates. In a crystal structure, a segment with no density is usually left out. The AlphaFold team found a large difference in pLDDT between resolved and unresolved residues of PDB sequences (Tunyasuvunakool et al., 2021). A gap in the deposited chain and a low-pLDDT stretch in the prediction often mark the same region.

Global and local resolution: cryo-EM#

The resolution reported for a single-particle cryo-EM map comes from a Fourier shell correlation. It is "a single number" averaged over the entire reconstructed volume, and substantial local variations in resolution can occur within one map. Lower-than-average local resolution can mean a mobile component, incomplete occupancy, or several conformational states averaged together (Cardone et al., J. Struct. Biol. 2013). ResMap, which assigns a resolution to every voxel, measured variation across a 4–40 Å range in its four test datasets. In a ribosome map published at 4.5 Å, it placed the 60S subunit at 4.5–5.5 Å and the 40S subunit at 4.5–9 Å (Kucukelbir et al., Nature Methods 2013).

So a "3 Å cryo-EM structure" can model a well-ordered core at close to that resolution next to a peripheral domain built into much weaker density. For the question in front of you, the local resolution at your residues is the number that matters, not the headline.

One trap: the B-factor column#

Predicted structure files put pLDDT in the B-factor column. AlphaFold DB does this (Varadi et al., 2021), and so does ESMFold on .bloom*. The two numbers run in opposite directions. A high pLDDT means confident. A high crystallographic B-factor means disordered. A viewer colouring "by B-factor" will colour both files without complaint, so check which kind of file you loaded before reading the colours.

Good Enough for What? A Task-by-Task Guide#

"Good enough" is a property of the question, not of the model. The same structure can settle one question and mislead on the next. Map each task to the precision it needs, then check the confidence signal over the residues that task actually reads.

The side-chain row carries the most weight in drug discovery. For residues the predictor was confident about, Terwilliger et al. found between 7% and 20% of side chains incompatible with the experimental data or different from the deposited model. A pocket's shape is set by a handful of side chains, so a small per-residue error rate becomes a large chance that at least one lining residue is wrong.

When to Use Each#

Use a predicted structure when#

• No experimental structure exists. Experimental structures covered 17% of residues in human protein sequences when the AlphaFold human proteome was released. The predictions brought confident coverage (pLDDT > 70) to 58% of residues (Tunyasuvunakool et al., 2021).
• The question is about fold, domains or constructs, and the residues involved are confidently predicted.
• You are choosing an experiment, not replacing one. Ranking targets, picking a construct boundary, or deciding which domain to express. A wrong model then costs a failed plate, not a wrong conclusion.
• The sequence is new: a designed protein, an engineered variant, a metagenomic sequence. ESMFold folds from a single sequence, so it doesn't need homologs to exist.

Use an X-ray structure when#

• Side chains or ligand contacts matter. At good resolution they are fitted to measured density, not inferred, and where a prediction and the deposited model disagree, the density sides with the deposited model.
• You need a specific ligand-bound state. A co-crystal with your chemotype or a close analogue shows the pocket in the conformation that matters.
• Read it before you trust it: resolution, R-free, the RSRZ outliers near your site, and the B-factors of the residues you care about.

Use a cryo-EM structure when#

• The protein is large, membrane-embedded or part of a complex that doesn't crystallise, and the bound partners are part of the question.
• Several conformational states matter, and the deposition separates them.
• Read the local resolution at your residues, not the headline number. A peripheral domain can be built into density far weaker than the reported resolution.

Use both#

When a prediction and a deposited structure exist for the same protein, compare them. Agreement over the confident region is a useful check on the deposition and on the prediction. Disagreement in a confident region is the interesting case: it usually points to a ligand, a partner, a crystal contact or a second state that the prediction doesn't represent.

Limitations to Keep in View#

• A prediction shows one state. Chakravarty and Porter tested AlphaFold2 on 98 fold-switching proteins. 94% of predictions captured one experimentally determined conformation but not the other, while confidence was moderate-to-high for 74% of the fold-switching residues (Chakravarty & Porter, Protein Sci. 2022). High pLDDT does not mean that state is the only one.
• Low confidence is often disorder, and disorder is not a structure. The very-low-confidence regions of the human proteome predictions largely overlap intrinsically disordered regions. Ruff and Pappu caution against reading "structures" or interactomes into them (Ruff & Pappu, J. Mol. Biol. 2021). The AlphaFold team adds that low confidence can also mark regions that are structured only in complex.
• Predictions leave out the environment. Ligands, cofactors, covalent modifications and partners are not in a single-chain prediction, and the conformations they induce are not either.
• Experimental structures are models too. Loops and side chains can be unclear in density, crystal contacts can bend a surface loop, and a cryo-EM periphery can be over-interpreted. These limits are the reason to read resolution, B-factors and local resolution. They aren't a reason to prefer a prediction where the data are good.
• The best-measured numbers are AlphaFold's. The density-map comparison and the pLDDT band definitions were measured on AlphaFold. ESMFold's published comparisons are against deposited structures on CAMEO and CASP14. Don't assume an ESMFold band carries AlphaFold's error rate until you have checked it against an experimental structure of your own target.

How to Run It on .bloom*#

On .bloom* you can get both kinds of structure for the same protein and put them side by side. Ask for the whole comparison in one request and Ivy plans the run, chains the tools and writes up the result (how orchestration works). Or run each step on its own:

1. Predict the structure with ESMFold. esmfold_predict takes either a sequence (10–2,500 residues, the 20 standard amino acids) or a UniProt accession, plus an optional number of recycles (1–10, default 3). It runs as a GPU job and usually completes in a minute or two (ESMFold inputs).
2. Know which model you got. With a UniProt accession, the tool checks AlphaFold DB first by default. If the protein is there, it returns the AlphaFold DB model instead of running ESMFold, and the result's source field reads alphafold. Set try_alphafold_first to false when you want a fresh ESMFold prediction, for example to compare the two predictors.
3. Read what an ESMFold run returns. You get the predicted PDB with pLDDT on a 0–100 scale in the B-factor column, plus the per-residue pLDDT list, its mean, median, minimum and maximum, the percentage of residues in each band (≥ 90, 70–90, 50–70, < 50), and pTM. The result card shows the structure coloured by pLDDT (ESMFold results). An ESMFold run here doesn't return a PAE matrix, so for inter-domain placement look up the AlphaFold DB entry.
4. Look up the AlphaFold DB entry. alphafold_fetch takes a UniProt accession and returns links to the AlphaFold DB model in PDB, mmCIF and BinaryCIF, its model version and creation date, and the PAE image and PAE JSON (AlphaFold DB lookup). It retrieves a stored prediction and runs no model.
5. Download the experimental structure. pdb_download retrieves a deposited entry from the RCSB PDB by its ID (PDB docs). Read its resolution, method and validation before you treat it as the reference.
6. Compare them. Run the superposition in a sandbox Python step, which has Biopython and gemmi available. Superpose on the residues with pLDDT above 90, then report the Cα deviation per pLDDT band, the way the table above does. Then look at the residues your question depends on, one by one.

What you leave with is the predicted PDB, its per-residue confidence, the deposited entry, and a per-band deviation table for your own protein. For how far ESMFold results have been shown to hold up against the bench, see how ESMFold results compare to the bench.

Key Takeaways#

• Very-high-confidence is close, not equal. For AlphaFold residues with pLDDT above 90, the median Cα error against crystallographic density was 0.6 Å, and about 10% were off by more than 2 Å. Two crystal forms of one protein differ by a median of 0.3 Å.
• Side chains are the weak point. 7–20% of confidently predicted side chains disagreed with the experimental data or the deposited model.
• Read each method's own error signal. pLDDT and pTM for a prediction, PAE for domain placement, resolution, B-factors, R-free and RSRZ for a crystal structure, local resolution for a cryo-EM map.
• pLDDT in the B-factor column is not a B-factor. High means confident in one file and disordered in the other.
• Fitness depends on the task. Fold, domains, constructs and mutation mapping usually work on a confident prediction. Ligand placement, interfaces, disorder and second conformations don't.
• ESMFold trails AlphaFold2 most on hard targets (mean TM-score 0.68 vs 0.85 on CASP14), and the density-map numbers were measured on AlphaFold.

Treat the prediction as the hypothesis and the deposited structure as the test. Then predict a structure, pull the deposited entry and compare them, band by band, on the .bloom* platform.

References#

1. Terwilliger, T.C., Liebschner, D., Croll, T.I. et al. "AlphaFold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination." Nature Methods 21, 110–116 (2023). doi:10.1038/s41592-023-02087-4
2. Jumper, J., Evans, R., Pritzel, A. et al. "Highly accurate protein structure prediction with AlphaFold." Nature 596, 583–589 (2021). doi:10.1038/s41586-021-03819-2
3. Tunyasuvunakool, K., Adler, J., Wu, Z. et al. "Highly accurate protein structure prediction for the human proteome." Nature 596, 590–596 (2021). doi:10.1038/s41586-021-03828-1
4. Lin, Z., Akin, H., Rao, R. et al. "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science 379, 1123–1130 (2023). doi:10.1126/science.ade2574 · preprint: doi:10.1101/2022.07.20.500902 · code: facebookresearch/esm
5. Mariani, V., Biasini, M., Barbato, A. & Schwede, T. "lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests." Bioinformatics 29, 2722–2728 (2013). doi:10.1093/bioinformatics/btt473
6. Varadi, M., Anyango, S., Deshpande, M. et al. "AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models." Nucleic Acids Research 50, D439–D444 (2021). doi:10.1093/nar/gkab1061
7. Wlodawer, A., Minor, W., Dauter, Z. & Jaskolski, M. "Protein crystallography for non-crystallographers, or how to get the best (but not more) from published macromolecular structures." The FEBS Journal 275, 1–21 (2007). doi:10.1111/j.1742-4658.2007.06178.x
8. Gore, S., Sanz García, E., Hendrickx, P.M.S. et al. "Validation of Structures in the Protein Data Bank." Structure 25, 1916–1927 (2017). doi:10.1016/j.str.2017.10.009
9. Cardone, G., Heymann, J.B. & Steven, A.C. "One number does not fit all: Mapping local variations in resolution in cryo-EM reconstructions." Journal of Structural Biology 184, 226–236 (2013). doi:10.1016/j.jsb.2013.08.002
10. Kucukelbir, A., Sigworth, F.J. & Tagare, H.D. "Quantifying the local resolution of cryo-EM density maps." Nature Methods 11, 63–65 (2013). doi:10.1038/nmeth.2727
11. Chakravarty, D. & Porter, L.L. "AlphaFold2 fails to predict protein fold switching." Protein Science 31, e4353 (2022). doi:10.1002/pro.4353
12. Ruff, K.M. & Pappu, R.V. "AlphaFold and Implications for Intrinsically Disordered Proteins." Journal of Molecular Biology 433, 167208 (2021). doi:10.1016/j.jmb.2021.167208