How ESM-3 results compare to the bench

You have a result from an ESM-3 surface. Here is what it supports.

• ⊘ The model served here is not ESM-3. Every surface labelled ESM-3 runs a smaller Meta ESM-2 protein language model, and the size options do not select between the sizes the documentation advertises. Read every published ESM-3 figure on this page as an upper bound on this endpoint rather than a description of it. The embedding dimensions you actually receive also differ from the documented ones, so read the dimension off the array you are given.
• The one number that should change your reading. On the same 201 ProteinGym deep-mutational-scanning assays, scored the same way by the same independent group: ESM-2 35M reaches Spearman ρ = 0.333, ESM-2 650M 0.428, and ESM-3 0.468. The default configuration here is the weakest row in that table, about 0.135 Spearman below the model the page is named after.
• ⊘ The variant-effect recipe these docs tell you to use has no published concordance at all. Every DMS figure for every ESM model grades the masked log-likelihood ratio at a mutated position. outputs and quickstart tell you to compare wild-type and mutant embeddings by cosine distance. That is a different estimator and no study grades it. If you want variant effects, use esm_scan_predict, which runs ESM-2 650M masked marginals — the method the benchmark actually measures — is honestly labelled, and costs 2 credits instead of 42.
• Structure and function are not here. task: "structure_prediction" is redirected to ESMFold and task: "function_prediction" to InterPro, by design — neither was ever implemented on this endpoint. There is no function track in the deployed compute at all, so no GO/EC figure on this page would describe anything that runs.
• Attention maps and contact prediction do not exist. return_attention and return_contacts are accepted and read by nothing; no H x L x L tensor is ever computed. The only "attention" in the compute is the tokenizer's padding mask.
• esm3_inverse_fold fails every call and is still billable. It raises rather than fabricate a sequence, which is the right behaviour for an unwired model and the wrong behaviour for a listed tool.
• Nothing produced here has been expressed or assayed. Every wet-lab validation of ESM-3 generation in the literature used real ESM-3 (7B or 1.4B) with structure-conditioned prompts. What runs here is position-by-position ESM-2 masked-LM sampling, capped at 256 residues, with no published wet-lab result of any kind.
• Confirm anything with the bench. For a variant effect, that is a deep mutational scan or a single-variant assay. For a designed sequence, it is expression plus a functional assay. No embedding substitutes.

Jump to agreement with wet lab, against other in silico methods, or the defect list. The general framing for every tool is in reading an in silico result against the bench.

How in silico is used currently

Protein language models are used for four jobs, and the field keeps them separate because they are graded against four different things.

• Representation. Turn a sequence into a vector and feed it to something else — a classifier, a clustering run, a similarity search. This is what the platform's esm3_embeddings surface returns, and it is graded indirectly: by how well the downstream model does, never by comparing the vector to a measurement.
• Variant effect. Score every possible point mutation and rank them. This is the one PLM task with a large, settled wet-lab comparator: deep mutational scanning. It is graded as Spearman correlation per assay against measured fitness.
• Structure. Predict coordinates. ESM-3 has a structure track; this platform does not expose it, and ESMFold is where that question is graded here.
• Generation. Propose sequences that do not exist yet. The only honest comparator is expression plus a functional assay, and the published examples are counted in single digits.

What nobody does, with validation they report honestly, is treat an embedding distance as a measurement. A cosine similarity of 0.82 between two vectors is a statement about a model's internal geometry. It becomes a statement about biology only after something downstream has been graded against an experiment.

Against other in silico methods

On zero-shot variant effect, the independent 201-assay evaluation tabulated in agreement with wet lab is the head-to-head, and the ordering by mean Spearman ρ is: ProSST 0.506, S3F 0.473, fine-tuned ESM-2 + structure 0.469, ESM-3 0.468, SaProt 0.462, S2F 0.460, ProtSSN 0.453, ESM-IF 0.440, ESM-2 650M 0.428, ESM-2 3B 0.421, ESM-2 150M 0.400, MIF-ST 0.400, MIF 0.382, ESM-2 35M 0.333, MULAN 0.323. ESM-3 is mid-table, four hundredths behind the leader and one thousandth behind a fine-tuned ESM-2. The platform's default configuration is second from last. (ESM-IF is the shape to watch: 0.440 on the average but 0.623 on stability, third-highest in the set behind only ProSST's 0.651 and ESM-3's 0.640 — a model can be mediocre overall and still the right tool for one readout class.)

On viral variant effect, the released ESM-3 loses to a non-deep-learning baseline. ESM3-open's 0.17 mean Spearman over 28 viral DMS landscapes sits "at the level of a BLOSUM prior" and "substantially worse than EVMutation", a Markov random field over a per-protein alignment. The vendor's own explanation is a clustering-threshold difference — EVMutation uses lower sequence-diversity thresholds for viral MSAs while ESM-2 and ESM-3 cluster training data at fixed 50% and 70% identity — but the mechanism does not change the number.

On contact prediction from representations, the margin ESM-3 is credited with is thin and not uniform. With a LoRA-finetuned MLP head, precision at L on the same held-out sets:

The vendor's phrasing is "slightly outperforming ESM2 (3B)". At 1.4B that is +0.01 on CAMEO with fully overlapping 95% confidence intervals, and −0.01 on CASP14. Read it as parity. Note also that this is a finetuned-head number: it does not describe what you get from a frozen pooled embedding, which is what the platform returns.

On embedding geometry — the platform's actual headline use — the one independent measurement is ambiguous. Clustering 10 recent SARS-CoV-2 lineages (1,000 sequences each) by Spike embeddings, ESM-3 reached ARI 0.8750 / NMI 0.8964, behind ESM-C's 0.9001 / 0.9143 and barely ahead of a transformer trained from scratch (0.8556 / 0.8983). Correlating pairwise cosine distance between lineage-mean embeddings against JTT-corrected evolutionary distance across 1,237 lineages, ESM-3 is reported at Pearson r = 0.797, Spearman ρ = 0.741.

⚠︎ That second pair must not be read as a win. In the source table, the ESM-3 row and the "transformer from scratch" row are identical to six decimal places (0.796673 / 0.740814 in both). Either the table duplicates a row or the coincidence is extraordinary; in neither case can the figure support "ESM-3 embeddings beat a small scratch-trained model at recovering evolutionary distance." The clustering table in the same paper has ESM-3 ahead of scratch by 0.019 ARI, which is the safer statement.

One finding from that work has a direct bearing on how these docs describe the model. The FAQ presents ESM-3 as multimodal. When the same group gave the pretrained model the Spike structure as an additional input, the zero-shot per-position amino-acid distributions improved "substantially compared to sequence-only inference" — attributable, with no parameter updates, "solely to the input modality." The platform's embedding path accepts sequence only, so the multimodal advantage the docs describe is unreachable here even if ESM-3 were loaded.

On memorization, the model's own paper gives the warning. Unconditional ESM-3 generations include "a cluster of generations with very high sequence identity to the training set; these correspond to antibody sequences, with the framework regions accounting for the high sequence identity."

In silico vs. wet lab

Two capabilities are exposed here, so there are two comparators.

For variant effect, the standard wet-lab approach is a deep mutational scan: build a library covering every substitution, apply a selection or a reporter, sequence the survivors, and read out a fitness score per variant. It is the comparator every figure in the next section uses. ⚠︎ DMS has no published commercial price of any kind — it appears in no row of the shared wet-lab cost basis, which is itself built from academic core-facility rate cards because no CRO publishes a list price for any technique on these pages. Do not read a cost figure into that cell that is not there.

For generated sequences, the comparator is expression plus a functional assay — synthesize the gene, express it, and measure the activity you designed for. Also unpriced publicly.

Read the accuracy column by cell, never down the column. A DMS Spearman is a rank correlation against a measured selection. "No figure describes this configuration" is not a low score; it is an absent score, which is worse, because you cannot bound it.

On that reading, the honest position is uncomfortable: the ESM-3 surfaces cost 42 credits to return a vector no study has graded, while a 2-credit sibling tool runs the one method the literature does grade, on a model twenty times larger than the ESM-3 default. If your question is a variant effect, the pricing and the evidence both point away from this tool.

Agreement with wet lab

The four tracks have four different comparators and four very different amounts of evidence. Do not blend them.

Variant effect, graded against deep mutational scanning

This is the only track with a substantial wet-lab comparator, and the measurement is Spearman ρ per assay against a DMS fitness landscape, reported across many assays.

Independently measured, over ProteinGym reduced to 201 assays (the original 217 minus proteins over 1,000 residues, because most baselines cap at 1,024 tokens), split by primary readout — 39 activity, 12 binding, 16 expression, 69 fitness, 66 stability:

Four things in that table matter more than the ordering.

The spread across readout classes is larger than the spread across models. ESM-3 runs from 0.391 on fitness to 0.640 on stability — a 0.249 range within one model, against a 0.173 range between the weakest and strongest model on the average. A single headline ρ hides which question you are asking. Binding is the hardest class for every model in the table (ESM-3 0.395, ESM-2 650M 0.369), and binding is what a drug-discovery reader usually wants.

Scaling is not the lever the docs imply. ESM-2 3B scores below ESM-2 650M on the average (0.421 vs 0.428). The FAQ advice to reach for a larger variant "where embedding quality is critical" is not supported even for the real ESM model family, and on this platform esm3_lg is byte-identical to esm3_md anyway.

ESM-3's advantage over ESM-2 650M is real but modest — 0.040 Spearman — and a fine-tuned ESM-2 erases it. The same paper reports that fine-tuning ESM-2 650M with structural inputs reaches 0.469, which it states as "comparable performance to ESM3 (0.468), despite ESM3 being a 1.4B parameter model trained with far greater resources."

No per-assay distribution is published for ESM-3. Both sources report group means. The vendor's figure panel shows the viral and non-viral distributions but prints no values, and the independent paper puts per-assay results in supplementary figures. So the honest statement about spread is the category breakdown above, plus the viral/non-viral split below — not a standard deviation, because none is published.

Author-reported, by the vendor, over 217 ProteinGym assays (28 viral landscapes, 189 non-viral), scored as the logit difference between variant and wild-type allele at a masked position. The released model is the one that matters, and the released model was deliberately degraded:

• Non-viral, 189 assays: mean Spearman ρ = 0.45 for ESM3-open, against 0.46 for the unfiltered compute-matched model — "on par with ESM2".
• Viral, 28 assays: mean Spearman ρ = 0.17 for ESM3-open, down from 0.28 before data filtering. The vendor's own words: this falls "to the level of a BLOSUM prior, which incorporates no knowledge of the statistics of the protein family. This indicates that the model contains essentially no information about the evolutionary statistics of the evaluated viral protein families."
• ESM3-open "performs substantially worse than EVMutation" on viral fitness — and EVMutation is a Markov random field over an alignment, not a deep-learning model at all.

That 0.17 is a deliberate safety mitigation, not a bug, and it is a property of the open weights specifically. ESM3-open was also trained without OAS antibody sequences, and the vendor states it "removed the capability for the model to follow prompts related to viruses and toxins." Any viral or antibody task is outside what the released model retains. None of this reaches the platform today, because the released model is not loaded either — but it is what you would inherit the day the relabelling lands.

Supervised, independently measured. Fine-tuning changes the question and the answer. On a SARS-CoV-2 Spike–ACE2 binding DMS set of 4,003 variants split 15% held-out test, ESM-3 with LoRA adapters (rank 4) reached Pearson r = 0.876 between predicted and measured DMS score, well above a transformer trained from scratch. Read that as a statement about transfer learning from ESM-3 embeddings, which is the platform's documented feature-extraction use case — and note it required the full model plus fine-tuning on labelled assay data, not a zero-shot embedding comparison. The same work records that ESM-3 "is the most memory-demanding model in our benchmark and exceeds 50 GB in our training setup".

⊘ None of these figures describes the recipe in these docs. Every one scores a mutation by a likelihood ratio at a masked position, or by fitting a supervised head. outputs and quickstart instead instruct you to "compare embeddings from wild-type and mutant sequences to estimate variant effects" and offer cosine-similarity bands for it. Searches for a published concordance between embedding cosine distance and DMS fitness return nothing for any ESM model. That recipe is ungraded, and a single point mutation moves a pooled 480-dimensional mean-pooled vector very little, so the bands on the outputs page are not calibrated for it either.

Hayes T, Rao R, Akin H, Sofroniew NJ, Oktay D, Lin Z, Verkuil R, Tran VQ, Deaton J, Wiggert M, Badkundri R, Shafkat I, Gong J, Derry A, Molina RS, Thomas N, Khan YA, Mishra C, Kim C, Bartie LJ, Nemeth M, Hsu PD, Sercu T, Candido S, Rives A. Simulating 500 million years of evolution with a language model. Science. 2025;387(6736):850-858. doi: 10.1126/science.ads0018

Xu J, Shi Y, Lang L, Cui T, Zhang Z, Chen G, Qiu J, Heng PA. InstructPLM-mu: protein fitness prediction from structure instructions. Briefings in Bioinformatics. 2026;27(5):bbag400. doi: 10.1093/bib/bbag400

Notin P, Kollasch AW, Ritter D, van Niekerk L, Paul S, Spinner H, Rollins N, Shaw A, Orenbuch R, Weitzman R, Frazer J, Dias M, Franceschi D, Gal Y, Marks DS. ProteinGym: large-scale benchmarks for protein fitness prediction and design. bioRxiv. 2023. doi: 10.1101/2023.12.07.570727

Fei T, Li S, Yang Z, Zhou K, Li Y, Zeng T. Decoding viral protein sequences by large language models. Briefings in Bioinformatics. 2026;27(4):bbag377. doi: 10.1093/bib/bbag377

Structure, graded against experimental structure

This track is not exposed on this platform. A structure_prediction request is rewritten to an ESMFold job at the API boundary, with the stated reason "ESM3 in this deployment does not perform standalone structure prediction; routed to ESMFold." So the figures here are context for why that redirect is correct, not a description of a result you can hold.

The metric is LDDT against experimentally determined structures, on 902 CAMEO targets released May 2020 – Aug 2023 held out temporally from training, plus CASP14 (71 targets) and CASP15 (70 targets) held out from a PDB training cutoff of 1 May 2020. Author-reported, iterative decoding:

The main text's claim is that "ESM3 98B surpasses ESMFold (0.880 vs 0.861 mean local distance difference test, LDDT, CAMEO test set)". That is true of the 98B model, which is not released and not available here. The released 1.4B model scores 0.801, below ESMFold's 0.861 and 0.102 below AlphaFold2. Quoting the headline sentence next to an open-weights deployment inverts its meaning. The ESM3 structure track also degrades on exactly the material the open model was filtered against: average LDDT 0.56 on viral structures purged from the PDB training set, down from 0.60 for the unfiltered compute-matched model.

For single-sequence structure prediction on this platform, read ESMFold's translatability page, which carries the full accuracy, pLDDT-calibration and downstream-docking accounting and is not restated here. ESM-3's own paper uses ESMFold as its structural evaluator throughout the GFP work.

Independently measured — one protein, and it went badly. The only independent evaluation of ESM-3 structure prediction found is a single-protein diagnostic against the IsPETase crystal structure, comparing homology modelling (SWISS-MODEL), AlphaFold 3 and ESM-3. All three reproduced the overall fold and the catalytic triad geometry, but ESM-3 "showed elevated steric clashes and reduced hydrogen bond counts", fluctuated more in molecular dynamics particularly in loops, and — the load-bearing result — "blind docking simulations revealed that the ESM-3 active site was sterically occluded, rendering it inaccessible to the PET dimer. This inaccessibility persisted even after targeted energy minimization." A fold-correct model with an unusable active site is the specific failure mode a drug-discovery reader should fear, and n = 1.

Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, Smetanin N, Verkuil R, Kabeli O, Shmueli Y, dos Santos Costa A, Fazel-Zarandi M, Sercu T, Candido S, Rives A. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science. 2023;379(6637):1123-1130. doi: 10.1126/science.ade2574

Orçun Yener B, Göl Ş, Kutlu B, Öztürk Ö. Comparative benchmarking of template-based, evolutionary-diffusion, and generative language models for IsPETase structure prediction. Journal of Bioinformatics and Computational Biology. 2026;24(3):2651002. doi: 10.1142/S0219720026510029

Generation, graded against expression and a functional assay

This is the track where the only honest comparator is the bench, and the published record is small enough to count. All of it used real ESM-3. None of it describes what runs here.

The flagship result: esmGFP. ESM-3 was prompted to generate a green fluorescent protein, and one design was expressed and found to fluoresce. What was actually measured:

• The generating model was the base pretrained 7B ESM-3 — not the released 1.4B, and not anything on this platform.
• The prompt was not blank. It carried the sequence of 7 template residues (Met1, Thr62, Thr65, Tyr66, Gly67, Arg96, Glu222) plus structure tokens and backbone coordinates for 16 template residues (96, 222 and 58–71) taken from PDB 1QY3, a natural avGFP intermediate structure. The central alpha helix and the chromophore-forming site were handed to the model.
• The funnel: 38,000 generated backbones → 24,000 after a <1 Å chromophore-site RMSD filter → ~92,000 sequences → 320 ranked candidates → 88 synthesized on one 96-well plate. That round produced one distant hit, well B8, which was dim. A second 96-well plate of designs continued the chain of thought from B8, and its best member, well C10, is esmGFP.
• Two plates, 184 designs, one bright distant protein. That is the denominator.
• Measurement was fluorescence of E. coli lysate on a plate reader (485 nm excitation, 515 nm emission) plus flow cytometry, not purified-protein photophysics. esmGFP "takes longer to mature than the known GFPs that we measured, but achieves a comparable brightness after two days." Excitation peak 496 nm against EGFP's 489 nm; both emit at 512 nm. No quantum yield and no extinction coefficient are reported, so "brightness in the distribution of natural GFPs" is a lysate comparison.
• Causality was tested properly: T65G and Y66G chromophore knockouts reduced fluorescence to background, in a replicate run by a commercial vendor.
• Distance, with its convention. esmGFP is 58% identical to tagRFP (96 mutations), its nearest neighbour in both BLAST against nr and an MMseqs search of ESM-3's own training set. tagRFP is a designed variant; the closest wild-type sequence is eqFP578, a red fluorescent protein, at 53% identity (107 positions). Against Aequorea victoria GFP it is 36% identical. Identity is computed from a MAFFT alignment as matching non-gap residues divided by the minimum ungapped length.
• The "500 million years" figure is an extrapolation from six species. A line was fitted between sequence identity and millions of years to last common ancestor for six anthozoan GFPs, forced through the point (identity 1.0, 0 MY), and esmGFP's identity to its nearest neighbour was read off that line. It is a rhetorical framing, not a measurement, and it should never be quoted as one.

⚠︎ No independent group has expressed or re-assayed esmGFP. PubMed returns zero title/abstract hits for esmGFP other than the source, and a Europe PMC search returns 11 full-text hits — all reviews, surveys and commentary that cite it. The plasmids were offered; no replication has been published.

Two independent wet-lab validations of ESM-3 generation do exist, and both are conditional infills rather than free design.

Non-ribosomal peptide synthetase thiolation domains. ESM-3, ProteinMPNN and EvoDiff were used to regenerate a masked T-domain while the flanking A and C domains were held fixed as sequence and structure context, then tested in a live E. coli production assay against wild-type titre. In round 1, all four ESM-3-derived variants supported detectable product, and AI-2 exceeded wild type at ~115%; only 2 of 7 ProteinMPNN-derived variants were functional. Designs were held to ≥50% identity to wild type and spanned ~50–75%. Across three rounds of 11, 25 and 30 designs the fraction above a 50% activity threshold rose steadily. In the reconstituted full-length assembly line, all eight top designs produced detectable peptide and two beat wild type — AI-27 at 257% (7.4 mg L⁻¹) and AI-38 at 144%. AI-2 also expressed and stayed soluble where wild type was predominantly insoluble. Note the honest caveat the authors attach to their own readout: titres "depend strongly on cultivation conditions" and should be read "as condition-dependent functional labels rather than context-independent measures of sequence performance."

Poly(U) polymerase. ESM-3 with Gibbs sampling generated 500 sequences seeded on characterized functional variants, filtered to 495, screened in silico, clustered, and reduced to 10 candidates for the bench. All 10 expressed. Exactly 1 of 10 was functional. That one, PUPdel2, carries 16 mutations and delivered a 3.4 °C gain in thermal stability, a 3.7-fold increase in expression yield, and 5.4× / 3.5× / 1.4× / 1.2× incorporation efficiency for 3′-O-allyl-UTP / ATP / GTP / CTP over the benchmark. A 10% functional hit rate at 100% expression is the most useful single statistic on this page for anyone planning a design campaign.

And one that never reached the bench. An ESM-3-designed hemoglobin variant, "esmHb", is reported with a full stack of ΔG, MM-GBSA/PBSA and DFT numbers and no experimental measurement at all — its authors close by noting that "integrating ESM3 with experimental benchmarking could accelerate" the work. Treat titles of the form "engineering a variant using a protein language model" as a claim about a pipeline until you find the assay.

⊘ What runs on this platform is none of the above. esm3_generate performs ESM-2 iterative masked-language-model unmasking — one forward pass per residue, sampling from the 20 canonical amino acids — unconditional, with no structure or function conditioning. The function_conditioned mode, which is the mechanism esmGFP's chain of thought depends on, always raises, with the explicit reason that ESM-2 "has no function-conditioning capability at all" and that falling back "would report a fabricated result as SUCCESS for a mode that never ran." There is no published wet-lab validation of ESM-2 iterative masked-LM unconditional generation at any length, let alone at the 256-residue cap imposed here. Searches for one return nothing. That is an absent figure, not a low one.

Bülbül EF, Bang S, George K, Bianchi G, Raj P, Chung S, Pauline V, Hochstrasser R, Minas HA, Elgaher WAM, Kany AM, Hirsch AKH, Schmitt S, Heinz DW, Kalinina OV, Klakow D, Bozhüyük KAJ. Generative AI designs functional thiolation domains for reprogramming non-ribosomal peptide synthetases. Nature Communications. 2026;17(1):10084. doi: 10.1038/s41467-026-77963-6

Yang L, He Y, Cao F, Qin Y, Wang Y, Zhang H, Han W, Yang M. Computational evolution of poly(U) polymerase for efficient and controlled RNA oligonucleotide synthesis. Nucleic Acids Research. 2026;54(3):gkaf1529. doi: 10.1093/nar/gkaf1529

Shityakov S, Novikov AS, Skorb EV, Dandekar T. Engineering a hemoglobin variant with enhanced stability and oxygen affinity using a protein language model. Frontiers in Chemistry. 2026;14:1898272. doi: 10.3389/fchem.2026.1898272

Function annotation — no figure describes this platform, and the vendor's figure grades a label, not a measurement

The function track does not exist in the deployed compute. No GO term, EC number, InterPro accession or ESM-3 function token is produced anywhere; the loaded ESM-2 checkpoints have no function head, and ESM-2 has no function track to have one. A function_prediction request is rewritten to an InterPro search, which is HMM profile matching — so any accuracy number you could attach to that result describes InterPro, not a language model. The section is kept because the overview and FAQ both advertise "zero-shot function prediction" and GO-term inference, and a reader needs to know the claim is unbacked here.

The vendor's own figure deserves a second look even so, because it is the clearest case on these pages of the rule in property predictors are graded one step removed from the bench. ESM3-open is reported at mean Average Precision 0.81 across all function keywords and 0.89 for the top 1,000 most prevalent keywords, on "a validation set derived from UniRef and annotated with InterProScan."

Ask what that label encodes. It is not a curated GO or EC assignment and it is certainly not an assay. It is the output of a profile-matching program, run over the same UniRef and MGnify corpora the model trained on, then vectorized by TF-IDF and hashed into 8 tokens of 8 bits each. The model is being graded on recovering another program's annotation of its own training distribution. That is a provenance label in the sense the index page warns about — a category something assigned, not a quantity anyone measured — and the 0.81 is a statement about agreement with InterProScan, which you could obtain directly by running InterProScan.

⚠︎ Two further cautions on that figure. The macro/micro labels are internally inconsistent in the source: the main function-prediction figure caption assigns "micro" to the full keyword set and "macro" to the top-1,000 average, while the open-model section says the opposite ("0.81 (macro average)" for all keywords, "0.89 (micro average)" for the top 1,000). One of the two is wrong and the source does not say which. And the single independent benchmark found that evaluates ESM-3 on function-related tasks — a protocol chapter running ESM-2, ESM-3, ProstT5 and SaProt through semantic-similarity, ontology-based function prediction, drug-target-family and binding-affinity tasks — states no figures in its abstract and is not in PMC, so no independent function number is quotable at all.

Çevrim E, Yiğit MG, Ulusoy E, Yılmaz A, Doğan T. A benchmarking platform for assessing protein language models on function-related prediction tasks. Methods in Molecular Biology. 2025;2947:241-268. doi: 10.1007/978-1-0716-4662-5_14

What the figure doesn't cover

• Everything above is for a model that is not running. The gap is not a version mismatch of the kind the index page warns about; it is a different model family from a different lab. ESM-2 (Meta, 2022, sequence-only) and ESM-3 (EvolutionaryScale, 2024, multimodal) do not share weights, a tokenizer, a function track or a structure track. The relabelling is a known issue and the decision is pending.
• Training overlap is not stated in a usable form for the variant-effect figures. The structure benchmarks are clean — CAMEO targets from May 2020 onward, CASP14/15 held out against a 1 May 2020 PDB cutoff, 902 / 71 / 70 targets. The ProteinGym assays are not date-split: ESM-2 trained on UniRef50 and ESM-3 on UniRef, MGnify90, JGI and OAS at 50%/70%/90% clustering, and ProteinGym's proteins are natural proteins that sit in those corpora. Neither source reports identity clustering between the DMS proteins and the training set. The independent paper notes only that "additional generalization tests after removing potential training-test overlap are summarized in Supplementary Table S4" — the main table is not that test. Treat every ρ above as an upper bound for a protein with close homologs, and assume nothing about an orphan.
• ESM-3's published DMS mean is a single number over a hidden distribution. The vendor publishes two means (0.45 non-viral, 0.17 viral) and plots the rest; the independent paper publishes five category means and puts per-assay results in supplementary figures. No standard deviation, interquartile range or per-assay list is available for ESM-3 in either. A model at mean ρ = 0.47 includes assays near zero.
• The documented context window exceeds the model's trained context. Both loaded checkpoints declare max_position_embeddings: 1026 with rotary position embeddings, so ESM-2 was trained at a 1,024-token context. The platform validates against 2,048. Sequences between roughly 1,024 and 2,048 residues run outside the trained range with no warning and no published characterization. This is a known issue.
• The embedding dimension you receive is not documented anywhere. It is read off the tensor at runtime. For the default it is 480; for esm3_md/esm3_lg it is 1,280. The outputs table's 1,536 / 2,560 / 6,144 describe the real ESM-3 family, and the FAQ's warning not to mix embeddings across model sizes is right for the wrong reason: esm3_md and esm3_lg are the same checkpoint, so those two are freely comparable, while esm3_sm differs in both model and dimension.
• Cosine-similarity bands on the outputs page are not calibrated for any model. The ">0.9 = same family / 0.7–0.9 = superfamily" scale cites no source, and no study on these pages establishes it for ESM-2 35M, ESM-2 650M or ESM-3. It is a convention, and it moves with the model, the pooling strategy and the dimension.
• pooling: "cls" returns something meaningful, but not what the name implies. The served model does have a prepended special token at index 0, so the option works. Real ESM-3 has no CLS token — it uses BOS/EOS — so the parameter has no counterpart in the model the docs describe.
• No measured runtime exists for any ESM-3 surface. The runtimes quoted in these docs are estimates, not observations, and the platform's own internal figures do not agree with each other. The container itself sets no timeout, so the platform default silently governs. The ESM-2 generation path runs one forward pass per residue, so generation latency scales with length and the 256-residue cap is the real ceiling on it. Do not plan against any latency figure on these pages.
• Independent evidence is thin by track. One independent ProteinGym evaluation. One independent supervised-DMS and embedding-geometry study, with a suspect table row. One independent single-protein structure case study, abstract-only. Two independent wet-lab design campaigns, both conditional infills. Zero independent function figures. Zero replications of esmGFP. For a model released in 2024 and published in Science in 2025, that is a small base, and every vendor figure on this page remains unreproduced.
• The vendor is the author. EvolutionaryScale's technical report is the sole source for every ESM-3 figure labelled author-reported here, including all of Tables S8 and S9, the function-keyword mAP and every esmGFP measurement. The released weights also carry a non-commercial licence, which an independent group noted when building on them.

Known issues on this tool

⊘ The model served here is not ESM-3, and the variant selector does not change it as documented. Every surface labelled ESM-3 runs a smaller Meta ESM-2 protein language model instead, and the size options do not select between the sizes the documentation advertises. Treat every published ESM-3 figure on this page as an upper bound on what this endpoint gives you, not a description of it — the independent comparison above shows the gap directly: on 201 deep-mutational-scanning assays, real ESM-3 reaches Spearman ρ = 0.468 while the model class actually served here spans 0.333 to 0.428.

Three further consequences for reading your own result: the function track does not exist — no GO, EC or InterPro output is produced; the embedding dimensions you receive do not match the documented ones; and a number of accepted request parameters do not change the computation. Relabelling this tool is pending, and the published figures elsewhere in these docs will be corrected with it.

What to do in the meantime: use this endpoint for sequence embeddings and masked-position scoring, read the dimension off the array you receive rather than from the documentation, and do not compare its output against a published ESM-3 benchmark.

To the compute's credit, the two places where a fabricated result would have been easiest — inverse folding and function-conditioned generation — refuse instead of falling back. No path here reports success unconditionally; terminal status is branched per outcome inside real error handling.

Go / no-go

Go — act on the result when:

• You want a cheap, deterministic, alignment-free vector to feed a downstream model you will train and validate yourself, and you will grade that model against your own held-out measurements. The embedding is an input, not an answer, and its provenance being ESM-2 rather than ESM-3 changes the feature quality but not the workflow.
• You are clustering or triaging a large sequence set and a rough grouping is enough. On the one independent test, ESM-3 embeddings recovered viral lineage structure at ARI 0.875 — close to a small transformer's 0.856, so set your expectations by the easy baseline, not by the model's name.
• You pick esm3_md (or equivalently esm3_lg) rather than the default, which moves you from ESM-2 35M to ESM-2 650M and, on the independent DMS table, from ρ = 0.333 to 0.428.
• You are using the masked-fill path to propose a small number of substitutions at positions you chose, and you will test them. A 1-in-10 functional hit rate at full expression is the published rate for real ESM-3 design; expect no better from ESM-2 masked-LM sampling, for which no rate is published at all.

No-go — do not act on the result when:

• You need a variant effect. Use esm_scan_predict, which runs the masked-marginal method the ProteinGym figures actually grade, on the 650M checkpoint, at 2 credits. Do not use embedding cosine distance for this; no study grades it.
• You need a function annotation, a GO term or an EC number. Nothing here produces one. Run InterPro directly rather than through a redirect, so the provenance of the answer is visible in the result.
• You need a structure, a contact map or an attention map. Use ESMFold for structure and read its translatability page first. Contacts and attention are not produced here at all.
• You are designing a sequence you intend to express. The published successes used real ESM-3 at 7B or 1.4B with structure-conditioned prompts and a filtering funnel of tens of thousands of candidates; the flagship result was one bright protein out of 184 synthesized designs across two plates. What runs here is a smaller, sequence-only model doing iterative masked-LM sampling with the conditioning mode disabled, and it has no published wet-lab record.
• Your protein is viral, or an antibody. The released ESM-3 was filtered to remove viral and toxin prompting and trained without antibody repertoire data, dropping viral DMS performance to a BLOSUM baseline. That constraint will arrive with the relabelling. ESM-2 has its own limits here, and neither model has a published figure for this platform's configuration.
• Your sequence is longer than about 1,000 residues. Both loaded checkpoints were trained at a 1,024-token context while the platform accepts 2,048. Embed domains separately instead.
• You are comparing embeddings produced before and after a configuration change. Model identity, dimension and pooling all move the vector space, two of the three advertised sizes are secretly identical, and the result does not report which layer or which model produced it.
• You need the number established rather than proposed. That requires the scan, the assay, or the structure. No embedding, perplexity or confidence value on this platform substitutes for one.

Caveats on reading this page: every figure above is quoted from the cited source with its original metric and sample. Figures from different studies are not directly comparable, because test sets, assay inclusion criteria, scoring conventions and model variants differ between them. Figures labelled author-reported come from EvolutionaryScale's own technical report — a vendor document — and have not been independently reproduced on those sets. Figures labelled independent come from groups with no stake in the model. The platform facts were read in the deployed compute and the live pricing table, and should be re-derived rather than trusted, because both move.