---
title: How Humatch results compare to the bench
description: >-
  What a Humatch humanness score has been shown to agree with experimentally, and what
  happened when a Humatch-humanized antibody was finally expressed.
tags:
  - humatch
  - validation
  - translatability
---

**You have a Humatch result. Here is what it supports.**

- **Humatch publishes no correlation coefficient against clinical immunogenicity at all.** Its anti-drug antibody (ADA) analysis is a three-bin separation over 211 therapeutics, not a regression, and its own paper claims only that rival scorers "offer similar separation." So Humatch cannot be placed above or below the **R² = 0.34** ceiling that [BioPhi's page](/docs/biophi/translatability#humanness-against-clinical-immunogenicity) establishes for this class — there is no comparable number to place. The only published coefficient for the same statistic Humatch uses is its predecessor's: **R² = 0.31**.
- **Its headline accuracies are V-gene family assignments, not immunogenicity predictions.** Humatch reports per-gene classifiers at ROC AUC 1.000 for almost every class — separating human repertoire sequences from mouse and rhesus repertoire sequences. That is a species-and-germline label. The index explains the general form of this caveat in [property predictors are graded one step removed from the bench](/docs/translatability#property-predictors-are-graded-one-step-removed-from-the-bench); here the label is the sequence's species of origin.
- ★ **One Humatch-humanized antibody has been expressed and fully assayed, by an independent group, and the result is split.** Affinity fell from **KD 0.47 nM** (chimeric parent) to **20.6 nM** by BLI — roughly 44-fold — with a ~24-fold rise in competitive-ELISA IC50. Thermostability *improved*: the humanized variant showed an apo Tm of **80.4 °C** and none of the parent's low-temperature instability. The programme carried the chimeric variant into its in vivo work. This is more wet-lab evidence than exists for any [BioPhi](/docs/biophi/translatability) design, and it is still **n = 1**.
- ⊘ **The paired CNN — Humatch's one distinctive contribution — is not running on this platform.** The Modal app loads the heavy and light models only and scores **one chain per call**. Eight platform surfaces nonetheless label the output "Joint VH/VL humanness," and `vl_sequence` is silently dropped at the default `chain_type: "H"`. See [known issues](#what-the-figure-doesnt-cover).
- ⊘ **`humanness_score` is not "probability this is human."** It is the largest softmax probability over a *single* human V-gene family, with the non-human class zeroed out of the argmax and filtered out of the result. Heavy chains are scored over 7 classes and light chains over 17, so a VH and a VL number are not on the same scale.
- **The authors state the limit themselves:** "Currently, Humatch, and all other computational humanization tools lack experimental validation."
- **Confirm immunogenicity with** ADA assays in a human trial, de-risked beforehand with T-cell proliferation / PBMC assays, MHC class II binding and DC-T (EpiScreen-type) assays. [BioPhi's page](/docs/biophi/translatability#the-in-vitro-surrogates-and-why-they-are-not-a-better-answer-than-they-look) covers how weak that surrogate cascade is, and that analysis applies here unchanged. Confirm any humanized variant by expressing it and measuring binding, titre and thermal unfolding against the parent.

Jump to [agreement with wet lab](#agreement-with-wet-lab) or [against other in silico methods](#against-other-in-silico-methods). The general framing for every tool is in [reading an in silico result against the bench](/docs/translatability).

**This page is the sibling of [BioPhi's](/docs/biophi/translatability).** The two tools answer overlapping questions, so the clinical-endpoint analysis, the independent R² = 0.02 survey, the CDR-epitope mapping and the in-vitro surrogate figures are all stated there and are not repeated here. This page covers what is specific to Humatch, and says where it differs.

## How in silico is used currently

Humatch covers three jobs with three different evidential standings, and the platform exposes only the first.

- **Humanness scoring as a triage filter.** A score orders candidates and localises the problem — which framework positions look least like the closest human germline — so a campaign knows where to intervene. This is the use the figures support, and `humatch_score` is the only Humatch mode on this platform.
- **Germline-targeted humanization as a sequence-proposal step.** Humatch's own emphasis is humanization: iterative single-point mutation guided by its classifiers toward one chosen V gene, in about 33 seconds per antibody. Those sequences then enter the normal funnel — express, measure binding, measure thermostability, pick. **None of this is available here;** [BioPhi](/docs/biophi) is the humanization route on this platform.
- **VH/VL pairing assessment.** Humatch is the first humanization tool to score heavy and light chains *jointly*, on the argument that separate optimisation "risks disrupting expression, lowering stability, and creating immunogenic epitopes that span both chains." It is the paper's distinctive claim and the reason to prefer it over Hu-mAb or Sapiens. **It is also not running here** (see [known issues](#what-the-figure-doesnt-cover)).

Humatch is also used in practice as a pass/fail gate on *generated* sequences — a 2026 light-chain generative model used it to filter its outputs, where it called **99.73%** of generated light chains human. Read that as a ceiling effect, not a validation: a filter that passes everything has not been shown to discriminate. ⚠︎ That model is from the same Oxford group as Humatch, so it is not an independent use.

> Capel HL, Ellmen I, Murray CJ, Mignone G, Black M, Clarke B, Breen C, Tierney S, Dougan P, Buick RJ, Greenshields-Watson A, Deane CM. LICHEN enables light-chain immunoglobulin sequence generation conditioned on the heavy chain and experimental needs. *Communications Biology*. 2026;9(1):468. doi: [10.1038/s42003-026-09727-3](https://doi.org/10.1038/s42003-026-09727-3)

What no figure located for this page supports is a humanness score used as a *decision* instrument about a patient's immune response. Humatch publishes no coefficient for that relationship, and the independent reading of the benchmark it uses is that the benchmark measures species.

## Against other in silico methods

**On its own humanization benchmark, Humatch leads — and the comparison is author-run.** The Table 2 head-to-head above is the full result: Humatch at 0.77 / 0.82 expert-mutation overlap against Hu-mAb 0.67 / 0.77, Sapiens 0.57 / 0.69 and AbNatiV 0.57 / 0.65. Three qualifications belong with it.

- **Hu-mAb is the same laboratory's previous tool.** Both papers are from the Oxford group, with a shared senior author. A method beating its own lab's predecessor is a self-comparison, and this page labels it as one.
- **Hu-mAb's numbers were taken from Hu-mAb's paper; Sapiens and AbNatiV were run by Humatch's authors** with "default and paper-recommended parameters." Two of the three competitors were operated by the group they were being compared against.
- **The metric is not the one BioPhi reports.** Humatch's 0.57 for Sapiens is a mean per-chain exact-match overlap; BioPhi reports Sapiens at 77% mutation precision, pooled across chains and divided by mutations the method made. [The sibling page flags this](/docs/biophi/translatability#humanization-against-experimental-humanization) — 0.57 and 77% are not the same quantity, and the ranking flips depending on who defines the metric. Neither number is wrong. Quoting either without its definition is.

**On mixed-gene rejection, Humatch is genuinely differentiated.** Sequences that sit "between" V genes are not observed naturally and are argued to carry immunogenic risk. Constructing artificial mixed-gene designs, the authors report that Humatch ranks **51%** of them as non-human, against AbNatiV **28%**, Hu-mAb **8%** and OASis **1%** (stated in the body; the analysis is in the paper's SI Section 15). That is a large margin on a real design pathology, and it is the best argument for using Humatch alongside OASis rather than instead of it. ⚠︎ It is also graded against *constructed* negatives — the label is how the sequence was built, not an immunogenicity measurement — and 51% means Humatch misses half of its own artificial mixed-gene set.

**On the pairing task, the one independent benchmark puts plain germline information ahead of Humatch.** The 2025 germline-aware benchmark evaluated three published pairing models — p-IgGen, Humatch and ImmunoMatch — across random, V-gene and germline splits against a Topline and Bottomline reference. Its findings, quoted as directions because the numbers are in a figure and a supplementary table (see the review note above):

- "Humatch and ImmunoMatch ... perform consistently across splits, achieving strong AUC-ROC scores, particularly on v-gene, but **fall short of Topline performance in F1 Score and Accuracy, highlighting limitations in binary classification calibration.**"
- On the external PairedAbNGS dataset: "**germline-based models (V and VDJ) consistently achieved the highest accuracy, closely followed by Humatch.** These findings confirm the robustness of germline information as a key determinant of pairing compatibility."

**That is the germline-baseline result for Humatch's distinctive claim, and it reads the same way the germline-content baseline reads for OASis:** a plain germline model matches or beats the deep-learning method on the classification task, and the deep-learning method's advantage is elsewhere. Humatch's "elsewhere" is the thermostability correlation reported under [agreement with wet lab](#agreement-with-wet-lab), where it does beat the germline baseline — r = 0.25 against 0.16 — with both effects small.

A 2025 diffusion-based humanization model independently reproduced the *shape* of Humatch's ADA result — stratifying therapeutics by score and finding high-ADA antibodies concentrated in the low-score bin, noting that "the trends observed in our model outputs mirror those reported in previous investigations about ADA evaluation, including Hu-mAb and Humatch." It reports no Humatch figure, so it corroborates the direction and adds no number.

> Liu D, Hao X, Fan L. HuAbDiffusion: a discrete language diffusion model used for antibody humanization. *Briefings in Bioinformatics*. 2025;26(6):bbaf658. doi: [10.1093/bib/bbaf658](https://doi.org/10.1093/bib/bbaf658)

## In silico vs. wet lab

**There are two comparators, because Humatch makes two kinds of claim.**

For **humanness and immunogenicity**, the wet-lab standard is **ADA incidence measured in human trials** — a tiered screening, confirmatory and titre/neutralising cascade on patient samples, which comes with the trial rather than being purchasable per candidate. The earlier in-vitro surrogates are **T-cell proliferation / PBMC assays**, **MHC class II binding assays** and **DC-T (EpiScreen-type) assays**.

For the **pairing** claim, the comparators are unusually concrete and were actually measured: **Fab melting temperature by differential scanning fluorimetry** and **expression titre in HEK cells**, both available for 137 clinical-stage antibodies from Jain et al.

For **humanization**, the comparator is an **experimental humanization campaign** — graft, back-mutation scan, then expression and assay of each variant for retained affinity, titre and thermostability. The agreement metric there is not a score; it is whether the variant still binds and still folds.

**None of those appears in the shared cost basis**, which covers crystallography, SPR, hERG, Ames and microsomal stability. This page therefore quotes **no wet-lab cost figure at all** rather than deriving one. Read [what the wet-lab cost figures mean](/docs/translatability#what-the-wet-lab-cost-figures-mean) for why the shared table is the only sanctioned basis.

| | Time to answer | Cost basis | Accuracy, and its metric |
|---|---|---|---|
| **Standard wet lab, immunogenicity** — tiered ADA assays in a human trial | The duration of the trial; months to years | **Not in the shared cost basis, and no figure is derived here.** It is not a per-candidate purchasable service. [What this means](/docs/translatability#what-the-wet-lab-cost-figures-mean) | Ground truth, and assay-dependent — ADA incidence for the same molecule moves with assay sensitivity, drug tolerance and sampling schedule |
| **Standard wet lab, humanization** — graft plus back-mutation scan, expressed and assayed | Weeks to months per campaign | **Not in the shared cost basis, and no figure is derived here.** | Ground truth for that variant: measured KD, expression titre, Tm |
| **Standard wet lab, pairing and stability** — DSF melting temperature, HEK expression titre | Days to weeks per panel | **Not in the shared cost basis, and no figure is derived here.** | Ground truth: Tm in °C, titre in mg/L |
| **In-vitro immunogenicity surrogates** — T-cell proliferation / PBMC, MHC-II binding, DC-T | Weeks, per candidate, per donor panel | **Not in the shared cost basis, and no figure is derived here.** | Donor response frequency across a PBMC panel. Specific but roughly half-sensitive and format-dependent — [the figures are on BioPhi's page](/docs/biophi/translatability#the-in-vitro-surrogates-and-why-they-are-not-a-better-answer-than-they-look) |
| **Other in silico humanness scores** — Hu-mAb, AbNatiV, OASis ([BioPhi](/docs/biophi) is the second scorer here), T20, Z-score, AbLSTM, MG score, IgReconstruct, germline content | Seconds to minutes | BioPhi here: 1 credit | Against clinical ADA on 211–217 therapeutics, R² 0.16–0.34; Humatch's own paper reports these as offering "similar separation" to Humatch and quotes no coefficient for any of them |
| **Combined tooling here** — `humatch_score`, one chain per call, then [BioPhi](/docs/biophi) and [developability](/docs/antibody-developability) | 52 s padded in pricing; "2–3 minutes" in the tool definition; ~132 s recorded as real. See the runtime note below | **1 credit** per run | Per-V-gene classification: PR AUC 0.848–1.000, F1 0.587–1.000 across 24 human V-gene classes on OAS repertoire data. Against clinical ADA: **no coefficient published**. Against measured thermostability, independently: **Pearson r = 0.25** (n = 137). **No affinity figure, because no affinity is computed** |

**Read the accuracy column by cell, never down the column.** An ADA incidence from a trial, a measured KD, a Tm in °C, a donor response frequency, and a multiclass V-gene F1 are five different kinds of number. What the table supports is comparing *time and cost at a stated accuracy*.

On that reading, the honest position for this tool is: **seconds instead of weeks, for a germline-family assignment — and the assignment has never been regressed against the clinical question at all.** Humatch is a fast, well-characterised classifier of what species and which V gene a sequence resembles. That is a real and useful thing to know. It is not an immunogenicity forecast, and unlike BioPhi it does not even come with a weak one.

**The runtime this price is derived from is shorter than the runtime the tool actually needs.** The **1 credit** price assumes a runtime of about **52 seconds**. The tool definition states `estimatedDuration: "2-3 minutes"`, and a dated comment immediately above it records the measured figure as **"the real ~132s runtime"** after an earlier "10-30 seconds" estimate caused live jobs to be swept as stale. Two older catalog rows still carry that superseded "10-30 seconds" / "10-60s" text. **Where they conflict, trust the ~132 s comment**: it is the only one of the five written as a measurement rather than an estimate, and it is dated. The padded 52 s the price is derived from is less than half of it.

⚠︎ **And the Modal container timeout is 120 seconds**, shorter than the ~132 s the tool records as real. A job that genuinely takes the recorded runtime is killed by its own container. Treat this as a known issue.

**Two asymmetries matter.** A clinical ADA rate is not a property of the molecule alone — it moves with indication, dosing, co-medication and assay, which is why the same antibody is reported at different incidences in different trials. And a sequence can become more germline-like while losing affinity; the score cannot see that, because binding is not in it. The one published expression experiment on a Humatch design is exactly that case.

## Agreement with wet lab

This section is in four parts, because Humatch's outputs are graded against four different things and **none of the gradings is a bench measurement of immunogenicity**.

### What the classifier's labels actually are

Humatch is three convolutional networks. CNN-H and CNN-L are multiclass: each outputs a probability vector over one non-human class plus the human V-gene families (HV1–7 for heavy; LV1–10 and KV1–7 for light). CNN-P is binary over "true" and "fake" pairs. All three were trained on the Observed Antibody Space repertoire database, not on therapeutics.

| Classifier | Positive classes | Negative class | Training volume |
|---|---|---|---|
| CNN-H | 7 human heavy V-gene families | Non-human heavy repertoire sequences, including rhesus | 8.26 m human, 3.77 m non-human |
| CNN-L | 10 lambda + 7 kappa V-gene families | Non-human light repertoire sequences | 12.73 m human, 1.41 m non-human |
| CNN-P | Naturally paired human VH/VL from OAS | **Artificially "badly" paired** human chains, joined across B-cell classes and across OAS studies | 1.67 m true pairs, 5.01 m artificial pairs, no non-human sequences |

**This is a materially different label set from BioPhi's, and the difference runs in Humatch's favour on one axis and against it on another.** BioPhi's and Hu-mAb's classification task puts *therapeutic* antibodies that successfully completed clinical humanization into the negative class, so their AUC partly reproduces a manufacturing provenance label — [the sibling page states that caveat in full](/docs/biophi/translatability#what-the-figure-doesnt-cover). Humatch's unpaired negatives are not humanized therapeutics; they are genuine non-human repertoire sequences. Its classification is therefore a cleaner species question. But that is also the problem: **a cleaner species classifier is further from, not closer to, an immunogenicity prediction.** Nothing in the human-versus-mouse-and-rhesus label touches the clinical endpoint.

**CNN-P's negative class has the sharper label problem, and the authors say so.** The artificial bad pairs were built by joining chains from different B-cell classes and different studies, which, verbatim, "means that the chains are unlikely, but not guaranteed, to match poorly." The label is co-occurrence provenance, not a measured pairing failure. The authors add that "a lack of ground truth 'badly' paired sequence data for training CNN-P could result in both false positive and negative predictions," and the independent benchmark below makes the same point about its own evaluation — "we cannot test the data on the negative (experimentally validated unpaired) sequences." **No dataset of experimentally confirmed badly-paired antibodies exists.** Every pairing figure anywhere in this literature is graded against synthetic negatives.

**Author-reported classification performance, read in the primary full text (Table 1).** Two rows are worth carrying; the rest are 0.99+:

| Class | PR AUC | F1 | MCC | ROC AUC | Balanced accuracy |
|---|---|---|---|---|---|
| KV7 (the sparsest gene, 4,122 training sequences) | **0.848** | **0.587** | 0.642 | 1.000 | 0.993 |
| True pairs (CNN-P) | 0.995 | 0.966 | 0.954 | 0.998 | 0.983 |
| All 22 other V-gene classes | 0.992–1.000 | 0.897–1.000 | 0.900–1.000 | 1.000 | 0.990–1.000 |

**Read the PR AUC and the F1, not the ROC AUC.** ROC AUC is 1.000 for every V-gene class including KV7, whose F1 is 0.587 — a class the model identifies barely better than half the time by harmonic mean of precision and recall. Under the extreme class imbalance of a 24-way repertoire problem, ROC AUC saturates and stops carrying information. A Humatch figure quoted as "ROC AUC 1.000" is not wrong and is not useful. The authors are candid about the cause: KV7 and LV9 are data-sparse, and "gene classes with the lowest scores tend to have little training data available."

⚠︎ **Humatch was trained on complete variable domains and requires them.** Sequences lacking cysteines at IMGT 23 and 104, or missing the first or last position, were removed from training, and the authors state that "users should only input complete sequences for reliable results." The platform's only length gate is a 90–130 residue window and a dominant-residue check; it does not verify the conserved cysteines. A truncated or mis-trimmed domain can score, and the score means nothing.

> Chinery L, Jeliazkov JR, Deane CM. Humatch - fast, gene-specific joint humanisation of antibody heavy and light chains. *mAbs*. 2024;16(1):2434121. doi: [10.1080/19420862.2024.2434121](https://doi.org/10.1080/19420862.2024.2434121)

### Humanness against clinical immunogenicity

**Humatch reports no correlation coefficient, no R², and no regression against ADA.** What it reports is a binning. Its authors took the Marks et al. set of **217 therapeutics with recorded ADA levels**, found sequences for **211**, scored each with all three CNNs, and took **the minimum of the three scores** — "reasoning that if one of the CNNs ranked a therapeutic poorly, then it should be classed as higher risk." The result, verbatim:

> Using a loose Humatch cutoff of 0.01 to retain most low ADA therapeutics removed 89% of high ADA (red), 59% of medium ADA (yellow), and just 23% of low ADA (green) therapeutics.

And immediately after, the comparison: "Alternative humanness classifiers, including Hu-mAb, AbNatiV, and OASis, offer similar separation of low and high ADA antibodies." **Humatch's own paper therefore claims parity with the methods on the R² = 0.16–0.34 band, not an improvement on them**, and declines to publish the coefficient that would let you check.

**That matters because the minimum-of-scores construction has a published coefficient — from the same laboratory's previous tool.** Hu-mAb's paper, read in its primary full text, assessed "the correlation between the percentage of patients who develop ADAs and the minimum humanness score of a therapeutic's VH and VL chains, as the least human chain is expected to dictate the level of immunogenicity, across 217 therapeutics," and found: "the correlation was weak with an **R² of 0.31**." Humatch uses the identical statistic on the identical dataset, extended from two chains to three CNNs, and reports a bar chart instead. **A page that quoted "Humatch correlates with ADA" without a number would be reporting Hu-mAb's 0.31 by implication.** This page does not do that: the honest statement is that **Humatch's position against the 0.34 ceiling is unmeasured in its own publication**, and the two nearest anchors are its predecessor's 0.31 and BioPhi's recomputation of Hu-mAb at 0.34.

**An independent group argues the whole benchmark measures species.** Re-examining the same Marks et al. set, they concluded, verbatim: "Rather than measuring the ability of a model to predict immunogenicity, it **indirectly measures the ability of the model to distinguish between species**." Their test is a stratification: "if we consider either humanized antibodies, mouse antibodies, chimeric antibodies or human antibodies in this dataset, Hu-mAb scores show **no statistically significant correlation with ADA in any group**." And the mechanism: "a model that assigns low scores to mouse or chimeric antibodies will correlate with ADA even though it cannot predict whether a mouse or a humanized antibody is more or less likely than other mouse or humanized antibodies to be immunogenic." The same group reports that "SAM, BioPhi, AbNativ, and Hu-mAb achieve statistically equivalent results for correlation with ADA."

⚠︎ **That critique predates Humatch and does not name it.** It is a critique of the *dataset and the framing*, published in May 2024 against a benchmark Humatch adopted in November 2024 — the same 217-therapeutic set, the same minimum-of-scores logic, and a model whose explicit negative class is non-human species. It transfers by dataset identity, not by citation, and this page says so rather than claiming the authors tested Humatch.

**Humatch's own data makes the same point from the other direction.** Its classifiers rank some antibodies that human experts humanized and regulators approved as non-human, and the paper notes the predecessor's version of this: "Marks et al. find Hu-mAb ranks only ~66% of experimentally humanized VH/VL pairs as human" — a figure confirmed in Hu-mAb's primary text as "two-thirds of the humanized therapeutics had both VH and VL classified as human." A score that rejects a third of the molecules that passed the clinic is not reading clinical risk.

For the full picture of what *is* known about humanness against ADA — the independent 88-product survey at R² = 0.02, the non-monotonic per-class incidences, the fully human antibody at 76% ADA, and the within-molecule spread that exceeds the between-class spread — read [BioPhi's agreement section](/docs/biophi/translatability#humanness-against-clinical-immunogenicity). Every one of those findings constrains Humatch exactly as it constrains OASis, and none is restated here.

> Marks C, Hummer AM, Chin M, Deane CM. Humanization of antibodies using a machine learning approach on large-scale repertoire data. *Bioinformatics*. 2021;37(22):4041-4047. doi: [10.1093/bioinformatics/btab434](https://doi.org/10.1093/bioinformatics/btab434)
>
> Parkinson J, Wang W. For antibody sequence generative modeling, mixture models may be all you need. *Bioinformatics*. 2024;40(5):btae278. doi: [10.1093/bioinformatics/btae278](https://doi.org/10.1093/bioinformatics/btae278)

### The paired model against something measured

**This is the one place Humatch connects to a bench measurement, and the connection is thin in its own paper and small when measured independently.**

Humatch's argument for CNN-P is that joint optimisation protects expression and stability. The evidence offered for it is one group comparison, stated in the body and computed in the supplement: using **137 therapeutics with melting temperature data from Jain et al.**, "those with high CNN-P predictions (above 0.5) had higher melting temperatures than those with lower predictions (p < 0.001)." The discussion restates it as "high CNN-P predictions are shown to correlate with higher melting temperatures."

**Three things about that sentence.** It gives a p-value and no effect size, so the body reports no correlation coefficient and no ΔTm. It splits at **0.5**, while the threshold Humatch recommends and uses for humanization is **0.95** — the validated split point is not the operating point. And the authors note they "could only find stability data for 13 of the 20 experimental endpoints predicted to pair well and none for the five predicted to pair less well," so the subgroup that would falsify the claim had no data at all.

**An independent group has since measured the effect size, on the same 137 antibodies.** A 2025 germline-aware VH–VL pairing benchmark computed Pearson's r and Spearman's ρ between each model's pairing score and the Jain et al. measurements, for HEK expression titre and Fab melting temperature. Its Table 3, read in the primary text, with bold marking p < 0.05:

| Model | Expression, Pearson | Expression, Spearman | Thermostability, Pearson | Thermostability, Spearman |
|---|---|---|---|---|
| **Humatch** | −0.15 | 0.01 | **0.25** | **0.24** |
| p-IgGen | **0.17** | 0.15 | 0.02 | 0.04 |
| ImmunoMatch | −0.11 | −0.12 | −0.03 | −0.02 |
| Their own model, VDJ-germline negatives | 0.14 | 0.11 | **0.16** | **0.18** |
| Their own model, V-germline negatives | −0.11 | −0.14 | −0.15 | 0.06 |
| Their own model, random negatives | −0.06 | −0.13 | 0.06 | 0.04 |

**Humatch has the strongest thermostability association of any model tested, and it is r = 0.25 — about 6% of the variance.** The authors' verdict: "The observed correlations with developability properties are modest. Humatch shows the strongest association with thermostability, while p-IgGen displays the clearest trend with expression. However, the effect sizes remain small, indicating that current AI models struggle to capture developability signals from VH–VL pairing alone." They flag the n = 137 limit themselves.

**On expression — the other property the paired model is argued to protect — Humatch's correlation is −0.15 Pearson and 0.01 Spearman, neither significant.** The direction is wrong and the magnitude is nil. Nothing published supports reading a pairing score as an expression forecast.

> Jain T, Sun T, Durand S, Hall A, Houston NR, Nett JH, et al. Biophysical properties of the clinical-stage antibody landscape. *PNAS*. 2017;114(5):944-949. doi: [10.1073/pnas.1616408114](https://doi.org/10.1073/pnas.1616408114)
>
> Joubbi S, D'Arco E, Maccari G, Milazzo P, Micheli A. Germline-aware deep learning models and benchmarks for predicting antibody VH-VL pairing. *mAbs*. 2025;17(1):2570749. doi: [10.1080/19420862.2025.2570749](https://doi.org/10.1080/19420862.2025.2570749)

### Humanization against experimental humanization

**No variant in the Humatch paper was expressed or assayed, and the authors say so without hedging:**

> Currently, Humatch, and all other computational humanization tools lack experimental validation. Unfortunately, only the experimental end-points are known to be sufficiently safe to administer to patients. Our designs, and those from Hu-mAb, AbNatiV, and Sapiens are only predicted to be safe. However, validating their true immunogenic risks is complex and expensive. Instead, future efforts may first focus on testing all designs' affinity and melting temperatures to ensure that suggested mutations do not disrupt binding or affect stability.

The benchmark is the same **25 precursor / experimentally-humanized pairs** that BioPhi used, curated by the Hu-mAb authors. The grading is **mutation overlap**: what fraction of the mutations the method proposed were also made by the human expert. Author-reported, read in the primary full text (Table 2), where overlap of 1.00 would mean every proposed mutation was also made experimentally:

| Method | H overlap | L overlap | H edits | L edits | Time (s) |
|---|---|---|---|---|---|
| **Humatch** | **0.77** | **0.82** | 20.6 | 13.0 | 33 |
| Hu-mAb | 0.67 | 0.77 | 14.8 | 10.6 | 1,093 |
| AbNatiV | 0.57 | 0.65 | 11.5 | 9.3 | 218 |
| Sapiens (BioPhi) | 0.57 | 0.69 | 21.3 | 17.8 | 3 |
| Experimental (the human experts) | 1.00 | 1.00 | 26.0 | 19.2 | — |

**This is a real result about proposal quality and nothing more.** Humatch's overlap is the highest of the four while making the second-largest number of mutations, which the authors reasonably read as evidence that it "does not simply find the 'easy' mutations." The number of mutations it proposes tracks the number the experts made at Pearson 0.89. On a throughput test over 1,000 non-human paired OAS sequences, 12 pairs failed to reach target within the 60-edit cap and the mean was 36 s per pair on 16 CPUs.

⚠︎ **Humatch mutates Vernier and interface positions by design, at the same rate the experts did.** Its mutation profile puts **19.3% of heavy and 12.6% of light** mutations inside Vernier and interface zones, against **17.9% and 14.6%** experimentally. The authors offer this as a match, and as a profile match it is one. But a *proportion* match is not a *position* match, and Vernier positions are the framework residues that support the CDR loops and most often cost affinity. [BioPhi's page documents two independent lines of evidence](/docs/biophi/translatability#expressed-and-assayed-two-antibodies-three-variants-both-negative) that this is where framework humanization loses binding — a Vernier mutation at VH F27/N28 costing an anti-CD99 scFv its affinity, and Vernier-zone agreement with experts of only 40–49% on these same 25 antibodies. The one Humatch design that has been expressed lost 44-fold affinity, and its authors attributed that to framework residues adjacent to the CDRs. **Three independent lines now point at the same residues.** Humatch lets users fix Vernier and interface positions; it does not do so by default.

**The one germline-only baseline the paper runs is graded on Humatch's own scores.** The authors ablated the CNN guidance, making only the top germline mutations with the same edit counts, and report that "no baseline-designed paired sequences matched Humatch's designs exactly and only two designs passed all three CNN target thresholds of 0.95." **That grades the germline baseline against Humatch's own classifiers, not against the experimental-overlap metric on which Humatch's headline 0.77 / 0.82 is reported.** It is the same self-grading trap BioPhi's authors identified when Hu-mAb-optimized sequences were scored with Hu-mAb. Since Humatch's humanization *begins* with a germline-frequency lookup — mutating to a target germline-likeness of 0.40 before any CNN is consulted, and the paper notes "the majority of these mutations were to common germline residues" — **how much of the 0.77 / 0.82 overlap a plain germline lookup would have achieved on its own is not reported.** That is precisely the question the germline-content baseline answered for OASis, where it was not beaten.

### Expressed and assayed: one antibody, one variant, split result

**This is the comparison the bench actually produces, and for Humatch it exists — once.** An independent group with no stake in the method humanized a murine anti-xylazine antibody with Humatch, expressed it in Expi293 cells, purified it, and characterised it by BLI, competitive ELISA, DSF and a cross-reactivity panel.

**Affinity fell substantially.** BLI, read in the primary text: KD **0.32 nM** for the murine parent, **0.47 nM** for the chimeric, and **20.6 nM** for the Humatch-humanized variant — roughly 44-fold worse than the chimeric and 64-fold worse than the murine parent, computed from the paper's own dissociation constants. Competitive ELISA agreed: "humanization resulted in approximately a **24-fold increase** in the IC50 reflecting the reduction in affinity observed by BLI." The programme carried the **chimeric** variant into its in vivo efficacy work.

**Thermostability improved, clearly.** The murine and chimeric variants "displayed high initial fluorescence in the absence of ligand, which is indicative of exposed hydrophobic regions in the apo state" and no resolvable apo Tm. The humanized variant "demonstrated no instability at low temperatures and a relatively high intrinsic Tm of **80.4 °C** in the apo state, indicative of stabilization of the antibody scaffold upon humanization," rising to 87.2 °C with ligand. The authors' summary: "despite the loss in affinity, the humanized construct displayed high biophysical stability, indicating a stable and well-folded scaffold suitable for manufacturability."

**Specificity changed in both directions.** The humanized variant lost the chimeric's cross-reactivity with tizanidine and naphazoline — an improvement — but gained measurable binding to free fentanyl, which the chimeric did not have. For an antibody intended for polysubstance overdose, a new opioid cross-reactivity is not a neutral change.

★ **The authors' mechanism is the same one BioPhi's failures point at.** Verbatim: "Structural analysis revealed that xylazine binds in a deep aromatic and hydrophobic pocket, which also involves **framework regions in close proximity to the Complementarity Determining Regions** (CDRs). We hypothesize that minor structural changes to the framework regions altered the binding pocket, leading to the observed decrease in affinity. This is well-documented in antibody engineering, where CDR grafting often **disrupts Vernier zone residues**." A 2.0 Å crystal structure of the parent in complex with xylazine shows the binding pocket forming at the VH/VL interface, which is the thing Humatch's paired model is meant to protect — and which was not running in the version the platform exposes.

**Read the limits honestly.** n = 1 antibody and 1 Humatch variant. It is a small-molecule hapten binder with an unusually buried, framework-adjacent pocket, which is close to the worst case for framework humanization and not representative of a protein-antigen antibody. No expression titre was reported for the humanized variant, no immunogenicity measurement of any kind was made on it, and the humanized variant was not the one taken in vivo. **The direction on affinity is unfavourable, the direction on stability is favourable, and one antibody settles neither.** What it does settle is that the comparison is no longer absent — which is one antibody more than [BioPhi could say until recently](/docs/biophi/translatability#expressed-and-assayed-two-antibodies-three-variants-both-negative), and which went the same way on affinity.

> Kang J, Xu R, Lee S, Luengas D, Kim CM, Marecki C, Gallant J, Moeller N, Shi K, Jahan R, Hicks D, Runyon S, Aihara H, Pravetoni M. Discovery, Structural Characterization, and Preclinical Evaluation of Monoclonal Antibodies against Xylazine Poisoning. *ACS Pharmacology & Translational Science*. 2026;9(7):1881-1892. doi: [10.1021/acsptsci.6c00117](https://doi.org/10.1021/acsptsci.6c00117)

## What the figure doesn't cover

- **No published figure relates a Humatch score to clinical immunogenicity by any coefficient.** Not R², not Pearson, not AUC against an ADA label. The only published statistic is a three-bin separation, and the paper's own comparison claim is parity with methods whose ceiling is R² = 0.34. Treat a low score as a reason to look harder and a high score as no assurance at all.
- **The classification label is species and V-gene family, not outcome.** The negatives are mouse and rhesus repertoire sequences. A model that perfectly reproduced that label would tell you what animal a sequence came from. This is the index's [one-step-removed](/docs/translatability#property-predictors-are-graded-one-step-removed-from-the-bench) caveat at its sharpest: there is no assay anywhere in the label.
- **ROC AUC 1.000 is a saturation artefact, not a result.** Read PR AUC and F1. KV7 sits at F1 0.587 with ROC AUC 1.000.
- **No independent evaluation of Humatch's humanness classification exists.** A full-text search of PMC for "Humatch" returns eight articles, which is the whole census this page could reach; of those, one is the paper itself, one independently benchmarks the *pairing* model, one expressed a humanized design, three use the score as a filter or a citation, and none re-grades the humanness classifier on an external set. PMC does not index every journal, so read that as a floor.
- ⊘ **The paired CNN is not loaded on this platform — known issue.** The Modal app loads `heavy.weights.h5` and `light.weights.h5` only and selects exactly one per job by `chain_type`. There is no paired code path. Humatch's own position is that it "as standard uses the outputs of all three CNNs for paired VH/VL humanness classification," so **the integration runs a configuration the paper treats as the non-default**, and the distinctive claim — the only one with a link to a measured property — is absent. The fix is to load the paired model and add a joint mode.
- ⊘ **Eight platform surfaces describe the output as "Joint VH/VL" — known issue.** The result card's description and primary-metric tooltip, the pipeline recipe line, and the catalog rows all say joint, and one catalog description goes further: "Humatch evaluates antibody humanness by jointly scoring paired VH/VL chains ... providing a more accurate assessment than single-chain methods." Every clause of that is false for what runs. The tool's own `description` is the one surface that gets it right — "scores one chain at a time." The fix is to correct the description in all eight places, including the generated mirrors; until then, read the tool description, not the card.
- ⊘ **`vl_sequence` is accepted and silently dropped at the default `chain_type` — known issue.** Submit both chains without setting `chain_type: "L"` and only the heavy chain is scored, with no warning and status `SUCCESS`. To score a light chain you must make a second call with `chain_type: "L"`. The fix is to reject or warn on an unused chain.
- ⊘ **`humanness_score` is max P(one human V-gene family), not P(human) — known issue.** The backend takes the largest softmax probability after zeroing the non-human class out of the argmax, and then **filters the non-human probability out of `v_gene_scores` entirely**, so no consumer can recover it. Two consequences. A sequence whose probability mass is spread across several human families scores low even when its non-human probability is near zero. And heavy chains are scored over 7 classes while light chains are scored over 17, so the light-chain scale is systematically compressed and a VH and a VL number are not comparable — while the same fixed cutoffs are applied to both. The fix is to emit the non-human probability and to document the per-chain denominator.
- ⊘ **The `is_human` cutoff is a fixed 0.5, and the card's colours disagree with it — known issue.** `score_threshold` defaults to 0.5 in the Modal schema and is not exposed in the tool schema, so it can never be changed from a prompt. No in-repo provenance ties 0.5 to any published Humatch distribution, and it is far below the **0.95** Humatch recommends for all three classifiers "to ensure high precision and recall." Meanwhile the result card paints the score `success` at ≥ 80%, `warning` at ≥ 60% and `error` below — so a sequence at 0.55 is `is_human: true` in the result and red on the card. The 0.50–0.60 band is a straight pass/fail inversion between two layers of the same platform, and neither cutoff is a published boundary. Read the number, not the colour, and not the label.
- ⊘ **`scheme` is accepted and explicitly never applied — known issue.** The Modal schema accepts `imgt`, `chothia` and `kabat`; the code states that "numbering is always IMGT" and records the request as `requested_scheme`, which is "recorded, not applied." `scheme_used` is therefore always `imgt`. Humatch is IMGT-only by construction — it pads to 200 canonical IMGT positions. The parameter is also not exposed in the tool schema at all, yet [the inputs page](/docs/humatch/inputs) documents it as a user-settable `Param`. The fix is to remove it from the schema and correct the inputs page.
- ⊘ **Three further documented parameters are unreachable — known issue.** `score_threshold`, the `sequences` batch (documented as "One to 50 per job") and `sequence_names` all exist on the Modal side and none is exposed by `humatch_score`, which always submits exactly one sequence. [The inputs page](/docs/humatch/inputs) presents all three as user inputs. Treat that page's parameter list as describing the Modal app, not the tool.
- ⊘ **`positional_scores` is not per-position — known issue.** [The outputs page](/docs/humatch/outputs) describes it as "per-position humanness, on the numbering scheme used." The backend writes at most one key into it, `germline_likeness`, and marks the location "legacy." No per-position score is ever emitted. What *is* measured is `germline_likeness` — the mean, over the aligned sequence, of how often each observed residue appears at that position in the predicted V gene's germline frequency table — and `flagged_positions`, which are framework IMGT positions (CDRs excluded) whose residue differs from the most-observed germline residue, ranked by germline-frequency gain. Those are germline-frequency statistics, not humanness scores, and they are the honest fields to read.
- ⊘ **Four of the result card's metrics are dead by construction — known issue.** The "VH", "VL", "Germline" and "Deviations" metrics read thirteen field names that no Humatch code path emits, so they never render; and `top_human_germline`, which *is* emitted, is not read by the "Germline" branch — it appears only inside a tooltip. The live metrics are the primary humanness score, germline likeness, flagged framework positions and the failure count. [The outputs page's](/docs/humatch/outputs) promise of a "per-chain read" on the card is not met. The fix is to point the extractors at the emitted field names.
- **Five runtime figures, one of them shorter than the container timeout — known issue.** 52 s padded in pricing, "2–3 minutes" in the tool definition, "~132s" in the tool's dated measurement comment, and "10-30 seconds" / "10-60s" in two older catalog rows, against a hard Modal `timeout=120`. A job at the recorded real runtime exceeds its own timeout. The price derives from the shortest figure.
- **Several fields a result can carry are not documented anywhere.** `germline_likeness`, `germline_likeness_error`, `flagged_position_details`, `flagged_positions_error`, `partial`, `requested_scheme`, `humatch_git_rev`, `germline_likeness_method` and `flagged_positions_method` are all emitted and none appears on [the outputs page](/docs/humatch/outputs). `humatch_git_rev` is the useful one: it records the exact upstream commit, which is the version a figure needs to be reproducible.
- **A partial job returns `SUCCESS`.** When germline likeness cannot be computed, the row carries `germline_likeness: null` with an error string and `partial: true`, while `humanness_score` is still reported. The platform is honest about it — and a card can therefore show a humanness number with no germline measurement behind it. Where a sequence fails outright the job fails rather than returning an empty success, which is the right behaviour and was a deliberate fix.
- **No humanization, and no humanness improvement measurement.** The platform scores; it does not rewrite. Every humanization figure on this page — the 0.77 / 0.82 overlap, the edit distances, the 33-second runtime — describes a capability you cannot reach from here. [BioPhi](/docs/biophi/translatability) is the humanization route, and its own expression evidence is unfavourable.
- **The version is pinned, which is unusual and good.** The Modal image installs Humatch from a fixed upstream commit, and an image-build assertion fails the build unless a reference sequence reproduces a known germline-likeness value. A figure quoted with `humatch_git_rev` is reproducible. That is the opposite of BioPhi's unpinned git HEAD.
- **Nothing here measures binding, expression or developability.** Humanness and germline likeness are blind to all three. The one expressed Humatch design lost 44-fold affinity while gaining thermostability. Run [antibody developability](/docs/antibody-developability) on any candidate, and read [its translatability page](/docs/antibody-developability/translatability) first, because the platform's descriptors there are hand-written rules.

## Go / no-go

**Go — act on the result when:**

- You are ordering or triaging a candidate list by resemblance to the human germline repertoire, and you want the relative rank. That is what the classification figures support, and the classifier is well characterised on that task.
- You want to know **which** human V gene a sequence is closest to, and **where** it departs from that gene's germline consensus. `top_human_germline`, `germline_likeness` and `flagged_positions` are real measurements of exactly that, computed on the IMGT-aligned sequence.
- You are cross-checking a sequence that another tool humanized. A score cannot grade a sequence it generated, and Humatch did not generate yours — it is the independent metric against a [BioPhi](/docs/biophi) output, and that cross-check is the one lesson in this literature with a measured demonstration behind it.
- You are screening for mixed-gene designs. Rejecting 51% of artificial between-gene sequences against OASis's 1% is the largest margin Humatch has over any competitor.
- You quote the figure with its chain type, its denominator and `humatch_git_rev`, and you read `humanness_score` rather than `is_human` or the card's colour.

**No-go — do not act on the result when:**

- You are predicting whether a candidate will provoke ADA in patients, or comparing two candidates' clinical immunogenicity risk. **No published coefficient supports either for Humatch**, and the benchmark its paper uses is independently argued to measure species.
- You are treating a high score as clearance, or a low score as a liability. The score drops for long or heavily mutated CDR3s and for sequences between germlines — and it rejected a third of the humanized antibodies that reached the clinic, in its predecessor's measurement.
- You are comparing a VH number to a VL number. They are maxima over 7 and 17 classes respectively.
- You are reading the output as a joint VH/VL assessment. The paired model is not loaded, and a result from a single call describes one chain.
- You passed `vl_sequence` without setting `chain_type: "L"`. Only the heavy chain was scored.
- You are relying on `scheme`, `score_threshold`, `sequence_names` or a multi-sequence batch to have shaped the result. None of them reaches the compute path.
- You are reading `positional_scores` as per-position humanness, or the card's absent "VH" / "VL" / "Germline" / "Deviations" metrics as missing data rather than as metrics that cannot render.
- You intend to ship a humanized sequence — from any tool — without expressing it and measuring binding, titre and thermostability. The one Humatch design ever expressed lost roughly 44-fold affinity, and its authors traced it to framework residues next to the CDRs.
- You need an immunogenicity risk assessment for a regulatory filing. That requires the assay cascade — MHC-II and T-cell assays, then clinical ADA monitoring — and no sequence score substitutes for it.

**Caveats on reading this page:** every figure above is quoted from the cited source with its original metric, threshold and sample, and every **quantitative** figure was read in the primary full body text rather than an abstract. Author-reported, self-compared and independent figures are labelled as such throughout: the classification and humanization tables are author-reported, the Hu-mAb comparison is a same-laboratory self-comparison, and the thermostability correlation, the expression correlation, the pairing benchmark and the expressed-and-assayed result are independent. Figures from different rows are not comparable, because label sets, thresholds, metrics and denominators differ between them. Where a comparison was not located — notably **any** coefficient relating a Humatch score to clinical ADA, any independent re-grading of its humanness classifier, any expression titre for a Humatch design, and any immunogenicity measurement of any kind on one — this page says so rather than substituting a figure from a different method, a different tool or a different version.