How DeepPurpose results compare to the bench

You have a DeepPurpose result. Here is what it supports.

• It is a sequence-only guess, with no structure anywhere in it. The job encodes your SMILES and your amino-acid sequence and runs them through one pretrained checkpoint. No pose, no pocket, no structure is used or produced. Compare MM-GBSA, which attacks the same affinity question from a structure.
• Know which checkpoint ran, because the number is meaningless without it. With no pretrained_model set, this platform loads MPNN_CNN_DAVIS — an MPNN drug encoder and a CNN target encoder, trained on the Davis kinase panel, emitting pKd. A KIBA checkpoint instead emits a raw KIBA score, which is not pKd and not a log scale, in the same field.
• ⊘ A model containing no chemistry at all scores the same. In the only independent re-evaluation of DeepPurpose, a configuration model that sees nothing but how many annotations each node has — no SMILES, no sequence — matched or beat it: AUROC 0.83 vs 0.82 on unseen edges and 0.77 vs 0.76 on unseen targets. Randomly shuffling the SMILES and sequences in training left test performance essentially unchanged (AUROC 0.86 → 0.84). See agreement with wet lab.
• ⊘ The split is the whole story. On Davis, the same method's concordance index runs 88.3 on a random pair split and 67.0 when both the drug and the target are new (50 is random). Independently retrained, DeepDTA's Pearson r on Davis falls from 0.841 to 0.477 once similar proteins are kept out of training. Any figure quoted without its split is unusable.
• ⚠︎ Read the error, not the ranking metric. Concordance index and ROC-AUC flatter this task badly, and this page documents three cases where a ranking metric and a magnitude metric give opposite verdicts on the same model and split.
• Confirm with a binding assay — SPR, ITC, MST, a radioligand displacement, or an enzymatic IC50/Ki determination. Nothing here substitutes for one.
• There is no kinome panel. One drug–target pair per call. The "kinome-wide off-target read" in the tool copy is a loop of single calls, not a mode.

Jump to agreement with wet lab or against other in silico methods. The general framing for every tool is in reading an in silico result against the bench.

How in silico is used currently

Sequence-based affinity prediction is used to order a list of things you were going to test anyway, not to decide whether a compound binds. In a live program that means three things:

• Panel triage. You have a compound and a list of candidate off-targets, and you want an order in which to run the assay. This is the use the platform wires DeepPurpose into, and the one its benchmarks come closest to supporting — on the explicit condition that the ordering is checked, not trusted.
• Repurposing shortlists. You have an approved-drug library and a target with no known ligand, and you want fifty names instead of six thousand. The published validation for this use is almost entirely retrospective: the model recovers interactions that are already in a database.
• Filling in a sparse matrix. Large kinase panels are incomplete, and a model trained on the measured cells is used to guess the missing ones. This is the task the benchmark datasets were built from, and it is the only one where the random-split figures are the right prior — because the question really is "predict a held-out cell for a drug and a target you have both seen before."

What sequence-based DTI is not used for, by anyone reporting their validation honestly, is deciding that a novel compound binds a novel target. That is exactly the regime where the published numbers fall to chance.

Against other in silico methods

DeepPurpose's own paper publishes no number. Its body text reports that "many DeepPurpose models achieve comparable prediction performance on two benchmark datasets, DAVIS and KIBA" and that they "achieve state-of-the-art prediction performance on DTI benchmark datasets," with the values living only in a figure panel and the supplement. The split is not stated anywhere in the text, there is no cold-start evaluation, and the pretrained model's output units are not given. The library offers eight compound encoders and seven protein encoders in "over 50 DL models," and compares itself against KronRLS, DeepDTA and GraphDTA. Everything quantitative on this page about DeepPurpose therefore comes from other groups.

On a random pair split, the deep models barely beat the kernels. From DeepDTA's own table, Davis and KIBA, random 1-in-6 split:

This is the third metric pair that inverts. On Davis, CI ranks DeepDTA first by 0.006 over SimBoost — while r²ₘ ranks SimBoost first, 0.644 against 0.630. And CI puts the no-deep-learning kernel within 0.007 of the deep model while r²ₘ puts it 0.223 behind. Whichever direction you want, a concordance index will give it to you.

On classification, logistic regression is close too. MolTrans's author-reported table over three datasets, random pair split:

On the imbalanced Davis set, ROC-AUC says a logistic regression gets 92% of the way; PR-AUC says it gets 57%. Same model, same split, two metrics.

Under cold splits, the ordering between the deep methods stops being measurable. MolTrans's own BIOSNAP cold-split table, ROC-AUC, mean of five runs:

The gaps between methods in the cold rows are 0.003 to 0.006 AUROC, against standard deviations of 0.007 to 0.029. The authors claim only "competitive performance," which is the correct claim: the differences are inside the noise.

MolTrans also has an independent inductive number, and it is the harshest on this page. The AI-Bind group measured it at transductive AUROC 0.952 ± 0.041 / AUPRC 0.887 ± 0.087 on BindingDB, falling to 0.572 ± 0.104 / 0.432 ± 0.105 inductive — a transductive figure better than anything else here, and an inductive figure within noise of a coin flip. On their own network-derived negatives, inductive: AI-Bind's VecNet 0.75 ± 0.032 AUROC / 0.718 ± 0.029 AUPRC, DeepPurpose 0.646 ± 0.023 / 0.576 ± 0.009, MolTrans 0.612 ± 0.028 / 0.478 ± 0.034.

GraphDTA has no citable author-reported figure. It has no PMC deposit, and its PubMed abstract contains no numbers, no metrics and no dataset names — only the claim that graph neural networks "not only predict drug-target affinity better than non-deep learning models, but also outperform competing deep learning methods." The only GraphDTA numbers on this page are the independent retrain in the clustered-split table above, where it is the best model on the random split and the worst on the clustered one.

For contrast, the structure-based lane behaves the same way and has been measured more sharply. On CASF2016, a leakage-filtered training set moved a retrained Pafnucy from RMSE 1.046 to 1.484 — "approaching the level of the simple search algorithms." Those simple algorithms are worth knowing: a nearest-neighbour lookup over the five most similar training complexes, with no learning at all, reaches Pearson 0.716 / RMSE 1.517, and over the five most similar ligands only, ignoring the protein entirely, Pearson 0.707 / RMSE 1.539. A graph model with all protein information removed reached RMSE 1.424 against 1.308 with it, and those ligand-only models "make more accurate predictions than the scoring function of AutoDock Vina and even outperform several published deep-learning-based scoring functions." That is a different task from this one — it needs a structure, and MM-GBSA is the platform's route into it — but the pattern is identical.

Cold-drug and cold-target do not rank consistently, and that is itself worth knowing. On Davis the cold-target setting costs little (CI 88.3 → 86.1) because the 442 kinases resemble each other, while cold-drug costs a lot (→ 74.8). On 119 GPCRs the ordering reverses: cold-ligand costs "approximately 0.10 AUROC units" while cold-target costs "0.19 to 0.32 units," taking AUROC from 0.88–0.91 down to 0.584–0.637 and Matthews correlation from 0.607 to 0.128. The common ground is cold-pair, which is worst in every study here. Note too that in the GPCR evaluation plain XGBoost on concatenated embeddings was the best model on unseen proteins (0.637), beating every cross-attention variant.

Öztürk H, Özgür A, Ozkirimli E. DeepDTA: deep drug-target binding affinity prediction. Bioinformatics. 2018;34(17):i821-i829. doi: 10.1093/bioinformatics/bty593

Huang K, Fu T, Glass LM, Zitnik M, Xiao C, Sun J. DeepPurpose: a deep learning library for drug-target interaction prediction. Bioinformatics. 2021;36(22-23):5545-5547. doi: 10.1093/bioinformatics/btaa1005

Huang K, Xiao C, Glass LM, Sun J. MolTrans: Molecular Interaction Transformer for drug-target interaction prediction. Bioinformatics. 2021;37(6):830-836. doi: 10.1093/bioinformatics/btaa880

Nguyen T, Le H, Quinn TP, Nguyen T, Le TD, Venkatesh S. GraphDTA: predicting drug-target binding affinity with graph neural networks. Bioinformatics. 2021;37(8):1140-1147. doi: 10.1093/bioinformatics/btaa921

Graber D, Stockinger P, Meyer F, Mishra S, Horn C, Buller R. Resolving data bias improves generalization in binding affinity prediction. Nat Mach Intell. 2025;7(10):1713-1725. doi: 10.1038/s42256-025-01124-5

In silico vs. wet lab

The standard wet-lab approach for the question DeepPurpose answers is a binding assay — surface plasmon resonance (SPR), isothermal titration calorimetry (ITC) or microscale thermophoresis (MST) for a direct dissociation constant, a competition binding assay for a panel, or an enzymatic IC50/Ki where the target has a catalytic readout. These report a number in molar units. They are the thing every affinity prediction is graded against, and for the kinase panels in particular they are the thing that built the benchmarks: the Davis set is a competition binding assay of 72 kinase inhibitors against 442 kinases, covering over 80% of the human catalytic kinome.

Agreement here has no single settled threshold. The field reports correlation (Pearson r or Spearman ρ, for magnitude), error (RMSE or MSE in pKd units), and concordance index (CI, the share of compound pairs ranked in the correct order). They answer different questions, and a method can look strong on one and useless on another — which is this method class's situation. One pKd unit is 1.364 kcal/mol at 298 K, so an RMSE in pKd converts directly if you need to compare against the energy-based literature.

Read the accuracy column by cell, never down the column. The wet-lab cell is a reproducibility figure — the floor below which no prediction can be graded. The conventional cell is a ranking claim on a random split. The SOTA cell is an energy error for a method with a different input class. The last cell is an admission.

On that reading, the honest position for this chain is: seconds instead of weeks, at an accuracy that is good on the benchmark it was built for and indistinguishable from a chemistry-free baseline outside it. The cheapness is real. The transferability is the open question, and the published answer is unflattering.

Two asymmetries matter beyond the table. A binding assay needs purified, behaving protein and a compound in hand, which is most of why sequence-based prediction exists at all — it is the only method here that needs neither a structure nor a sample. And an assay reports what is, at a stated buffer and temperature, while this model reports what a kinase panel from 2011 implies. The two are not interchangeable at any correlation coefficient.

Agreement with wet lab

Read every figure below with its split. The order of the four regimes is the single most useful thing on this page, and the field's names for them come from one paper: S1, both the drug and the target were seen in training; S2, the target was seen but the drug is new (cold drug); S3, the drug was seen but the target is new (cold target); S4, neither was seen (cold pair).

The split, quantified

Author-reported, KronRLS with nested cross-validation, concordance index ×100 (50 is a random predictor):

⚠︎ Now the same runs, same splits, read as AUC-PR instead — and the verdict inverts:

On binarized Davis the concordance index falls from 95.2 to 70.0 across those four regimes — a number you could still talk yourself into. AUC-PR on the identical runs falls from 67.0 to 17.2. One of those readings says the model degrades; the other says it stops working. The authors' own summary is that "the practically more realistic settings resulted in reduced accuracies," and that previously published near-perfect figures came from "the oversimplified formulation of the drug–target prediction problem, as well as unrealistic evaluation of the model performance."

Independently retrained, with the similar proteins removed

A separate group retrained three published sequence models themselves and compared a random 80:20 split against clustered cross-validation, which holds protein sequence identity between train and test below 50% (40% on Davis):

This is the second metric pair that inverts. Read the CI columns and these models lose 0.12 to 0.22 and still look respectable. Read the Pearson columns and GraphDTA loses 63% of its correlation — 0.844 to 0.314. "The MSE increases by at least 2 times on going from random splitting to clustered cross validation," and of the best random-split model the authors wrote that it "seems to be rewarding overfitting."

What a model with no chemistry in it achieves

This is the finding that decides how to read every number above.

DeepPurpose itself has one independent re-evaluation. A competing group retrained it on BindingDB (10,416 ligands, 1,391 targets, binary labels, 5-fold CV) and tested it in three regimes they define as transductive (both the protein and the ligand appear in training), semi-inductive (only the ligand appears), and inductive (neither appears). Alongside it they ran a duplex configuration model, in which "the probability of observing a link is determined only by the degrees of its end nodes" — it sees no SMILES, no sequence, no chemistry of any kind:

The degree-only model beats DeepPurpose in the first two regimes, and in the third, where it is pinned at chance by construction, DeepPurpose reaches 0.61 ± 0.074 — about a tenth of an AUROC above a coin flip, with a standard deviation of 0.074.

Three further controls from the same work:

• Shuffling the chemistry changed nothing. Randomly reassigning the SMILES strings and amino-acid sequences across the training set, while keeping each node's positive and negative annotation counts, moved AUROC from 0.86 ± 0.005 to 0.84 ± 0.004 and AUPRC from 0.64 ± 0.009 to 0.62 ± 0.004 — "an operation that did not change the test performance."
• The predictions track popularity. Protein degree ratio and DeepPurpose's predictions correlate at Spearman r = 0.94. The top 100 false positives are high-degree proteins; the false negatives are low-degree ones.
• The mechanism is annotation imbalance, not chemistry. In that training data 3,084 of 10,416 ligands carry only positive annotations and 6,539 only negative ones; just 793 carry both. The authors' conclusion is that "DeepPurpose leverages network topology as a learning shortcut and fails to generalize predictions to proteins and ligands beyond the training data, indicating that we must use inductive testing to evaluate the true performance of ML models."

Which half of the pair carries the signal has also been measured, and it is not the protein. An independent evaluation of sequence-based architectures on 152,831 GPCR–ligand pairs permuted each input in turn: "Permuting the MolFormer embeddings or the labels reduced AUROC to near-random levels for every model on the unseen-ligands test set, whereas permuting the ProtBERT embeddings produced only a modest decrease across all the models." Their reading: "models retain most of their predictive power without correct protein information but lose nearly all of it without correct ligand information," and "sequence-based architectures learn predominantly from ligand features, with protein encodings contributing less."

The masked-input experiment gives a different answer on Davis and KIBA, and that difference is real. The group that retrained the sequence models also trained ligand-only and protein-only variants by masking out the other half of the input:

On PDBbind, where the model sees a structure, half the input is nearly enough: "the ligand-only and protein-only performances are very comparable to the performance of the full complex, with the Pearson correlation dropping only by around 0.1 … the model does not necessarily focus on protein–ligand interactions for making binding affinity predictions." On Davis and KIBA, the sequence datasets DeepPurpose uses, masking does hurt — the authors read that as "the absence of such a bias in these data sets."

So the two traps are not the same trap. The masked-input shortcut is a structure-benchmark phenomenon, and on Davis and KIBA it is refuted. The annotation-topology shortcut is the one that holds for DeepPurpose, and the evidence for it is a chemistry-free baseline that beats the model outright. Note also that even on Davis the ligand-only model reaches CI 0.75 against the full model's 0.88 and a random 0.50 — the concordance index compresses a 0.36-point collapse in Pearson r into something that reads like a near-miss.

Where the labels come from, and what they encode

Neither benchmark label is a clean measurement of the thing you want.

Davis is censored, and most of it sits on the floor. Affinities are used as pKd, and "the peak at pKd value 5 (10 000 nM) constitutes more than half of the dataset (20 931 out of 30 056)" — those are pairs that either bound more weakly than 10 µM or were never observed in the primary screen. Just under 70% of the dataset is a single value representing "no useful binding." A constant predictor returning exactly 5.0 is exactly right on 69.6% of Davis, which is arithmetic on the authors' own distribution, not a published benchmark result — but it is the reason an MSE on this dataset flatters, and the reason the platform's binding-class cutoffs do not work (see the defects).

KIBA is not a measurement at all. The KIBA score is a model-based integration of IC50, Ki and Kd values across assays, introduced precisely because those quantities disagree: the authors' own warning is that ignoring "data heterogeneity and representation may lead to biased modeling of drugs' polypharmacological effects as well as to unrealistic evaluation of computational strategies for the prediction of drug-target interaction networks." A KIBA figure grades a model against another model's output. This is the general hazard described in property predictors are graded one step removed from the bench, in its sharpest form on this platform: ask what the label encodes, and here it encodes a curation decision.

And the targets, not the drugs, are the ones that repeat. An interventional study of this bias found that on BindingDB "target label frequencies exhibit extreme bimodal concentrations at 0 and 1 (Pt = 0.000), while drug label frequencies center near 0.5 with no significant deviation (Pd = 1.000)," and that "targets exhibit significantly higher overlap across training, validation, and test sets compared to drugs." Their summary is that "models can achieve high performance by merely memorizing the target."

Chatterjee A, Walters R, Shafi Z, Ahmed OS, Sebek M, Gysi D, et al. Improving the generalizability of protein-ligand binding predictions with AI-Bind. Nat Commun. 2023;14(1):1989. doi: 10.1038/s41467-023-37572-z

Pahikkala T, Airola A, Pietilä S, Shakyawar S, Szwajda A, Tang J, et al. Toward more realistic drug-target interaction predictions. Brief Bioinform. 2015;16(2):325-337. doi: 10.1093/bib/bbu010

Kanakala GC, Aggarwal R, Nayar D, Priyakumar UD. Latent biases in machine learning models for predicting binding affinities using popular data sets. ACS Omega. 2023;8(2):2389-2397. doi: 10.1021/acsomega.2c06781

Dey S, Lu P, Wallqvist A, AbdulHameed MDM. Evaluation of deep learning architectures for predicting ligand interactions with neurologically relevant GPCRs. J Cheminform. 2026;18(1):139. doi: 10.1186/s13321-026-01258-7

Lin G, Zhang X, Ren Z, Zou Q, Tiwari P, Zhou C, et al. TAPB: an interventional debiasing framework for alleviating target prior bias in drug-target interaction prediction. Nat Commun. 2025;16(1):10867. doi: 10.1038/s41467-025-66915-1

Davis MI, Hunt JP, Herrgard S, Ciceri P, Wodicka LM, Pallares G, et al. Comprehensive analysis of kinase inhibitor selectivity. Nat Biotechnol. 2011;29(11):1046-1051. doi: 10.1038/nbt.1990

Tang J, Szwajda A, Shakyawar S, Xu T, Hintsanen P, Wennerberg K, et al. Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis. J Chem Inf Model. 2014;54(3):735-743. doi: 10.1021/ci400709d

Ross GA, Lu C, Scarabelli G, Albanese SK, Houang E, Abel R, Harder ED, Wang L. The maximal and current accuracy of rigorous protein-ligand binding free energy calculations. Commun Chem. 2023;6(1):222. doi: 10.1038/s42004-023-01019-9

What the figure doesn't cover

• No published figure describes this configuration. Every number above was measured on a model retrained for the benchmark. This platform loads a fixed pretrained checkpoint and applies it to whatever protein you send. Nothing in the literature grades that, and when the target is not a kinase in the Davis panel, the cold-target and cold-pair rows are the applicable prior — not the random-split rows that the tool's own documentation implicitly invokes.
• No prospective validation exists for this tool. No published study has run DeepPurpose on this platform's configuration, nominated compound–target pairs, and reported the measured affinities. Retrospective recovery of database entries is not the same claim.
• A ranking metric will not tell you the model stopped working. Three independent cases on this page have a concordance index or an ROC-AUC reading "degraded" where AUC-PR, Pearson r or r²ₘ on the identical runs reads "stopped working." If you are given one number for a DTI model, it is overwhelmingly likely to be the flattering one.
• The benchmarks are kinase-shaped. Davis is 442 kinases and KIBA is 467. A pretrained checkpoint has seen almost nothing else. Applying it to a GPCR, an ion channel, a protease or a protein–protein interface is an extrapolation with no figure attached, and the one GPCR evaluation on this page puts unseen-target AUROC at 0.58–0.64.
• Censoring, not just leakage. Nearly 70% of Davis sits at exactly pKd 5.0 by construction. Low predictions from a Davis-trained model are not measurements of weak binding; they are the model reproducing a detection limit.
• A KIBA figure grades the model against another model. The score is a cross-assay integration, not a measured quantity, and it does not convert to a Kd.
• The output has no error bar. The number returned carries no confidence interval, and the confidence the platform attaches is computed from fixed constants rather than from the model (see below). Against the independent cold-split figures, the appropriate uncertainty on a novel target is close to the full range of the scale.
• The literature is overwhelmingly author-reported, and the pattern is consistent. Every architecture on this page looks strong in its own paper and weaker when someone else retrains it — DeepDTA's Davis Pearson 0.841 → 0.477, GraphDTA 0.844 → 0.314, MolTrans transductive 0.952 → inductive 0.572, DeepPurpose matched by a degree-only baseline. Treat an author-reported DTI figure as the ceiling of what the method can do, never as an estimate of what it will do.
• "Reviews are not evidence" applies hard here. This field has a very large survey literature that restates benchmark CI values without their splits. Every figure on this page is from a primary comparative study, with its split named; the two claims carried without a figure are marked in place above.

Known issues on this tool

These are known issues, and a fix is scoped for the calibration.

⊘ The binding-strength label is not calibrated to the model you selected. This tool buckets a numeric prediction into STRONG / MODERATE / WEAK / INACTIVE using one fixed ladder, but the checkpoints it serves were trained on datasets with different scales and different floors — one is censored at the bottom of its range, so the weakest class is close to unreachable and the next class up absorbs everything a real panel would have called a non-binder. Some checkpoints also emit a different affinity scale entirely, and one emits a score that is not an affinity at all, with no unit marked on the result.

What to do: read the numeric prediction, not the label, and confirm which scale the checkpoint you chose reports on before comparing values across runs. Treat the class names as presentational until the calibration is derived from the loaded checkpoint rather than a single fixed ladder.

⚠︎ Separately, several pretrained models and screening datasets named in the surrounding documentation are not available here, and a few request parameters — including your encoder choice — do not change the computation. The parameter list this tool accepts is wider than the set it acts on.

Go / no-go

Go — act on the result when:

• You are ordering a list of kinases for a panel assay you were going to run anyway, your compound resembles a kinase inhibitor, and you treat the order as a suggestion to be checked rather than a result.
• Both your compound and your target are well represented in the training data — a known kinase, a known chemotype — which is the transductive regime, and the only one where the strong figures apply.
• You are filling in missing cells of a panel you already partly measured, which is the task the benchmarks were built from.
• You read the number with its checkpoint and its units, and you have confirmed which checkpoint ran. With no pretrained_model set, that is MPNN_CNN_DAVIS, emitting pKd.
• The cost of being wrong is one assay well. At 4 credits and 44 seconds, that is a reasonable trade, and it is the honest case for this tool.

No-go — do not act on the result when:

• Your target is novel. On unseen proteins and ligands the independent AUROC for this architecture is 0.61 against a chance level of 0.50, and a degree-only model with no chemistry in it is pinned at 0.50 by construction and still beats it in the easier regimes.
• Your target is not a kinase, and you intend to read the number as an affinity. There is no figure for that, and the one non-kinase evaluation here puts unseen-target AUROC at 0.58–0.64.
• You want a Kd, a nanomolar potency, or a "strong binder" verdict. The binding-class buckets are miscalibrated against the censored training distribution, the units differ silently between checkpoints, and no error bar is attached.
• You are comparing across checkpoints. A pKd from a Davis model, a pIC50 from a BindingDB model and a raw KIBA score are three different scales in one field.
• A low score is your evidence of selectivity. Nearly 70% of the training labels are a censored floor, so a low prediction reproduces a detection limit rather than measuring a non-interaction. Negative selectivity claims need the panel assay.
• The ranking is the deliverable and nobody will check it. Detection without confirmation is this method class's documented failure mode, and the published cold-split figures say the ordering is the first thing to go.

Caveats on reading this page: every figure above is quoted from the cited source with its original metric, dataset and split, and labelled author-reported or independent. Figures from different studies are not directly comparable — datasets, split definitions, label thresholds and metrics all differ between them. The two claims carried without a number are marked in place, because the figures behind them sit in supplements or image-only panels that were not read.

Read next