Score a DeepPurpose DTI model with MSE, Pearson r and concordance index or AUROC and AUPRC, under k-fold and cold_drug or cold_protein splits.

DeepPurpose supports comprehensive evaluation for drug-target interaction (DTI) prediction models. This guide covers metrics, cross-validation, and evaluation best practices.

Evaluation modes

DeepPurpose supports two prediction modes with different evaluation metrics:

Mode	Task	Primary Metrics
Regression	Predict binding affinity (Kd, Ki, IC50)	MSE, Pearson r, Concordance Index
Classification	Predict binding/non-binding	AUROC, AUPRC, Accuracy

Cross-validation

from DeepPurpose import DTI as models
from DeepPurpose import utils, dataset

# Load dataset
X_drugs, X_targets, y = dataset.load_process_DAVIS(
    path='./data',
    binary=False,
    convert_to_log=True
)

# K-fold cross-validation
from sklearn.model_selection import KFold
import numpy as np

kf = KFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = []

for fold, (train_idx, test_idx) in enumerate(kf.split(X_drugs)):
    X_drug_train = [X_drugs[i] for i in train_idx]
    X_drug_test = [X_drugs[i] for i in test_idx]
    X_target_train = [X_targets[i] for i in train_idx]
    X_target_test = [X_targets[i] for i in test_idx]
    y_train = [y[i] for i in train_idx]
    y_test = [y[i] for i in test_idx]

    train, _, _ = utils.data_process(
        X_drug_train, X_target_train, y_train,
        drug_encoding="Morgan", target_encoding="CNN",
        split_method='no_split'
    )
    test, _, _ = utils.data_process(
        X_drug_test, X_target_test, y_test,
        drug_encoding="Morgan", target_encoding="CNN",
        split_method='no_split'
    )

    config = utils.generate_config(
        drug_encoding="Morgan", target_encoding="CNN",
        train_epoch=50
    )
    model = models.model_initialize(**config)
    model.train(train, test, test)

    preds = model.predict(test)
    # Calculate fold metrics...
    cv_scores.append(fold_mse)

print(f"CV MSE: {np.mean(cv_scores):.4f} +/- {np.std(cv_scores):.4f}")

Regression metrics

For binding affinity prediction (continuous values):

from sklearn.metrics import r2_score, mean_squared_error
from scipy.stats import pearsonr
import numpy as np

preds = model.predict(test)
y_true = [sample[2] for sample in test]  # Extract labels

# Mean Squared Error
mse = mean_squared_error(y_true, preds)
print(f"MSE: {mse:.4f}")

# Root Mean Squared Error
rmse = np.sqrt(mse)
print(f"RMSE: {rmse:.4f}")

# R-squared
r2 = r2_score(y_true, preds)
print(f"R2: {r2:.4f}")

# Pearson correlation coefficient
pearson_r, p_value = pearsonr(y_true, preds)
print(f"Pearson r: {pearson_r:.4f}")

# Concordance Index (CI) - commonly used in DTI
def concordance_index(y_true, y_pred):
    """Calculate the concordance index for ranking pairs."""
    from itertools import combinations
    pairs = list(combinations(range(len(y_true)), 2))
    concordant = 0
    for i, j in pairs:
        if (y_true[i] > y_true[j] and y_pred[i] > y_pred[j]) or \
           (y_true[i] < y_true[j] and y_pred[i] < y_pred[j]):
            concordant += 1
        elif y_true[i] == y_true[j]:
            concordant += 0.5
    return concordant / len(pairs)

ci = concordance_index(y_true, preds)
print(f"Concordance Index: {ci:.4f}")

Classification metrics

For binary binding prediction:

from sklearn.metrics import (
    roc_auc_score, average_precision_score,
    accuracy_score, precision_score, recall_score, f1_score
)

preds = model.predict(test)
y_true = [sample[2] for sample in test]

# For probability outputs, binarize at threshold
y_pred_binary = [1 if p > 0.5 else 0 for p in preds]

# AUROC (Area Under ROC Curve)
auroc = roc_auc_score(y_true, preds)
print(f"AUROC: {auroc:.4f}")

# AUPRC (Area Under Precision-Recall Curve)
auprc = average_precision_score(y_true, preds)
print(f"AUPRC: {auprc:.4f}")

# Accuracy, Precision, Recall, F1
accuracy = accuracy_score(y_true, y_pred_binary)
precision = precision_score(y_true, y_pred_binary)
recall = recall_score(y_true, y_pred_binary)
f1 = f1_score(y_true, y_pred_binary)

print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1 Score: {f1:.4f}")

Calibration

For classification, bin predictions and plot reliability curves.
Consider temperature scaling on a held-out validation set to reduce over/under-confidence.

Cold split evaluation

For realistic evaluation, use cold splits that test generalization to unseen drugs or targets:

# Cold drug split - test on drugs not seen during training
train, val, test = utils.data_process(
    X_drugs, X_targets, y,
    drug_encoding="Morgan", target_encoding="CNN",
    split_method='cold_drug',
    frac=[0.7, 0.1, 0.2]
)

# Cold target split - test on proteins not seen during training
train, val, test = utils.data_process(
    X_drugs, X_targets, y,
    drug_encoding="Morgan", target_encoding="CNN",
    split_method='cold_protein',
    frac=[0.7, 0.1, 0.2]
)

Cold splits typically show lower performance but better reflect real-world drug discovery scenarios.

Benchmark comparison

Compare against published benchmarks on standard datasets:

Dataset	Metric	Typical Range	Notes
DAVIS	MSE	0.2 - 0.4	Log-transformed Kd values
DAVIS	CI	0.85 - 0.90	Concordance Index
KIBA	MSE	0.1 - 0.3	KIBA scores
BindingDB	AUROC	0.85 - 0.95	Binary classification

Logging recommendations

Save metrics per epoch and per fold to outputs/metrics.csv.
Record dataset splits, encoders, and random seeds alongside results.
Add a short SUMMARY.json with best metrics and model path for quick scanning.
Track encoder configurations to ensure reproducibility across experiments.

Evaluation & metrics