---
title: Evaluation & metrics
description: Score a DeepPurpose DTI model with MSE, Pearson r and concordance index or AUROC and AUPRC, under k-fold and cold_drug or cold_protein splits.
---

[DeepPurpose](https://github.com/kexinhuang12345/DeepPurpose) supports comprehensive evaluation for drug-target interaction (DTI) prediction models. This guide covers metrics, cross-validation, and evaluation best practices.

## Evaluation modes

DeepPurpose supports two prediction modes with different evaluation metrics:

| Mode | Task | Primary Metrics |
|------|------|-----------------|
| **Regression** | Predict binding affinity (Kd, Ki, IC50) | MSE, Pearson r, Concordance Index |
| **Classification** | Predict binding/non-binding | AUROC, AUPRC, Accuracy |

## Cross-validation

```python
from DeepPurpose import DTI as models
from DeepPurpose import utils, dataset

# Load dataset
X_drugs, X_targets, y = dataset.load_process_DAVIS(
    path='./data',
    binary=False,
    convert_to_log=True
)

# K-fold cross-validation
from sklearn.model_selection import KFold
import numpy as np

kf = KFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = []

for fold, (train_idx, test_idx) in enumerate(kf.split(X_drugs)):
    X_drug_train = [X_drugs[i] for i in train_idx]
    X_drug_test = [X_drugs[i] for i in test_idx]
    X_target_train = [X_targets[i] for i in train_idx]
    X_target_test = [X_targets[i] for i in test_idx]
    y_train = [y[i] for i in train_idx]
    y_test = [y[i] for i in test_idx]

    train, _, _ = utils.data_process(
        X_drug_train, X_target_train, y_train,
        drug_encoding="Morgan", target_encoding="CNN",
        split_method='no_split'
    )
    test, _, _ = utils.data_process(
        X_drug_test, X_target_test, y_test,
        drug_encoding="Morgan", target_encoding="CNN",
        split_method='no_split'
    )

    config = utils.generate_config(
        drug_encoding="Morgan", target_encoding="CNN",
        train_epoch=50
    )
    model = models.model_initialize(**config)
    model.train(train, test, test)

    preds = model.predict(test)
    # Calculate fold metrics...
    cv_scores.append(fold_mse)

print(f"CV MSE: {np.mean(cv_scores):.4f} +/- {np.std(cv_scores):.4f}")
```

## Regression metrics

For binding affinity prediction (continuous values):

```python
from sklearn.metrics import r2_score, mean_squared_error
from scipy.stats import pearsonr
import numpy as np

preds = model.predict(test)
y_true = [sample[2] for sample in test]  # Extract labels

# Mean Squared Error
mse = mean_squared_error(y_true, preds)
print(f"MSE: {mse:.4f}")

# Root Mean Squared Error
rmse = np.sqrt(mse)
print(f"RMSE: {rmse:.4f}")

# R-squared
r2 = r2_score(y_true, preds)
print(f"R2: {r2:.4f}")

# Pearson correlation coefficient
pearson_r, p_value = pearsonr(y_true, preds)
print(f"Pearson r: {pearson_r:.4f}")

# Concordance Index (CI) - commonly used in DTI
def concordance_index(y_true, y_pred):
    """Calculate the concordance index for ranking pairs."""
    from itertools import combinations
    pairs = list(combinations(range(len(y_true)), 2))
    concordant = 0
    for i, j in pairs:
        if (y_true[i] > y_true[j] and y_pred[i] > y_pred[j]) or \
           (y_true[i] < y_true[j] and y_pred[i] < y_pred[j]):
            concordant += 1
        elif y_true[i] == y_true[j]:
            concordant += 0.5
    return concordant / len(pairs)

ci = concordance_index(y_true, preds)
print(f"Concordance Index: {ci:.4f}")
```

## Classification metrics

For binary binding prediction:

```python
from sklearn.metrics import (
    roc_auc_score, average_precision_score,
    accuracy_score, precision_score, recall_score, f1_score
)

preds = model.predict(test)
y_true = [sample[2] for sample in test]

# For probability outputs, binarize at threshold
y_pred_binary = [1 if p > 0.5 else 0 for p in preds]

# AUROC (Area Under ROC Curve)
auroc = roc_auc_score(y_true, preds)
print(f"AUROC: {auroc:.4f}")

# AUPRC (Area Under Precision-Recall Curve)
auprc = average_precision_score(y_true, preds)
print(f"AUPRC: {auprc:.4f}")

# Accuracy, Precision, Recall, F1
accuracy = accuracy_score(y_true, y_pred_binary)
precision = precision_score(y_true, y_pred_binary)
recall = recall_score(y_true, y_pred_binary)
f1 = f1_score(y_true, y_pred_binary)

print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1 Score: {f1:.4f}")
```

## Calibration

- For classification, bin predictions and plot reliability curves.
- Consider temperature scaling on a held-out validation set to reduce
  over/under-confidence.

## Cold split evaluation

For realistic evaluation, use cold splits that test generalization to unseen drugs or targets:

```python
# Cold drug split - test on drugs not seen during training
train, val, test = utils.data_process(
    X_drugs, X_targets, y,
    drug_encoding="Morgan", target_encoding="CNN",
    split_method='cold_drug',
    frac=[0.7, 0.1, 0.2]
)

# Cold target split - test on proteins not seen during training
train, val, test = utils.data_process(
    X_drugs, X_targets, y,
    drug_encoding="Morgan", target_encoding="CNN",
    split_method='cold_protein',
    frac=[0.7, 0.1, 0.2]
)
```

Cold splits typically show lower performance but better reflect real-world drug discovery scenarios.

## Benchmark comparison

Compare against published benchmarks on standard datasets:

| Dataset | Metric | Typical Range | Notes |
|---------|--------|---------------|-------|
| DAVIS | MSE | 0.2 - 0.4 | Log-transformed Kd values |
| DAVIS | CI | 0.85 - 0.90 | Concordance Index |
| KIBA | MSE | 0.1 - 0.3 | KIBA scores |
| BindingDB | AUROC | 0.85 - 0.95 | Binary classification |

## Logging recommendations

- Save metrics per epoch and per fold to `outputs/metrics.csv`.
- Record dataset splits, encoders, and random seeds alongside results.
- Add a short `SUMMARY.json` with best metrics and model path for quick
  scanning.
- Track encoder configurations to ensure reproducibility across experiments.