---
title: How CRISPR guide results compare to the bench
description: 'What a Rule Set 3 on-target score has been shown to agree with, why off-target needs AUC-PR not ROC AUC, and why the near-match scan is not genome-wide.'
tags:
  - crispr
  - validation
  - translatability
---

**You have a CRISPR result. Here is what it supports.**

- ★ **This is the one tool here with large-scale matched wet-lab ground truth.** Tens of thousands of guides have been measured for editing efficiency and hundreds of thousands of candidate sites assayed for off-target cleavage, so this page carries real concordance figures rather than hedges. It also means the honest numbers are lower than the headline ones, and you can see exactly how much lower.
- ⊘ **Keep the two questions apart.** *Will this guide cut its intended site?* is answered here by **Rule Set 3**, a published, independently-benchmarked model. *Is this guide safe elsewhere in the genome?* is **not answered here at all** — see the next bullet.
- ⊘ **`crispr_offtarget` is not a genome-wide off-target search and not an off-target predictor.** It takes the first protospacer it finds in the sequence *you* submit and scans **that same sequence** (plus an optional FASTA you upload) for near-matches. No reference genome is consulted, and there is no genome index or alignment step. The result emits `genome_wide: false` and `off_target_scope: "user_supplied_sequence"`. **"0 off-targets" here means "no near-match inside the sequence you pasted"** and says nothing about genomic safety. Use it for a plasmid, amplicon or transgene; for a genome-wide answer use a genome-aware designer, and for a therapeutic claim use an empirical assay.
- ⊘ **The on-target score is an unbounded z-score, not a 0–1 probability.** Rule Set 3 predicts a Yeo-Johnson-transformed, z-scored, sign-flipped activity value: in its own training set (n = 51,087) the mean is 0.000, the sd 0.9995, the range −3.80 to +4.95, and **47.1% of all values are negative**. More positive is more active. **Rank guides against each other within your target; never threshold on an absolute band.** Any "0.7 = high, below 0.3 = poor" table belongs to Rule Set 2 / Azimuth and is wrong here — see [known defects](#known-defects-on-this-platform).
- ⊘ **Read AUC-PR, never ROC AUC, for anything off-target.** Positives are 0.03–3% of candidate sites, so ROC AUC saturates and *orders methods wrongly*. The MIT/Hsu-Zhang score records **ROC AUC 0.770 beside AUC-PR 0.044** on the same test set. The same trap is live on the [P2Rank page](/docs/p2rank/translatability#the-apo-versus-holo-problem-is-the-whole-story), where accuracy and F1 did not move at all across a 38% AUPRC collapse, and on the [Paragraph page](/docs/paragraph/translatability), where a positional-frequency lookup table beats a graph neural network on ROC AUC.
- **Confirm on-target efficiency with** amplicon deep sequencing of the edited locus (indel frequency), or a reporter/surrogate assay. **Confirm off-target safety with** an empirical assay — GUIDE-seq, DISCOVER-seq or TTISS in cells; CIRCLE-seq, CHANGE-seq or SITE-seq on naked DNA, read as an over-calling screen rather than an answer.

Jump to [agreement with wet lab](#agreement-with-wet-lab) or [against other in silico methods](#against-other-in-silico-methods). The general framing for every tool is in [reading an in silico result against the bench](/docs/translatability).

## How in silico is used currently

Guide design is the one place in this doc set where in silico prediction is not optional — nobody picks a 20-mer by hand. But the field uses it for two different jobs with very different standards of evidence.

- **Ranking guides within a target window.** You have a region and need the three or four guides most likely to cut. This is what on-target scoring is for, and it is used as a *rank* over candidates in one locus, not as an absolute efficiency estimate. Every published model is evaluated as a correlation precisely because that is the claim being made.
- **Clearing a guide for specificity.** Here in silico work is a **triage step that narrows a candidate list for an empirical assay**, never a verdict. A sequence-similarity search enumerates thousands of homologous sites; a score ranks them; an assay decides. The FDA's own recommendation, quoted in a 2026 *Nature Genetics* review, is to use "multiple methods (e.g., in silico, biochemical, cellular-based assays) that include a genome-wide analysis…to reduce bias in identification of potential off-target sites."
- **Library design for pooled screens.** On-target models were largely *trained* on screen data, and this is the use they fit best: picking 4–6 guides per gene across thousands of genes, where per-guide error averages out.

What nobody does, reporting honestly, is declare a guide safe from a computational prediction. The clearest published illustration is Casgevy: as the review above records, "the original in silico predicted off-targets were not observed in cell-based assays, but additional off-targets were observed and validated using a variant-aware in silico off-target prediction tool called CRISPRme."

> Stewart AA, Ahrens-Nicklas RC, Tsai SQ, Musunuru K, Giannikopoulos P, Clelland CD. Measurement and clinical interpretation of CRISPR off-targets. *Nature Genetics*. 2026;58(1):20-27. doi: [10.1038/s41588-025-02428-3](https://doi.org/10.1038/s41588-025-02428-3)

## Against other in silico methods

⊘ **First, separate predictors from searches** — the same distinction the [AiZynthFinder page](/docs/aizynthfinder/translatability#a-score-is-not-a-search) makes for retrosynthesis. **Cas-OFFinder is an enumerator, not a predictor.** Its paper's title says it "searches for potential off-target sites"; its output is "chromosome number, position, direction, number of mismatched bases and potential off-target DNA sequences"; the string `scor` does not appear anywhere in its body text. **CCTop is both** — a sequence-similarity search plus a mismatch-position heuristic score. **CFD, MIT/Hsu-Zhang, CRISTA, DeepCRISPR, Elevation, CRISPR-Net and CRISOT are predictors.** Listing Cas-OFFinder beside CFD as a competing "predictor" is a category error, and in practice Cas-OFFinder *supplies the denominator* that makes ROC AUC misleading: CHANGE-seq, Elevation and CnnCrispr all built their negative class with it.

> Bae S, Park J, Kim JS. Cas-OFFinder: a fast and versatile algorithm that searches for potential off-target sites of Cas9 RNA-guided endonucleases. *Bioinformatics*. 2014;30(10):1473-1475. doi: [10.1093/bioinformatics/btu048](https://doi.org/10.1093/bioinformatics/btu048)

**On-target, head to head.** Rule Set 3 beat Rule Set 2, CRISPRon, DeepSpCas9, DeepCRISPR and VBC on all three of its own validation screens when the tracrRNA was matched — a self-comparison, and the screens are tabulated in [agreement with wet lab](#agreement-with-wet-lab). Independently, over 13 external datasets, CRISPRon and DeepSpCas9 are the strongest of the earlier generation (0.197–0.703 and 0.180–0.617) with Rule Set 2 behind them (0.104–0.512), and conventional gradient-boosted models "achieve at least comparable performance to the deep learning ones." An independent review's verdict on the whole class: "all design tools performed worse on the datasets that differed from the ones they were trained on."

**Off-target, head to head.** Among the conventional scores, CFD is consistently best and the ordering is stable across independent and author-reported evaluations alike: CFD **ROC AUC 0.91** against MIT **0.87** independently; and author-reported Pearson correlation with GUIDE-seq read counts across 402 off-target sites of **CFD 0.40, CCTop 0.31, Hsu-Zhang 0.26**. Stratified by mismatch count, the same work put CFD at AUC 0.82–0.98, Hsu-Zhang at 0.61–0.87 and CCTop at 0.64–0.77. The learned models beat all of them on AUC-PR — but read the ordering from AUC-PR, not ROC AUC, for the reason set out under [agreement with wet lab](#agreement-with-wet-lab): on a target set where almost every site is a true negative, ROC AUC flatters a model that AUC-PR does not.

⚠︎ **Two cautions on the learned-model literature.** First, **data leakage is documented**: one 2023 benchmark found that "the training datasets of the DLcrispr and CRISPRnet models contained sgRNAs that were in the testing Guide-seq datasets," and greyed those bars out. Second, **the two most on-point independent benchmarks of off-target predictors are paywalled with no open deposit** — including one titled, precisely, "Data imbalance in CRISPR off-target prediction." Their abstracts support the conclusion of this page ("the benchmark of CRISPR off-target prediction should be properly evaluated and not overestimated by considering data imbalance issue"; "Most methods showed excellent performance on balanced datasets but have much room for improvement on moderate- and severe-imbalanced datasets"), but their figures could not be read.

**Also worth knowing: a landscape of ROC/PR pairs exists but is review-tabulated.** An independent 2023 review compiles ROC AUC and AUC-PR for roughly 19 off-target methods, ranging from 0.071 to 0.978 AUC-PR at ROC AUCs of 0.80–0.995. Those rows are author-reported figures gathered by a third party, with conditions differing per row, so this page does not lean on them — read them as a landscape, not as verified figures.

> Sherkatghanad Z, Abdar M, Charlier J, Makarenkov V. Using traditional machine learning and deep learning methods for on- and off-target prediction in CRISPR/Cas9: a review. *Briefings in Bioinformatics*. 2023;24(3):bbad131. doi: [10.1093/bib/bbad131](https://doi.org/10.1093/bib/bbad131)

## In silico vs. wet lab

**There are two wet-lab comparators here, and they are not interchangeable.**

For **on-target efficiency**, the standard is **amplicon deep sequencing of the intended locus**, reported as indel frequency — or a surrogate: a lentivirally integrated synthetic target read out by sequencing, a fluorescent reporter read by flow cytometry, or a pooled-screen log-fold-change. The agreement metric is a **Spearman correlation between predicted score and measured activity**, which is why this section can carry real numbers.

For **off-target risk**, the standard is an **empirical genome-wide assay**. These split into two families whose answers differ by two orders of magnitude, and that split is the most important thing on this page after the metric question: **in-cell** assays (GUIDE-seq, DISCOVER-seq, TTISS) and **cell-free / in vitro** assays on purified DNA (CIRCLE-seq, CHANGE-seq, SITE-seq, Digenome-seq). See [the in-vitro versus in-cell gap](#the-in-vitro-versus-in-cell-gap).

| | Time to answer | Cost basis | Accuracy, and its metric |
|---|---|---|---|
| **Standard wet lab, on-target** — amplicon deep sequencing of indel frequency | Days to weeks per locus panel | **No basis available.** The [shared wet-lab cost table](/docs/translatability#what-the-wet-lab-cost-figures-mean) has **no row for amplicon sequencing**, and no commercial list price was found | Ground truth. Replicates of the same assay in the same cell type agree at Spearman **0.71–0.77**, which is the ceiling on any prediction |
| **Standard wet lab, off-target** — GUIDE-seq / CIRCLE-seq / CHANGE-seq | Weeks, plus targeted validation sequencing | **No basis available.** The shared table has **no row for GUIDE-seq or any off-target assay** | Assay-dependent, not definitional. As nominators against targeted deep sequencing: GUIDE-seq precision **0.225**, SITE-seq **0.014**, CIRCLE-seq **0.007** |
| **Conventional in silico** — Rule Set 2 / Azimuth on-target; CFD + Cas-OFFinder off-target | Seconds | Free, open source | On-target Spearman **0.104–0.512** across 10 independent datasets. Off-target CFD **ROC AUC 0.91** independently — which is **~1.2% precision** at the authors' own recommended threshold |
| **Combined tooling here** — `crispr_design_guides` (Rule Set 3, sequence model) + `crispr_offtarget` (local near-match scan) | 1 credit, padded runtime **80 s** | 1 credit per run, for either job type | On-target: Rule Set 3's own validation gives mean per-gene Spearman **0.579 / 0.388 / 0.469** on three A375 screens by tracrRNA. **Off-target: no figure, because no off-target prediction is performed** — the scan has no genome and uses no published scoring matrix |

**Read the accuracy column cell by cell, never down the column.** Deep sequencing's accuracy is definitional. A Spearman correlation is a rank agreement bounded above by assay noise at about 0.7. An assay precision is the fraction of nominated sites that were really edited. An AUC-PR is a classifier's average precision under extreme class imbalance. Nothing meaningful comes from comparing those to each other.

On that reading, the honest position for this chain is: **seconds instead of weeks for a guide ranking that is genuinely state-of-the-art, and nothing at all for genome-wide specificity.** The second half of that sentence is the whole scope of this tool, and it is why the page spends as much space on what is absent as on what is measured.

## Agreement with wet lab

### On-target: Rule Set 3, and the spread that matters more than any single number

This platform runs the **sequence** model of Rule Set 3, with the tracrRNA set to either `Hsu2013` or `Chen2013`. Scope verified at 100% of the model's 133,674 published training and test rows: **SpCas9 only, NGG PAM only, 20-nt spacers only**, Pol III (U6) transcription.

**The model's own validation**, author-reported and a self-comparison. Conditions: A375 melanoma cells stably expressing SpCas9, lentiviral tiling library of 84,609 sgRNAs over 201 essential and 198 non-essential genes, 3 biological replicates, ≥500× representation, harvested 21 days post-transduction, readout = **screen log2 fold-change**, Spearman computed within each essential gene (n = 200 genes per screen):

| Model | Hsu-tracr screen | Chen-tracr screen | DeWeirdt-tracr screen |
|---|---|---|---|
| **Rule Set 3 (Sequence + Target), matched tracr** | **0.579** | **0.388** | **0.469** |
| Rule Set 3 (Sequence only), matched tracr | 0.522 | 0.277 | 0.363 |
| CRISPRon | 0.500 | 0.172 | 0.234 |
| DeepSpCas9 | 0.473 | 0.171 | 0.231 |
| **Rule Set 2 / Azimuth** | 0.448 | 0.168 | 0.212 |
| DeepCRISPR | 0.116 | 0.064 | 0.082 |

The authors state that "Rule Set 3 (Sequence + Target) significantly outperformed all other models (t-test p value < 0.002) when the correct tracrRNA was specified," with p = 4.99 × 10⁻²⁷ against Rule Set 2 on the Hsu screen.

⚠︎ **Two things that table does not tell you.** First, the platform runs **Sequence only** — the row below the headline, not the headline. Rule Set 3's **Target** model is additive and never loaded here; its authors report it adds "an average improvement of 6.7%" in Spearman. Second, the readout is **screen log-fold-change**, not indel frequency, and the two comparators diverge sharply: across the model's six held-out datasets, the one scored by **indel frequency at integrated targets** (Kim 2019, n = 518) plots in a band around **0.5–0.9**, while the genome-wide **screen-LFC** sets (Behan 2019 n = 7,025; Shalem 2014 n = 3,968) plot around **0.16–0.28**. A single headline correlation for Rule Set 3 would misrepresent both.

> DeWeirdt PC, McGee AV, Zheng F, Nwolah I, Hegde M, Doench JG. Accounting for small variations in the tracrRNA sequence improves sgRNA activity predictions for CRISPR screening. *Nature Communications*. 2022;13(1):5255. doi: [10.1038/s41467-022-33024-2](https://doi.org/10.1038/s41467-022-33024-2)

**★ The spread across datasets and cell types, independently measured.** This is the part a vendor page omits. The single best source is a third-party evaluation of ten predictors its authors did not build, over 13 external datasets, with guides overlapping each method's training data removed and training cells marked:

| Predictor | Independent Spearman range | Datasets | On its own training data |
|---|---|---|---|
| **CRISPRon** | **0.197 – 0.703** | 13 | — |
| **DeepSpCas9** | **0.180 – 0.617** | 13 | 0.77 (10-fold CV) |
| **Rule Set 2 / Azimuth** | **0.104 – 0.512** | 10 | 0.482 – 0.727 |
| CRISPRedict | 0.253 – 0.571 | 12 | 0.602 |
| Chari | 0.145 – 0.495 | 12 | 0.452 |
| CRISPRscan / Moreno-Mateos | −0.197 – 0.357 | 12 | 0.579 |
| WU-CRISPR / Wong | 0.037 – 0.299 | 11 | 0.287, 0.412 |

The same pattern holds in every independent matrix published. **DeepCRISPR is the cautionary case**: it reads **0.787 and 0.706 on its own training data and −0.106 to 0.160 across seven datasets it did not train on.** And a model's own held-out split is not an independent number either — DeepHF reports **0.867** on a held-out partition of its own library and **0.129–0.594** across 18 external datasets, in the same paper.

> Zarate OA, Yang Y, Wang X, Wang JP. BoostMEC: predicting CRISPR-Cas9 cleavage efficiency through boosting models. *BMC Bioinformatics*. 2022;23(1):446. doi: [10.1186/s12859-022-04998-z](https://doi.org/10.1186/s12859-022-04998-z)
>
> Kim HK, Kim Y, Lee S, Min S, Bae JY, Choi JW, et al. SpCas9 activity prediction by DeepSpCas9, a deep learning-based model with high generalization performance. *Science Advances*. 2019;5(11):eaax9249. doi: [10.1126/sciadv.aax9249](https://doi.org/10.1126/sciadv.aax9249)
>
> Wang D, Zhang C, Wang B, Li B, Wang Q, Liu D, et al. Optimized CRISPR guide RNA design for two high-fidelity Cas9 variants by deep learning. *Nature Communications*. 2019;10(1):4284. doi: [10.1038/s41467-019-12281-8](https://doi.org/10.1038/s41467-019-12281-8)

**★ The ceiling is about 0.7, so treat any figure above it as in-distribution.** Replicates of the same assay in the same cell type correlate at Spearman **0.71–0.77**, and the authors who measured that conclude "a correlation of about 0.7 constitutes an upper limit of any prediction." Independent reproductions agree: 0.73–0.82 between K562 and Jurkat replicates, and 0.72–0.77 between synthetic integrated targets and endogenous sites. Below that, **cell type alone costs you a large share of the remainder** — identical guide sequences measured in different cell lines correlate at Spearman **0.462–0.560** (n = 40–1,675), 0.752 for HL60/KBM-7 but 0.53–0.63 for RPE-1. No sequence-only model can beat that bound across cell types.

**⊘ Promoter and delivery dominate everything else, and this is the boundary most likely to bite you.** Rule Set 3's authors say so themselves: "in vitro transcribed sgRNAs used especially in zebrafish systems are poorly predicted by models trained on results from mammalian cell models. Similarly, we expect Rule Set 3 to generalize less well to sgRNAs transcribed from a Pol II promoter." The measured size of that effect, with printed values: Azimuth's mean Spearman across four U6 datasets is **0.373** and across four T7 datasets **0.036**. In one group's own newly generated data, Rule Set 2 goes from **0.415** on U6-in-cells to **−0.304** on T7-transcribed guides injected into mouse embryos. **Never carry a U6 figure into a T7, IVT or synthetic-RNP context.**

> Haeussler M, Schönig K, Eckert H, Eschstruth A, Mianné J, Renaud JB, et al. Evaluation of off-target and on-target scoring algorithms and integration into the guide RNA selection tool CRISPOR. *Genome Biology*. 2016;17(1):148. doi: [10.1186/s13059-016-1012-2](https://doi.org/10.1186/s13059-016-1012-2)
>
> Konstantakos V, Nentidis A, Krithara A, Paliouras G. CRISPRedict: a CRISPR-Cas9 web tool for interpretable efficiency predictions. *Nucleic Acids Research*. 2022;50(W1):W191-W198. doi: [10.1093/nar/gkac466](https://doi.org/10.1093/nar/gkac466)
>
> Konstantakos V, Nentidis A, Krithara A, Paliouras G. CRISPR-Cas9 gRNA efficiency prediction: an overview of predictive tools and the role of deep learning. *Nucleic Acids Research*. 2022;50(7):3616-3637. doi: [10.1093/nar/gkac192](https://doi.org/10.1093/nar/gkac192)

**⊘ Independent validation of Rule Set 3 specifically is thin and off-domain — say it plainly.** No independent group has published a Rule Set 3 Spearman on a human U6-transcribed screen benchmark in retrievable body text. The two independent point estimates that exist are both outside the model's domain:

- **Human iPSCs, chemically synthesized crRNA:tracrRNA duplexes, lipofected** (n = 78 guides, readout a composite of cell death plus editing by amplicon sequencing): Rule Set 3 was the **best** of the tools tested at Pearson **r = 0.42 with the Chen tracrRNA and r = 0.31 with Hsu** — in a paper whose verdict on the whole field is "All tools provide far from a satisfactory prediction." This is synthetic, not U6-transcribed, which is precisely the boundary the model's authors warned about.
- ***Nicotiana benthamiana* leaves**, transient expression, amplicon indel readout, two datasets of **20 and 32 guides**: Rule Set 3 was among tools showing "significantly higher Spearman's r" than CRISPRedict and CCTop, in a group reaching r > 0.8 on the 20-guide set and about 0.5 on the 32-guide set.

> Riesenberg S, Kanis P, Karlic R, Maricic T. Robust prediction of synthetic gRNA activity and cryptic DNA repair by disentangling cellular CRISPR cleavage outcomes. *Nature Communications*. 2025;16(1):4717. doi: [10.1038/s41467-025-59947-0](https://doi.org/10.1038/s41467-025-59947-0)
>
> Gong Z, Chen M, Zhang H, Mortimer JC, Botella JR. Evaluation of computational tools for the prediction of CRISPR/SpCas9 gRNA activity in plants. *Plant Cell Reports*. 2026;45(5):123. doi: [10.1007/s00299-026-03820-x](https://doi.org/10.1007/s00299-026-03820-x)

**The tracrRNA setting is not cosmetic.** Only `Hsu2013` and `Chen2013` exist in the model; there is no third setting, and the authors use the Chen setting as a proxy for a third variant. Choosing wrong costs real accuracy (0.579 → 0.489 on the Hsu screen), and the per-guide difference between the two settings has standard deviation **0.915 z-units** — comparable to the entire dynamic range of the score. Screen-level, the variant moves essential-vs-non-essential ROC AUC from **0.76 (Hsu) to 0.82 (Chen and DeWeirdt)**.

### Off-target: the metric question comes first

⊘ **The positive class is tiny and the negative class is enormous, so ROC AUC is actively misleading and AUC-PR is the honest metric.** This is not an editorial preference; it is stated outright in the primary literature. One group abandoned the metric mid-paper: "Due to the highly unbalanced nature of the data set, it was easy for the model to obtain a high auROC value. Therefore, we gave up the comparison of auROC values and focused on the comparison results of auPRC and Recall value on the test set." Another spells out the mechanism: "The AUROC metric is less sensitive to imbalanced class distributions, and its calculation relies on the False Positive Rate (FPR), which may be influenced by the overwhelming number of negative instances." A third: "Since the off-target datasets were highly imbalanced, the PR-AUC results rather than ROC-AUC were more convincing."

**The denominators, which explain why.** On one shared candidate universe (all genomic sites within 6 mismatches, hg38, SpCas9/NGG), positives as a fraction of candidates:

| Assay | Setting | sgRNAs | Candidate sites | Detected | Positives : candidates |
|---|---|---|---|---|---|
| CIRCLE-seq, U2OS | in vitro | 5 | 92,113 | 2,844 | 1 : 32 |
| CHANGE-seq | in vitro | 110 | 1,384,929 | 21,645 | 1 : 64 |
| SITE-seq | in vitro | 12 | 153,478 | 2,047 | 1 : 75 |
| **GUIDE-seq** (Tsai) | **in cell** | 10 | 158,169 | **341** | **1 : 464** |
| **TTISS** | **in cell** | 59 | 669,648 | **866** | **1 : 773** |
| **GUIDE-seq** (Listgarten) | **in cell** | 23 | 187,934 | **53** | **1 : 3,546** |

★ **In-cell assays are 5–55× more imbalanced than in-vitro assays on the same candidate universe.** Published imbalance ratios across ten off-target datasets run from **1.10 to 6,846.6**. At 1:1,000, a classifier can hold a 43% false-positive rate and still post a 0.91 ROC AUC.

**The figures, with ROC AUC and AUC-PR side by side.** Independent of the MIT score's authors, on the DeepCRISPR off-target set (29 sgRNAs, HEK293T and K562, SpCas9/NGG, >650 positives among >160,000 candidate loci, about 1:250):

| Predictor | ROC AUC | **AUC-PR** |
|---|---|---|
| CnnCrispr (self-comparison) | 0.975 | **0.679** |
| DeepCRISPR | **0.981** | **0.497** |
| CFD | 0.942 | 0.316 |
| CNN_std | **0.947** | **0.208** |
| **MIT / Hsu-Zhang** | **0.770** | **0.044** |

★ **The orderings disagree, and that is the real finding.** DeepCRISPR *beats* CnnCrispr on ROC AUC (0.981 vs 0.975) and loses by 18 AUC-PR points (0.497 vs 0.679). CNN_std *beats* CFD on ROC AUC (0.947 vs 0.942) and loses by 11 AUC-PR points. And the MIT/Hsu-Zhang score — still the default specificity number in much tooling — posts a respectable-looking 0.770 ROC AUC alongside **4.4% average precision**. If you grade an off-target method by ROC AUC you will rank the methods wrongly, not merely optimistically.

**The strongest form of the argument is a controlled ablation.** In the same paper, **three deliberately degraded variants of the model each scored a *higher* ROC AUC than the full model** — 0.987, 0.986 and 0.985 against 0.975 — while scoring lower AUC-PR and 5–25 points lower recall. A fourth, fully broken variant, which the authors state classified "the entire test set … as negative samples," recorded ROC AUC 0.500 and AUC-PR 0.504 — a model that predicts nothing reads as a coin flip on both, which is the vacuity control for this table.

The same ROC/PR divergence recurs wherever both are reported: one 2024 model records **ROC AUC ≈ 0.994 beside AUC-PR 0.70**, with a comparator at ROC AUC 0.9655 beside AUC-PR 0.46; a 2023 fingerprint model records **ROC AUC > 0.984 beside PR-AUC 0.46–0.70**. An independent re-run of six models on one rig makes the compression visible: **ROC AUC squeezes all six into a 0.77–0.85 band while AUC-PR spreads them from 0.40 to 0.62**, and the best result on validated data is ROC AUC 0.7682 / PR-AUC 0.3958 — far below the 0.995 figures the same methods report for themselves.

> Liu Q, Cheng X, Liu G, Li B, Liu X. Deep learning improves the ability of sgRNA off-target propensity prediction. *BMC Bioinformatics*. 2020;21(1):51. doi: [10.1186/s12859-020-3395-z](https://doi.org/10.1186/s12859-020-3395-z)
>
> Sun J, Guo J, Liu J. CRISPR-M: predicting sgRNA off-target effect using a multi-view deep learning network. *PLoS Computational Biology*. 2024;20(3):e1011972. doi: [10.1371/journal.pcbi.1011972](https://doi.org/10.1371/journal.pcbi.1011972)
>
> Zhang Y, Wang M, Zhang J, Jiang H, Zhang M, Xu T, et al. Genome-wide CRISPR off-target prediction and optimization using RNA-DNA interaction fingerprints. *Nature Communications*. 2023;14(1):7107. doi: [10.1038/s41467-023-42695-4](https://doi.org/10.1038/s41467-023-42695-4)
>
> Zhang G, Luo Y, Xie H, Dai Z. Crispr-SGRU: prediction of CRISPR/Cas9 off-target activities with mismatches and indels using stacked BiGRU. *International Journal of Molecular Sciences*. 2024;25(20):10945. doi: [10.3390/ijms252010945](https://doi.org/10.3390/ijms252010945)

**What a good ROC AUC is worth in precision, concretely.** The best-regarded conventional score, CFD, was independently measured at **ROC AUC 0.91** against 143 validated off-targets among 26,034 candidates (hg19, up to four mismatches), with MIT at 0.87. At the authors' own recommended operating point — a CFD cutoff of 0.023 — they report a **98% true-positive rate with a 43% false-positive rate**. On their published numbers that is roughly 140 true positives against 11,100 false positives: **a precision of about 1.2%, or one real site in eighty.** That is what a 0.91 ROC AUC buys on this task.

### The in-vitro versus in-cell gap

⊘ **A sequence-only off-target prediction cannot see chromatin, and in-vitro assays over-call relative to what happens in cells.** Both halves are measured, and both are large.

**Sites called per guide, matched guides in one lab** (8 gRNAs, HEK293T expressing Cas9, SpCas9/NGG, hg38, triplicate): GUIDE-seq nominated "an average of 40 sites per gRNA, the lowest average among the three assays," while CIRCLE-seq and SITE-seq nominated "an average of 5,244 and 2,093 sites per gRNA." **In vitro exceeds in cell by 131× and 52×.**

**What those extra calls are worth.** Graded against targeted deep sequencing of 66,165 homology-nominated candidates at ≥500× depth, of which 51 were sequence-confirmed as edited:

| | Edited sites nominated | Unedited sites nominated | Precision |
|---|---|---|---|
| **GUIDE-seq** (in cell) | 50 of 51 | 172 | **0.225** |
| **SITE-seq** (in vitro) | 50 of 51 | 3,585 | **0.014** |
| **CIRCLE-seq** (in vitro) | 49 of 51 | 6,602 | **0.007** |

★ **The in-vitro assays are 16–32× less precise and buy essentially no recall** — all three land at 96–98% recall inside the homology space. Only 1 of the 51 confirmed sites was missed by every assay, and the five missed by at least one replicate "were all edited at low frequencies &lt;0.5%."

**Confirmation rates run the same way.** Of 648 CHANGE-seq-detected sites examined by targeted tag sequencing in primary human T cells, 278 (42.9%) validated — but split by overlap, **98.3% of sites detected by both CHANGE-seq and GUIDE-seq confirmed, against 84 of 427 (18.3%) of sites detected by CHANGE-seq alone.** For CIRCLE-seq, **24 of 98 (24.5%)** CIRCLE-seq-only sites showed integration in cells, at frequencies of 0.003–0.2%. In vivo it is starker: for a therapeutically realistic orthogonal guide in mouse liver, the authors "comprehensively examined 181 of the 182 off-target CIRCLE-seq cleavage sites" and found that "only the gM on-target site showed significant evidence of indels" — **0 of 181 in-vitro calls confirmed.** The regulatory review's summary: "Apparent false positive discovery rates for naked DNA methods have been reported between 27% and 95%, while cellular methods tend to have less than 20%."

**★ Chromatin is the mechanism, and the controlled experiment is decisive.** One study placed **identical protospacers in open and closed chromatin** and assayed the same guides at both. For a perfectly matched on-target guide, indel frequency differed by only **1.6 ± 0.2-fold (HEK293T) and 1.3 ± 0.2-fold (HeLa)**. For guides with 1–2 mismatches — that is, for off-target cleavage — indel frequency at the open site exceeded the closed site "by up to **530-fold in HEK 293T cells and 1100-fold in HeLa cells**." In one worked pair, two sites with the *same DNA sequence* differed by factors of **68 and 43**. The authors' conclusion: "Cas9 on-target interactions are strong enough to cause DNA cleavage … regardless of the chromatin state, whereas Cas9 off-target interactions are in general much weaker and, thereby, are hindered by closed chromatin."

**The variance-explained figure is the one to remember.** In the same study, cleavage scores from **naked DNA** predicted cellular indel frequency at **R² = 0.07**; the identical assay run on **native chromatin** predicted it at **R² = 0.72**. Validation rates followed: 27% for the chromatin assay against 8.3% for the naked-DNA assay on the same sgRNA, and "no indels were detectably induced at 34 in vitro cleavage sites detected by Digenome-seq using histone-free DNA but missed by DIG-seq using chromatin DNA."

Independent, cell-matched epigenomics agrees on direction: cellular off-target sites are "about 2 to 4 times more likely to be found in chromatin states associated with active promoters, strong transcription, active enhancers and open chromatin," enriched for H3K4me3 and H3K27ac with no enrichment in H3K27me3 or H3K9me3 — with the caution that "chromatin openness is not the only epigenetic determinant." And the blind spot has been counted directly: in one curated cleavage database, **8,922 of 230,274 data points (3.9%) are sequence-identical guide–target pairs with different measured cleavage activity**, and "such a pair looks identical to purely sequence-based prediction algorithms. They might therefore not predict dangerous off-target effects."

⚠︎ **Be fair to the counter-evidence, because it changes the prescription.** Adding accessibility features to a predictor has repeatedly failed to help. One group tried DNase I data and dropped it: improvement in only one of four model variants in one of three cell types, never consistently best, and cell-type-averaged DNase I "did not increase the model performance in any of the cell types" — so they shipped sequence-only. Another found its **sequence-only variant had the lowest mean squared error** even though the full model recovered more of the true pairwise difference, and warned that "a low mean squared error does not necessarily go hand in hand with the model drawing the correct conclusions from environmentally sensitive features." The defensible statement is therefore: **the chromatin effect on off-target cleavage is large and well measured, but feeding accessibility into a predictor has so far produced inconsistent gains, largely because cell-type-matched epigenomic data exists for almost none of the training guides.** The practical consequence is unchanged — a sequence-only score cannot distinguish your open site from your closed one, so the assay is not optional.

> Chaudhari HG, Penterman J, Whitton HJ, Spencer SJ, Flanagan N, Zhang MCL, et al. Evaluation of homology-independent CRISPR-Cas9 off-target assessment methods. *The CRISPR Journal*. 2020;3(6):440-453. doi: [10.1089/crispr.2020.0053](https://doi.org/10.1089/crispr.2020.0053)
>
> Lazzarotto CR, Malinin NL, Li Y, Zhang R, Yang Y, Lee G, et al. CHANGE-seq reveals genetic and epigenetic effects on CRISPR-Cas9 genome-wide activity. *Nature Biotechnology*. 2020;38(11):1317-1327. doi: [10.1038/s41587-020-0555-7](https://doi.org/10.1038/s41587-020-0555-7)
>
> Tsai SQ, Nguyen NT, Malagon-Lopez J, Topkar VV, Aryee MJ, Joung JK. CIRCLE-seq: a highly sensitive in vitro screen for genome-wide CRISPR-Cas9 nuclease off-targets. *Nature Methods*. 2017;14(6):607-614. doi: [10.1038/nmeth.4278](https://doi.org/10.1038/nmeth.4278)
>
> Akcakaya P, Bobbin ML, Guo JA, Malagon-Lopez J, Clement K, Garcia SP, et al. In vivo CRISPR editing with no detectable genome-wide off-target mutations. *Nature*. 2018;561(7723):416-419. doi: [10.1038/s41586-018-0500-9](https://doi.org/10.1038/s41586-018-0500-9)
>
> Kim D, Kim JS. DIG-seq: a genome-wide CRISPR off-target profiling method using chromatin DNA. *Genome Research*. 2018;28(12):1894-1900. doi: [10.1101/gr.236620.118](https://doi.org/10.1101/gr.236620.118)
>
> Listgarten J, Weinstein M, Kleinstiver BP, Sousa AA, Joung JK, Crawford J, et al. Prediction of off-target activities for the end-to-end design of CRISPR guide RNAs. *Nature Biomedical Engineering*. 2018;2(1):38-47. doi: [10.1038/s41551-017-0178-6](https://doi.org/10.1038/s41551-017-0178-6)
>
> Störtz F, Mak JKW, Minary P. piCRISPR: physically informed deep learning models for CRISPR/Cas9 off-target cleavage prediction. *Artificial Intelligence in the Life Sciences*. 2023;3:100075. doi: [10.1016/j.ailsci.2023.100075](https://doi.org/10.1016/j.ailsci.2023.100075)

### The assays do not fully agree with each other either

Matched guides, one lab, hg38: CIRCLE-seq and SITE-seq each recovered about **86%** of GUIDE-seq's calls, and CIRCLE-seq recovered **78%** of SITE-seq's. Between the two in-vitro assays, downsampled to equal depth across 10 targets, CHANGE-seq and CIRCLE-seq overlapped **70.3–97.3%**.

★ **But reproducibility and validity run in opposite directions, which is worth internalising.** Across three replicates, only **30%** of GUIDE-seq's nominated sites were reproducibly nominated by all three, against **60.4%** for SITE-seq and **72%** for CIRCLE-seq; read-count reproducibility runs the same way (R² 0.54–0.67, 0.90–0.91 and 0.80–0.83 respectively). **The least reproducible assay is the most precise, and the most reproducible is among the least precise.** Reproducibility measures an assay's noise floor, not its agreement with cellular reality.

## What the figure doesn't cover

- **The score is a rank, not an efficiency.** Rule Set 3 returns a z-scored value centred on zero. It does not predict a percentage of edited alleles, and no published figure converts it to one. Two guides scoring +1.2 and +0.4 in different genes are not comparable in any calibrated sense.
- **Training overlap inflates most published on-target numbers, and the effect is measurable.** Every independent matrix marks the cells where a score was graded against its own training data, and those cells are consistently the high ones: Rule Set 2 reads 0.482–0.727 on its own data and 0.104–0.512 elsewhere. For Rule Set 3 specifically, **47.4% of its training sgRNAs come from exogenous integrated target sites in HEK293T scored by amplicon indel frequency**, not from pooled screens — so it is trained on both comparators in roughly equal measure, z-scored onto one axis. If your readout is one of those two, you are closer to its training distribution than the other.
- **The comparator you grade against changes the answer by more than the choice of model does.** Screen log-fold-change, integrated-target indel frequency, endogenous-locus indel frequency and reporter flow cytometry are four different measurements. One independent reporter-assay evaluation of five predictors found Pearson r of just **0.04 to 0.20** — including 0.118 for an earlier Doench rule set — on 430 sgRNAs in HEL cells, and concluded they showed "modest performance for individual sgRNA/target design" while remaining useful for library generation. That is the same class of model that reads 0.5+ on screen data.
- **Nothing here is validated for anything but SpCas9 with an NGG PAM.** Rule Set 3's training and test sets are 100% 20-nt spacers with NGG PAMs. The app refuses other nucleases for on-target scoring rather than mis-scoring them, which is the right behaviour — but see the defects below for where that refusal is missing.
- **No chromatin, no accessibility, no cell type.** The platform calls the **sequence** model only. Rule Set 3's companion **Target** model — which adds phyloP conservation, amino-acid composition, protein domains and relative cut position, and is additive with the sequence score for an average 6.7% Spearman improvement — is never loaded. Note that the Target model is *protein-coding-sequence* features, not chromatin; no component of this tool models accessibility at all.
- **The off-target scan has no applicability domain, because it is not a model.** It emits `is_heuristic: true` and `scoring_method: "mismatch_count_heuristic"`, and its score is a flat per-position penalty, not the published CFD mismatch-weight matrix. No published concordance figure describes it, and none could — the literature grades off-target predictors against genome-wide assays, and this scan has no genome.
- **No prospective validation of a platform result exists.** No study has taken a guide ranked by this chain, run it at the bench, and reported the outcome. Every figure on this page describes the underlying models, not this configuration.

### Known issues on this tool

These are known issues. A documentation rewrite is planned for the first.

⊘ **The rest of the CRISPR documentation is stale against what is deployed, and its score-interpretation tables are wrong.** The compute was rewritten to use Rule Set 3 and real MAGeCK, and that version is live; the sibling pages still describe the version before it. [inputs](/docs/crispr/inputs) names the older scoring model and advertises a genome-wide coordinate search; [outputs](/docs/crispr/outputs) lists genomic-coordinate fields that are never emitted, because no genome is consulted; [the overview](/docs/crispr) and [FAQ](/docs/crispr/faq) state that the on-target score is a composition heuristic and that MAGeCK is not run, both now false; and [quickstart](/docs/crispr/quickstart) shows invented coordinates. **Until that rewrite lands, trust this page and the tool's own output fields over them.**

⊘ **Do not threshold the on-target score — rank it.** The published interpretation band (0.7–1.0 "high", below 0.3 "poor") is a 0–1 shape applied to a score that is **not bounded to 0–1**. Against the model's real training distribution (n = 51,087) that band would reject **57.5%** of guides as "poor", capture only **10.3%** as "high", and leave **17.5%** above the table and every negative score — **47.1%** — below it. The package's own two-guide example returns −0.90 and +1.11. The band should be deleted rather than rescaled: **order your guides by score and take the top of your own list.** Those breakpoints have no peer-reviewed source — they are in neither the Rule Set 2 paper nor the Azimuth documentation, and even Rule Set 2's `[0,1]` describes its *training label*, with its own documentation warning that predictions *"are not explicitly normalized"* and may fall outside the range.

## Go / no-go

**Go — act on the result when:**

- You are **ranking candidate guides within one target region** for SpCas9 with an NGG PAM, delivered from a U6 promoter in mammalian cells, and you treat the score as an ordering. This is the configuration every published Rule Set 3 figure describes.
- You are **designing a pooled screen library**, picking several guides per gene across many genes. Per-guide error averages out, and this is the use the model was trained for.
- You are **checking a guide against a specific construct** — a plasmid, an amplicon, a transgene, or a FASTA of regions you care about. That is exactly what `crispr_offtarget` does, and it does it correctly.
- You are **analyzing a pooled screen count table** and you can name the control and treatment columns. This path runs real `mageck test` and returns MAGeCK's own RRA statistics.
- You will **confirm efficiency by amplicon deep sequencing** of the locus before committing to a guide.

**No-go — do not act on the result when:**

- You need to know whether a guide is **genome-wide safe**. Nothing here searches a genome. Use a genome-aware designer to enumerate candidate sites, then an empirical assay — and for a therapeutic programme, an in-cell assay, because in-vitro assays over-call by 52–131× at 16–32× worse precision.
- You are **thresholding on an absolute on-target score**, or carrying a score between target regions. Nearly half of all real scores are negative and no absolute band is meaningful.
- You are **grading any off-target method by ROC AUC, accuracy or F1.** Use AUC-PR. ROC AUC inverts the ranking of methods on this task, and a deliberately broken model still scores 0.50 on both.
- Your guides are **not SpCas9/NGG**, or are **not U6-transcribed** — in vitro transcribed, chemically synthesized, RNP-delivered, or driven by a Pol II promoter. The model's authors expect it to generalize worse to Pol II, measured promoter transfer collapses Azimuth from 0.373 to 0.036, and the only independent Rule Set 3 figures on synthetic guides are Pearson 0.31–0.42 in a study concluding that no tool was satisfactory.
- You are working in a **cell type or organism far from the training set**, and you intend to treat the ranking as reliable. Identical guides correlate at only 0.46–0.56 between human cell lines, which is a hard ceiling no sequence model can cross.
- You expect **genomic coordinates, nearest-gene annotation or exon context** in the output. None are computed. The documentation that promises them is stale.
- You read **"0 off-targets" as a specificity result.** It means no near-match inside the sequence you submitted.

**Caveats on reading this page:** every figure above is quoted from the cited source with its original metric, sample and conditions. Figures from different studies are not comparable, because assay type, promoter, delivery method, cell type, nuclease, candidate-site universe and genome build all differ between them — and several near-identical-looking figures are genuinely different quantities. Genome builds behind the off-target figures are hg38 for the matched-assay comparison, CHANGE-seq, TTISS and the imbalance table; hg19 for the CFD evaluation and the chromatin experiment; and unstated in the CIRCLE-seq 2017 report. Do not carry a figure across any of those boundaries.