Pull large ZINC exports in 50k-100k pages, stream each shard to disk, merge and gzip them, and record query parameters and shard counts in a metadata.json.

This page shows how to pull large ZINC exports without exhausting memory or storage — chunking requests into pages, streaming to disk, merging shards, and recording provenance. Requesting smaller pages and streaming responses is what most affects whether a bulk download completes reliably. These scripts pull straight from ZINC's CartBlanche22 service: ask Ivy to run them in the chat's code workspace, where files land under /workspace/scratch/, or run them on your own machine.

Chunk large requests

Request smaller pages (e.g., 50k–100k compounds per call).
Stream responses to disk to avoid high memory usage.

import requests

def stream_chunk(offset: int, limit: int) -> None:
    params = {"subset": "in-stock", "count": limit, "start": offset}
    with requests.get("https://cartblanche22.docking.org/substances.txt", params=params, stream=True, timeout=120) as r:
        r.raise_for_status()
        outfile = f"/workspace/scratch/zinc_{offset}_{offset+limit}.csv"
        with open(outfile, "w", encoding="utf-8") as f:
            for chunk in r.iter_content(chunk_size=8192):
                if chunk:
                    f.write(chunk.decode("utf-8"))

for start in range(0, 200000, 50000):
    stream_chunk(start, 50000)

Merge shards

cat /workspace/scratch/zinc_*.csv > /workspace/scratch/zinc_full.csv

Provenance

Store query parameters in /workspace/scratch/metadata.json.
Record counts per shard and final merged count.
Keep the original, unfiltered shards alongside filtered outputs.

Storage tips

Compress CSVs after filtering: gzip zinc_full.csv.
If producing SDF, chunk by 10k–20k molecules to keep files manageable.
Validate a random sample of SMILES from each shard before downstream screening.

Canvas Visualization

When a ZINC lookup returns, a result card appears on the canvas.

Card size: 350x300px
Border color: Blue-grey (#607D8B)
Visualization: Compound grid with vendor availability indicators

Tier 2 Metrics

Compounds found count
Average molecular weight
Available (with vendor count)
Search time
Query string

Tanimoto Coefficient

The ZINC lookup itself finds analogs by edit distance, not Tanimoto. When you ask Ivy to filter results by Tanimoto similarity, it computes it with RDKit in the code workspace; read the threshold like this:

1.0: Identical fingerprints
>=0.7: Highly similar (recommended for analog finding)
0.5-0.7: Moderate similarity (scaffold hopping candidates)
<0.5: Low similarity (diverse chemical space)

Downloads & Batching