---
title: Downloads & Batching
description: 'Pull large ZINC exports in 50k-100k pages, stream each shard to disk, merge and gzip them, and record query parameters and shard counts in a metadata.json.'
---

This page shows how to pull large [ZINC](/docs/zinc) exports without exhausting memory or storage — chunking requests into pages, streaming to disk, merging shards, and recording provenance. Requesting smaller pages and streaming responses is what most affects whether a bulk download completes reliably. These scripts pull straight from ZINC's CartBlanche22 service: ask Ivy to run them in the chat's [code workspace](/docs/sandbox), where files land under `/workspace/scratch/`, or run them on your own machine.

## Chunk large requests

- Request smaller pages (e.g., 50k–100k compounds per call).
- Stream responses to disk to avoid high memory usage.

```python
import requests

def stream_chunk(offset: int, limit: int) -> None:
    params = {"subset": "in-stock", "count": limit, "start": offset}
    with requests.get("https://cartblanche22.docking.org/substances.txt", params=params, stream=True, timeout=120) as r:
        r.raise_for_status()
        outfile = f"/workspace/scratch/zinc_{offset}_{offset+limit}.csv"
        with open(outfile, "w", encoding="utf-8") as f:
            for chunk in r.iter_content(chunk_size=8192):
                if chunk:
                    f.write(chunk.decode("utf-8"))

for start in range(0, 200000, 50000):
    stream_chunk(start, 50000)
```

## Merge shards

```bash
cat /workspace/scratch/zinc_*.csv > /workspace/scratch/zinc_full.csv
```

## Provenance

- Store query parameters in `/workspace/scratch/metadata.json`.
- Record counts per shard and final merged count.
- Keep the original, unfiltered shards alongside filtered outputs.

## Storage tips

- Compress CSVs after filtering: `gzip zinc_full.csv`.
- If producing SDF, chunk by 10k–20k molecules to keep files manageable.
- Validate a random sample of SMILES from each shard before downstream
  screening.

## Canvas Visualization

When a ZINC lookup returns, a result card appears on the canvas.

- **Card size**: 350x300px
- **Border color**: Blue-grey (#607D8B)
- **Visualization**: Compound grid with vendor availability indicators

### Tier 2 Metrics

- Compounds found count
- Average molecular weight
- Available (with vendor count)
- Search time
- Query string

### Tanimoto Coefficient

The ZINC lookup itself finds analogs by edit distance, not Tanimoto. When you ask Ivy to filter results by Tanimoto similarity, it computes it with RDKit in the code workspace; read the threshold like this:
- **1.0**: Identical fingerprints
- **>=0.7**: Highly similar (recommended for analog finding)
- **0.5-0.7**: Moderate similarity (scaffold hopping candidates)
- **<0.5**: Low similarity (diverse chemical space)