---
title: Batch Processing
description: >-
  Submit, monitor, and manage batch job collections with parallel execution for
  high-throughput screening.
tags:
  - batch
  - parallel
  - screening
  - automation
---

  intermediate
  Authenticated CLI (`ivybloom auth login`), Familiarity with job submission (`ivybloom run`)

The ivybloom CLI is the scriptable surface for the platform, and batch processing is how you submit, monitor, and manage whole collections of jobs with parallel execution. Reach for `batch` instead of individual `run` commands when a script needs high-throughput screening — running many jobs at once and managing them as a single unit you can pause, resume, or cancel. It's the workhorse for reproducible, large-scale pipelines; for the full command surface, see the [CLI overview](/docs/cli).

## Batch Processing

Submit, monitor, and manage batch job collections with parallel execution for high-throughput screening.

> `batch` is a hidden command: `ivybloom batch` runs it, but `ivybloom --help` does not list it. Run `ivybloom batch --help` to see its subcommands.

---

## When to Use Batch

Use `batch` instead of individual `run` commands when you need to:

- Screen a compound library through ADMET prediction
- Dock multiple ligands against a protein target
- Fold a family of related protein sequences
- Run any tool across many inputs with parallel execution

A single `batch submit` replaces dozens of manual `run` invocations with built-in parallelism, checkpointing, and result aggregation.

---

## Batch File Format

Batch jobs are defined in YAML or JSON files. Here is the full schema:

```yaml
name: "ADMET Screen"
description: "Screen drug candidates for ADMET properties"
tool: admet                          # Default tool for all jobs
variables:                           # ${var} substitution
  target_ph: "7.4"
  project: "proj_abc123"
defaults:                            # Shared params for all jobs
  timeout: 3600
jobs:
  - name: "aspirin"
    parameters: { smiles: "CC(=O)Oc1ccccc1C(=O)O" }
  - name: "ibuprofen"
    parameters: { smiles: "CC(C)Cc1ccc(cc1)C(C)C(=O)O" }
  - name: "caffeine"
    parameters: { smiles: "Cn1c(=O)c2c(ncn2C)n(C)c1=O" }
options:
  parallel: 4                        # Max concurrent workers (default: 3)
  stop_on_error: false               # Continue on individual failures
  retry_failed: 2                    # Retry failed jobs up to N times
  checkpoint: true                   # Enable checkpointing for resume
  output_dir: "./results"            # Output directory for artifacts
```

### Variable Substitution

Use `${var}` syntax in parameters to reference variables defined at the top level:

```yaml
variables:
  receptor: "1ABC"
  ph: "7.4"
jobs:
  - name: "dock-aspirin"
    parameters:
      receptor_pdb: "${receptor}"
      ph: "${ph}"
      ligand_smiles: "CC(=O)Oc1ccccc1C(=O)O"
```

Variables can be overridden at submit time with the `--var` flag.

### Per-Job Tool Overrides

Each job can override the default `tool`:

```yaml
tool: admet
jobs:
  - name: "fold-first"
    tool: esmfold                    # Override default tool
    parameters: { protein_sequence: "MKLVFF..." }
  - name: "admet-aspirin"
    parameters: { smiles: "CC(=O)Oc1ccccc1C(=O)O" }
```

---

## Submitting Batches

```bash
# Basic submission
ivybloom batch submit screen.yaml

# With parallelism override
ivybloom batch submit screen.yaml --parallel 8

# Wait for all jobs to complete
ivybloom batch submit screen.yaml --wait

# Dry-run: validate the batch file without submitting
ivybloom batch submit screen.yaml --dry-run

# Override variables at submit time
ivybloom batch submit screen.yaml --var target_ph=8.0 --var receptor=2XYZ
```

- `--parallel` (int) —
Maximum number of concurrent jobs (`-P` shorthand).

- `--wait` (bool) —
Block until all jobs complete (`-w` shorthand).

- `--dry-run` (bool) —
Validate the batch file and parameters without submitting (`-d` shorthand).

- `--var` (string) —
Override a variable: `--var key=value`. Can be repeated (`-v` shorthand).

---

## Monitoring

### Check Status

```bash
ivybloom batch status BATCH_ID
```

### Real-Time Monitor

The `monitor` subcommand provides a live TUI dashboard showing job progress:

```bash
# Real-time dashboard (default: poll every 3s)
ivybloom batch monitor BATCH_ID

# Custom poll interval
ivybloom batch monitor BATCH_ID --interval 5

# JSON stream output (for piping)
ivybloom batch monitor BATCH_ID --json
```

- `--interval` (int) —
Poll interval in seconds.

- `--json` (bool) —
Output JSON stream instead of TUI dashboard.

### Listing Batches

```bash
ivybloom batch list
ivybloom batch list --format json --limit 20
```

---

## Execution Control

### Pausing

```bash
# Pause execution (running jobs continue, no new jobs start)
ivybloom batch pause BATCH_ID

# Pause and cancel all pending jobs
ivybloom batch pause BATCH_ID --cancel-pending
```

### Resuming

```bash
# Resume a paused batch
ivybloom batch resume BATCH_ID

# Re-run all jobs (not just pending/failed)
ivybloom batch resume BATCH_ID --force-all

# Resume from a specific job index
ivybloom batch resume BATCH_ID --from-index 5

# Override variables on resume
ivybloom batch resume BATCH_ID --var target_ph=8.0
```

- `--force-all` (bool) —
Re-run all jobs, not just pending or failed ones.

- `--from-index` (int) —
Resume from a specific job index (0-based).

- `--var` (string) —
Override variables on resume.

### Cancelling

```bash
# Cancel specific jobs within a batch
ivybloom batch cancel JOB_ID1 JOB_ID2
```

---

## Collecting Results

### Individual Job Results

```bash
# Download results for specific jobs
ivybloom batch results JOB_ID1 JOB_ID2 --format json
ivybloom batch results JOB_ID1 --output-dir ./out/
```

- `--format` (string) —
Output format: `table`, `json`, `csv`, `yaml` (`-f` shorthand).

- `--output-dir` (string) —
Directory to save result files (`-o` shorthand).

### Aggregate Batch Results

The `collect` command aggregates results from all jobs in a batch:

```bash
# Collect all results as JSON
ivybloom batch collect BATCH_ID --output results.json

# CSV format, completed jobs only
ivybloom batch collect BATCH_ID --format csv --completed-only

# Filter by confidence threshold
ivybloom batch collect BATCH_ID --confidence 0.8

# Include in-progress partial results
ivybloom batch collect BATCH_ID --partial
```

**JSON**

```json
{
  "batch_id": "batch_abc123",
  "total": 3,
  "completed": 2,
  "failed": 0,
  "results": [
    {"job": "aspirin", "status": "completed", "score": 0.92},
    {"job": "ibuprofen", "status": "completed", "score": 0.87},
    {"job": "caffeine", "status": "running"}
  ]
}
```

**CSV**

```
job,status,score
aspirin,completed,0.92
ibuprofen,completed,0.87
caffeine,running,
```

- `--output` (string) —
Output file path (`-o` shorthand).

- `--format` (string) —
Output format: `json`, `csv` (`-f` shorthand).

- `--completed-only` (bool) —
Only include completed jobs.

- `--confidence` (float) —
Minimum confidence threshold for results (`-c` shorthand).

- `--partial` (bool) —
Include partial results from in-progress jobs.

---

## Batch Execution Flow

 Pool
    subgraph Pool["Job Pool (--parallel N)"]
      direction LR
      J1["J1<br/>running"]
      J2["J2<br/>running"]
      J3["J3<br/>pending"]
      J4["J4<br/>pending"]
    end
    Pool --> Mon["batch monitor (real-time)"]
    Pool --> PR["batch pause / resume"]
    Pool --> Col["batch collect (aggregate results)"]`}
  caption="Submitted jobs run in a pool of N parallel slots; monitor, pause/resume and collect operate on the batch."
/>

---

## CI/CD Integration

> For CI/CD automation, add `--agent` for structured JSON output. See [Agent Mode](/docs/cli/agent-mode) for the full envelope schema.

### GitHub Actions Example

```yaml
- name: Run ADMET screen
  run: |
    BATCH_ID=$(ivybloom batch submit screen.yaml --wait --agent | jq -r '.data.batch_id')
    ivybloom batch collect $BATCH_ID --format csv --output results.csv
  env:
    IVYBLOOM_API_KEY: ${{ secrets.IVYBLOOM_API_KEY }}
```

### Exit Codes

| Code | Meaning |
|------|---------|
| `0` | All jobs completed successfully |
| `1` | One or more jobs failed |

---

## Examples

### Screen 100 Molecules Through ADMET

```yaml
# admet-screen.yaml
name: "ADMET Library Screen"
tool: admet
options:
  parallel: 8
  checkpoint: true
jobs:
  - name: "mol-001"
    parameters: { smiles: "CC(=O)Oc1ccccc1C(=O)O" }
  - name: "mol-002"
    parameters: { smiles: "CC(C)Cc1ccc(cc1)C(C)C(=O)O" }
  # ... up to 100 entries
```

```bash
ivybloom batch submit admet-screen.yaml --wait
ivybloom batch collect BATCH_ID --format csv --output admet-results.csv
```

### Dock a Ligand Library Against a Target

```yaml
# docking-screen.yaml
name: "DiffDock Library Screen"
tool: diffdock
defaults:
  receptor_pdb_id: "1ABC"
options:
  parallel: 4
jobs:
  - name: "aspirin"
    parameters: { ligand_smiles: "CC(=O)Oc1ccccc1C(=O)O" }
  - name: "ibuprofen"
    parameters: { ligand_smiles: "CC(C)Cc1ccc(cc1)C(C)C(=O)O" }
```

### Fold a Protein Family

```yaml
# family-fold.yaml
name: "Kinase Family Fold"
tool: esmfold
options:
  parallel: 3
jobs:
  - name: "EGFR"
    parameters: { protein_sequence: "MRPSGTAGAALLALLAALCPA..." }
  - name: "BRAF"
    parameters: { protein_sequence: "MAALSGGGGGGAEPGQALFNG..." }
  - name: "ABL1"
    parameters: { protein_sequence: "MLEICLKLVGCKSKKGLSSSS..." }
```

---

## Read next

- [Job Management](/docs/cli/jobs) — Monitor individual jobs within a batch
- [Compounds](/docs/cli/compounds) — Build compound libraries for batch screening
- [Recipes](/docs/cli/recipes) — End-to-end workflow tutorials using batch
- [Agent Mode](/docs/cli/agent-mode) — Automate batch workflows in CI/CD