Job Failures & Stuck Runs

Job status values

Every job passes through a defined lifecycle. Understanding these statuses helps you diagnose where things went wrong.

Older jobs and clients may show the legacy spellings RUNNING and FAILED. Jobs only move forward through this lifecycle, so a failed job can't become completed. To re-run one, submit it again (see Resubmitting jobs below). The full status list is in Jobs.

Job stuck in "pending"

A job that stays in PENDING for longer than expected is usually waiting for GPU resources, not broken.

Expected wait times by tool tier:

Common causes and fixes:

• Modal cold starts. If no warm GPU worker exists, Modal provisions a new container. This adds roughly 30 seconds on top of normal queue time. Cold starts are most common during off-peak hours or after platform updates.
• GPU queue congestion. During peak hours, GPU workers may all be occupied. The job will start as soon as a worker becomes available. You do not need to resubmit.
• Concurrent-job limit reached. Each plan caps how many jobs run at once, and new jobs queue behind the running ones. The limit for each plan is on Pricing.
• Large input payload. Jobs with very large input files (100MB+ PDB files, large SDF libraries) take longer to upload to the worker before computation begins. The upload happens during the PENDING phase.
• Region-specific delays. GPU availability varies by region and time of day. If you are submitting during US business hours (9am-6pm ET), expect slightly longer queues.

What to do: Wait at least 5 minutes before assuming a job is stuck. If it exceeds the maximum expected time for its tier, check platform health (curl https://www.ivybiosciences.com/api/health). If every check is healthy, cancel and resubmit.

Monitoring PENDING jobs:

• ivybloom jobs follow JOB_ID shows live status, phase and logs in your terminal.
• The job's canvas card shows its current status while it waits.

Job failed with error code

When a job fails, the error message often includes a numeric harness error code. Here is what each code means and how to respond.

Other common error patterns:

• "Out of memory" in logs: Your input exceeded the GPU memory of the assigned worker. Split into smaller batches or reduce the input size. Common with large protein complexes (>5000 residues) or batch docking of thousands of compounds.
• "Connection reset" or "Worker terminated": The Modal container was preempted or crashed. Resubmit the job; it will be assigned a new worker. This is typically a transient issue.
• "Rate limit exceeded": You've submitted too many requests in a short window. Wait for the Retry-After interval and retry. This is separate from your plan's concurrent-job limit.
• "Invalid input format": The file you uploaded does not match the expected format for the selected tool. Double-check the tool's documentation for accepted formats.
• "Dependency unavailable": An external service that the tool depends on (e.g., a public database) was temporarily unreachable. Retry after a few minutes.

Reading job logs:

1. Run ivybloom jobs follow JOB_ID to stream the job's status and logs, or ivybloom jobs status JOB_ID for a one-off check.
2. Look for error lines. They carry the actual failure message, and warnings can explain degraded performance.
3. Copy the log output into your support ticket.

The job detail page (/jobs/[jobId]) shows the job's inputs, outputs and manifest.

Job completed but results are empty

A COMPLETED status with no visible results usually means the tool ran successfully but produced no output for your specific input.

Common causes:

• Sequence too short. Protein folding tools (ESMFold, OpenFold) require a minimum sequence length, typically at least 16 residues. Shorter peptides may produce empty or degenerate structures.
• Sequence too long. Some tools cap input length (typically up to roughly 2500 residues). Longer sequences are silently truncated or may produce no output.
• No hits found. Similarity searches, BLAST, or substructure queries return empty when no match exceeds the confidence threshold. Try lowering the threshold or broadening the query.
• Invalid SMILES. If you pass a malformed SMILES string to a chemistry tool, the tool may parse it as empty input rather than erroring. Validate your SMILES using the built-in SMILES validator or an external tool like the RDKit SMILES parser.
• Wrong file format. Uploading a .mol2 file to a tool expecting .sdf may parse without error but yield zero molecules.
• Filter too strict. Some tools have built-in quality filters. For example, docking tools may discard poses with scores worse than a threshold. If all poses are below the threshold, the result set is empty.

What to do: Open the job detail page and check the raw output JSON. If the result object exists but contains empty arrays, the tool ran but found nothing. Adjust your inputs and resubmit.

Job results look wrong

When results exist but seem incorrect or unexpected, the issue is usually parameter misconfiguration or misinterpretation of output scores.

Parameter misconfiguration:

• Docking box too large or too small. A search box that does not cover the binding site will produce poor poses. Use the recommended auto-detection or define the box around known binding residues. A box that is too large wastes computation and reduces accuracy.
• Wrong protonation state. Molecular docking and ADMET tools are sensitive to pH. Check the pH the tool assumed (physiological pH, 7.4, is the usual default). Acidic or basic compounds may need explicit protonation.
• Incorrect chain selection. For a multi-chain PDB file, specify the chain to dock against. If you don't, the tool may pick the first chain, which might not be the biologically relevant one.
• Exhaustiveness too low. For docking tools, low exhaustiveness (search effort) values produce faster but less accurate results. Increase exhaustiveness for production runs.

Interpreting confidence scores:

• pLDDT (protein folding): Values above 70 are generally reliable. Below 50 indicates disordered regions where the prediction is uncertain. Values between 50 and 70 should be treated with caution.
• Docking scores (kcal/mol): More negative values indicate stronger predicted binding. Scores above -4 kcal/mol are generally considered weak. Scores below -8 kcal/mol suggest strong binding. These are predictions, not experimental measurements.
• ADMET predictions: These are model scores, not measurements. A toxicity score of 0.6 ranks the compound as more likely toxic than one at 0.3, but it isn't a calibrated 60% probability or a definitive classification. Compare several ADMET models and look for agreement.
• QED (drug-likeness): Ranges from 0 to 1. Above 0.5 is considered drug-like. Values below 0.3 suggest the compound is unlikely to be an oral drug.
• BLAST E-values: Lower is better. E-values below 1e-5 are generally significant. Values above 0.01 should be treated with skepticism.

Resubmitting jobs

Not every failure warrants a simple retry. Here is when to retry versus when to adjust.

Retry without changes when:

• The error was a timeout or worker termination (infrastructure issue).
• A platform health check was failing and is now healthy again.
• The error code was a transient network or rate limit error.

Adjust parameters when:

• Validation failed (error 1002) -- fix the flagged input field.
• Execution failed (error 1003) -- review logs for the specific runtime error.
• Results were empty or wrong -- revisit input format and parameters.

How to resubmit: In the jobs panel of your project workspace, use Retry on the failed job. This creates a new job with the same parameters. To change parameters, run ivybloom jobs clone JOB_ID with your overrides, or call POST /api/v1/jobs/{jobId}/resubmit.

Credit implications: Retrying a job consumes credits again. The original failed job's credits are not automatically refunded unless the failure was caused by a platform issue. See the Billing troubleshooting page for refund details.

Batch job failures

Batch jobs submit multiple items (compounds, sequences) as a single unit. When a batch partially fails:

• Check the results summary. ivybloom batch status BATCH_ID shows how many items succeeded and how many failed, with per-item errors.
• Partial results are kept. Successful items aren't discarded when others fail. Download them with ivybloom batch results JOB_ID.
• Resubmit only failures. Resubmit the failed items as a new, smaller batch so the completed ones don't run again.
• Credit handling. Each item that runs is charged as its own job.

Common batch-specific issues:

• Mixed formats in a CSV. If one row has an invalid SMILES but others are valid, the invalid row fails while the rest proceed. The per-item error will indicate which row had the problem.
• Batch size limits. POST /api/v1/jobs/batch accepts up to 100 jobs per request. Split larger sets into several batches.
• Timeout scaling. Batch timeout is per-item, not total. A batch of 100 docking jobs does not need to complete in the same time as a single job.
• Memory scaling. Very large batches (500+ items) may exceed worker memory. If a batch fails with "out of memory," split it into smaller batches of 100-200 items.

Cancelling jobs

You can cancel a PENDING or RUNNING job at any time.

• From the UI: use Cancel on the job in your project workspace's jobs panel.
• From the CLI: run ivybloom jobs cancel JOB_ID.
• Cancellation is not instant. The signal goes to the worker, which stops at its next check.

Job notifications

A job's status shows in three places without a refresh: the jobs panel in your project workspace, the chat transcript, and the job's canvas card.

• Terminal. ivybloom jobs follow JOB_ID streams one job's status and logs, and ivybloom jobs watch watches your jobs as they run.
• Automation. ivybloom jobs stream JOB_ID --agent prints the job's events as JSON lines for scripts and agents.
• Email. The platform doesn't send job emails. Use one of the options above for long GPU jobs.

Viewing job history

Your past jobs are listed in the jobs panel of each project workspace, and from the terminal with ivybloom jobs list.

• Filtering. Filter by status, tool name, project, or date range.
• Bulk actions. Select several jobs to archive, cancel, delete or retry them together.
• Search. ivybloom jobs search finds jobs by name, tool or input.

Include in support tickets

If you need to contact support about a job issue, include:

• Job ID(s) and tool name
• The exact error message or error code
• Your input file format and approximate size
• Steps to reproduce the issue
• Your plan (Seed, Grow, Flourish, Thrive, Team or Enterprise)
• Screenshots of the job detail page showing the error
• Job logs (from ivybloom jobs follow JOB_ID)