---
title: RL Loops & Curriculum
description: Steer a REINVENT run — random_seed, Murcko-scaffold diversity filters with minsimilarity and bucket_size, the replay buffer and curriculum stages.
---

This page configures how [REINVENT](/docs/reinvent) steers generation over time — seeds and reproducibility, diversity filters, the experience-replay buffer, and multi-stage curriculum. Your curriculum design and diversity settings most affect whether the run converges toward high-scoring, novel molecules or collapses onto repeated scaffolds.

## Reinforcement Learning Training in REINVENT 4

REINVENT 4, developed by AstraZeneca and published in the Journal of Cheminformatics (2024), uses reinforcement learning to fine-tune generative models (RNN or transformer architectures) toward desired molecular properties. The RL loop iteratively samples molecules from the agent, scores them, and updates the model to favor high-scoring structures.

For implementation details, see the [REINVENT 4 GitHub repository](https://github.com/MolecularAI/REINVENT4) (Apache 2.0 license).

## Seeds and reproducibility

- Set `random_seed` at the top of your config.
- Pin prior/agent checkpoints and record their paths in job metadata.

## Diversity filters

Common choices:

- `IdenticalMurckoScaffold`: penalize repeating scaffolds
- `NoFilter`: disable filtering for small tests
- `IdenticalTopologicalScaffold`: stricter topology constraints

Key params: `minsimilarity`, `minscore`, `bucket_size`.

## Memory buffer

- `replay_buffer_size`: number of sequences to keep for experience replay.
- For small GPUs, lower `batch_size` and `replay_buffer_size` together.

## Curriculum example

```yaml
reinforcement_learning:
  random_seed: 42
  curriculum:
    - name: warmup
      n_steps: 25
      scoring_function:
        name: qed
        weight: 1.0
    - name: main
      n_steps: 75
      scoring_function:
        name: custom
        parameters: [] # see scoring page
```

Curriculum stages let you start with simple objectives (e.g., QED) and
transition to multi-term scores.

## Monitoring

- Log mean total score, validity %, and uniqueness per step.
- Track top-N SMILES per epoch; persist to `/workspace/outputs/top.csv`.
- If validity drops, relax filters or lower learning rate.

## REINVENT 4 running modes

REINVENT 4 supports multiple generation modes, each with specific RL loop configurations:

- **Reinforcement learning**: Standard RL optimization against scoring functions
- **Transfer learning**: Fine-tune prior models on focused datasets before RL
- **Sampling**: Generate molecules without optimization (for baseline analysis)
- **Curriculum learning**: Multi-stage training with progressively complex objectives

## Architecture considerations

REINVENT 4 offers both RNN and transformer-based generators:

- **RNN models**: Lower memory footprint, faster training per step
- **Transformer models**: Better long-range dependencies, improved scaffold control

Select the architecture based on your computational resources and design requirements (de novo design, scaffold hopping, R-group replacement, or linker design).