RL Loops & Curriculum

This page configures how REINVENT steers generation over time — seeds and reproducibility, diversity filters, the experience-replay buffer, and multi-stage curriculum. Your curriculum design and diversity settings most affect whether the run converges toward high-scoring, novel molecules or collapses onto repeated scaffolds.

Reinforcement Learning Training in REINVENT 4

REINVENT 4, developed by AstraZeneca and published in the Journal of Cheminformatics (2024), uses reinforcement learning to fine-tune generative models (RNN or transformer architectures) toward desired molecular properties. The RL loop iteratively samples molecules from the agent, scores them, and updates the model to favor high-scoring structures.

For implementation details, see the REINVENT 4 GitHub repository (Apache 2.0 license).

Seeds and reproducibility

• Set random_seed at the top of your config.
• Pin prior/agent checkpoints and record their paths in job metadata.

Diversity filters

Common choices:

• IdenticalMurckoScaffold: penalize repeating scaffolds
• NoFilter: disable filtering for small tests
• IdenticalTopologicalScaffold: stricter topology constraints

Key params: minsimilarity, minscore, bucket_size.

Memory buffer

• replay_buffer_size: number of sequences to keep for experience replay.
• For small GPUs, lower batch_size and replay_buffer_size together.

Curriculum example

reinforcement_learning:
  random_seed: 42
  curriculum:
    - name: warmup
      n_steps: 25
      scoring_function:
        name: qed
        weight: 1.0
    - name: main
      n_steps: 75
      scoring_function:
        name: custom
        parameters: [] # see scoring page

Curriculum stages let you start with simple objectives (e.g., QED) and transition to multi-term scores.

Monitoring

• Log mean total score, validity %, and uniqueness per step.
• Track top-N SMILES per epoch; persist to /workspace/outputs/top.csv.
• If validity drops, relax filters or lower learning rate.

REINVENT 4 running modes

REINVENT 4 supports multiple generation modes, each with specific RL loop configurations:

• Reinforcement learning: Standard RL optimization against scoring functions
• Transfer learning: Fine-tune prior models on focused datasets before RL
• Sampling: Generate molecules without optimization (for baseline analysis)
• Curriculum learning: Multi-stage training with progressively complex objectives

Architecture considerations

REINVENT 4 offers both RNN and transformer-based generators:

• RNN models: Lower memory footprint, faster training per step
• Transformer models: Better long-range dependencies, improved scaffold control

Select the architecture based on your computational resources and design requirements (de novo design, scaffold hopping, R-group replacement, or linker design).