Transformers FAQ

These answers cover how to choose among the biomedical NLP models in the Transformers tool and what to expect from them: PubMedBERT versus BioBERT versus ChemBERTa, how they handle clinical text and long documents, inferring relationships from NER output, how accurate entity extraction is, and how text embeddings differ from ESM-3 protein embeddings.

Frequently asked questions

Which model should I use for biomedical NER?

PubMedBERT is the recommended default for PubMed abstract analysis. It was pre-trained exclusively on PubMed text with a domain-specific vocabulary, giving it the best performance on standard biomedical NER benchmarks. Use BioBERT if your text includes PMC full-text content or a broader mix of biomedical sources. Use ChemBERTa specifically for chemical entity recognition or chemical text analysis.

What is the difference between BioBERT and PubMedBERT?

Both are BERT-based models fine-tuned for biomedical text, but they differ in pre-training:

• BioBERT starts from the general BERT checkpoint and continues pre-training on PubMed and PMC articles. It retains some general English understanding.
• PubMedBERT is trained from scratch exclusively on PubMed abstracts with a domain-specific vocabulary. It typically achieves higher accuracy on PubMed-specific tasks but may perform worse on non-biomedical text.

Can these models handle clinical notes or electronic health records?

The available models are optimized for biomedical research text (abstracts, papers). Clinical notes have different language patterns (abbreviations, sentence fragments, misspellings) that these models handle less well. For clinical NLP, specialized models trained on clinical text (e.g., ClinicalBERT) would be more appropriate. The current Transformers tool focuses on research literature mining.

How long can my input text be?

BERT-based models accept up to 512 tokens (~300-400 words) per input. Longer documents are automatically split into overlapping chunks of 512 tokens with 50 tokens of overlap between adjacent chunks. Entity extraction works across chunks, but entities spanning a chunk boundary may occasionally be missed. For best results with long documents, keep paragraphs focused on single topics.

Can I run NER and relation extraction in the same job?

There is no separate relation-extraction task — task accepts embeddings, ner, classification, zero_shot, qa, or summarization, one per job. Run NER to get entity spans, then infer relationships from co-occurring entities, or ask a targeted question with qa (e.g., "what does gefitinib inhibit?") against the same text.

How accurate is entity extraction?

On standard biomedical NER benchmarks, PubMedBERT achieves F1 scores of 85-93% depending on the entity type and dataset. Drug and disease entities tend to have higher accuracy than gene entities, which are often ambiguous with protein names. In practice, accuracy depends on how similar your text is to the training data. Well-written PubMed abstracts yield the best results.

Can I fine-tune models on my own labeled data?

The current Transformers tool uses pre-trained models and does not support custom fine-tuning through the webapp interface. If you need models tuned for specific entity types or relation categories not covered by the defaults, contact support for custom model training options.

What is the difference between text embeddings from Transformers and protein embeddings from ESM-3?

These are fundamentally different representations:

• Transformers embeddings: Encode the meaning of natural language text (abstracts, documents). Useful for document similarity, search, and classification.
• ESM-3 embeddings: Encode the biological properties of protein sequences. Useful for protein similarity, function prediction, and clustering.

They operate on different input types and produce embeddings in different vector spaces. However, both can be used as features in downstream ML models.

Read next