Structured Multi-extraction Prompting Improves {LLM}-Based Biomedical Relation Extraction Without Fine-Tuning

Annotated text data is essential for structuring biomedical knowledge. State-of-the-art machine learning models achieve F1 scores >90\% for named entity recognition (NER), but relation extraction (RE) remains challenging, with F1 scores of 60\% (up to 75–80\% on binary datasets using additional strategies). These models require fine-tuning on costly manually annotated datasets.

This paper investigates structured multi-extraction prompting as a technique for enabling Large Language Models (LLMs) such as Mistral and OpenAI models to perform biomedical relation extraction (RE) without fine-tuning, reducing manual annotation burdens. We evaluate this approach across zero-shot and few-shot settings and compare extracting relations individually versus in batches, assessing whether multi-extraction with structured generation can serve as a viable, cost-effective alternative to fine-tuned models.

Our results show that multi-extraction prompting with structured generation significantly boosts LLM performance on biomedical relation extraction (RE) tasks, roughly matching fine-tuned models on binary classification in zero-shot settings and approaching their performance on multi-class tasks with few-shot examples but the gap remains larger in those cases. By providing the model with a list of entities and leveraging structured, dynamic generation, we harness LLMs’ in-context learning to enhance both efficiency and accuracy. This method improves performance by up to 19\% while reducing token usage by up to 550\%, lowering computational costs comparing it to a single extraction baseline.

In summary, structured multi-extraction prompting enables modern LLMs to perform binary RE tasks without fine-tuning, matching or approaching the performance of small specialised models, while substantially reducing token usage and computational cost—making them viable annotators for low-resource scenarios where little to no training data exists.