# Testing prompt engineering methods for knowledge extraction from text

Article · Fina Polat, Ilaria Tiddi, Paul Groth

- Archive id: doi:10.3233/sw-243719
- Published: 2024-09-20
- Venue: Semantic Web
- DOI: https://doi.org/10.3233/sw-243719
- Citations: 22
- Source URL: https://doi.org/10.3233/sw-243719
- PDF: https://journals.sagepub.com/history/e9b90887-d8d4-4bad-9107-dcdbfcd9f67f/sw-243719.17864871.pdf
- Provenance: openalex

## Abstract

The capabilities of Large Language Models (LLMs,) such as Mistral 7B, Llama 3, GPT-4, present a significant opportunity for knowledge extraction (KE) from text. However, LLMs’ context-sensitivity can hinder obtaining precise and task-aligned outcomes, thereby requiring prompt engineering. This study explores the efficacy of five prompt methods with different task demonstration strategies across 17 different prompt templates, utilizing a relation extraction dataset (RED-FM) with the aforementioned LLMs. To facilitate evaluation, we introduce a novel framework grounded in Wikidata’s ontology. The findings demonstrate that LLMs are capable of extracting a diverse array of facts from text. Notably, incorporating a simple instruction accompanied by a task demonstration – comprising three examples selected via a retrieval mechanism – significantly enhances performance across Mistral 7B, Llama 3, and GPT-4. The effectiveness of reasoning-oriented prompting methods such as Chain-of-Thought, Reasoning and Acting, while improved with task demonstrations, does not surpass alternative methods. This suggests that framing extraction as a reasoning task may not be necessary for KE. Notably, task demonstrations leveraging examples selected via retrieval mechanisms facilitate effective knowledge extraction across all tested prompting strategies and LLMs.

_Abstract reproduced under CC-BY (publisher license, via OpenAlex)._

Topics: [Reasoning Models & Test-Time Compute](https://space-kitty.com/topics/reasoning-models)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.