{"id":"doi:10.3233/sw-243719","kind":"article","title":"Testing prompt engineering methods for knowledge extraction from text","authors":["Fina Polat","Ilaria Tiddi","Paul Groth"],"author_count":3,"year":2024,"published":"2024-09-20","venue":"Semantic Web","cited_by":22,"doi":"10.3233/sw-243719","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.3233/sw-243719","archive_url":"https://space-kitty.com/papers/doi:10.3233/sw-243719","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4402678047","pdf_url":"https://journals.sagepub.com/history/e9b90887-d8d4-4bad-9107-dcdbfcd9f67f/sw-243719.17864871.pdf","abstract":"The capabilities of Large Language Models (LLMs,) such as Mistral 7B, Llama 3, GPT-4, present a significant opportunity for knowledge extraction (KE) from text. However, LLMs’ context-sensitivity can hinder obtaining precise and task-aligned outcomes, thereby requiring prompt engineering. This study explores the efficacy of five prompt methods with different task demonstration strategies across 17 different prompt templates, utilizing a relation extraction dataset (RED-FM) with the aforementioned LLMs. To facilitate evaluation, we introduce a novel framework grounded in Wikidata’s ontology. The findings demonstrate that LLMs are capable of extracting a diverse array of facts from text. Notably, incorporating a simple instruction accompanied by a task demonstration – comprising three examples selected via a retrieval mechanism – significantly enhances performance across Mistral 7B, Llama 3, and GPT-4. The effectiveness of reasoning-oriented prompting methods such as Chain-of-Thought, Reasoning and Acting, while improved with task demonstrations, does not surpass alternative methods. This suggests that framing extraction as a reasoning task may not be necessary for KE. Notably, task demonstrations leveraging examples selected via retrieval mechanisms facilitate effective knowledge extraction across all tested prompting strategies and LLMs.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Topic Modeling"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Testing prompt engineering methods for knowledge extraction from text\" by Fina Polat, Ilaria Tiddi, Paul Groth, https://doi.org/10.3233/sw-243719, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified.","topics":[{"slug":"reasoning-models","name":"Reasoning Models & Test-Time Compute","domain_slug":"ai"}],"cited_in_dossiers":[]}