Group discussions improve reliability and validity of rated categories based on qualitative data from systematic review

20703 6 110177 2 333896 document 110177 journal.pone.0326166.pdf application/pdf 95e0cbd360df636a0aaffa1638c62c34 MD5 1630873 2025-06-23 14:32:13 https://pure.iiasa.ac.at/id/eprint/20703/1/journal.pone.0326166.pdf 20703 1 1 application/pdf text en public cc_by_4

journal.pone.0326166.pdf

published 110193 1 333917 document 110193 indexcodes.txt text/plain eae06c4241d10b99f70f6f2996b91b62 MD5 15597 2025-06-23 14:42:23 https://pure.iiasa.ac.at/id/eprint/20703/2/indexcodes.txt 20703 2 2 text/plain other Generate index codes conversion from text to indexcodes en public

indexcodes.txt

lightbox.jpg

preview.jpg

medium.jpg

small.jpg

http://eprints.org/relation/isVersionOf https://pure.iiasa.ac.at/id/document/110177 http://eprints.org/relation/isVolatileVersionOf https://pure.iiasa.ac.at/id/document/110177 http://eprints.org/relation/issmallThumbnailVersionOf https://pure.iiasa.ac.at/id/document/110177 archive 353 disk0/00/02/07/03 2025-06-23 14:33:21 2025-06-23 14:33:21 2025-06-23 14:33:21 article show Beher J. 2380 Treml E. Wintle B. Group discussions improve reliability and validity of rated categories based on qualitative data from systematic review pub BNR BEC The number of literature reviews in the fields of ecology and conservation has increased dramatically in recent years. Scientists conduct systematic literature reviews with the aim of drawing conclusions based on the content of a representative sample of publications. This requires subjective judgments on qualitative content, including interpretations and deductions. However, subjective judgments can differ substantially even between highly trained experts that are faced with the same evidence. Because classification of content into codes by one individual rater is prone to subjectivity and error, general guidelines recommend checking the produced data for consistency and reliability. Metrics on agreement between multiple people exist to assess the rate of agreement (consistency). These metrics do not account for mistakes or allow for their correction, while group discussions about codes that have been derived from classification of qualitative data have shown to improve reliability and accuracy. Here, we describe a pragmatic approach to reliability testing that gives insights into the error rate of multiple raters. Five independent raters rated and discussed categories for 23 variables within 21 peer-reviewed publications on conservation management plans. Mistakes, including overlooking information in the text, were the most common source of disagreement, followed by differences in interpretation and ambiguity around categories. Discussions could resolve most differences in ratings. We recommend our approach as a significant improvement on current review and synthesis approaches that lack assessment of misclassification. 2025-06-18 published Public Library of Science (PLoS) doi:10.1371/journal.pone.0326166 3456 public PLoS ONE 20 6 e0326166 1 TRUE 1932-6203 Koch Frank H. info:eu-repo/semantics/article