A Tool for Selecting and Writing Texts for Assessing Reading: Reader-Friendly Informative Text Rubric (ODBIMDEPA)

Abstract

This study focuses on the rubrics used for text evaluation, with the aim of introducing the Reader-Friendly Informative Text Rubric (ODBIMDEPA). This rubric, developed and validated through a reliability study, evaluates the reader-friendliness of informative texts used both in educational settings and in exams administered by institutions such as the Ministry of National Education (MoNE), the Assessment, Selection and Placement Center (ÖSYM), and the Yunus Emre Institute (YEE). The study also seeks to share insights gained from the analysis of evaluated texts. To achieve these goals, the document review method was employed, resulting in the creation of two distinct datasets. The first dataset, analyzed during the initial stage of the study, comprised 50 informative texts taken from the 5th, 6th, 7th, and 8th-grade textbooks used in MoNE schools in 2018. The second dataset, analyzed in the subsequent stage, included 37 informative texts selected from the YKS (University Entrance) exams administered by ÖSYM in 2023 and 2024. In the first stage, two independent academic raters scored the texts, while in the second stage, 11 expert raters for 4 subjects, 13 raters for 1 subject assessed them across eight dimensions defined by ODBIMDEPA. In the initial analysis, the weighted kappa coefficient was used to evaluate inter-rater reliability, indicating that the weighted average was acceptable and demonstrating that ODBIMDEPA exhibited significant consistency. In the second stage, the dataset was analyzed using the many-facet Rasch model, revealing a rater reliability of 0.95. This high reliability underscores the robustness of the developed rubric. Based on these findings, ODBIMDEPA provides a reliable framework for quantifying the reader-friendliness of informative texts used in assessments administered by institutions such as MoNE and ÖSYM, as well as those in educational textbooks. The rubric facilitates a quantitative assessment of whether these texts are suitable for use in exams or educational settings, offering insights into text selection and evaluation.