Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment

GRAY, ANDREW

doi:10.23889/SUThesis.71898

E-Thesis 29 views 7 downloads

Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment / ANDREW GRAY

Swansea University Author: ANDREW GRAY

PDF | E-Thesis – open access

Copyright: the author, Andy Gray, 2026. Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).
Download (10.73MB)

DOI (Published version): 10.23889/SUThesis.71898

Abstract

Assessment is a cornerstone of education, yet traditional marking methods can be inconsistent, biased, and cognitively demanding for educators. Comparative Judgement (CJ)oﬀers an alternative by ranking student work through pairwise comparisons but faces challenges around transparency, eﬃciency, and...

Full description

Published:	2026
Institution:	Swansea University
Degree level:	Doctoral
Degree name:	Ph.D
Supervisor:	Rahat, A., Crick, T., Pearson, J., and Lindsay, S.
URI:	https://cronfa.swan.ac.uk/Record/cronfa71898

first_indexed	2026-05-14T13:40:21Z
last_indexed	2026-05-15T05:41:59Z
id	cronfa71898
recordtype	RisThesis
fullrecord	<?xml version="1.0"?><rfc1807><datestamp>2026-05-14T14:52:18.0613692</datestamp><bib-version>v2</bib-version><id>71898</id><entry>2026-05-14</entry><title>Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment</title><swanseaauthors><author><sid>70200df2c37f5d14360e8660d93f57a8</sid><firstname>ANDREW</firstname><surname>GRAY</surname><name>ANDREW GRAY</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2026-05-14</date><abstract>Assessment is a cornerstone of education, yet traditional marking methods can be inconsistent, biased, and cognitively demanding for educators. Comparative Judgement (CJ)oﬀers an alternative by ranking student work through pairwise comparisons but faces challenges around transparency, eﬃciency, and biases in pair selection. This thesis pro-poses Bayesian Comparative Judgement (BCJ) as a structured, data-driven alternative to address these limitations.BCJ integrates entropy-driven active learning to select the most informative comparisons, systematically improving ranking accuracy while avoiding the model deterioration seen in traditional CJ. By generating complete predictive rank distributions, BCJ also enables probabilistic grading aligned with assessors’ decisions. Experiments using syn-thetic and real-world datasets, including GCSE essays, demonstrate BCJ’s strong performance and eﬃciency against existing CJ methods. BCJ also introduces methods to quantify assessor agreement, reinforcing reliability and accountabilityTo enhance performance insights, the thesis extends BCJ to a multi-criteria frame-work (MBCJ), aligning with rubric-based assessment by evaluating individual learning outcomes (LOs) alongside overall performance. This approach preserves CJ’s eﬃciency while providing granular, criterion-speciﬁc insights.The practical implementation of BCJ and MBCJ is evaluated through studies with professional markers in higher education, focusing on transparency, usability, and trust.Findings show that while traditional marking is familiar, BCJ and MBCJ reduce subjectivity, improve alignment with target rankings, and maintain educational integrity, oﬀering a viable alternative for assessment.Overall, this research demonstrates how Bayesian methods can reﬁne CJ to enhance transparency, fairness, and eﬃciency in assessment. The thesis supports adopting structured CJ methods to improve feedback and reduce educator workload, contributing to fairer and more transparent assessment practices across diverse educational contexts.</abstract><type>E-Thesis</type><journal/><volume/><journalNumber/><paginationStart/><paginationEnd/><publisher/><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint/><issnElectronic/><keywords>Artificial Intelligence, Machine Learning, Active Learning, Comparative Judgement, Education, Assessment, Transparency, Bayesian Statistics.</keywords><publishedDay>25</publishedDay><publishedMonth>3</publishedMonth><publishedYear>2026</publishedYear><publishedDate>2026-03-25</publishedDate><doi>10.23889/SUThesis.71898</doi><url/><notes/><college>COLLEGE NANME</college><CollegeCode>COLLEGE CODE</CollegeCode><institution>Swansea University</institution><supervisor>Rahat, A., Crick, T., Pearson, J., and Lindsay, S.</supervisor><degreelevel>Doctoral</degreelevel><degreename>Ph.D</degreename><degreesponsorsfunders>EPSRC Enhancing Human Interactions and Collaborations with Data and Intelligence-Driven Systems</degreesponsorsfunders><apcterm/><funders>EPSRC Enhancing Human Interactions and Collaborations with Data and Intelligence-Driven Systems</funders><projectreference/><lastEdited>2026-05-14T14:52:18.0613692</lastEdited><Created>2026-05-14T14:16:03.1112166</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>ANDREW</firstname><surname>GRAY</surname><order>1</order></author></authors><documents><document><filename>71898__36730__76636d07e8b04878aa93036aa82f0da1.pdf</filename><originalFilename>2026_Gray_A.final.71898.pdf</originalFilename><uploaded>2026-05-14T14:35:18.2839406</uploaded><type>Output</type><contentLength>11255427</contentLength><contentType>application/pdf</contentType><version>E-Thesis – open access</version><cronfaStatus>true</cronfaStatus><documentNotes>Copyright: the author, Andy Gray, 2026. Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>https://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling	2026-05-14T14:52:18.0613692 v2 71898 2026-05-14 Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment 70200df2c37f5d14360e8660d93f57a8 ANDREW GRAY ANDREW GRAY true false 2026-05-14 Assessment is a cornerstone of education, yet traditional marking methods can be inconsistent, biased, and cognitively demanding for educators. Comparative Judgement (CJ)oﬀers an alternative by ranking student work through pairwise comparisons but faces challenges around transparency, eﬃciency, and biases in pair selection. This thesis pro-poses Bayesian Comparative Judgement (BCJ) as a structured, data-driven alternative to address these limitations.BCJ integrates entropy-driven active learning to select the most informative comparisons, systematically improving ranking accuracy while avoiding the model deterioration seen in traditional CJ. By generating complete predictive rank distributions, BCJ also enables probabilistic grading aligned with assessors’ decisions. Experiments using syn-thetic and real-world datasets, including GCSE essays, demonstrate BCJ’s strong performance and eﬃciency against existing CJ methods. BCJ also introduces methods to quantify assessor agreement, reinforcing reliability and accountabilityTo enhance performance insights, the thesis extends BCJ to a multi-criteria frame-work (MBCJ), aligning with rubric-based assessment by evaluating individual learning outcomes (LOs) alongside overall performance. This approach preserves CJ’s eﬃciency while providing granular, criterion-speciﬁc insights.The practical implementation of BCJ and MBCJ is evaluated through studies with professional markers in higher education, focusing on transparency, usability, and trust.Findings show that while traditional marking is familiar, BCJ and MBCJ reduce subjectivity, improve alignment with target rankings, and maintain educational integrity, oﬀering a viable alternative for assessment.Overall, this research demonstrates how Bayesian methods can reﬁne CJ to enhance transparency, fairness, and eﬃciency in assessment. The thesis supports adopting structured CJ methods to improve feedback and reduce educator workload, contributing to fairer and more transparent assessment practices across diverse educational contexts. E-Thesis Artificial Intelligence, Machine Learning, Active Learning, Comparative Judgement, Education, Assessment, Transparency, Bayesian Statistics. 25 3 2026 2026-03-25 10.23889/SUThesis.71898 COLLEGE NANME COLLEGE CODE Swansea University Rahat, A., Crick, T., Pearson, J., and Lindsay, S. Doctoral Ph.D EPSRC Enhancing Human Interactions and Collaborations with Data and Intelligence-Driven Systems EPSRC Enhancing Human Interactions and Collaborations with Data and Intelligence-Driven Systems 2026-05-14T14:52:18.0613692 2026-05-14T14:16:03.1112166 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science ANDREW GRAY 1 71898__36730__76636d07e8b04878aa93036aa82f0da1.pdf 2026_Gray_A.final.71898.pdf 2026-05-14T14:35:18.2839406 Output 11255427 application/pdf E-Thesis – open access true Copyright: the author, Andy Gray, 2026. Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0). true eng https://creativecommons.org/licenses/by/4.0/
title	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
spellingShingle	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment ANDREW GRAY
title_short	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
title_full	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
title_fullStr	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
title_full_unstemmed	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
title_sort	Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment
author_id_str_mv	70200df2c37f5d14360e8660d93f57a8
author_id_fullname_str_mv	70200df2c37f5d14360e8660d93f57a8_***_ANDREW GRAY
author	ANDREW GRAY
author2	ANDREW GRAY
format	E-Thesis
publishDate	2026
institution	Swansea University
doi_str_mv	10.23889/SUThesis.71898
college_str	Faculty of Science and Engineering
hierarchytype
hierarchy_top_id	facultyofscienceandengineering
hierarchy_top_title	Faculty of Science and Engineering
hierarchy_parent_id	facultyofscienceandengineering
hierarchy_parent_title	Faculty of Science and Engineering
department_str	School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science
document_store_str	1
active_str	0
description	Assessment is a cornerstone of education, yet traditional marking methods can be inconsistent, biased, and cognitively demanding for educators. Comparative Judgement (CJ)oﬀers an alternative by ranking student work through pairwise comparisons but faces challenges around transparency, eﬃciency, and biases in pair selection. This thesis pro-poses Bayesian Comparative Judgement (BCJ) as a structured, data-driven alternative to address these limitations.BCJ integrates entropy-driven active learning to select the most informative comparisons, systematically improving ranking accuracy while avoiding the model deterioration seen in traditional CJ. By generating complete predictive rank distributions, BCJ also enables probabilistic grading aligned with assessors’ decisions. Experiments using syn-thetic and real-world datasets, including GCSE essays, demonstrate BCJ’s strong performance and eﬃciency against existing CJ methods. BCJ also introduces methods to quantify assessor agreement, reinforcing reliability and accountabilityTo enhance performance insights, the thesis extends BCJ to a multi-criteria frame-work (MBCJ), aligning with rubric-based assessment by evaluating individual learning outcomes (LOs) alongside overall performance. This approach preserves CJ’s eﬃciency while providing granular, criterion-speciﬁc insights.The practical implementation of BCJ and MBCJ is evaluated through studies with professional markers in higher education, focusing on transparency, usability, and trust.Findings show that while traditional marking is familiar, BCJ and MBCJ reduce subjectivity, improve alignment with target rankings, and maintain educational integrity, oﬀering a viable alternative for assessment.Overall, this research demonstrates how Bayesian methods can reﬁne CJ to enhance transparency, fairness, and eﬃciency in assessment. The thesis supports adopting structured CJ methods to improve feedback and reduce educator workload, contributing to fairer and more transparent assessment practices across diverse educational contexts.
published_date	2026-03-25T06:41:59Z
_version_	1865231860415594496
score	11.106

Bayesian Active Learning for Comparative Judgement: A New Paradigm for Educational Assessment / ANDREW GRAY

Similar Items