No Cover Image

Journal article 236 views 54 downloads

Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system

Ashley Mattheis, Ashton Kingdon

Policy & Internet, Volume: 15, Issue: 4, Pages: 478 - 497

Swansea University Author: Ashley Mattheis

  • 65253.VOR.pdf

    PDF | Version of Record

    © 2023 The Authors. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

    Download (3.75MB)

Check full text

DOI (Published version): 10.1002/poi3.381

Abstract

Due to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or servi...

Full description

Published in: Policy & Internet
ISSN: 1944-2866 1944-2866
Published: Wiley 2023
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa65253
Tags: Add Tag
No Tags, Be the first to tag this record!
first_indexed 2023-12-08T17:05:52Z
last_indexed 2023-12-08T17:05:52Z
id cronfa65253
recordtype SURis
fullrecord <?xml version="1.0" encoding="utf-8"?><rfc1807 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><bib-version>v2</bib-version><id>65253</id><entry>2023-12-08</entry><title>Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system</title><swanseaauthors><author><sid>20bd641e721999fbea309db74f2d60c5</sid><firstname>Ashley</firstname><surname>Mattheis</surname><name>Ashley Mattheis</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2023-12-08</date><deptcode>ONDF</deptcode><abstract>Due to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or service. But technologically deterministic approaches are not infallible and can be predictable based on their actions. This opens the door for manipulation of media content and technological affordances to become tactical options for actors seeking to subvert regulation. Existing discussions often neglect the topic of manipulation of content, algorithms, or platform affordances as a primary aspect of the strategies used by extremists in relation to the difficulties of moderation from a policy perspective. This paper argues that it is essential to understand how extremists and conspiracy theorists use manipulation tactics to “game” the current policy, regulatory, and legislative systems of content moderation. Developing approaches that attend to manipulation as a strategy and focus on platform and context-specific tactics will generate more effective policies, platform rules, AI developments, and moderation procedures. This paper analyses and demystifies three primary tactics, which the authors categorise as numerology, borderlands, and merchandising, regularly used by extremists online in their strategies to ‘game’ content moderation. We provide case examples from a variety of ideologies including far-right, QAnon, and male supremacism to highlight the tactics rather than ideological nature of such manipulation. We conclude with a discussion of how demystification processes could be incorporated into content moderation settings. This paper contributes new insights about evasion tactics to the content moderation discussion and expands current understanding of how platforms can develop sociotechnical remedial measures.</abstract><type>Journal Article</type><journal>Policy &amp; Internet</journal><volume>15</volume><journalNumber>4</journalNumber><paginationStart>478</paginationStart><paginationEnd>497</paginationEnd><publisher>Wiley</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint>1944-2866</issnPrint><issnElectronic>1944-2866</issnElectronic><keywords>Artificial intelligence (AI), content moderation, digital culture, extremism, machine learning. manipulation tactics. propaganda. 内容审核 极端主义 人工智能(AI) 机器学习 操纵策略 （政治）宣传 数字文化Moderación de contenidos. extremismo. inteligencia artificial (IA). aprendizaje automático. tácticas de manipulación , propaganda, cultura digital</keywords><publishedDay>1</publishedDay><publishedMonth>12</publishedMonth><publishedYear>2023</publishedYear><publishedDate>2023-12-01</publishedDate><doi>10.1002/poi3.381</doi><url>http://dx.doi.org/10.1002/poi3.381</url><notes/><college>COLLEGE NANME</college><department>Other/Subsidiary Companies - Not Defined</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>ONDF</DepartmentCode><institution>Swansea University</institution><apcterm>SU Library paid the OA fee (TA Institutional Deal)</apcterm><funders/><projectreference/><lastEdited>2024-04-04T16:08:33.3350213</lastEdited><Created>2023-12-08T13:50:57.9386574</Created><path><level id="1">Faculty of Humanities and Social Sciences</level><level id="2">Hilary Rodham Clinton School of Law</level></path><authors><author><firstname>Ashley</firstname><surname>Mattheis</surname><order>1</order></author><author><firstname>Ashton</firstname><surname>Kingdon</surname><order>2</order></author></authors><documents><document><filename>65253__29240__33d5a81e29bf48caad7454759107f397.pdf</filename><originalFilename>65253.VOR.pdf</originalFilename><uploaded>2023-12-08T17:06:12.1880055</uploaded><type>Output</type><contentLength>3937305</contentLength><contentType>application/pdf</contentType><version>Version of Record</version><cronfaStatus>true</cronfaStatus><documentNotes>© 2023 The Authors. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>http://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807>
spelling v2 65253 2023-12-08 Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system 20bd641e721999fbea309db74f2d60c5 Ashley Mattheis Ashley Mattheis true false 2023-12-08 ONDF Due to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or service. But technologically deterministic approaches are not infallible and can be predictable based on their actions. This opens the door for manipulation of media content and technological affordances to become tactical options for actors seeking to subvert regulation. Existing discussions often neglect the topic of manipulation of content, algorithms, or platform affordances as a primary aspect of the strategies used by extremists in relation to the difficulties of moderation from a policy perspective. This paper argues that it is essential to understand how extremists and conspiracy theorists use manipulation tactics to “game” the current policy, regulatory, and legislative systems of content moderation. Developing approaches that attend to manipulation as a strategy and focus on platform and context-specific tactics will generate more effective policies, platform rules, AI developments, and moderation procedures. This paper analyses and demystifies three primary tactics, which the authors categorise as numerology, borderlands, and merchandising, regularly used by extremists online in their strategies to ‘game’ content moderation. We provide case examples from a variety of ideologies including far-right, QAnon, and male supremacism to highlight the tactics rather than ideological nature of such manipulation. We conclude with a discussion of how demystification processes could be incorporated into content moderation settings. This paper contributes new insights about evasion tactics to the content moderation discussion and expands current understanding of how platforms can develop sociotechnical remedial measures. Journal Article Policy & Internet 15 4 478 497 Wiley 1944-2866 1944-2866 Artificial intelligence (AI), content moderation, digital culture, extremism, machine learning. manipulation tactics. propaganda. 内容审核 极端主义 人工智能(AI) 机器学习 操纵策略 （政治）宣传 数字文化Moderación de contenidos. extremismo. inteligencia artificial (IA). aprendizaje automático. tácticas de manipulación , propaganda, cultura digital 1 12 2023 2023-12-01 10.1002/poi3.381 http://dx.doi.org/10.1002/poi3.381 COLLEGE NANME Other/Subsidiary Companies - Not Defined COLLEGE CODE ONDF Swansea University SU Library paid the OA fee (TA Institutional Deal) 2024-04-04T16:08:33.3350213 2023-12-08T13:50:57.9386574 Faculty of Humanities and Social Sciences Hilary Rodham Clinton School of Law Ashley Mattheis 1 Ashton Kingdon 2 65253__29240__33d5a81e29bf48caad7454759107f397.pdf 65253.VOR.pdf 2023-12-08T17:06:12.1880055 Output 3937305 application/pdf Version of Record true © 2023 The Authors. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. true eng http://creativecommons.org/licenses/by/4.0/
title Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
spellingShingle Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
Ashley Mattheis
title_short Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
title_full Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
title_fullStr Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
title_full_unstemmed Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
title_sort Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system
author_id_str_mv 20bd641e721999fbea309db74f2d60c5
author_id_fullname_str_mv 20bd641e721999fbea309db74f2d60c5_***_Ashley Mattheis
author Ashley Mattheis
author2 Ashley Mattheis
Ashton Kingdon
format Journal article
container_title Policy & Internet
container_volume 15
container_issue 4
container_start_page 478
publishDate 2023
institution Swansea University
issn 1944-2866
1944-2866
doi_str_mv 10.1002/poi3.381
publisher Wiley
college_str Faculty of Humanities and Social Sciences
hierarchytype
hierarchy_top_id facultyofhumanitiesandsocialsciences
hierarchy_top_title Faculty of Humanities and Social Sciences
hierarchy_parent_id facultyofhumanitiesandsocialsciences
hierarchy_parent_title Faculty of Humanities and Social Sciences
department_str Hilary Rodham Clinton School of Law{{{_:::_}}}Faculty of Humanities and Social Sciences{{{_:::_}}}Hilary Rodham Clinton School of Law
url http://dx.doi.org/10.1002/poi3.381
document_store_str 1
active_str 0
description Due to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or service. But technologically deterministic approaches are not infallible and can be predictable based on their actions. This opens the door for manipulation of media content and technological affordances to become tactical options for actors seeking to subvert regulation. Existing discussions often neglect the topic of manipulation of content, algorithms, or platform affordances as a primary aspect of the strategies used by extremists in relation to the difficulties of moderation from a policy perspective. This paper argues that it is essential to understand how extremists and conspiracy theorists use manipulation tactics to “game” the current policy, regulatory, and legislative systems of content moderation. Developing approaches that attend to manipulation as a strategy and focus on platform and context-specific tactics will generate more effective policies, platform rules, AI developments, and moderation procedures. This paper analyses and demystifies three primary tactics, which the authors categorise as numerology, borderlands, and merchandising, regularly used by extremists online in their strategies to ‘game’ content moderation. We provide case examples from a variety of ideologies including far-right, QAnon, and male supremacism to highlight the tactics rather than ideological nature of such manipulation. We conclude with a discussion of how demystification processes could be incorporated into content moderation settings. This paper contributes new insights about evasion tactics to the content moderation discussion and expands current understanding of how platforms can develop sociotechnical remedial measures.
published_date 2023-12-01T16:08:29Z
_version_ 1795417240758124544
score 11.016235

Similar Items