Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Blame Permalink
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
slides-notes.md 16.39 KiB

% You shall not publish: Edit filters on EN Wikipedia % Master Thesis Defence % Lusy

Notes: Gandalf

  • on title slide? / as a logo?

Overview

  • What is an edit filter
  • Motivation
  • Research Questions
  • Approach: Trace Ethnography
  • Analysis sources
    • State of the literature/Literature: What does the scientific community know
    • Documentation: What is an edit filter and why was it introduced according to Wikipedia's/MediaWiki pages/Wikipedia's community?
    • Data Analysis: Edit filters on English Wikipedia
  • Findings
  • Open questions/Directions for future studies
  • Contributions and Limitations

What is an edit filter (live)

Notes: vlt swap with motivation Ask audience:

  • how many have a Wikipedia account/ who has edited Wikipedia?
  • has anyone encountered a filter? been prevented from publishing their changes?

Then demo:

  • edit, receive a warning
  • have a look at the corresponding filter's detailed page

Sumup:

  • MediaWiki extension
  • regex based filtering of edits and other actions (e.g. account creation, page deletion or move, upload)
  • triggers before an edit is published
  • different actions can be defined: warn, tag, disallow most common

Motivation

Rise and decline in numbers of editors on EN Wikipedia Source: Halfaker et al. "The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to popularity is causing its decline" vandalism example in the Veganism article on EN Wikipedia, Jan 2019 Screenshot of an edit filter disallow message displayed to an editor

Notes why is it relevant? decription of state of the art

  • wikipedia is a complex socio technical system
  • we have the luck it's "open", so we can study it and learn how things work and apply the insights to less open systems
  • "anyone can edit": increasing popularity in 2006; -> increasing need for quality control
  • edit filters a one particular mechanism for quality control among several, and one previously unstudied
  • seem relevant to understand how and what they do since they make it possible to disallow edits (and other actions, but above all edits) from the very beginning

Research questions

Q1: What is the role of edit filters among existing algorithmic quality-control mechanisms on Wikipedia (bots, semi-automated tools, ORES, humans)?

Q2: Edit filters are a classical rule-based system. Why are they still active today when more sophisticated ML approaches exist?

Q3: Which type of tasks do filters take over?

Q4: How have these tasks evolved over time (are they changes in the type, number, etc.)?

Approach: Trace Ethnography

combines the richness of participant-observation with the wealth of data in logs so as to reconstruct patterns and practices of users in distributed sociotechni- cal systems Source: R Stuart Geiger and David Ribes. "Trace ethnography: Following coordination through documentary practices."

Note: It extends classic documentary ethnography which can rely on any written artefact such as archive records, diaries, manuals and handbooks, correspondence, standards, protocols, or trading records, by the extensive use of logs and other records generated by digital environments in an automated manner.

Analysis Sources

Literature Documentation Data

Notes:

  • Literature
  • Documentation
  • Data (Edit filter patterns, DB log table)

State of the Scientific Literature

Funnel diagramm of all vandal fighting mechanisms according to the current state of scientific literature (no filters)

Note:

  • One thing is ostentatiously missing: edit filters
  • TODO discuss mechanisms with the help of the summary table from Chapter 2

Q1: What is the role of edit filters among existing algorithmic quality-control mechanisms on Wikipedia (bots, semi-automated tools, ORES, humans)?

Funnel diagramm of all vandal fighting mechanisms according to me

Notes:

  • 1st mechanism activated to control quality (at the beginning of the funnel)

Consider pasting the comparison table from chapter 4 here

Notes:

  • historically: faster, by being a direct part of the core software: disallow even before publishing
  • can target malicious users directly without restricting everyone (<-> page protection)
  • introduced to take care of obvious but cumbersome to remove vandalism
  • people fed up with bot introduction and development processes (poor quality, no tests, no code available for revision in case of problems) (so came up with a new approach)
  • disallow certain types of obvious pervasive (perhaps automated) vandalism directly
  • takes more than a single click to revert
  • human editors can use their time more productively elsewhere

Q2: Edit filters are a classical rule-based system. Why are they still active today when more sophisticated ML approaches exist?

Timeline with the introductions of different quality control mechanisms

"The idea is to automatically deal with the very blatant, serial pagemovers for instance, freeing up human resources to deal with the less blatant stuff. SQLQuery me! 20:13, 9 July 2008 (UTC)" "This extension is not designed to catch the subtle vandalism, because it's too hard to identify directly. It's designed to catch the obvious vandalism to leave the humans more time to look for the subtle stuff. Happy‑melon 16:35, 9 July 2008 (UTC)" This is quite different from, say, an anti-vandalism adminbot. The code is private, and, in any case, too ugly for anybody to know how to use it properly [...] and the bot is controlled by a private individual, with no testing.

(Source https://en.wikipedia.org/wiki/Wikipedia_talk:Edit_filter/Archive_1)

Notes:

  • introduced before most vandalism fighting ML systems came along; so they were there first historically; still work well; don't touch a running system^^
  • a gap was perceived in the existing system which was filled with filters
    • in functionality: disallow cumbersome vandalism from the start
    • in governance: bots are poorly tested, communication and updates are difficult
  • volunteer system: people do what they like and can (someone has experience with this types of tech and implemented it that way)
  • rule-based systems are more transparent and accountable
  • and easier to work with (easier to add yet another rule than tweak paremeters in an obscure ML based approach)
  • allows for finer levels of control than ML: i.e. disallowing specific users
  • filter allow more easily for collaboration

Q3: Which type of tasks do filters take over?

  • overall distribution There are 954 edit filters on EN Wikipedia: roughly 21% of them are active, 16% are disabled, and 63% are deleted There are 954 edit filters on EN Wikipedia: roughly 21% of them are active, 16% are disabled, and 63% are deleted
  • in total most filters are hidden: so implemented with the purpose of taking care of cumbersome vandalism by specific malicious users
  • actions distribution
Filter actions for all enabled filters Filter actions for all enabled filters Filter actions for enabled public filters Filter actions for enabled public filters Filter actions for enabled hidden filters Filter actions for enabled hidden filters
  • manual tags distribution
Distribution of manually assigned labels for all filters Distribution of manually assigned labels for enabled filters * vandalism/good faith/maintenance <- briefly explain coding?

Q4: How have these tasks evolved over time (are they changes in the type, number, etc.)?

Number of filter hits per month, Mar 2009-Jan 2019 Number of filter hits per month, according to manuall assigned labels Number of filter hits per month, according to causing editor's action Number of filter hits per month, according to filter action Number of reverts per month, Jul 2001-Jul 2017

Notes:

  • filter hit numbers are of the same magnitude as reverts (way higher than initially expected)
  • beginning: more good faith, later more vandalism hits (somewhat unexpected)
  • surge in 2016 and a subsequently higher baseline in hit numbers (explaination?)
  • overall number of active filters stays the same (condition limit)
  • most active filters of all times are quite stable through the years

Directions for future studies

Data

  • Only EN Wikipedia
  • abuse_filter_history was missing
  • no access to hidden filters

Process

  • manual filter classification only conducted by me

  • no ethnographic analysis: repercussions on editors/bots vs filters

  • chaotic: instead a lot of interesting directions, hard to decide which way to go

  • Verify results

    • abuse_filter_history was missing
    • manual filter classification only conducted by me

  • What proportion of quality control work do filters take over?

  • To implement a bot or to implement a filter?

  • What are the repercussions on affected editors?

  • Is there a qualitative difference between the tasks/patterns of public and hidden filters?

  • What are the differences between how filters are governed on EN Wikipedia compared to other language versions?

  • Are edit filters a suitable mechanism for fighting harassment?

  • (How) has the notion of ``vandalism'' on Wikipedia evolved over time?

  • What are the urgent situations in which edit filter managers are given the freedom to act as they see fit and ignore best practices of filter adopt

Open Questions

maybe just an exceprt here?

  • How have edit filters's tasks evolved over time? : should be easier to look into it with the abuse_filter_history When a dump becomes available, an extensive investigation of filters' actions, creation and activation patterns, as well as patterns they have targeted over time will be possible.
  • What proportion of quality control work do filters take over?: Filter hits can be systematically compared with the number of all edits and reverts via other quality control mechanisms.
  • Is it possible to study the filter patterns in a more systematic fashion?
  • What can be learnt from this?: For example, it has come to attention that 1/5 of all active filters discriminate against new users via the \verb|!("confirmed" in user_groups)| pattern. Are there other tendencies of interest?
  • Is there a qualitative difference between the tasks/patterns of public and hidden filters?: According to the guidelines for filter creation, general filters should be public while filters targeting particular users should be hidden. Is there something more to be learnt from an examination of hidden filters' patterns? Do they actually conform to the guidelines? %One will have to request access to them for research purposes, sign an NDA, etc.
  • How are false positives handled?: Have filters been shut down regularly, because they matched more false positives than they had real value? Are there big amounts of false positives that corrupt the filters hit data and thus the interpretations offered by the current work?
  • To implement a bot or to implement a filter?: An ethnographic inquiry into if an editor is simultaneously an edit filter manager and a bot operator when faced with a new problem, how do they decide which mechanism to employ for the solution?
  • What are the repercussions on affected editors? An ethnographic study of the consequences of edit filters for editors whose edits are filtered. Do they experience frustration or alienation? Do they understand what is going on? Or do they experience for example edit filters' warnings as helpful and appreciate the hints they have been given and use them to improve their collaboration?
  • What are the differences between how filters are governed on EN Wikipedia compared to other language versions?: Different Wikipedia language versions each have a local community behind them. These communities vary, sometimes significantly, in their modes of organisation and values. It would be very insightful to explore disparities between filter governance and the types of filters implemented between different language versions.
  • Are edit filters a suitable mechanism for fighting harassment?: A disturbing rise in online personal attacks and harassment is observed in a variety of online spaces, including Wikipedia~\cite{Duggan2014}. The Wikimedia Foundation sought to better understand harassment in their projects via a Harassment Survey conducted in 2015~\cite{Wikimedia:HarassmentSurvey}. According to the edit filter noticeboard archives~\cite{Wikipedia:EditFilterNoticeboardHarassment}, there have been some attempts to combat harassment by means of filters. The tool is also mentioned repeatedly in the timeline of Wikipedia's Community Health Initiative~\cite{Wikipedia:CommunityHealthInitiative} which seeks to reduce harassment and disruptive behaviour on Wikipedia. An evaluation of its usefulness and success at this task would be really interesting.
  • (How) has the notion of vandalism'' on Wikipedia evolved over time?: By comparing older and newer filters, or respectively updates in filter patterns, it could be investigated whether there has been a qualitative change in the interpretation of the vandalism'' notion on Wikipedia.
  • What are the urgent situations in which edit filter managers are given the freedom to act as they see fit and ignore best practices of filter adoption?: (i.e. switch on a filter in log only mode first and announce it on the notice board so others can have a look)? Who determines they are urgent? These cases should be scrutinised extra carefully since ``urgent situations'' have historically always been an excuse for cuts in civil liberties.

Contributions

  • initial systematic investigation of filters and their role among other mechanisms
  • codebook for classifying filters according to their task
  • openly accessible artefacts: scripts for data analysis, etc.
  • abuse_filter_history table
  • questions for future research

Current Limitations

Data

  • Only EN Wikipedia
  • abuse_filter_history missing
  • no access to hidden filters

Process

  • manual filter classification only conducted by me
  • no ethnographic analysis; can answer valuable questions (i.e. bot vs filter?)
  • chaotic, oftentimes difficult to organise information in a coherent, systematic fashion; constant feeling that I'm missing a significant point
  • Evaluation: what would I do differently?/what went not so well
    • Start writing after getting hold of all the data

Outlook

blackout German Wikipedia March 2019 https://upload.wikimedia.org/wikipedia/commons/c/c5/Blackout_of_wikipedia.de_by_Wikimedia_Deutschland_-_March_2019.png

Thank you!

These slides are licensed under the CC BY-SA 4.0 License.

by sa

Questions? Comments? Thoughts?