Newer
Older
\chapter{Edit Filters As Part of Wikipedia's Socio-Technical Infrastructure}
``Abuse Filter is enabled'' reads the title of one of the eight stories of the 23 March 2009 issue of English Wikipedia's community newspaper, The Signpost~\cite{Signpost2009}.
``The extension allows all edits to be checked against automatic filters and heuristics, which can be set up to look for patterns of vandalism including page move vandalism and juvenile-type vandalism, as well as common newbie mistakes,'' the article proclaims.
The extension, or at least its end user facing parts, was later renamed to ``edit filter'' in order to not characterise false positives as ``abuse'' and thus alienate good faith editors striving to improve the encyclopedia~\cite{Wikipedia:EditFilter},~\cite{Wikipedia:EditFilterTalkArchiveNameChange}.
The aim of this chapter is to understand how edit filters work, who implements and runs them and above all, how and why they were introduced in the first place and what the qualitative difference is between them and other algorithmic quality control mechanisms.
The analysed data is presented in the following section.
Section~\ref{sec:4-def} defines what an edit filter is.
The AbuseFilter MediaWiki extension is introduced in section~\ref{sec:mediawiki-ext}.
After this common understanding of the state of the art of edit filters has been established, section~\ref{section:4-history} looks back and traces the historical debate that led to the filters' introduction.
Sections~\ref{sec:filters-internal} and~\ref{sec:filters-external} take respectively an internal (edit filter managers') and an external (all other users') perspective towards the edit filters.
The findings are then compared with the results from chapter~\ref{chap:background} and discussed in section~\ref{sec:filters-role-qc-system}.
\section{Data}
The foundations for the present chapter lie in EN Wikipedia's policies and guidelines.
Following pages were analysed in depth: \\
\url{https://en.wikipedia.org/wiki/Wikipedia:Edit_filter} \\
\url{https://en.wikipedia.org/wiki/Wikipedia:Edit_filter/Documentation} \\
\url{https://en.wikipedia.org/wiki/Wikipedia:Edit_filter/Instructions} \\
\url{https://en.wikipedia.org/wiki/Wikipedia:Edit_filter_noticeboard} \\
\url{https://en.wikipedia.org/wiki/Wikipedia_talk:Edit_filter/Archive_1}
%************************************************************************
%Regarding the controls that can be set, we can briefly state that:
Every edit filter defines a pattern
\footnote{These patterns consist of one or more conditions, e.g. matching the edit's content against a regular expression or checking the usergroups of the contributing editor.}
against which every edit made to Wikipedia is checked.
If there is a match, the edit in question is logged and potentially, additional actions such as tagging the edit summary, issuing a warning or disallowing the edit are invoked.
Both the patterns and the possible edit filter actions are investigated in greater detail in the following sections.
According to EN Wikipedia's own definition, an edit filter is ``a tool that allows editors in the edit filter manager group to set controls mainly to address common patterns of harmful editing''~\cite{Wikipedia:EditFilter}.
Who is in the edit filter manager group and how did they become part of it?
What controls exactly can be set?
What does ``mainly'' mean, are there other patterns addressed?
And what are the patterns of harmful editing addressed by the filters?
At least the ``mainly'' question is swiftly answered by the paragraph itself, since there is a footnote stating that ``[e]dit filters can and have been used to track or tag certain non-harmful edits, for example addition of WikiLove''~\cite{Wikipedia:EditFilter}.
The controls that can be set are looked into in the sections that follow.
The edit filter manager group and its members are discussed in section~\ref{section:who-can-edit} and the patterns of harmful editing (as well as some further non-harmful edit patterns) are inspected in detail in the next chapter.
For illustration purposes, let us have a closer look at what a single edit filter looks like.
Edit filter with ID 365 is public
\footnote{There are also private (hidden) filters. The distinction is discussed in more detail in sections~\ref{section:4-history} and \ref{sec:public-hidden}.}
This means the filter is working and everyone interested can view the filter's details.
Its description reads ``Unusual changes to featured or good content''.
\begin{verbatim}
"page_namespace == 0 &
!(""confirmed"" in user_groups) &
old_size > 20000 & (
""#redirect"" in lcase(added_lines) |
edit_delta < -15000 |
edit_delta > 15000
) &
old_wikitext rlike
""\{\{([Ff]eatured|[Gg]ood)\s?article\}\}"""
\end{verbatim}
And the currently configured filter actions are: ``disallow''.
So, if a user whose status is not confirmed
\footnote{A confirmed user can do following five things a non-confirmed user cannot: create pages; move pages; edit semi-protected pages; upload files; vote in certain elections (a different minimum edit count can be required for certain elections).
An account can be explicitly confirmed, most accounts are autoconfirmed though.
Generally, accounts are autoconfirmed when they have been registered for at least four days and have made a minimum of ten edits~\cite{Wikipedia:About}.
The requirements are adjustable.}
yet tries to edit a page in the article namespace which contains ``Featured'' or ``Good article'' and they either insert a redirect, delete 3/4 of the content or add 3/4 on top, the edit is automatically disallowed.
Note that an edit filter editor can easily change the action of the filter. (Or the pattern, as a matter of fact.)
All these details can be viewed on the filter's detailed page~\cite{Wikipedia:EditFilter365}
or on the screenshot thereof (figure~\ref{fig:filter-details}) that I created for convenience.
Further information the filter detailed page displays is:
some statistics (the average time the filter takes to check an edit, percentage of hits and how many conditions from the condition limit it consumes
\footnote{According to various community comments, both of these numbers are not particularly reliable and should be treated with caution~\cite{Mediawiki:AbuseFilterConditions}.}
);
comments (left by edit filter managers, generally to log and explain changes);
flags (``Hide details of this filter from public view'', ``Enable this filter'', ``Mark as deleted'');
links to last modified (with diff and user who modified it), the edit filter's history and a tool for exporting the filter to another wiki;
and actions to take when the filter's pattern matches.
\includegraphics[width=.8\paperwidth,keepaspectratio]{pics/detailed-page-filter365-no-boarder.png}
\caption{Detailed page of edit filter \#365}~\label{fig:filter-details}
\end{figure}
%TODO graphic still looks weird: above all quality is poor
%************************************************************************
\section[The AbuseFilter MediaWiki Extension]{The AbuseFilter\footnote{Note that the user facing elements of this extension were renamed to ``edit filter'', however the extension itself, as well as its corresponding permissions, database tables etc. still reflect the original name.} Mediawiki Extension}
At the end, from a technical perspective, Wikipedia's edit filters are a MediaWiki plugin that allows every edit (and some other editor's actions) to be checked against a specified pattern before it is published.
The extension introduces following database tables where all data generated by it is stored: \emph{abuse\_filter}, \emph{abuse\_filter\_log}, \emph{abuse\_filter\_action},\\
and \emph{abuse\_filter\_history} \cite{gerrit-abusefilter-tables}.
\emph{abuse\_filter} contains detailed information about every filter.
\emph{abuse\_filter\_action} stores the currently configured actions for each filter and their corresponding parameters.
Every update of a filter action, pattern, comments or other flags (whether the filter is enabled, hidden, deleted), etc. is recorded in \emph{abuse\_filter\_history}.
And every time a filter matches, the editor's action that triggered it as well as further data such as the user who triggered the filter, their IP address, a diff of the edit (if it was an edit), a timestamp, the title of the page the user was looking at, etc. are logged in \emph{abuse\_filter\_log}.
Most frequently, edit filters are triggered upon new edits, there are however further editor's actions that can trip an edit filter.
As of 30 June 2019, these include: \emph{edit}, \emph{move}, \emph{delete}, \emph{createaccount}, \emph{autocreateaccount}, \emph{upload}, \emph{stashupload}\footnote{See line 181 in \url{https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/AbuseFilter/+/refs/heads/master/includes/special/SpecialAbuseLog.php}}.
%TODO explain what the actions are, especially the less obvious ones such as `autocreateaccount'
Historically, further editor's actions such as \emph{feedback}, \emph{gatheredit} and \emph{moodbar} could trigger an edit filter.
These are in the meantime deprecated. %TODO explain why? I have the guess that these are not available in the software anymore (generally, not only for the edit filters)
When a filter's pattern is matched, beside logging the event in the\\
\emph{abuse\_filter\_log} table (the only filter action which cannot be switched off), a further filter action may be invoked as well.
The plugin defines following possible filter actions:
\emph{tag}, \emph{throttle}, \emph{warn}, \emph{blockautopromote}, \emph{block}, \emph{degroup}, \emph{rangeblock}, \emph{disallow}\footnote{See line 2808 in \url{https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/AbuseFilter/+/refs/heads/master/includes/AbuseFilter.php}}.
%TODO verify that none of the actions are deprecated; I have my doubts that for instance `revoking auto-promoted groups' may not be available anymore -- as far as I can see they are available in the software. However, there never was a community consensus to use them
The documentation of the AbuseFilter extension provides us comprehensive definitions for these~\cite{Mediawiki:AbuseFilterActions}:
\item \emph{tag}: The contribution is tagged with a specific tag (which can be defined and styled by the edit filter manager) which then appears on Recent Changes, contributions, logs, history pages, etc. and allows aggregations of lists for dashboards and similar.
\item \emph{throttle}: The filter is activated upon the tripping of a rate limit. Configurable parameters are the allowed number of actions, the period of time in which these actions must occur, and how those actions are grouped. Actions can be grouped by user, IP address, /16 IP range, creation date of the user account, page, site, the edit count of the user or a combination thereof. (A simple example for throttling is something like ``do this if page X is edited more than Y times in Z seconds''.)
\item \emph{warn}: A warning is displayed that the edit may not be appreciated. (The warning message is configurable by the edit filter manager.) The editor who tripped the filter is provided with the opportunity to revise their edit and re-submit it. A link to the false positives page~\cite{Wikipedia:EditFilterFalsePositives} is also provided.
\item \emph{blockautopromote}: The user whose action matched the filter's pattern is banned from receiving extra groups from \emph{\$wgAutopromote} for a random period of 3 to 7 days. %TODO what is wgAutopromote?
\item \emph{block}: The user who triggered the filter is blocked indefinitely. An error message is displayed to inform the user of this action.
\item \emph{degroup}: The user whose action matched the filter's pattern is removed from all privileged groups (sysop, bureaucrat, etc). An error message is displayed to inform them of this action.
\item \emph{rangeblock}: The entire /16 IP range from which the filter was triggered is blocked for a week.
\item \emph{disallow}: An error message is shown to the editor informing them their edit was considered unconstructive and will not be saved. They are provided the opportunity to report a false positive.
\emph{rangeblock}, \emph{block}, \emph{degroup} have never been used on the EN Wikipedia, at least according to the logs.
Those severer actions were discussed controversially by the community before introducing the extension and a lot of Wikipedians felt uncomfortable with a fully automated mechanism blocking users indefinitely or removing them from privileged groups~\cite{Wikipedia:EditFilterTalkArchive1}, see also section~\ref{section:4-history}.
As far as I can tell, the functionality has been implemented but never activated (at least on the EN Wikipedia).
The last time filter actions other than \emph{log}, \emph{tag}, \emph{warn} or \emph{disallow} were triggered on the EN Wikipedia was in 2012 and these were \emph{blockautopromote} and \emph{aftv5flagabuse} %TODO Refer to data analysis or make a quarry to back this
% Following 4 filters have blockautopromote as an action (note that this could have changed since 2012!): 115, 267, 334, 21;
\footnote{\emph{aftv5flagabuse} is a deprecated action related to the now deprecated Article Feedback MediaWiki extension (or Article Feedback Tool, Version 5) whose purpose was to involve readers more actively in article quality assessment~\cite{Wikipedia:ArticleFeedback}.
However, during the testing phase the majority of reader feedback was found not particularly helpful and hence the extension was discontinued.}.
%TODO: why? look for talk page archives around the last time they were used. Maybe there was a particular incident; nothing concerning autopromote in the EdiFilter talk page around 2012
Guidelines specifically call for careful use of \emph{disallow}.
Only severe cases for which ``substantially all good-faith editors would agree are undesirable'' or specific cases for which consensus has been reached should be disallowed~\cite{Wikipedia:EditFilter}.
Following new user permissions are introduced by the AbuseFilter plugin: %TODO quote \url{https://www.mediawiki.org/wiki/Extension:AbuseFilter} in a sensible manner, once it's decided on refs vs inline urls
\item \emph{abusefilter-modify}: ``Modify abuse filters''
\item \emph{abusefilter-view}: ``View abuse filters''
\item \emph{abusefilter-log}: ``View the abuse log''
\item \emph{abusefilter-log-detail}: ``View detailed abuse log entries''
\item \emph{abusefilter-private}: ``View private data in the abuse log''
\item \emph{abusefilter-modify-restricted}: ``Modify abuse filters with restricted actions''
\item \emph{abusefilter-modify-global}: ``Create or modify global abuse filters''
\item \emph{abusefilter-revert}: ``Revert all changes by a given abuse filter''
\item \emph{abusefilter-view-private}: ``View abuse filters marked as private''
\item \emph{abusefilter-log-private}: ``View log entries of abuse filters marked as private''
\item \emph{abusefilter-hide-log}: ``Hide entries in the abuse log''
\item \emph{abusefilter-hidden-log}: ``View hidden abuse log entries''
\item \emph{abusefilter-private-log}: ``View the AbuseFilter private details access log''
For additional reference, the format for the rules~\cite{Mediawiki:AbuseFilterRules}, the general documentation of the extension~\cite{Mediawiki:AbuseFilterExtension}, as well as its source code~\cite{gerrit-abusefilter-source} can be consulted.
%************************************************************************
%TODO make sure there is enough explanation in the sections before to be able to follow the narrative here
Now that there is a general understanding of what edit filters look like today, let us take a step back and investigate how they came to be this way.
In order to comprehend the consensus building on the functionality of the extension, I sifted through the archives of the Edit Filter talk page~\cite{Wikipedia:EditFilterTalkArchive1}
for the period between the announcement that the extension is planned up until the voting process preceding its introduction.
For a while at the beginnings of the discussion, there was some confusion among editors regarding the intended functionality of the edit filters.
Participants invoked various motivations for the introduction of the extension (which sometimes contradicted each other) and argued for or against the filters depending on these.
The discussion reflects a mix of ideological and practical concerns.
The biggest controversies lay along the lines of filters being public vs. private (hidden from public view)
\footnote{The terms ``private'' and ``hidden'' are used interchangeably for such filters throughout the thesis.}
and the actions the filters were to invoke upon a match.
An automated rights revocation or a block of the offending editor with no manual confirmation by a real person were of particular concern to a lot of editors (they were worried that the filters would not be able to understand context thus resulting in too many false positives and blocking many legitimate edits and editors).
As far as I understood, these features were technically implemented but never really used on English Wikipiedia.
As to the public vs. private debate, the initial plan was that all filters are hidden from public view and only editors with special permissions (the edit filter managers) were supposed to be able to view and modify the patterns and consult the logs.
The core developer of the extension was reasoning that its primary purpose was to fend off really persistent vandals with reasonable technical understanding who were ready to invest time and effort to circumvent anti-vandal measures
and that it was therefore unwise to make circumvention easier to them by allowing them to view the pattern according to which their edits were suppressed.
This was however met with serious resistance by the community who felt that such secret extension was contradicting Wikipedia's values of openness and transparency.
Besides, opponents of the filters being completely private were concerned that the tool was quite powerful and hiding everything will prevent the community from monitoring for errors and abuse.
Related to the above discussion, there was also disagreement regarding who was to have access to the newly developed tool.
Some felt access had to be granted more broadly in order for the tool to be effectively used.
They were reasoning that at least all administrators should be able to use it, since they already had the trust of the community.
Others feared that the admin group was quite broad already and access should be granted as carefully as possible since the extension had the potential to cause quite a lot of damage if used maliciously and it was ``not beyond some of our more dedicated trolls to "work accounts up" to admins, and then use them for their own purpose''~\cite{Wikipedia:EditFilterTalkArchive1}.
This narrower option is how the right ended up to be governed
\footnote{Although motivated trolls can potentially work up an account to any user rights.}.
Another debated point was what the difference to bots (with admin rights) was and whether the extension was needed at all.
Apparently, there was some discontent with bot governance mirrored in the arguments for introducing the extension.
It was underlined that in contrast to admin bots the extension's source code was to be publicly available and well tested with more people (i.e. the edit filter managers) using and monitoring its functionality than the (usually) single bot operator responsible for a bot who, apparently, was oftentimes not responding to community concerns and emergencies fast enough (or at all).
On the other hand, there were yet again arguments, that the extension was supposed to indeed target the really malicious vandals not deterred by anti-vandalism measures already in force by preferably blocking them on the spot.
Others were asking what additional advantages the extension offered compared to semi-protection of pages which requires users to be autoconfirmed in order to be able to edit (normally meaning they have to have been registered for at least 4 days and have made at least 10 edits, but the restrictions are adjustable).
Here, User:Werdna was reasoning that the Edit Filters allow for fine-tuning of such restrictions and targeting offending editors in particular without making constraints unnecessarily strict for all users.
Although there were some diverging opinions on what the extension was supposed to target, in a nutshell, the motivation for its introduction seems to have been as follows:
Bots weren't reverting some kinds of vandalism fast enough, or, respectively, these vandalism edits required a human intervention and took more than a single click to get reverted.
These were mostly obvious but pervasive cases of vandalism (e.g. moving a lot of pages to some nonsensical name), possibly introduced in a (semi-)automated fashion, that took some time and effort to clean up.
The motivation of the extension's developers was that if a filter just disallows such vandalism, vandal fighters could use their time more productively and check less obvious cases for which more background knowledge/context is needed in order to decide whether an edit is vandalism or not.
According to the discussion archives, following types of edits were supposed to be targeted by the extension:\\
\url{https://en.wikipedia.org/wiki/Special:Contributions/Omm_nom_nom_nom}\\
\url{https://en.wikipedia.org/wiki/Special:Contributions/AV-THE-3RD}\\
\url{https://en.wikipedia.org/wiki/Special:Contributions/Fuzzmetlacker}\\
%************************************************************************
\section{Building a Filter: the Internal Perspective}
\label{sec:filters-internal}
Only edit filter managers have the permissions necessary to implement filters, but anybody can propose new ones.
Every editor who notices some problematic behaviour they deem needs a filter can raise the issue at the Edit Filter Requested page~\cite{Wikipedia:EditFilterRequested}.
The request can then be approved and implemented by an edit filter manager (mostly after a discussion/clarification of the details).
The Edit Filter Requested page asks users to go through the following checklist before requesting a filter:
\item problems with a single page are not suitable for an edit filter, since filters are applied to all edits;
\item filters, after adding up, make editing slower, so the usefulness of every single filter and condition has to be carefully considered;
\item in depth checks should be done by a separate software that users run on their own machines;
\item no trivial errors should be caught by filters (e.g. concerning style guidelines);
\item there are the Titles Blacklist~\cite{Mediawiki:TitleBlacklist} and the Link/Spam Blacklist \cite{Mediawiki:SpamBlacklist} which should be used if the issue at hand has to do with a problematic title or link.
For edit filter managers, the best practice way for introducing a new filter is described on the Edit Filter Instructions page~\cite{Wikipedia:EditFilterInstructions}.
According to it, these steps should be followed:
\item read the documentation~\cite{Mediawiki:AbuseFilterRules},
\item test with debugging tools: \url{https://en.wikipedia.org/wiki/Special:AbuseFilter/tools} (visible only for users who are already in the edit filter managers user group),
\item test with the batch testing interface (also available to edit filter managers only),
\item create a logging only filter,
\item announce the filter at the edit filter notice board~\cite{Wikipedia:EditFilterNoticeboard}, so other edit filter managers can comment on it,
\item finally, fully enable the filter by adding an appropriate additional edit filter action.
According to the documentation, step 4 from the checklist can be skipped in ``urgent situations'' and corresponding filters can have severer actions enabled directly.
In such case, the editor introducing the filter has to closely monitor the logs for potential filter misconduct.
However, the guidelines do not elaborate on what exactly constitutes a ``urgent situation''.
Unfortunately, investigating potential pitfalls of this provision is beyond the scope of the present work and one of the directions for further studies suggested in section~\ref{sec:further-studies}.
Edit filter managers often introduce filters based on some phenomena they have observed caught by other filters, other algorithmic quality control mechanisms or general experience.
As all newly implemented filters, these are initially enabled in logging only mode until enough log entries are generated to evaluate whether the incident is severe and frequent enough to need a filter.
%TODO this actually fits also in the patterns of new filters in chap.5; these are the filters introduced for couple of days/hours, then switched off to never be enabled again
It is not uncommon, that the action(s) a particular filter triggers change over time.
Sometimes, when a wave of particularly persistent vandalism arises, a filter is temporarily set to ``warn'' or ``disallow'' and the actions are removed again as soon as the filter is not tripped very frequently anymore. %TODO examples?
Such action changes, updates to an edit filter's pattern, or a warning template, as well as problems with filters behaviour are discussed on the Edit Filter Noticeboard~\cite{Wikipedia:EditFilterNoticeboard}.
Last but not least, performance seems to be fairly important for the edit filter system:
On multiple occasions, there are notes on recommended order of operations, so that the filter evaluates as resource sparing as possible~\cite{Wikipedia:EditFilterInstructions} or invitations to consider whether an edit filter is the most suitable mechanism for solving a particular issue at all~\cite{Wikipedia:EditFilter},~\cite{Wikipedia:EditFilterRequested}.
To optimise performance, the edit filter system uses the so-called condition limit.
According to the documentation~\cite{Wikipedia:EditFilterDocumentation}, the condition limit is a hard-coded treshold of total available conditions that can be evaluated by all active filters per incoming edit.
Currently, it is set to $1,000$.
The motivation for this heuristic is to avoid performance issues since every incoming edit is checked against all currently enabled filters which means that the more filters are active the longer the checks take.
However, the page also warns that counting conditions is not the ideal metric of filter performance, since there are simple comparisons that take significantly less time than a check against the \emph{all\_links} variable for example (which needs to query the database)~\cite{Wikipedia:EditFilterDocumentation}.
Nevertheless, the condition limit seems to still be the heuristic used for filter performance optimisation today.
In order to be able to set up an edit filter on their own, an editor needs to have the \emph{abusefilter-modify} permission (which makes them part of the edit filter manager group).
According to~\cite{Wikipedia:EditFilter} this right is given only to editors who ``have the required good judgment and technical proficiency''.
Further down on the page it is clarified that it is administrators who can assign the permission to users (also to themselves) and they should only assign it to non-admins in exceptional cases, ``to highly trusted users, when there is a clear and demonstrated need for it''.
If editors wish to be given this permission, they can hone and prove their skills by helping with requested edit filters and false positives~\cite{Wikipedia:EditFilter}.
The formal process for requesting the \emph{abusefilter-modify} permission is to raise the request at the Edit Filter Noticeboard~\cite{Wikipedia:EditFilterNoticeboard}.
A discussion is held there, usually for 7 days, before a decision is reached~\cite{Wikipedia:EditFilter}
\footnote{According to the documentation, the Edit Filter Noticeboard is also the place to discuss potential permission withdraws in cases of misuse where raising the issue directly with the editor concerned has not resolved the problem.}.
As of 2017, when the ``edit filter helper'' group was introduced (editors in this group have the \emph{abusefilter-view-private} permission)~\cite{Wikipedia:EditFilterHelper},
the usual process seems to be that editors request this right first and only later the full \emph{abusefilter-modify} permissions\footnote{That is the tendency observed at the Edit filter noticeboard~\cite{Wikipedia:EditFilterNoticeboard}.}.
According to the edit filter managers list for the EN Wikipedia~\cite{Wikipedia:EditFilterManagersList}, as of 10 May 2019 there are 154 users in this group
\footnote{For comparison, as of 9 March 2019 there are 1181 admins~\cite{Wikipedia:Admins}. The role does not exist at all on the German, Spanish and Russian Wikipedias where all administrators have the \emph{abusefilter\_modify} permission~\cite{Wikipedia:EditFilterDE}, \cite{Wikipedia:EditFilterES}, \cite{Wikipedia:EditFilterRU}.}.
Out of the 154 edit filter managers only 11 are not administrators (most of them have other privileged groups such as ``rollbacker'', ``pending changes reviewer'', ``extended confirmed user'' and similar though).
The edit filter managers group is quite stable, with only 4 users who have become an edit filter manager since November 2016 (according to the archives of the edit filter noticeboard where the permission is requested)~\cite{Wikipedia:EditFilterNoticeboard}.
Since the edit filter helper group has been created in September 2017, only 11 users have been granted the corresponding permissions and only one of them has been subsequently ``promoted'' to become an edit filter manager
\footnote{Interestingly, as of July 2019 there are 19 people in the edit filter helpers group, so apparently some of them have received the right although no records could be found on the noticeboard.}.
Moreover, a number of the 154 edit filter managers on English Wikipedia have a kind of ``not active at the moment'' banner on their user page, which leads to the conclusion that the edit filter managers group is aging.
Some of the edit filter managers are also bot operators.
The interesting patterns of collaboration between the two technologies are discussed in section~\ref{subsection:collaboration-bots-filters}.
%TODO leave this here or directly discuss it in the collaboration section?
%************************************************************************
\section{Filters during Runtime: the External Perspective}
%TODO: Flowchart of the filtering process!
So what happens when an editor's action matches the pattern of an edit filter? Do they notice this at all?
As described section~\ref{sec:mediawiki-ext}, a variety of different actions may occur when a filter's pattern matches.
Of these, only \emph{tag}, \emph{throttle}, \emph{warn}, and \emph{disallow} seem to be used today (and \emph{log}, which is always enabled).
If a filter is set to \emph{warn} or \emph{disallow}, the editor is notified that they hit a filter by a warning message (see figure~\ref{fig:screenshot-warn}).
These warnings describe the problem that occurred and present the editor with possible paths of action:
complain on the False Positives page~\cite{Wikipedia:EditFilterFalsePositives} in case of \emph{disallow} (the edit is not saved),
or, complain on the False Positives page
\footnote{Edit filter managers and other interested editors monitor the False Positives page and verify or disprove the reported incidents.
Edit filter managers use actual false positives to improve the filters, give advice to good faith editors who tripped a filter and discourage authors of vandalism edits who reported these as false positives from continuing with their disruption.
% who moderates the false positives page? where does the info come from that it is edit filter managers? I think this info comes from observation
}
and publish the change anyway in case of \emph{warn}.
(Of course, in case of a warning, the editor can modify their edit before publishing it.)
Possible alternative paths of action an editor may wish to consult are also listed.
On the other hand, when the filter action is set to \emph{tag} or \emph{log} only, the editor doesn't really notice they tripped a filter unless they are looking more closely.
Tagged edits are marked as such in the page's revision history for example (see figure~\ref{fig:tags-in-history})
and all edits that trigger an edit filter are listed in the Abuse Log~\cite{Wikipedia:AbuseLog} (see figure~\ref{fig:screenshot-abuse-log}).
%TODO How about throttling: the AbuseLog is currently timing out when I try to filter entries according to action(=throttle)
\includegraphics[width=1\columnwidth]{pics/screenshots-filter-trigger/Screenshot-tags-in-revision-history.png}
\caption{Tagged edits are marked as such in a page's revision history}~\label{fig:tags-in-history}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=1\columnwidth]{pics/screenshots-filter-trigger/Screenshot-abuse-log.png}
\caption{Abuse Log showing all filter matches by User:Schnuppi4223}~\label{fig:screenshot-abuse-log}
\includegraphics[width=1\columnwidth]{pics/screenshots-filter-trigger/Screenshot-warning-page-blank.png}
\caption[Editor gets two warnings upon erasing an entire page]{Editor gets two warnings upon erasing an entire page: one for page blanking and another one for removing all categories from an article. The warnings list possible pages the editor may want to consult and actions they can take.}~\label{fig:screenshot-warn}
%************************************************************************
\section{Edit Filters' Role in the Quality Control Ecosystem}
\label{sec:filters-role-qc-system}
The purpose of the present section is to review what has been learnt so far about edit filters and summarise how they fit in Wikipedia's quality control ecosystem.
As timeline~\ref{fig:timeline} shows, the time span in which algorithmic quality control mechanisms (first vandal fighting bots and semi-automated tools, and later filters) were introduced fits logically the period after the exponential growth of Wikipedia took off in 2006 (compare figures~\ref{fig:editors-development},~\ref{fig:edits-development}).
The surge in numbers of editors and contributions implied a rapidly increasing workload for community members dedicated to quality assurance
which could not be feasibly handled manually anymore and thus the community turned to technical solutions.
As shown elsewhere~\cite{HalGeiMorRied2013}, this shift had a lot of repercussions—one of the most severe of them being that newcomers' edits were reverted stricter than before (accepted or rejected on a yes-no basis with the help of automated tools, instead of manually seeking to improve the contributions and ``massage'' them into an acceptable form), which in consequence drove a lot of them away.
%TODO sounds ending abruptly; maybe a kind of a recap with historical background, compare introduction
\begin{table}
\begin{tabular}{ r | p{.8\textwidth}}
Oct 2001 & automatically import entries from Easton’s Bible Dictionary by a script \\
29 Mar 2002 & First version of \url{https://en.wikipedia.org/wiki/Wikipedia:Vandalism} (WP Vandalism is published) \\
Oct 2002 & RamBot \\
2006 & The Bot Approvals Group (BAG) was first formed \\
13 Mar 2006 & 1st version of Bots/Requests for approval is published: some basic requirements (also valid today) are recorded \\
28 Jul 2006 & VoABot II ("In the case were banned users continue to use sockpuppet accounts/IPs to add edits clearly rejected by consensus to the point were long term protection is required, VoABot may be programmed to watch those pages and revert those edits instead. Such edits are considered blacklisted. IP ranges can also be blacklisted. This is reserved only for special cases.") \\
21 Jan 2007 & Twinkle Page is first published (empty), filled with a basic description by beginnings of Feb 2007 \\
24 Jul 2007 & Request for Approval of original ClueBot \\
16 Jan 2008 & Huggle Page is first published (empty) \\
18 Jan 2008 & Huggle Page is first filled with content \\
23 Jun 2008 & 1st version of Edit Filter page is published: User:Werdna announces they're currently developing the extension \\
2 Oct 2008 & \url{https://en.wikipedia.org/wiki/Wikipedia_talk:Edit_filter} was first archived; its last topic was the voting for/against the extension which seemed to have ended end of Sep 2008 \\
Mar 2009 & The AbuseFilter extension is enabled on English Wikipedia \\
Jun 2010 & STiki initial release \\
20 Oct 2010 & ClueBot NG page is created \\
11 Jan 2015 & 1st commit to github ORES repository \\
30 Nov 2015 & ORES paper is published
\end{tabular}
\caption{Timeline: Introduction of algorithmic quality control mechanisms}~\label{fig:timeline}
\end{table}
\includegraphics[width=1\columnwidth]{pics/editors-development-total.png}
\caption[EN Wikipedia: Number of editors over the years]{EN Wikipedia: Number of editors over the years (source: \url{https://stats.wikimedia.org/v2/})}~\label{fig:editors-development}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=1\columnwidth]{pics/edits-development-total.png}
\caption[EN Wikipedia: Number of edits over the years]{EN Wikipedia: Number of edits over the years (source: \url{https://stats.wikimedia.org/v2/})}~\label{fig:edits-development}
\subsection{Wikipedia's Algorithmic Quality Control Mechanisms in Comparison}
As we can read from timeline~\ref{fig:timeline}, edit filters were introduced at a moment when bots and semi-automated tools were already in place.
Thus, the question arises: Why were they implemented when already these other mechanisms existed?
Here, the salient features of the different quality control mechanisms and the motivation for the filters' introduction are reviewed.
A concise summary of this discussion is offered in table~\ref{table:mechanisms-comparison}.
Since edit filters are a fully automated mechanism, above all a comparison to bots seems obvious.
The main argument for introducing the extension were the usecases it was supposed to take care of: the obvious persistent vandalism (often automated itself) which was easy to recognise but more difficult to clean up.
Filters were going to do the job more neatly than bots by reacting faster, since the extension was part of the core software,
and since they are triggered \emph{before} an edit is published–by not allowing abusive content to become public at all.
By being able to disallow such malicious edits from the beginning, the extension was to reduce the workload of other mechanisms and free up resources for vandal fighters using semi-automated tools or monitoring pages manually to work on less obvious cases that required human judgement, reasoned proponents of the filters.
%Structural/soft factors
The rest of the arguments for edit filters vs. bots touched on in the discussion prior to introducing filter~\cite{Wikipedia:EditFilterTalkArchive1} were more of infrastructural/soft nature. %TODO find a better description for this.
The plugin's developers optimistically announced that it was going to be open source, the code well tested, with framework for testing single filters before enabling them and edit filter managers being able to collaboratively develop and improve filters.
They viewed this as an improvement compared to (admin) bots which would be able to cover similar cases but whose code was mostly private, not tested at all, and with a single developer/operator taking care of them who was often not particularly responsive in emergency cases
\footnote{For the sake of completeness, it should be mentioned here that the most popular semi-automated anti-vandalism tools are also open sourced.
Their focus however lies somewhat differently, since a final human decision is required, and that is why probably they are not mentioned at all in this discussion.
ORES is open source as well, it is kind of a meta tool that can be employed by the other mechanisms though and that is a why a direct comparison is also not completely feasible.
Besides, it was introduced some 6-7 years after the edit filters, so obviously people were not discussing it at the time.
}.
% Comparison filters vs page protection
Another apparent comparison is the one between edit filters and MediaWiki's page protection mechanism~\cite{Mediawiki:PageProtection}.
As pointed out in section~\ref{sec:page-protection}, page protection is reasonable when a rise in disruptive activity on a particular page occurs.
Similarly to applying an edit filter aiming at the specific page, page protection would simply disallow edits to it from the start.
The difference however is that edit filters could target a specific malicious user (or users) directly, without imposing restrictions on the vast majority of editors.
%Who does all of this, how difficult is it to become involved
%TODO this seems a bit out of place, but no better placement found so far
From all the mechanisms, it is probably the hardest to become engaged with edit filters.
As signaled in section~\ref{section:who-can-edit}, the permissions are only granted to very carefully selected editors who have long history of participation on Wikipedia and mostly also various other special permissions.
The numbers also demonstrate that this is the most exclusive group:
as mentioned in section~\ref{section:who-can-edit}, there are currently 154 edit filter managers on EN Wikipedia,
compared to at least 232 bot operators~\cite{Wikipedia:BotOperators} (most likely not all bot operators are listed in the category~\cite{Wikipedia:FAQCategory})
and 6130 users who have the \emph{rollback} permission~\cite{Wikipedia:Rollback}.
As to the difficulty/competences needed, it is probably easiest to learn to use semi-automated tools where one ``only'' has to master the user interface of the software.
Bots require presumably most background knowledge since one has to not only be familiar with a programming language but also learn to interact with Wikipedia's API, etc.
Filters on the other hand, are arguably easier to use: here, ``only'' understanding of regular expressions is required.
%Different pitfalls and concerns are express
As already summarised in chapter~\ref{chap:background}, critical voices worry about various aspects of the individual quality control mechanisms (see also table~\ref{table:mechanisms-comparison}).
Concerns with filters resemble somewhat the concerns expressed about bots: namely, the apprehension of a fully-automated mechanism taking (potentially erroneous) decisions about excluding editors from participation.
In consequence, community consensus on using filter actions such as \emph{rangeblock}, \emph{block}, and \emph{degroup} never happened.
According to the discussion archives~\cite{Wikipedia:EditFilterTalkArchive1}, others feared that edit filters were placing a lot of power in the hands of very few people.
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
\begin{landscape}
{\small
\begin{longtable}{ | p{2cm} | p{2.5cm} | p{2.5cm} | p{2.5cm} | p{2.5cm} | p{2.5cm} | p{2.5cm} }
\hline
& Filters & Page protection & Blacklists & Bots & Semi-Automated tools & ORES \\
\hline
\multirow{7}{*}{Properties}& rule-based & per page & rule-based & rule/ML based & rule/ML based & ML framework \\
& part of the "software" (MediaWiki plugin) & part of MediaWiki & part of MediaWiki & run on user's infrastructure ("bespoke code") & extra infrastructure & not used directly, can be incorporated in other tools \\
& open source& MediaWiki is open source & & no requirement for code to be public & \raggedright most popular are open source (but not a hard requirement) & open source \\
& public filters directly visible for anyone interested & & & & heuristics obfuscated by the interface & \\
& trigger \emph{before} an edit is published & one cannot edit at all & \raggedright trigger before an edit is published & latency varies (ClueBot NG needs only a couple of seconds)& generally higher latency than bots & \\
& collaborative effort & & & \raggedright mostly single dev/operator (recently: bot frameworks) & few devs & few devs \\
\hline
Necessary & \raggedright \emph{abusefilter-modify} permission & admin access & admin access & \raggedright no special rights needed (except for admin bots) & \emph{rollback} permission needed & \\
permissions & & & bot gets a ``bot flag'' & & \\
\hline
People involved & \raggedright edit filter managers (EN Wiki)& admins & admins (anyone can report problems) & bot operators & \raggedright editors with \emph{rollback} permission & mostly Scoring Platform team \\
\hline
Hurdles to & gain community trust to become an edit filter manager & & & get approval from the BAG & get a \emph{rollback} permission& \\
participate & understand regexes & & understand regexes & programming knowledge, understand APIs, ... & \raggedright get familiar with the tool & understand ML \\
\hline
\multirow{3}{*}{Concerns} & \raggedright automated agents blocking/desysoping human users & & & ``botophobia'' & gamification & general ML concerns: hard to understand \\
& \raggedright hidden filters lack transparency and accountability& & & & \raggedright interface makes some paths of action easier than others & \\
& censorship infrastructure & & & & & \\
\hline
Areas of application & \raggedright persistent vandal with a known modus operandi and a history of circumventing prevention methods' demographic (obvious vandalism which takes time to clean up) & problems with single page & abusive titles/ link spam & mostly obvious vandalism & \raggedright less obvious cases that require human judgement & \\
\hline
\caption{Wikipedia's algorithmic quality control mechanisms in comparison}~\label{table:mechanisms-comparison-literature}
\end{longtable}
}
\end{landscape}
\begin{comment}
\begin{landscape}
\begin{longtable}{ | p{3cm} | p{4.5cm} | p{4.5cm} | p{4.5cm} | p{4.5cm} | }
\hline
& Filters & Bots & Semi-Automated tools & ORES \\
\multirow{7}{*}{Properties} & rule based & rule/ML based & rule/ML based & ML framework \\
& part of the "software" (MediaWiki plugin) & run on user's infrastructure ("bespoke code") & extra infrastructure & not used directly, can be incorporated in other tools \\
& extension is open source & no requirement for code to be public & most popular are open source (but it's not a hard requirement) & open source \\
& public filters directly visible for anyone interested & & heuristics obfuscated by the interface & \\
& trigger \emph{before} an edit is published & trigger after an edit is published & trigger after an edit is published & \\
& zero latency, trigger immediately & latency varies & generally higher latency than bots & \\
& collaborative effort & mostly single dev/operator (recently: bot frameworks) & few devs & few devs \\
\hline
\multirow{2}{*}{People involved} & edit filter managers (EN Wiki) & no special rights needed (except for admin bots) & \emph{rollback} permission needed & mostly Scoring Platform team \\
& & bot gets a ``bot flag'' & & \\
\hline
\multirow{2}{*}{Hurdles to participate} & gain community trust to become an edit filter manager & get approval from the BAG & get a \emph{rollback} permission& \\
& understand REGEXes & programming knowledge, understand APIs, ... & get familiar with the tool & understand ML \\
\hline
\multirow{2}{*}{Concerns} & automated agents blocking/desysoping human users & ``botophobia'' & gamification & general ML concerns: hard to understand \\
& hidden filters lack transparency and accountability & & interface makes some paths of action easier than others & \\
& censorship infrastructure & & & \\
Areas of application & persistent vandal with a known modus operandi and a history of circumventing prevention methods' demographic (obvious vandalism which takes time to clean up) & mostly obvious vandalism & less obvious cases that require human judgement & \\
\caption{Wikipedia's algorithmic quality control mechanisms in comparison}~\label{table:mechanisms-comparison}
| Filters | Bots | Semi-Automated tools | ORES
--------------------------------------------------------------------------------------------------------------------------------------------
| | | |
| - based on REGEXes | - rule/ML based | - rule/ML based | - ML framework
properties | - part of the "software"/ | - "bespoke code": run on | - heuristics obfusca- | - not used directly, but can
| platform (MediaWiki ext) | user's infrastructure | ted by the interface | instead be incorporated in
| - public filters are directly | - no requirement for the code | (but often configura- | other tools
| visible for everyone | to be made public | ble to an extent) |
| interested | - you can relatively easily | - mostly based on a |
| - trigger *before* an edit is | get all the filters; you | centralised queue |
| published | cannot easily get all bots | - trigger after an edit |
| - source code of the extension | - trigger after an edit is | is published |
| is publicly viewable | published | |
| - zero latency, triggered imme- | - latency varies | |
| diately | | |
| - collaborative effort | - mostly single dev/operator | |
| | (although in recent years a | |
| | lot of bots rely on bot | |
| | frameworks) | |
| | | |
---------------------|------------------------------------------------------------------------------------------------------------------------
| | | |
| - edit filter managers | - no special persmissions/ | - *rollback* perm | - mostly Scoring platform
| group (EN Wiki) | rights needed | | team (?)
Who does this? | (abusefilter-modify perm) | - a bot gets a bot flag | |
| | | |
---------------------|------------------------------------------------------------------------------------------------------------------------
| | | |
| - become an edit filter manager | - get an approval to run the | - learn the tool | - understand ML
What are the hurdles | - you have to *only* understand | bot from the BAG | - install Windows^^ | - formal requirements to
to participate | REGEXes (relatively simple? | - programming knowledge | (some don't support | develop ORES?
| although relatively fast quite | - understand APIs, .. | other OS) |
| confusing) | - (but there is a lot to | - get the rollback perm |
| | understand with all the | |
| | mechanisms) | |
| | | |
-----------------------------------------------------------------------------------------------------------------------------------------------
| - censorship infrastructure? | - "botophobia" | - gamification | - general ML concerns:
Concerns | - powerful, can in theory block | | | hard to understand
| editors based on hidden | | |
| filters | | |
| | | |
------------------------------------------------------------------------------------------------------------------------------------------------
|
Application areas |
|
------------------------------------------------------------------------------------------------------------------------------------------------
%*************************************************************
%\subsection{Collaboration with bots (and semi-automated tools)}
\label{subsection:collaboration-bots-filters}
So far, the single quality control mechanisms have been juxtaposed and separately compared.
It is however worth mentioning that they not only operate alongside each other but also cooperate on occasions.
Such collaborations are studied for instance by Geiger and Ribes~\cite{GeiRib2010} who go as far as describing them as ``distributed cognition''.
They follow a particular series of abuse throughout Wikipedia, along the traces the disrupting editor and the quality control mechanisms deployed against their edits left.
The researchers demonstrate how a bot (ClueBot), and several editors using the semi-automated tools Huggle and Twinkle all collaborated up until the malicious editor was banned by an administrator.
During the present study, I have also observed various cases of edit filters and bots mutually facilitating each other's work.
%TODO check whether there are other types of cooperations at all: what's the deal with Twinkle? and update here!
DatBot, Mr.Z-bot and MusikBot are all examples for bots conducting support tasks for filters.
DatBot~\cite{Wikipedia:DatBot} monitors the Abuse Log~\cite{Wikipedia:AbuseLog}
and reports users tripping certain filters to WP:AIV (Administrator intervention against vandalism)\cite{Wikipedia:AIV} and WP:UAA (usernames for administrator attention)~\cite{Wikipedia:UAA}.
It is the successor of Mr.Z-bot~\cite{Wikipedia:MrZBot}
which used to report users from the abuse log to WP:AIV, but has been inactive since 2016 and therefore recently deactivated.
MusikBot also has several tasks dedicated to monitoring different aspects of edit filter behaviour and compiling reports for anyone interested:
The FilterMonitor task ``[r]eports functional changes of edit filters to the watchable page User:MusikBot/FilterMonitor/Recent changes. The template\\ %End line since otherwise template name protrudes in margin
\verb|{{recent filter changes}}| formats this information and can be transcluded where desired''~\cite{Wikipedia:MusikBotFilterMonitor}.
The StaleFilter task ``[r]eports enabled filters that have not had any hits in over 30 days, as specified by \verb|/Offset|''~\cite{Wikipedia:MusikBotStaleFilters}.
The AbuseFilterIRC task ``[r]elays all edit filter hits to IRC channels and allows you to subscribe to notifications when specific filters are tripped. See \verb|#wikipedia-en-abuse-log-all| for the English Wikipedia feed''~\cite{Wikipedia:MusikBotAbuseFilterIRC}.
On the other hand, there are also examples for filters supporting bot work:
Filter 323 (``Undoing anti-vandalism bot'') tags edits reverting revisions by XLinkBot and ClueBot NG.
Although it is hidden, so no details can be viewed by a unauthorised user, filter 603 is named ``Special case of reverting XLinkBot reverts'' so it is probably safe to assume that is filtering what it claims to be.
And there are several filters (historically) configured to ignore particular bots: filter 76 (``Adding email address'') exempting XLinkBot, filter 28 (``New user redirecting an existing substantial page or changing a redirect'') exempting Anybot, filter 532 (``Interwiki Addition'') exempting Cydebot are some examples thereof.
There are also filters configured to ignore all bots: filter 368 (``Making large changes when marking the edit as minor''), filter 702 (``Warning against clipboard hijacking''), filter 122(``Changing Username malformed requests'').
On occasions, data from the Abuse Log is used for (semi-)protecting frequently disrupted pages.
And as discussed in chapter~\ref{chap:background}, ORES scores can be employed by bots or semi-automated tools as a heuristic to detect potentially harmful edits.
Note that edit filters cannot use ORES, since the service computes scores according to different models for already published revisions.
%*************************************************************
%Conclusion, resume, bottom line, lesson learnt, wrap up
In short, this chapter studied edit filters' documentation and community discussions and worked out the salient characteristics of this mechanism.
Moreover, the filters were compared to other quality control technologies on Wikipedia such as bots, semi-automated anti-vandalism tools and the machine learning framework ORES.
Edit filters were considered in the context and time of their introduction and it was concluded that the community implemented them as a means to fight obvious, particularly persistent, and cumbersome to remove vandalism by disallowing it on the spot.
Other ``softer'' arguments such as dissatisfaction with bot development processes (poorly tested, non-responsive operators) seemed to encourage the introduction as well.
It was found that the individual filters are implemented and maintained by edit filter managers, a special highly-restricted user group.
Revising the quality control ecosystem diagram~\ref{fig:funnel-no-filters} introduced in chapter~\ref{chap:background}, filters can now be properly placed on it (see figure~\ref{fig:funnel-with-filters}).
It seems that claims of the literature (see section~\ref{section:bots}) should be revised:
In terms of temporality not bots but edit filters are the first mechanism to actively fend off a disruptive edit.
\includegraphics[width=0.9\columnwidth]{pics/funnel-with-filters-new.png}
\caption[Edit filters' role in the quality control ecosystem]{Edit filters' role in the quality control ecosystem: They are the first mechanism to get active.}~\label{fig:funnel-with-filters}