* 8 is too much; no clear distinction. boil them down to 3 (vandalism, good faith, maintenance) again; however where do the "seo/pov problems" go then?
* does it make sense to have the subcategories (above all for vandalism)?
* Bots on Wikipedia: Unfolding their duties -> technical report?
* methods chapter vs explaining methodology in chapters?
* split chapter 5: manual classification in a separate chapter?
@@ -387,7 +387,7 @@ Based on grounded theory methodology presented in chapter~\ref{chap:methods}, I
Three big clusters of filters were identified, namely ``vandalism'', ``good faith'' and ``maintenance''. %TODO define what each of them are; I actually work with 8 main clusters in the end; Unify this
\subsection{Challenges with labeling}
\subsection{Labeling process and challenges}
It was not always a straightforward decision to determine what type of edits a certain filter is targeting.
This was of course particularly challenging for private filters where only the public comment (name) of the filter was there to guide the coding.
On the other hand, guidelines state up-front that filters should be hidden only in cases of particularly persistent vandalism, in so far it is probably safe to establish that all hidden filters target some type of vandalism.
...
...
@@ -395,6 +395,48 @@ However, the classification was difficult for public filters as well, since ofte
While there are cases of juvenile vandalism (putting random swear words in articles) or characters repetiton vandalism which are pretty obvious, that is not the case for sections or articles blanking for example.
For these, from the edit alone there is no way of knowing whether the deletion was malicious or the editor conducting it just wasn't familiar with say the correct procedure for moving an article.
\subsection{A few notes on the labels/labeling process}
I started coding strongly influenced by the coding methodologies applied by grounded theory scholars~\cite[42-71]{Charmaz2006} (described in more detail in chapter~\ref{chap:methods}) and mostly let the labels emerge during the process.
In addition to that, for vandalism related labels, I used some of the vandalism types identified by the community in~\cite{Wikipedia:VandalismTypes}.
However, I regarded the types more as an inspiration and haven't adopted the proposed typology 1:1 since I found some of the identified types quite general and more specific categories seemed to render more insights.
For instance, I haven't adopted the 'addition of text' category since it seemed more insightful(syn!) to have more specific labels such as 'hoaxing' or 'silly\_vandalism', see below for definition.
Moreover, I found some of the proposed types redundant.
For example, 'sneaky vandalism' seems to overlap partially with 'hoaxing' and partially with 'sockpuppetry', 'link vandalism' mostly overlaps with 'spam' or 'self\_promotion', although not always and for some reason, 'personal attacks' are listed twice.
I have labeled the dataset twice.
One motivation therefor was to return to it once I've gained better insight into the data and more detailed understanding of it and use this newly gained knowledge to re-evaluate ambiguous cases, i.e. re-label some data with codes that emerged later in the process.
This process (syn) of labeling is congrous with the simultaneous coding and data collection suggested by grounded theory scholars~\cite{}.
Another motivation for this second round of labeling was to ensure at least some intra-coder integrity, since, unfortunately, multiple coders were not available~\cite{LazFenHo2017}. %TODO add page num; I also need to elaborate on methodoly here
During the first labeling, I looked through the data paying special attention to the name of the filters ('af\_public\_comments'), the comments ("af\_comments"), as well as the regular expression pattern constituting the filter and identified one or several possible labels. %TODO reword? I also looked at the comments, name and regex the second time..
In ambiguous cases, I either labeled the filter with the code which I deemed most appropriate and a question mark, or assigned all possible labels (or both).
There were also cases for which I could not gather any insight relying on the name, comments and pattern, since the filters were hidden from public view and the name was not descriptive enough.
However, upon some further reflection, I think it is safe to assume that all hidden filters target a form of (more or less grave) vandalism, since the guidelines suggest that filters should not be hidden unless dealing with cases of persistent and specific vandalism where it could be expected that the vandalising editors will actively look for the filter pattern in their attempts to circumvent the filter\cite{Wikipedia:EditFilter}.
Therefore, during the second round of labeling I labeled all such cases 'hidden\_vandalism' (all of them where nothing more specific was found).
And then again, there were also cases where I could not determine any suitable label, since I didn't understand the regex pattern, none of the existing categories seemed to fit and I couldn't think of an insightful new category to assign.
During the 1st labeling, these were labeled 'unknown', 'unclear' or 'not sure'.
For the second round, I intend to unify them under 'unclear'.
For a number of filters, it was particularly difficult to determine whether they were targeting vandalism or good faith edits.
The only thing that would have distinguished between the two would have been the contributing editor's motivation, which we had no way of knowing.
During the first labeling session, I tended to label such filters with 'vandalism?', 'good\_faith?'.
For the cross-validation labeling (2nd time), I intend to stick to the "assume good faith" guideline\footnote{\url{https://en.wikipedia.org/w/index.php?title=Wikipedia:Assume_good_faith&oldid=889253693}} myself
and only label as vandalism cases where good faith can definitely be no longer assumed/out of the question.
One characteristic/feature which guided me here is the filter action which represents the judgement of the edit filter manager(s).
Since communication is crucial when assuming good faith, all ambiguous cases which have a less 'grave' filter action such as "warn" or "tag", will receive a 'good\_faith' label.
On the other hand, I will label all filters set to "disallow" as 'vandalism' or a particular type thereof, since the filter action is a clear sign that at least the edit filter managers have decided that seeking a dialog with the offending editor is no longer an option.
%TODO compare also with revising codes as the analysis goes along according to Grounded Theory
The second time, I labeled the whole data set again, this time using the here quoted compiled code book and assigned to every filter every label I deemed appropriate, without looking at the labels I assigned the first time around.
I then compared the labels from both coding sessions. %TODO And did what?; how big was the divergence between both coding sessions?; should I select one, most specific label possible? or allow for multiple labels?
%TODO quote M's methodology book
%TODO disclose links to 1st and 2nd labelling
First round of labeling is available under \url{https://github.com/lusy/wikifilters/blob/master/filter-lists/20190106115600_filters-sorted-by-hits-manual-tags.csv}.
%TODO I actually need a final document where I compare both and decide on final (at least for this work) labeling I rely upon
\subsection{Editors' motivation}
\begin{comment}
# Filter according to editor motivation
...
...
@@ -598,6 +640,7 @@ some concrete users/cases (hidden filters, e.g. 4,21) and sockpuppetry (16,17)
If discerning motivation is difficult, and, we want to achieve different results, depending on the motivation, that lead us to the question whether filtering is the proper mechanism to deal with disruptive edits.
%TODO doesn't really seem related, maybe get rid of it altogether
Well, on the other hand, I'd say there are filters that seem to be there in order to protect from malicious activity (the vandalism filters) and such that kind of enhance the MediaWiki functionality: by providing warning messages (with hopefully helpful feedback) or by tagging behaviours to be aggregated on dashboards for later examination
Here, a detailed overview of the labels\footnote{Here, I use the words "codes"/"tags"/"labels" interchangeably.} used for the manual tagging of edit filters is provided.
\subsection{A few notes on the labels/labeling process}
I started coding strongly influenced by the coding methodologies applied by grounded theory scholars~\cite[42-71]{Charmaz2006} (described in more detail in chapter~\ref{chap:methods}) and mostly let the labels emerge during the process.
In addition to that, for vandalism related labels, I used some of the vandalism types identified by the community in~\cite{Wikipedia:VandalismTypes}.
However, I regarded the types more as an inspiration and haven't adopted the proposed typology 1:1 since I found some of the identified types quite general and more specific categories seemed to render more insights.
For instance, I haven't adopted the 'addition of text' category since it seemed more insightful(syn!) to have more specific labels such as 'hoaxing' or 'silly\_vandalism', see below for definition.
Moreover, I found some of the proposed types redundant.
For example, 'sneaky vandalism' seems to overlap partially with 'hoaxing' and partially with 'sockpuppetry', 'link vandalism' mostly overlaps with 'spam' or 'self\_promotion', although not always and for some reason, 'personal attacks' are listed twice.
I have labeled the dataset twice.
One motivation therefor was to return to it once I've gained better insight into the data and more detailed understanding of it and use this newly gained knowledge to re-evaluate ambiguous cases, i.e. re-label some data with codes that emerged later in the process.
This process (syn) of labeling is congrous with the simultaneous coding and data collection suggested by grounded theory scholars~\cite{}.
Another motivation for this second round of labeling was to ensure at least some intra-coder integrity, since, unfortunately, multiple coders were not available~\cite{LazFenHo2017}. %TODO add page num; I also need to elaborate on methodoly here
During the first labeling, I looked through the data paying special attention to the name of the filters ('af\_public\_comments'), the comments ("af\_comments"), as well as the regular expression pattern constituting the filter and identified one or several possible labels. %TODO reword? I also looked at the comments, name and regex the second time..
In ambiguous cases, I either labeled the filter with the code which I deemed most appropriate and a question mark, or assigned all possible labels (or both).
There were also cases for which I could not gather any insight relying on the name, comments and pattern, since the filters were hidden from public view and the name was not descriptive enough.
However, upon some further reflection, I think it is safe to assume that all hidden filters target a form of (more or less grave) vandalism, since the guidelines suggest that filters should not be hidden unless dealing with cases of persistent and specific vandalism where it could be expected that the vandalising editors will actively look for the filter pattern in their attempts to circumvent the filter\cite{Wikipedia:EditFilter}.
Therefore, during the second round of labeling I labeled all such cases 'hidden\_vandalism' (all of them where nothing more specific was found).
And then again, there were also cases where I could not determine any suitable label, since I didn't understand the regex pattern, none of the existing categories seemed to fit and I couldn't think of an insightful new category to assign.
During the 1st labeling, these were labeled 'unknown', 'unclear' or 'not sure'.
For the second round, I intend to unify them under 'unclear'.
For a number of filters, it was particularly difficult to determine whether they were targeting vandalism or good faith edits.
The only thing that would have distinguished between the two would have been the contributing editor's motivation, which we had no way of knowing.
During the first labeling session, I tended to label such filters with 'vandalism?', 'good\_faith?'.
For the cross-validation labeling (2nd time), I intend to stick to the "assume good faith" guideline\footnote{\url{https://en.wikipedia.org/w/index.php?title=Wikipedia:Assume_good_faith&oldid=889253693}} myself
and only label as vandalism cases where good faith can definitely be no longer assumed/out of the question.
One characteristic/feature which guided me here is the filter action which represents the judgement of the edit filter manager(s).
Since communication is crucial when assuming good faith, all ambiguous cases which have a less 'grave' filter action such as "warn" or "tag", will receive a 'good\_faith' label.
On the other hand, I will label all filters set to "disallow" as 'vandalism' or a particular type thereof, since the filter action is a clear sign that at least the edit filter managers have decided that seeking a dialog with the offending editor is no longer an option.
%TODO compare also with revising codes as the analysis goes along according to Grounded Theory
The second time, I labeled the whole data set again, this time using the here quoted compiled code book and assigned to every filter every label I deemed appropriate, without looking at the labels I assigned the first time around.
I then compared the labels from both coding sessions. %TODO And did what?; how big was the divergence between both coding sessions?; should I select one, most specific label possible? or allow for multiple labels?
%TODO quote M's methodology book
%TODO disclose links to 1st and 2nd labelling
First round of labeling is available under \url{https://github.com/lusy/wikifilters/blob/master/filter-lists/20190106115600_filters-sorted-by-hits-manual-tags.csv}.
%TODO I actually need a final document where I compare both and decide on final (at least for this work) labeling I rely upon
Def
Example <-- examples so far come from the 1st round of labeling
This section provides a detailed overview of all the codes\footnote{Here, I use the words ``codes'', ``tags'' and ``labels'' interchangeably.} used for the manual tagging of edit filters.
%TODO put all the labels in a table?
...
...
@@ -65,10 +20,6 @@ Example <-- examples so far come from the 1st round of labeling
\subsubsection{Structure related}
'bot\_vandalism'
Def: Vandalism caused by an automated agent
Examples: 277 "possible vandalbot"; 276 "scripted anomtalk/spoofed IP vandalism"
'page\_move\_vandalism'
Def: vandalism involving moving a page (i.e. renaming the page), mostly to some nonsensical name
(Wikipedia typology: "Renaming pages (referred to as "page-moving") to disruptive, irrelevant, or otherwise inappropriate terms.")
...
...
@@ -136,6 +87,10 @@ Examples: 154 "Macedonia naming conflict 2"; 19 "Replacement of "partition of In
\subsubsection{General vandalism}
\textbf{'bot\_vandalism'}\\
Def: Vandalism caused by an automated agent\\
Examples: 277 "possible vandalbot"; 276 "scripted anomtalk/spoofed IP vandalism"\\
'general\_vandalism'
Def: vandalism for which none of the more specific tags applied
Example:
...
...
@@ -186,7 +141,7 @@ Note: according to Wikipedia this behaviour constitutes harassment: "Posting ano
Example:
\subsection{Spam/malware/etc.}
\subsubsection{Spam/malware/etc.}
'spam'
Def: There is a "Spam" type of vandalism in the Wikipedia Vandalism Typology. However, I've got the feeling that I'm mostly labeling the cases listed there as "self promotion" or similar (although maybe not; This is the def: " Adding text to any page that promotes an interest that benefits the user, except in user space in a manner allowable under Wikipedia's guidelines
...
...
@@ -203,8 +158,14 @@ Note: according to Wikipedia this behaviour constitutes harassment: "Posting ano
Def: Malware is explicitely mentioned in the filter's name %TODO maybe combine phishing and malware
Examples: 243 "WikiMedia Viewer possible malware"; 429 "Possible malware attack" <-- only two instances
\subsection{Good faith}
'good\_faith'
Def: In ambigous cases, e.g. editor blanking sections, we assume good faith as long as there are not any indicators to the contrary. One such indicator would be the filter action: filters set to "warn" try to communicate with the editors, point out potential pitfalls to them and give them the opportunity to update and publish the edit (or publish the edit regardles, if they think all is good). Filters set to "disallow" on the other hand, do not seek to guide an editor but rather protect the encyclopedia from harmful content.
Examples: 180 "Large unwikified new article"; 98 "Creating very short new article"; 657 "Adding an external link to a disambiguation page" (used to be labeled 'good\_faith?', but since actions are "warn,tag", according to my newly defined guidelines, this is a good\_faith filter)
\subsection{Disruptive editing which is not outright vandalism}
\subsubsection{Policy violations}
%\subsection{Disruptive editing which is not outright vandalism}
'copyright\_violation'
Def: Filters targeting potential copyright violations: e.g. images without license information, ..
...
...
@@ -242,14 +203,8 @@ Note: according to Wikipedia this behaviour constitutes harassment: "Posting ano
Def: Filters targeting SEO edits (mostly, explicitely mentioned in the filter name)
Examples: 36 "SEO push University of Atlanta"; 682 "SEO/Attack page"; 554 "top100 blog charts" (bc of this and the Daily Mail sources, I am contemplating creating a 'unreliable\_sources' label)
\subsection{Good faith}
'good\_faith'
Def: In ambigous cases, e.g. editor blanking sections, we assume good faith as long as there are not any indicators to the contrary. One such indicator would be the filter action: filters set to "warn" try to communicate with the editors, point out potential pitfalls to them and give them the opportunity to update and publish the edit (or publish the edit regardles, if they think all is good). Filters set to "disallow" on the other hand, do not seek to guide an editor but rather protect the encyclopedia from harmful content.
Examples: 180 "Large unwikified new article"; 98 "Creating very short new article"; 657 "Adding an external link to a disambiguation page" (used to be labeled 'good\_faith?', but since actions are "warn,tag", according to my newly defined guidelines, this is a good\_faith filter)
TODO: label cases additionally with scope/area the filter is targeting
%TODO: label cases additionally with scope/area the filter is targeting
