Which single Wikimedia metric would inspire you most?

The statistics we maintain within the Wikimedia movement broadly fall into two categories. Part is there to inspire us, with limited feedback to our daily activities. Part is there to reveal patterns or to signal mishaps, both meant to lead to actionable knowledge. Of course the boundary is not strict. Your qualification may depend on how you judge your own capacity to influence trends. Broadly speaking much of Wikistats currently falls inside the first category, I would say: numbers to inspire.

I do not mean to belittle the importance of general metrics. I would rate the superb talks by web celebrity Hans Rosling as being foremost inspirational, less actionable, for all of us normal earthlings, who seldom make decisions with global impact (except at the ballot). If you’re a dedicated contributor to one of our projects it can lighten you day when trends are favorable. It can awe the press and the public at large, it feeds into our fundraiser, even helps to open doors (be it to photo shoot opportunities, or to GLAM institutes). But other than that it is nice to know.

Much as I welcome more actionable metrics, still my focus in this blog post is on that inspirational aspect. Forget research, forget operations. As a thought experiment I want to raise this hypothetical question: If for one year we could only update one metric about any aspect of the Wikimedia movement, which one would help best to inspire you? It could be a complicated metric, but just one. Let me try to answer that myself. I’m side stepping issues on data gathering. For sake of argument let’s assume any metric is spot on. The issue is more how unambiguous each metric is and how much it tells about our success.

Would I choose page counts? Definitely not. We use this metric too often to impress our audience. Too often some of our projects play one-upmanship (or so it seems) with article creation bots. Also our definition of article is very permissive, and the barrier for importing data collections seems to lower steadily. Pardon the hyperbole: we could add millions of ‘articles’ from a star database any day. See also next metric.

Number of languages? We cite this everywhere so this must be an number which inspires. You may want to know that 83 of our wikis (of which 10 Wikipedias) are locked for updates, due to lack of editors. 61 or our 281 Wikipedias have less than 1000 articles. Many of those wikis exist for half a decade. And even some wikis with 2K+ articles may consist of year stubs mostly, at least that happened a lot in early years. There is a huge wikistats table showing article counts per wiki per month. Note how many wikis grew by 1K or 2K articles at once, early in their history, often bot induced stubs. That still leaves a whopping number of Wikipedia’s which are an unbelievable success. No need to stretch the imagination I would say.

Editors? Likely candidate, but the number is somewhat ambiguous. We could gain in editors but lose faster in average editor activity, for instance because of ever increasing competition of other web initiatives (social sites). Even a large influx of new editors by itself could overtax our veteran users, and ultimately cause burn-out and estrangement. While this seems to be a problem we can only dream of in many projects, it is already realistic in some others.

Edits? Attractive option. We already count edits by registered users on primary content (mostly namespace 0). Ideally we would substract reverts and reverted content (vandal fighters are arguably our most important lifeline, but if they would need to step up their activity due to increase in vandalism that would not exactly put smiles on our faces). Although edits on talk pages etc are very important, if they don’t lead to extra encyclopedic content, they are irrelevant, so let’s exclude those (like wikistats already does). This leads to ‘non reverting/reverted edits, by registered and anonymous users (thus excluding bots) on primary content’.

Page views? Better navigation tools could have a negative effect on our page view count, and we’d still welcome that. It has been said that Google strives to get people from their site asap, as that signals they found what they’re after faster. Only part of our traffic comes from intentional searches. There is also lots of random browsing, which is great fun and educational, but still I would rate that differently (see below).

Unique visitors? If for some reason the average number of visits per person dropped faster than the number of visitors rose, our usage would decline and from this one metric alone we wouldn’t know.

Completion of targets? I said our metric could be a complicated one. So what about a (weighed?) average of the percentage fulfillment for each of our strategic targets for 2015? While useful for general guidance, I personally think targets in many cases are more about how optimistic we were at the time they were set, and less about realistic expectations (to some degree the most realistic expectations about complex social movements are: no expectations at all, trends might be distinctly non-linear). Goals define a direction, targets pretend travel speed can be predicted, which works better in explored than in new territory. Also, for some of our targets we still have no operational definition, or method to measure them reliably (content quality, gender ratio).

Total amount donated? If we would raise funds (with lower intensity) all year round the amount donated monthly could seem a nice metric to measure appreciation by our readers, and thus our impact on the world. But the average amount donated could fluctuate because of external influences. Also the message to the public, and how this is delivered, is always in flux, banners and stories evolve, so the playing field to compare consecutive measurements is not level.

Number of donations? That comes closer. I assume external (economic) factors will influence the average amount donated more than the decision to donate at all. Just my guess.

Gender ratio of our editors? If that would improve it would be hugely inspiring. But not my first choice, if we can sneak peek so little from under our blindfolds. (It could even mean more male editors dropped out).

Total visits? This would be my candidate of choice today. I imagine most people visit Wikipedia foremost to get a specific question answered. After their initial curiosity has been fulfilled they might browse further and learn more, but their initial question is what drove them to visit. If someone visits us once a month Wikipedia clearly has a limited role in their lives. For that share of our visitors which visit us several times a week or even more (*) something essential has changed. They have learned that the answer to many of their questions is within reach, and affordable in terms of time invested. They might even have become more inquisitive in general. That I find very inspirational.

*According to comScore we received 2.45 M visits to all our projects in January 2012, and 2.50 M a year later, with somewhat lower values in between (mobile access may be underrepresented). On average that would be slightly over 5 visits per unique visitor per month. Like with other metrics above, there are complications: seasonality, uneven distribution, socially or geographically, and external influences (mainly rise of mobile, which actually will amplify this metric). 

Note these are my personal opinions, some based on limited anecdotal evidence.

I would love to hear your feedback.  Which metric would you choose?

 

 

 

 

 

 

 

 

 

 

This entry was posted in Musings. Bookmark the permalink.

13 Responses to Which single Wikimedia metric would inspire you most?

  1. That would inspire me to be more active? My first choice would be impact of my edits. That is, the amount of improvement I made in the lives of each of the readers who comes by, totaled by month. But that’s hard to measure.

    I’d settle for estimated impact of my edits. Which would be something like article views per month times estimated net contribution I made to the article. The nice thing about a contribution metric would be that contribution doesn’t have to be zero sum. There are all sorts of good things that someone can do to an article, and all of those could count.

  2. I like William Pietri’s idea – I’ve proposed similar things myself. The overall success of the project is nice to think about, but if you want to inspire people, give them a sense of how they matter.

    Another idea for personal metrics – have edits rated by other editors. Imagine a page where you get to see a random recent edit stripped of authorial information, and you can then “hot or not” the contribution. Include every kind of contribution – regular edits, reverts, talk pages, page deletions, etc. Then the score for each user is tallied. To avoid creating an incentive to game the system, keep the score private; maybe only the user can see their score, and perhaps some people who need to investigate troublesome users. Knowing your score would give incentive to people to the unsung heroes, and may gently nudge rude people or trigger-happy deletionists to reconsider their behavior.

    A more traditional stat – new Wikipedians. Quantify how many people are in the funnel towards becoming regular editors. Could be done with standard engagement metrics.

  3. ping pong says:

    Edits (edits – reverts) and active users.

    active users x edits / reverts x acconts

  4. Nemo says:

    My choice would surely be the number of words.
    It’s one of the most reliable numbers in wikistats, because it doesn’t oscillate too much, it’s not affected by bots or users doing tons of tiny changes, it discards (ideally) the non-linguistic automatically-added content (like new bot-created articles with an imported infobox and a category), it’s not affected by external influences like bot-visits or slashdot-effects, it balances the main focus of individual editors (boasting editcount) and wikis (articles) and finally it’s strictly measurable, unlike pageviews (requires lots of polishing) or unique visitors (statistical esteem)… if you can agree on what’s a word and what markup you should count.

  5. Erik says:

    Such wonderful and original ideas!

  6. Erik says:

    @William, @Neil,

    It would be neat to have page views stats on the wiki page itself.

    Pending that, perhaps someone could make a javascript macro which directly links on any wiki page to per article view stats on grok.se: say from http://en.wikipedia.org/wiki/Rembrandt to http://stats.grok.se/en/201303/rembrandt.

    A monthly report with say 10 most viewed pages where an editor contributed to seems doable in theory, even when in the current Wikistats context it would mean a huge sort merge batch job. I wonder should any edit be counted, even a typo fix? I guess not. And here it becomes fuzzy, how to make that selection of edits which would qualify? Fuzzy in how to define criteria and how to make this search perform well.

  7. Erik says:

    @Neil, I like your idea for peer reviewed edits. But it seems quite an undertaking to do this well, and even have it integrated into the wiki. Yes gaming the system could be become an issue with public feedback, but public recognition is also a big motivator. Then again, people could start edit/revert wars over unwelcome public feedback. Hmm, this seems subject matter for researchers (E3 team?).

    BTW not sure what you would want to see specifically about ‘New Wikipedians’? We publish in Wikistats on those already, and WMF did cohort analysis. What I will do is tally New Wikipedians deduplicated (based on merged editor counts for all projects).

  8. Erik says:

    @ping pong You may want to read my last blog post http://infodisiac.com/blog/2013/03/monthly-edits-on-wikimedia-wikis-still-on-the-rise/ which is not the definitive answer but a step in that direction.

  9. Erik says:

    @Nemo, I would also prefer word count over page count, for reasons given in the article. Yet it could become another victim to easy gamification, where larger texts are copied from external databases. And just ignore bot created articles would be rather course (most US town on English wikipedia started as bot created articles).

    Come to think it: how about weighing the articles by word count and (non bot) view count? This would factor in content depth and relevance to our audience.

  10. ping pong says:

    Sources!

    Number of tags

  11. Erik says:

    Ping pong, could you elaborate please? I don’t get it.

  12. ping pong says:

    Oh, lol – number of “ref tags” or “ref tags per article”

    tag with does not work (or just works) here 😉

    Of course many articles have “bibliography” section without “references”, but it could be (good) indicator of reliability

  13. jane023 says:

    Erik,
    I think a good metric for myself is measuring how many languages my articles are translated into. On the English Wikipedia, there is a whole complicated set of stub-start-A-B-C-GA-FA statuses for articles, but for me, if a stub I created gets translated into 3 languages, then I feel that someone has felt just as interested in the subject as I did. I also do look at what gets created and will occasionally merge stuff back into the English article if I feel it adds value to mine.

Leave a Reply

Your email address will not be published. Required fields are marked *