Occasionally Scott Alexander will get annoyed enough by a statistic that's going around Facebook to write up an in-depth piece on how it's completely missing the point. These are generally interesting and informative. The most recent one is no exception, but instead of simply talking about the flaws in the the statistics Scott goes on to say:

I have been debunking bad statistics for a long time. In medicine, in psychology, in politics. Click on the statistics tag of this blog if you don't believe me. Yet the feminist blogosphere is the only place where I consistently see things wrong by factors of several thousand get reblogged by thousands of usually very smart people without anyone ever bothering to think critically about them. Like, thirty five thousand feminists – including some who self-identify as rationalists! – saw an article that literally said a guy was more likely to get hit by a comet than get falsely accused of rape, and said "Yeah, sure, that sounds plausible."

So please permit me to keep griping just one moment longer. Do not trust anything that comes out of the feminist blogosphere. When you see something in the feminist blogosphere, your default assumption should be that it is approximately as honest as this Clymer article.

That's a strong claim, not that feminist blogs tend to present bad statistics but that they're much worse than everyone else. I've also looked into tricky statistics before and found problems (efficiency of meat, fair trade opposition) and these weren't feminist blogs. Without looking into it my guess would be that most groups making political claims present misleading statistics. Not because they're trying to mislead, but because not making mistakes is hard. Plus even if someone does notice a stat seems fishy they don't evaluate it because that seems like being a traitor to their cause, doing the work of the political adversaries.

But let's actually look into this, with as much of a survey as I can find time for. I've collected the first ten feminist blog posts I find that use statistics, and the first ten "men's rights" blogposts for the same, and then look into them to see how they compare for accuracy.

For finding feminist blog posts that use stats, Jezebel has a nice tag. I selected the first ten, excluding any that didn't present statistics. For men's rights posts went down r/MensRights and selected the first 10 self posts and links to blog posts that had positive upvote counts. The posts are:

Feminist

• Breaking News! Female Protagonists Are Still Rare in Gaming and Movies
• A State-by-State Look at Where Kids Are Not Being Vaccinated
• UK Women Under 50 Diagnosed with Breast Cancer in Record Numbers
• Show This Depressing Graph to the Rape Apologist in Your Life
• Teens Are Less Harlot-y Than Ever
• America's Favorite Songs Are Almost Always About Sex
• Why Earn Money When Your Husband Can Do It For You?
• Teen Birth Rate At Record Low
• Women Beat Men in Streaming-Teevee Piracy
• Report: Television Violence Against Women On The Rise

MRA

• Feminism is Not Mandatory
• UK's Sex-Crazed And Corrupt Justice System
• The Men's Human Rights Movement - Core Issues
• The (Silent) War On Men
• Update: 16 year-old schoolboy dies from self-immolation
• Men forced to penetrate as often as women were raped
• Women in tech: I make my own sandwiches, go bake your own pie.
• Some important findings from the ONS crime stats: Intimate & sexual violence
• Prostrate Cancer and The Wage Gap
• UK:CSEW 2012/13 released: Male victims of SOA 4.4.c-d still not counted

I'll save the rest for later and just look at the first one here: Breaking News! Female Protagonists Are Still Rare in Gaming and Movies. They present a few stats:

Women make up 45% of the gaming community and 4% of the main characters of the 25 biggest games of the year.

This isn't so good. For the "gaming community" they're including people who play "casual" games online or on phones, while the "biggest games" is measured in top sellers at retail which excludes those categories. If we figure popularity by players instead of money we have the problem that many popular games don't even really have "main characters"; consider Candy Crush Saga.

(On the other hand, the primary bird in Angry Birds, Red, would have made sense as either gender but they went with male.)

Men make up 35% of the cinema audience and 84% of the protagonists of the 25 biggest movies of the year.

This stat seems reasonable; we're comparing the same categories. Better would be something like "male protagonists as a fraction of movie-views" but that seems like a lot harder to calculate and probably not that different.

A third (more or less) of the top ten and a single blockbuster comedy is by no means equal, but it is an improvement over the 2012 field, which featured only two top-ten, female driven movies (The Hunger Games and Brave). It's also light years better than the 2011 field, which featured no female-driven movies in the top ten, unless you count Breaking Dawn, which would be perhaps too optimistic. The progress may be slow, but the "bottlenecks" ... are slowly being broken up by the egalitarian forces of the internet

That 2013 (3 of 10) was better than 2012 (2 of 10), which was better than 2011 (0 of 10) is not really enough to detemine that the internet is breaking down barriers. Let's make a larger chart, looking at the last 20 years of best selling movies and counting the ones where a woman is credited first on imdb's autocomplete:

Things are definitely improving, but over a longer timescale than you'd think from 2011: 0, 2012: 2, 2013: 3. They also shifted from "top 25" counting to "top 10" counting for the quote above. What happens if we look at the whole top 25?

It looks like much of this change in the top 10 is movies breaking the top-10-barrier, not an overall increase in female-led movies. Maybe we can attribute this to the internet but it's not as much of a change it looked like at first. This sets us up for a different feminist argument: men are most of the main characters and it's not getting better. It's less positive, but still in line with Jezebel's politics.

(Data is at the end, footnote [1]. Note that this data isn't ideal; first billing can be more about popularity than role in the film. For example, Hairspray billed John Travolta (Edna) first even though Nikki Blonsky (Tracy) was clearly the main character. This probably happened several times in the data, but I don't know movies well enough to go by "main character".)

Summary: three claims, one was fine, one was conflating two different populations, one was extrapolating from too little data and overstated recent change. Even the ones that were off were far from being off by "factors of several thousand".

[1] Movie data:

year	female-led movies in top 10
2013	Hunger Games: CF, Frozen, Gravity
2012	Hunger Games, Twilight: Breaking Dawn II, Brave
2011	Twilight: Breaking Dawn
2010	Alice in Wonderland, Twilight: Eclipse, Tangled
2009	Twilight: New Moon
2008	Twilight
2007	none
2006	none
2005	Narnia, King Kong
2004	none
2003	none
2002	My Big Fat Greek Wedding, Chicago
2001	none
2000	Scary Movie
1999	Runaway Bride, The Blair Witch Project
1998	none
1997	My Best Friend's Wedding
1996	Twister
1995	none
1994	none

year	female-led movies in rank 11-25
2013	The Heat
2012	Snow White and the Huntsman, Prometheus
2011	The Help, Bridesmaids, Super 8
2010	Salt, Valentine's Day, Black Swan
2009	Monsters vs Aliens, The Proposal, Cloudy with a Chance of Meatballs
2008	Sex and the City, Mamma Mia, Wanted, Four Christmases
2007	Juno, Enchanted
2006	The Devil Wears Prada, The Break-Up, Dreamgirls, Scary Movie 4, Charlotte's Web
2005	Flightplan, Monster-in-Law
2004	The Village, The Grudge, Million Dollar Baby, The Princess Diaries II
2003	Freaky Friday, Scary Movie 3, How to Lose a Guy in 10 Days
2002	Lilo and Stitch, The Ring, Sweet Home Alabama, Panic Room
2001	Tomb Raider, Spy Kids, The Princess Diaries, The Others, Legally Blonde, America's Sweethearts, Save the Last Dance
2000	Erin Brockovich, Charlie's Angels, Miss Congeniality
1999	Double Jeopardy, Pokemon: The First Movie
1998	Mulan, The Rugrats Movie, Shakespeare in Love, Stepmom
1997	Scream 2, Contact, I Know What You Did Last Summer
1996	The First Wives Club, Scream, The Hunchback of Notre Dame
1995	Dangerous Minds, While You Were Sleeping, Congo
1994	The Client