Critiquing the critics

A film’s fortunes hang on the opinion of a few newspaper critics. Should we trust them?

LABKULTUR has already covered one major challenge for creative industries businesses: how to estimate the demand for a product before it has been released. How can you estimate how many people will want to buy your DVD or see your film in the cinema. What the critics say makes a difference. According to the New York Times, an Oscar nomination is worth a 22% increase in sales, or $20 million dollars.

So it’s big business how your film is critically received. And exclusively for LABKULTUR I have crunched the numbers.Some film review syndication sites already give you an average of the score each reviewer gives a film, like Rotten Tomatoes does. This is a creditable approach for getting a reasonable figure.
But to understand how accurate critics are, the only benchmark you have is other critics. By measuring how much they disagree we can work out how accurate they are.

Crunching the data
Taking a sample of 20 recent releases and five reviewers, recording how they scored each film out of five, and tallying up the data has allowed me to conclude that in general - critics more or less agree on whether a film is dreadful or brilliant or not, but not much more. Of my sample of five, Empire Magazine is the most generous reviewer. On average, it scored 0.6 of a star higher than the meanest, which is Time Out London.

And on average, the standard deviation - the different between their scores and the average - of each review was about 0.7 of a star. This isn’t too bad. It means that if there are five audience members, one might rate it one star, one might rate it three stars, but most rate it two stars (as the critics did with Happy Feet II).

In other words, from readin any single review you can guess that if it scores 1 it’s a stinker, 5 it’s a good film, but anything in between and you’ll have to check another review too. And the critics agree on some films - Alvin and the Chipmunks: Chipwrecked was panned, The Sitter was universally agreed as mediocre.

But not on others, with Hugo and War Horse either great or mediocre, depending on who you ask. This chart shows the full list. The lower the score, the more critics agreed.

Which films do critics agree on?

Different directions

Of course, the critics may want it another way. It’s clear from the success of the tabloid press that accuracy is not a well regarded factor in British journalism today. If they wanted to be scandalous and interesting, then the standard deviation would be very different.

And there’s a school of thought that says that left-wing newspapers and right-wing newspapers will have different opinions. There was some evidence of this, with the pro-Thatcher Telegraph giving the Iron Lady 4 stars, while the anti-Thatcher Mirror gave it two. The famed theatre critic, Kenneth Tynan, said that “a critic is someone who knows the way but can’t drive the car”. It is worth remembering that critics have different destinations.

The full data: