50% of people think this is the best blog you will read this month
Did you know that 42% of Brits admit they cannot remember the last time they laughed out loud, while 32% believe they haven’t as much as giggled in the last month? Well, it’s true, according to Sky TV last year.
Or that one in five flute players say Lizzo has had an impact on their playing or attitude to the instrument? So says the British Flute Society.
How about that 51% of people consider a Jaffa Cake to be a biscuit and only 38% see it as a cake (thanks, YouGov)?
Journalists love a survey. It’s easy, often eye-catching copy that pretty much writes itself. For that reason, those looking for publicity love a survey too – it can hammer home a message, be an understandable way into a difficult or staid topic, or show you’re a bit of fun.
From a journalist perspective, the main hurdle is credibility. This obviously depends on the target and the context, but when it comes to largely serious publications like legal magazines and websites, the first question can simply be one of numbers.
The British Polling Council provides useful guidance to journalists on reporting surveys. The basic five questions to ask are: How and when was the poll conducted? Who was interviewed? How were they chosen? Who sponsored or paid for it? What questions were asked?
The Legal Services Consumer Panel is a good example of an organisation that ticks the boxes. It commissions a reputable polling company to conduct a survey every year that tracks attitudes to buying and using legal services.
This generally has a representative sample of around 3,500 people, more than enough, according to statistical theory, to produce a broadly accurate picture. Plus, because the panel has been doing it for more than a decade, you can see trends too.
But surveys like this are expensive and not many businesses, certainly in the legal sector, are willing to spend what is required (although there are ways to reduce the cost).
One way some businesses that are not sector specific try and get bang for their buck is to commission a survey with, say, 2,000 respondents, consisting of 100 people in each of 20 different industries, so that they can achieve coverage in a wide range of trade titles.
As a journalist, you have to be very careful with these, and indeed one of them was the spur for this blog.
I received a press release earlier this month with the headline ‘Almost two in five legal businesses admit to using ChatGPT for documents’. The company surveyed “2,000 workers across varying job levels and sectors”, to find out more about the use of ChatGPT in the workplace. At 38%, legal came behind only advertising in sector use.
It sounded like a good story – everyone’s writing about generative AI at the moment. There wasn’t much meat to the legal sector angle but fortunately there was a link to a spreadsheet with the results. Hopefully, there would be more in that.
Unfortunately, this indicated that they had drummed up a grand total of… eight legal sector respondents.
Yes, you read that right. Eight. Three of whom (ie, 37.5%) admitted using ChatGPT for documents.
I asked the (external) PR how they could justify sending this out. Perhaps unsurprisingly, despite my chasing, I have yet to receive a reply. Even less surprisingly, I didn’t report the story on Legal Futures.
This is an extreme example but I have ditched plenty of legal PR survey stories after digging into the sample. It’s usually a telling sign when the press release doesn’t mention the sample.
Proper polling companies (which also know how to ask the questions) will not allow customers to use their work this way. But, frankly, some PRs are banking on journalists being too busy, too desperate or too lazy to check. And more often than I would like to be the case, they are right.
In legal PR, you often see much smaller but, crucially, targeted samples – 100 law firm leaders or general counsel is a favourite.
The more targeted, the better, of course. Given me 100 leaders in the top 200 firms, or in personal injury firms, over 100 leaders in law firms of all sizes, because on many topics a magic circle firm has little in common with a high street practice.
But these have a decent chance of coverage, especially if the number of respondents is in three figures (100 just sounds so much better than 99). And you can sometimes get away with small numbers if the subject is itself sufficient niche or if it is a qualitative study that has made up for the sample by really digging into the topic and delivering interesting results.
Numbers aren’t the be all and end all – polls conducted via social media, say, can attract a big response but the self-selecting nature of respondents damages their credibility significantly.
The other main issue for me as an editor is the purpose of the research. Putting public bodies like the Legal Services Consumer Panel to one side, most businesses commission surveys to prove there’s a problem that, magically, they can solve. We get that – it’s the quid pro quo for the story.
But if you bash me over the head with that message, making the research so self-serving that the issue becomes too narrow, then I may well not cover it.
Surveys may be easy copy but they are not easy. Unless, as the headline to this blog demonstrates, you simply survey yourself and your spouse.
