Spare a thought for the scientific studies churned out by the champions of public health, they only have a life expectancy of 48 hours. They are born as press releases on the day before publication and live on in the media for another 24 hours before being dropped in favour of fresh health scares and new revelations. True, they survive in libraries and on the internet where they exist in a sort of half-life but it is not an existence any of us would relish. Here, they are ripped apart by less credulous critics and suffer the indignity of having their methodology mocked and their conclusions ridiculed.

Since the only motive behind these studies is to generate publicity towards the epidemic of smoking/drinking/bacon/salt/mobile phones (delete as appropriate) it is perhaps only humane that public health practitioners have sought to spare them the humiliation of a half-life by bypassing publication altogether.

The most notorious recent example of what Dr Michael Siegel calls "science by press release" came in September last year when Dr Jill Pell announced that Scotland's smoking ban had brought about a 17% reduction in heart attacks. Despite her conclusion defying everything we know about human biology, it was immediately picked up by news organisations around the world. I vividly remember waking up to Radio 4's Today programme as presenter John Humphrys failed to ask even the most basic questions of the health spokesman who was lauding the 'study'. (Humphries: "And you believe this was because of the smoking ban?" Interviewee: "Yes." Humphrys: "Thank you very much. And now the sport...") It was not what you would call a grilling.

On BBC television, Peter Donnelly, Scotland's Deputy Chief Medical Officer, was afforded an even easier ride. After talking at length about the number of lives that were saved thanks to the smoking ban, the intrepid BBC reporter asked the single question: "How pleased are you with this?" The answer, of course, was "We're absolutely delighted."

Jill Pell has since gone to ground and today refuses to talk about the study but at the time she was far less reticent. "The primary aim of smoking bans," she said, "is to protect non-smokers from the effects of passive smoking. Previous studies have not been able to confirm whether or not that has been achieved. What we were able to show is that among people who are non-smokers there was a 20% reduction in heart attack admissions. This confirms that the legislation has been effective in helping non-smokers." (1)

To The Times, she said: "The difference between our study and earlier ones is that we have been able to show an effect in people who have never smoked. That can only be due to lower levels of passive smoke."

The day after the Today programme first publicised the findings, the British press covered it en masse and in wide-eyed wonder:

The Daily Mail: "Further dramatic evidence emerged last night to show that banning smoking in public reduces the rate of heart attacks." (2)

The Times: "The ban on smoking in public places in Scotland is already beginning to have an impact on the nation's health."

The BBC: "The Scottish smoking ban has led to a significant advance in public health, the most detailed scientific study of the measure so far has suggested."

All this related to a Powerpoint presentation given at an Edinburgh conference in September 2007. The lead author of the report was Dr Jill Pell of the Department of Public Health Medicine at Greater Glasgow Health Board. The study was not published then and it has not been published since so what do we really know about it?

Firstly, the conference was called 'Towards a Smoke-Free Society' ("an exciting conference that will move action on smokefree public places a stage further", according to the programme). Dr Jill Pell is a Consultant in Public Health Medicine at the Greater Glasgow Health Board who appeared on BBC Television on March 24 2006 predicting that heart attack incidence would fall by 10% as a result of the Scottish smoking ban which came into force two days later. Her presentation was given on 10 September 2007 and was entitled 'Testing the Montana Hypothesis: Results from Scotland.' The 'Montana Hypothesis' is a reference to the much-derided American study which claimed that heart attacks fell by no less than 60% after the small town of Helena, Montana brought in a smoking ban. That study was also launched with a Powerpoint presentation at an anti-smoking conference (in Chicago, 2003). It was co-authored by the high priest of California's anti-smoking movement, Stanton Glantz, and was widely ridiculed for its tiny sample group - none of whom were asked any questions about smoking or secondhand smoke exposure. When it was published a year later in the British Medical Journal the 60% had been mysteriously dropped to 40%.

The Scottish study was devised shortly before the Scottish government implemented its nationwide smoking ban and its methodology was outlined in the March 2006 issue of the Journal of Public Health, in an article entitled 'Legislation on smoking in enclosed public places in Scotland: how will we evaluate the impact?' Jill Pell was a co-author of this document under the aegis of a body with the tenuous, finger-wagging acronym of STOPIT (STudy Of Public place Intervention on Tobacco exposure). I reads, in part:

"STOPIT is a prospective study designed to test the hypothesis raised by the Montana study that a reduction in ETS [Environmental Tobacco Exposure] exposure is accompanied by a rapid reduction in the incidence of acute coronary syndrome (ACS). STOPIT's prospective design allows patients' smoking status and levels of ETS exposure to be determined thus overcoming some of the methodological problems associated with the Montana study...

Aim: To determine change in the incidence of acute coronary syndrome

Design: Multi-centre prospective study of hospital admissions for acute coronary syndrome. Entry criteria: chest pain + raised troponin on admission/within 12 h[ours].

Data collection: Continuous May 2005-April 2007. Research nurse-administered questionnaire on smoking status and self-reported exposure; admission blood sample (for cotinine assay)."(3)

In the end it seems that the sample group came from nine Scottish hospitals. According to The Guardian, these included hospitals in Edinburgh, Glasgow, Paisley, Dundee and Lanarkshire and made up 63% of heart attack admissions in Scotland. All sources agreed that there were 3,235 heart attack admissions in these areas before the ban and 2,684 after it. Among non-smokers, the figure was 1,630 and 1,306. This worked out as a reduction of 19.9% amongst nonsmokers and 14.1% amongst smokers; 17% overall.

This much was agreed but elsewhere, confusion reigned. Scotland's smoking ban began on March 26 2006. According to The Guardian (who misspelt Pell's first name as Gill throughout) the researchers compared the 12 month period before the ban with the 12 months after the ban. The Times and The Scotsman disagreed, both claiming the test period was just 10 months.

According to The Guardian:

"The number of all heart attack admissions at these hospitals had fallen by more than 550, from 3,235 in the year to March 2006, to 2,684 in the year to March 2007." (4)

But according to The Times:

"In the ten months of the year leading up to the ban, there were 3,235 admissions, while in the matching period after the ban, the figure was 2,684." (5)

The confusion was shared by senior members in the Scottish public health movement. The country's deputy Chief Medical Officer told the BBC that the study had looked at two 12 month periods but Dr Lawrence Gruer, Pell's colleague at the Greater Glasgow Health Board and also talking to the BBC, referred to two 10 month periods. A small but significant discrepancy: 12 months would be the obvious length of any study of this kind. To pick anything else would suggest cherry-picking; arbitrarily excluding two months immediately raises the suspicion that these months did not contain the kind of data that supported the desired hypothesis. It would be strange if two independent journalists got it into their head that 10 month periods were involved and it is stranger still that Trish Groves - who had actually attended the conference - reported that the period in question was just 6 months in her account for the British Medical Journal:

"I walked into Edinburgh's impressive international conference centre just as a couple of minibuses pulled up outside, spilling a gaggle of protesters armed with cigarettes, Scottish flags, and placards decrying "junk science"... Another study from the Scottish evaluation, by Jill Pell and colleagues, reported a 17% reduction in admissions from acute myocardial infarction in nine hospitals in the six months after the ban." (6)

A further problem with the timeframe is that in the original proposal it was explicitly given as being May 2005 to April 2007. The first thing to mention is that the decision to begin the STOPIT study came in March 2006, almost a year after it was supposed to start! This strongly suggests that the 'study' was no more than a totting up of statistics from a selection of hospitals and not, as was later claimed, a more detailed study in which patients were asked specific questions relevant to the smoking ban (7). But also note that the ban came in at the end of March 2006. This allows 11 months before the ban (May '05-March '06) and 13 months after it (April '06-April '07). Since the ban came in on 26 March, it is plausible that the researchers did not count March 2006 as being either pre-ban or post-ban. If so, that would leave a pre-ban period of 10 months against a post-ban period of 13 months. Whichever way you slice it, they could not have compared two 12 month periods, despite what The Guardian and the Scottish government claimed on the day the results were announced (8). A 10 or 6 month period therefore appears to be more likely.

If there were doubts about the length of time involved, there were also questions about what was actually being studied. This should have been straightforward. The proposal outlined above specifically referred (3 times) to acute coronary syndrome but, according to the BMJ writer (above) the Edinburgh conference heard only about acute myocardial infarction. There is a big difference between the two; in simple terms, the former describes a range of chest pains while the latter is what is known to the layman as a heart attack. It is possible that the BMJ writer was mistaken - as she may have been about the six months - but the same team presented the same data two months later, this time under the unambiguous billing of 'Changes in myocardial infarction incidence and mortality following the Scottish smoke-free legislation'. It seems clear that the team set out to study acute coronary syndrome but ended up reporting on acute myocardial infarction. Why?

Finally, there is the question of the sample group. As The Guardian explained, Pell's study involved "nine hospitals, in areas including Edinburgh, Glasgow, Paisley, Dundee and Lanarkshire, [which] accounted for 63% of heart attack admissions in Scotland." Why was the study limited to these hospitals? It is clear Pell had access to the data from the whole country - how else could she know that these areas saw 63% of heart attacks? And if she didn't, she should have just waited a few weeks for them to be published to the whole world on the ISD Scotland website (9). As it turned out, it was this website that would be her undoing. These are the statistics it gives for acute myocardial infarction admissions in the whole of Scotland for the three months before and after the ban:

[Figures show admissions in 2005 compared with those in 2006]

January: 740/791 (a rise of 9.5%)

February: 687/678 (a fall of 1.3%)

March: 810/709 (a fall of 14.2%)

April: 747/689 (a fall of 8.4%)

May: 740/700 (a fall of 5.7%)

June: 703/627 (a fall of 12.1%)

Heart attack incidence has been falling in Scotland for many years, as it has in most other Western countries. The year-on-year rise in January is very unusual but it is the nature of statistics to go up and down, usually for no definable reason beyond the certainty of chance and there is no evidence of a sharper than usual fall in heart attacks after the smoking ban. Quite the reverse, in fact, since the biggest decline in this period came in March 2006 when there was no smoking ban (apart from the last 6 days of the month).

If the smoking ban had an immediate effect on heart attack rates, one would expect this to be seen in April and May. In February, March, April and May 2006 the number of heart attack admissions in (the whole of) Scotland were 687, 810, 747 and 740 respectively. Because some months are longer than others it is necessary to divide the admissions by the days in the month to see the real trend. The average daily number of admissions were as follows:

February 2006: 24.54

March 2006: 22.87

[Smoking ban]

April 2006: 22.97

May 2006: 23.33

It is clear from these figures that the smoking ban made no difference to the rate of heart attacks in the weeks that immediately followed the ban. On the contrary, admissions fell slightly just before the ban and rose slightly after it.

The nugget of truth in the Pell study is that the number of heart attacks in Scotland fell overall in the twelve months following the ban, just as they have done every year since the 1990s. She claims that admissions fell by 17% which, if true, would be a significant decline but the data for the whole of Scotland does not support this. ISD Scotland data shows that hospital admissions for acute myocardial infarction fell by just 9.7% in the six months following the ban compared to the same period in 2005. The biggest drop in admissions came in the summer months of June and July when heart attacks are traditionally less common and not in the months immediately after the ban. 9.7% is a significant decline but it is in line with the long-term trend and there was a steeper fall in 1999, long before the smoking ban. And it is a long way off the 17% claimed by Dr Pell.

Whether one looks at the figures for acute myocardial infarction or acute coronary syndrome and whether one looks at admissions or discharges two things are very clear. Firstly, there is a long-term downward trend. Secondly, the numbers are volatile and fluctuate wildly for no discernible reason. Heart attack discharges fell by 1% in the year to March 2004 but over the next twelve months they fell by 3.7%. In the year to March 2006, they fell by a further 6.8%. Can anything be read into this? Can we say that the rate of decline more than trebled in 2005 for any particular reason? Did something remarkable happen in 2006 for it to double again? No. The rate has been falling for years and in some years it falls more sharply than in others. In the year to March 1999 it fell by 11.7%. Good news for the people of Scotland but not the result of any identifiable change in policy. To pin an explanation on it to suit one's own agenda is cynical opportunism.

By confining her analysis to a selection of hospitals and picking her own time-frame, Jill Pell is following in the footsteps of the Helena study and others like it by highlighting a particular downward trend and ascribing her own reason for it. Even if heart attack admissions did fall by 17% in Pell's sample group there is no reason to believe tobacco smoke - or the absence of it - was responsible. Indeed, the very fact that the heart attack rate fell almost as sharply amongst smokers as it did amongst nonsmokers strongly suggests that another factor was at work. It may have been the warm summer of 2006, changes in diet or the widespread prescription of statins by Scottish doctors. It may have been all or none of these. It may have been something else entirely or nothing at all. Without supporting evidence, I would not care to guess and nor should Dr Pell. As far as we can tell from the media's coverage of this study, the hospital patients were not asked about their exposure to secondhand smoke or whether they were regular pub-goers. It appears that no information was elicited from them other than whether or not they were smokers. Without basic information of this kind, all that is left is a dubious statistic and a suggested explanation. That is not a study, that is a theory.

It is hard to imagine epidemiology sinking much lower than this. So glaring are the flaws in this press release/study that it has since been ridiculed by The Times in its end-of-the-year run down of 'The worst junk stats of 2007', where it appeared in a list of "classic idiocies". The BBC did not fall over itself to debunk the study but it did at least allow an article to appear online entitled 'The facts get in the way of a good story' (10) which left little doubt that all had no been as it seemed when they shouted about the Scottish miracle two months earlier. Welcome as they are, these two small rebuttals generated scant publicity compared to the fanfare which greeted the original reports and while they will have barely registered with the public, they might act as a warning to the anti-smoking number-crunchers that the media won't be fooled so easily next time.

And perhaps the lesson has been learnt. A recent press release announcing a 15% decline in heart attacks and strokes in France was not widely reported and the BBC ignored it altogether. But these barrel-scraping studies will doubtless multiply in the months ahead. The current tactic of the tobacco control movement seems to be to release so many studies of this kind that the public assume there to be no smoke without fire. It is now almost standard practice for these studies to be announced via the media so they do not have to be subjected to the peer-review process or leave their data and methodology exposed for analysis. In the world of newspapers, it is well-known that the vast majority of readers never make it past the headline and this fact alone will benefit the hit-and-run approach of the anti-smoking movement in their efforts to convince people that passive smoking is, in effect, responsible for 1 in 5 heart attacks.

The press release which announced the news from France emphasised that "these statistics show the same tendency professionals have already observed in Italy, Ireland and Scotland."(11) In fact, the studies that the "professionals" of Scotland and Ireland produced have never been published, even online, and the Italian study actually showed a 2% increase in heart attack admissions after the ban, as Dr Michael Siegel has shown. Few casual readers will be aware of this, of course, and as these unpublished, unreviewed and unverified 'studies' are churned out they will naturally come to believe that smoking bans do indeed immediately save lives. In a year or two there may even be a "scientific consensus" that this is the case and those who point out that the emperor is naked will be described as "denialists". For the time being, the message is "never mind the quality, look at the quantity". But there is no quantity while these papers remain unpublished. In the field of science they do not exist and cannot, therefore, be quantified. Their authors should publish and be damned, starting with Dr Jill Pell, wherever she may be.

Christopher J. Snowdon is the author of Velvet Glove, Iron Fist: A history of Anti-Smoking www.velvetgloveironfist.com

References:

(1) http://news.bbc.co.uk/1/hi/health/542150.stm

(2)http://www.dailymail.co.uk/pages/live/articles/health/healthmain.html?in_article_id=480973&in_page_id=1774

(3) http://jpubhealth.oxfordjournals.org/cgi/reprint/fdi080v1.pdf

(4) http://www.guardian.co.uk/uk/2007/sep/11/health.smoking

(5) http://www.timesonline.co.uk/tol/life_and_style/health/article2426743.ece

(6) http://blogs.bmj.com/bmj/2007/09/14/towards-a-smokefree-society-edinburgh-trish-groves/

(7) "Dr Colin Fishbacker, an ISD official, says the two studies used data from different sources. 'We wouldn't expect the sources to agree exactly. The routine data we publish is based on the discharge summary sent to the GP and medical records for coding. The StopIt study was a specific research exercise.'" http://news.bbc.co.uk/1/hi/magazine/7093356.stm

(8) http://www.scotland.gov.uk/News/Releases/2007/09/10081400

(9) www.isdscotland.org

(10) http://news.bbc.co.uk/1/hi/magazine/7093356.stm

(11)http://www.escardio.org/vpo/Press+Area/Press+Releases/2007-esc-press-releases/heart-attack-rates-fall.htm?hit=home

If you type 'Scotland heart attacks smoking' into www.bbc.co.uk/news (select 'news and sport') you can view the following television news broadcasts:

Jill Pell predicts a fall in heart attack incidence in a highly sympathetic piece called 'Smoke ban "could reduce heart attacks by 10%"'

Dr Lawrence Gruer lauds the Pell study and specifically refers to the 10 month study period in 'Effects of the smoking ban'

Scotland's Deputy Chief Medical Officer gets an easy ride in 'Smoking ban cuts heart attacks'

The agenda for the two Scottish health conferences referred to above can be found at:

http://www.sepho.org.uk/Download/Public/10728/5/Conference%20Programme%20.pdf

and

http://www.publichealthconferences.org.uk/scotland_2007/downloads/FinalPH07_reg_A42.pdf