Post by marchesarosa on Dec 26, 2009 13:27:01 GMT
Some Thoughts on Disclosure and Due Diligence in Climate Science
By Steve McIntyre
(This post seems to have caught a chord and has quickly become the most read posting on the site. It was was cited approvingly by Roger Pielke at his blog and re-printed with slight edits by National Post on Feb. 15, 2005.)
I have spent much of the past 2 years analyzing and re-constructing some of the basic studies used by the Intergovernmental Panel on Climate Change (IPCC) to support their conclusions about global warming and, in turn, to promote policies on climate change. It started as a hobby and it evolved into a full time avocation, resulting to date in 3 peer-reviewed publications, which Natuurwetenschap & Techniek, the National Post and the Wall Street Journal have recently reported on. Previously, I spent about 35 years in the mining and mineral exploration business. During the last 20 years of this, I worked in the micro-cap exploration business and have a great deal of practical experience in dealing with prospectus and securities issues from the company point of view. Concepts like audit trails, due diligence and full, true and plain disclosure become second nature when you work in such an environment.
I have not before stepped into academic disputes, where very different standards of disclosure and procedures for due diligence apply. I think that many non-academic people, who would be put off by technical questions like the validity of principal components algorithms, may very well be interested in what I have learned about these processes as they apply to modern climate studies. In a corporate world, there is simply no question about providing audit trails, and while they can take many different forms they all serve the purpose of ensuring the validity of information used for investment decisions. In addition to familiar forms of financial audit trails, the splitting and retention of drill cores is a form of audit trail in the exploration business. In my opinion, the absence of drill core at the Bre-X exploration site, if publicly known, would have alarmed investors long prior to the final demise. (See my notes Bre-X#1, Bre-X#2 and Bre-X #3)
The 2001 IPCC report produced findings that have guided investment decisions, which vastly exceed the sums involved in even the largest financial scandals of recent years. Since the IPCC leaned heavily on a novel approach called a “multiproxy climate study” and in particular the “hockey stick graph” of Mann et al., this is where I’ve focused my attention. An audit trail in this case is easily defined: the data in the form used by the authors and the computer scripts used to generate the results. In principle, these can be easily buttoned up and publicly archived. I think that most civilians would assume (as I did prior to starting my studies in this area) that such packages would be standard as part of a peer review process.
In fact, this is not the case. None of the major multiproxy studies have anything remotely like a complete due diligence packages and most have none at all. The author of one of the most quoted studies [Crowley and Lowery, 2000] told me that he has “mis-placed” his data.
In the case of the Mann et al [1998,1999] study, used for the IPCC’s “hockey stick” graph, Mann was initially unable to remember where the data was located, then provided inaccurate data, then provided a new version of the data which was inconsistent with previously published material, etc. The National Post has recently reported on my experience as this unfolded.
In addition to the lack of due diligence packages, authors typically refuse to make their source code and data available for verification, even with a specific request. Even after inaccuracies in a major study had been proven, when we sought source code, the original journal (Nature) and the original funding agency (the U.S. National Science Foundation) refused to intervene. In the opinion of the latter, the code is Mann’s personal commercial property. Mann recently told the Wall Street Journal that “Giving them the algorithm would be giving in to the intimidation tactics that these people employ”. My first request for source code was a very simple request and could in now way be construed as “intimidation”.
However, the issue neatly illustrates the disconnect.
IPCC proponents place great emphasis on the merit of articles that have been “peer reviewed” by a journal. However, as a form of due diligence, journal peer review in the multiproxy climate field is remarkably cursory, as compared with the due diligence of business processes. Peer review for climate publications, even by eminent journals like Nature or Science, is typically a quick unpaid read by two (or sometimes three) knowledgeable persons, usually close colleagues of the author.
It is unheard of for a peer reviewer to actually check the data and calculations. In 2004, I was asked by a journal (Climatic Change) to peer review an article. I asked to see the source code and supporting calculations. The editor said that no one had ever asked for such things in 28 years of his editing the journal. He refused to ask for source code; the author refused to provide supporting calculations. Out of my involvement, the journal ended up with a new data policy, which was all to the good. But there is nothing at the journal peer review stage in climate publications which is remotely like an audit.
It’s my view that this is all the more reason why source code and data should be archived. There is a great deal of public misconception of the forms of due diligence actually carried out by the IPCC. Although the IPCC and similar agencies have many meetings and committees (usually in nice places), they do not carry out any audit or verification activities. While this has long been known by insiders, it was recently admitted in written answers, see especially questions 30-40, by Michael Mann to the US Senate in fall 2003.
30. Did IPCC carry out any independent programs to verify the calculations that you made in MBH98 or MBH99? If so, please provide copies of the reports resulting from such studies. It is distinctly against the mission of the IPCC to "carry out independent programs", so the premise of the question is false. However, the IPCC’s author team did engage in a lively interchanges about the quality and overall consistency of all of the papers as the chapter was drafted and revised in the course of review.
Thus, if a paper has passed the cursory journal peer review process, there were no subsequent audit or verification steps prior to adoption by the IPCC. Ross McKitrick and I have demonstrated that there were serious calculation errors in the most famous IPCC graph–the 1000 year climate hockey stick. In this case, the methodology had been incorrectly described in the journal publication. I also found that there had been an influential but unreported alteration to a key data series, where the alteration had been disguised by a (perhaps unintentional) misrepresentation of the start date of the underlying data. We published these findings recently in Geophysical Research Letters and Energy & Environment. But the math involved is not particularly sophisticated: the errors would have been discovered long ago had there been even routine checking.
It still amazes me that for all the billions of dollars being spent on the climate change industry (which I suspect dwarfs the mineral exploration industry in dollar volume [SM Note, Feb. 16, 2005 - I'm thinking here of hard rock exploration, the business that I've mostly been in, rather than oil exploration. It's quite possible that the climate industry is (say) only half the size of hard rock exploration. But my impression and what I'm trying to convey is that it's not a small industry], and the thousands of people working full time on this issue just in Canada, it was nobody’s job to check if the IPCC’s main piece of evidence was right. The inattentiveness of IPCC to verification is exacerbated by the lack of independ
ence between authors with strong vested interests in previously published intellectual positions and IPCC section authors.
For example, Michael Mann had published an academic article announcing that the 1990s were the warmest decade in human history. He then became IPCC section author for the critical section surveying climate history of the last millennium, adopting the very graph used in his own paper on behalf of IPCC. For someone used to processes where prospectuses require qualifying reports from independent geologists, the lack of independence is simply breathtaking and a recipe for problems, regardless of the reasons initially prompting this strange arrangement. It seems to me that prospectus-like disclosure must become the standard in climate science, certainly for documents like IPCC reports (which are like scientific prospectuses), but even for journals. The American Economic Review last year adopted such a rule; I hope they enforce it.
In business, “full, true and plain disclosure” is a control on stock promoters. While it may not always be successful, it gives an enforcement mechanism. There is no such standard in climate science. In the Mann study there are important examples of pertinent adverse results, known to the authors, which were not reported. In fairness, the journals do not require authors to warrant full, true and plain disclosure and there is little guidance to such authors as to what is required reporting and what is not required.
I’ve found that scientists strongly resent any attempt to verify their results. One of the typical reactions is: don’t check our studies, do your own study. I don’t think that businesses like being checked either, but one of the preconditions of being allowed to operate is that they are checked. Many of the most highly paid professionals in our society–securities lawyers, auditors–earn much of their income simply by verifying other people’s results. Businesses developed checks and balances because other peoples’ money was involved, not because businessmen are more virtuous than academics.
Back when paleoclimate research had little implication outside academic seminar rooms, the lack of any adequate control procedures probably didn’t matter much. However, now that huge public policy decisions are based, at least in part, on such studies, sophisticated procedural controls need to be developed and imposed. Climate scientists cannot expect to be the beneficiaries of public money and to influence public policy without also accepting the responsibility of providing much more adequate disclosure and due diligence.
climateaudit.org/2005/02/14/some-thoughts-on-disclosure-and-due-diligence-in-climate-science/
By Steve McIntyre
(This post seems to have caught a chord and has quickly become the most read posting on the site. It was was cited approvingly by Roger Pielke at his blog and re-printed with slight edits by National Post on Feb. 15, 2005.)
I have spent much of the past 2 years analyzing and re-constructing some of the basic studies used by the Intergovernmental Panel on Climate Change (IPCC) to support their conclusions about global warming and, in turn, to promote policies on climate change. It started as a hobby and it evolved into a full time avocation, resulting to date in 3 peer-reviewed publications, which Natuurwetenschap & Techniek, the National Post and the Wall Street Journal have recently reported on. Previously, I spent about 35 years in the mining and mineral exploration business. During the last 20 years of this, I worked in the micro-cap exploration business and have a great deal of practical experience in dealing with prospectus and securities issues from the company point of view. Concepts like audit trails, due diligence and full, true and plain disclosure become second nature when you work in such an environment.
I have not before stepped into academic disputes, where very different standards of disclosure and procedures for due diligence apply. I think that many non-academic people, who would be put off by technical questions like the validity of principal components algorithms, may very well be interested in what I have learned about these processes as they apply to modern climate studies. In a corporate world, there is simply no question about providing audit trails, and while they can take many different forms they all serve the purpose of ensuring the validity of information used for investment decisions. In addition to familiar forms of financial audit trails, the splitting and retention of drill cores is a form of audit trail in the exploration business. In my opinion, the absence of drill core at the Bre-X exploration site, if publicly known, would have alarmed investors long prior to the final demise. (See my notes Bre-X#1, Bre-X#2 and Bre-X #3)
The 2001 IPCC report produced findings that have guided investment decisions, which vastly exceed the sums involved in even the largest financial scandals of recent years. Since the IPCC leaned heavily on a novel approach called a “multiproxy climate study” and in particular the “hockey stick graph” of Mann et al., this is where I’ve focused my attention. An audit trail in this case is easily defined: the data in the form used by the authors and the computer scripts used to generate the results. In principle, these can be easily buttoned up and publicly archived. I think that most civilians would assume (as I did prior to starting my studies in this area) that such packages would be standard as part of a peer review process.
In fact, this is not the case. None of the major multiproxy studies have anything remotely like a complete due diligence packages and most have none at all. The author of one of the most quoted studies [Crowley and Lowery, 2000] told me that he has “mis-placed” his data.
In the case of the Mann et al [1998,1999] study, used for the IPCC’s “hockey stick” graph, Mann was initially unable to remember where the data was located, then provided inaccurate data, then provided a new version of the data which was inconsistent with previously published material, etc. The National Post has recently reported on my experience as this unfolded.
In addition to the lack of due diligence packages, authors typically refuse to make their source code and data available for verification, even with a specific request. Even after inaccuracies in a major study had been proven, when we sought source code, the original journal (Nature) and the original funding agency (the U.S. National Science Foundation) refused to intervene. In the opinion of the latter, the code is Mann’s personal commercial property. Mann recently told the Wall Street Journal that “Giving them the algorithm would be giving in to the intimidation tactics that these people employ”. My first request for source code was a very simple request and could in now way be construed as “intimidation”.
However, the issue neatly illustrates the disconnect.
IPCC proponents place great emphasis on the merit of articles that have been “peer reviewed” by a journal. However, as a form of due diligence, journal peer review in the multiproxy climate field is remarkably cursory, as compared with the due diligence of business processes. Peer review for climate publications, even by eminent journals like Nature or Science, is typically a quick unpaid read by two (or sometimes three) knowledgeable persons, usually close colleagues of the author.
It is unheard of for a peer reviewer to actually check the data and calculations. In 2004, I was asked by a journal (Climatic Change) to peer review an article. I asked to see the source code and supporting calculations. The editor said that no one had ever asked for such things in 28 years of his editing the journal. He refused to ask for source code; the author refused to provide supporting calculations. Out of my involvement, the journal ended up with a new data policy, which was all to the good. But there is nothing at the journal peer review stage in climate publications which is remotely like an audit.
It’s my view that this is all the more reason why source code and data should be archived. There is a great deal of public misconception of the forms of due diligence actually carried out by the IPCC. Although the IPCC and similar agencies have many meetings and committees (usually in nice places), they do not carry out any audit or verification activities. While this has long been known by insiders, it was recently admitted in written answers, see especially questions 30-40, by Michael Mann to the US Senate in fall 2003.
30. Did IPCC carry out any independent programs to verify the calculations that you made in MBH98 or MBH99? If so, please provide copies of the reports resulting from such studies. It is distinctly against the mission of the IPCC to "carry out independent programs", so the premise of the question is false. However, the IPCC’s author team did engage in a lively interchanges about the quality and overall consistency of all of the papers as the chapter was drafted and revised in the course of review.
Thus, if a paper has passed the cursory journal peer review process, there were no subsequent audit or verification steps prior to adoption by the IPCC. Ross McKitrick and I have demonstrated that there were serious calculation errors in the most famous IPCC graph–the 1000 year climate hockey stick. In this case, the methodology had been incorrectly described in the journal publication. I also found that there had been an influential but unreported alteration to a key data series, where the alteration had been disguised by a (perhaps unintentional) misrepresentation of the start date of the underlying data. We published these findings recently in Geophysical Research Letters and Energy & Environment. But the math involved is not particularly sophisticated: the errors would have been discovered long ago had there been even routine checking.
It still amazes me that for all the billions of dollars being spent on the climate change industry (which I suspect dwarfs the mineral exploration industry in dollar volume [SM Note, Feb. 16, 2005 - I'm thinking here of hard rock exploration, the business that I've mostly been in, rather than oil exploration. It's quite possible that the climate industry is (say) only half the size of hard rock exploration. But my impression and what I'm trying to convey is that it's not a small industry], and the thousands of people working full time on this issue just in Canada, it was nobody’s job to check if the IPCC’s main piece of evidence was right. The inattentiveness of IPCC to verification is exacerbated by the lack of independ
ence between authors with strong vested interests in previously published intellectual positions and IPCC section authors.
For example, Michael Mann had published an academic article announcing that the 1990s were the warmest decade in human history. He then became IPCC section author for the critical section surveying climate history of the last millennium, adopting the very graph used in his own paper on behalf of IPCC. For someone used to processes where prospectuses require qualifying reports from independent geologists, the lack of independence is simply breathtaking and a recipe for problems, regardless of the reasons initially prompting this strange arrangement. It seems to me that prospectus-like disclosure must become the standard in climate science, certainly for documents like IPCC reports (which are like scientific prospectuses), but even for journals. The American Economic Review last year adopted such a rule; I hope they enforce it.
In business, “full, true and plain disclosure” is a control on stock promoters. While it may not always be successful, it gives an enforcement mechanism. There is no such standard in climate science. In the Mann study there are important examples of pertinent adverse results, known to the authors, which were not reported. In fairness, the journals do not require authors to warrant full, true and plain disclosure and there is little guidance to such authors as to what is required reporting and what is not required.
I’ve found that scientists strongly resent any attempt to verify their results. One of the typical reactions is: don’t check our studies, do your own study. I don’t think that businesses like being checked either, but one of the preconditions of being allowed to operate is that they are checked. Many of the most highly paid professionals in our society–securities lawyers, auditors–earn much of their income simply by verifying other people’s results. Businesses developed checks and balances because other peoples’ money was involved, not because businessmen are more virtuous than academics.
Back when paleoclimate research had little implication outside academic seminar rooms, the lack of any adequate control procedures probably didn’t matter much. However, now that huge public policy decisions are based, at least in part, on such studies, sophisticated procedural controls need to be developed and imposed. Climate scientists cannot expect to be the beneficiaries of public money and to influence public policy without also accepting the responsibility of providing much more adequate disclosure and due diligence.
climateaudit.org/2005/02/14/some-thoughts-on-disclosure-and-due-diligence-in-climate-science/
