Summary: We havea rare moment when even laypeople can understand a major issue in a cutting edge scientific debate. Not only does this give a hint (but no more) about the strength of the two “sides” (global warming proponents vs skeptics), but also illustrates one of the most important effects of the Internet: opening previously closed social processes to public view, thereby stripping elites of their control of information — and thereby reducing their power. As Glen Reynolds (the Instapundit) says, we have created an “Army of Davids.”
Modern debates in the sciences are by their nature difficult for non-specialists to understand, let alone evaluate. Occasionally one facet will become so clear that even laypeople can judge for themselves. We have such a moment in one of the most important debates of our time — about climate trends.
The primary complaint of skeptics has been about the availability of data and methods used to produce the results on which advocates urge major public policy changes. Without this replication is impossible for scientists or interested “outsiders” to the establishment.
The response has been a grudging of data, often partial and poorly documented. The battle has been surprisingly difficult because most journals’ require documentation and release of this information at publication. Even more striking, much (most?) of this research is publicly funded — and hence keeping it secret usually violates regulations of the funding agencies.
We saw the battle lines reform with the publication — to a massive publicity campaign — of “Warming of the Antarctic ice-sheet surfacesince the 1957 International Geophysical Year”, Steig, E.J., D.P. Schneider, S.D. Rutherford, M.E. Mann, J.C. Comiso, and D.T. Shindell. Nature, January 2009, pp 459-462. (the abstract for Steig 2009 is here; you can see the full text here).
Steig 2009 was immediately greeted with requests for the supporting documentation. The replies tell us much about the state of climate science, and the foundation it provides for large public policy action. Read and decide for yourself; the tale is quite clear. Note the dates!
Contents
- Have the authors of Steig 2009 made their methods and data available to the public?
- Reminder: some scientists were skeptical after reading Steig 2009
- Significance of the debate over Steig 2009
- Nature’s policy about data availability
- Afterword
- Where to go for more information
1. Have the authors of Steig 2009 made their methods and data available to the public?
Nature, like most professional journals in the Sciences, has clear policies requiring supporting data to be available to the public. See section 4 below for details. How well have Steigandhis co-authors complied with these?
(1) Email from co-author Eric Steig to Steve McIntyre on January 23 (source):
“I have always intended to provide all the material on line; I wasn’t allowed to do this before the paper was published. I would have done it already but have been busy answering emails. I should have these up on line next week.”
(2) From Josefino Comiso (NASA Goddard Space Flight Center), in reply to request for data by McIntyre on January 23 (source):
“Thanks for your request for AVHRR {satellite} surface temperature IR data. I am actually planning to have the entire data set archived in the near future and as soon as I get the associated document that describes the data and discusses the errors and caveats completed. The data are indeed on a gridded monthly basis. I will let you know how to access them in the web as soon as they are archived and ready to be downloaded. Best Wishes”
*** The first two replies said that the data would soon be available. But then the story changes from “will be made available” to “the data was available.”
(3) From Steig’s website at the U of Washington (no date posted):
“All of the data used in the temperature reconstructions are from publically available data sources.”
(4) Steig replying to a comment at RealClimate on February 2:
[Response: ALL of the data that were used in the paper, and EXACTLY the code used in our paper have been available for a long time, indeed, long before we published our paper. This is totally transparent, and attempts to make it appear otherwise are disingenuous. This has always been clear to anyone that asked. If you wanted to do the work yourself, for legitimate reasons, you could do so. If the point is to “audit” our work, it makes no sense whatsoever to provide all the intermediate products used in our analysis. That would defeat the purpose of the supposed “audit”.–eric]
(5) Steig replying to a comment at RealClimate on February 2:
“… Excluding your study, would it be your opinion that code and data archiving practices are adequate in the climate science community, overall? This is a common complaint on sites like CA.”
[Response: Yes, I think we do a fine job. We could do better, but we do very very well. I’ve never had trouble getting data that I need from others. Indeed, our Nature study was based entirely on freely available data and code. –eric]
(6) Steig replying to a comment at RealClimate on February 3:
[Response: Nope I don’t think it is a bit strong. I released an electronic version of our data and links to all the original data and code almost as soon as our paper was published. Anyone paying attention would know that. … -eric]
*** Up to this point the replies on RealClimate have been clear and unequivocal: the data is available to the public. Now the story shifts again. Its’ available to some people, or its proprietary NASA data, or it will soon be available.
(7) Steig replying to a comment at RealClimate on February 3; bold emphasis added:
“… What I don’t understand is simply, why don’t you routinely publish those intermediate steps. Most people don’t have the scientific background, or the free hours, to truly dig into the data. However most people with real-world experience know transparency when they see it. …”
[Response: I do routinely make all our data available, as does everyone else that I know. In this particular case, anyone legitimate who has asked for all our data, including the intermediate steps, has received it. To continue with the analogy with financial auditing, let me very clear on what I mean by legitimate: In the business world, auditors
- don’t publicly accuse a company of withholding data prior to requesting said data;
- are not self-appointed;
- have to demonstrate integrity and competence;
- are regulated.
On this point, if you are suggesting that Steve McIntyre be regulated by an oversight committee, and have his auditor’s license revoked when he breaks ethical rules, then we may have something we can agree on.–eric]
(8) Gavin Schmidt (NASA Goddard Institute for Space Studies; see his bio) replying to comment #38 by by Nicolas Nierenberg on February 9, posted at RealClimate (post titled “On Replication”; bold emphasis added):
“Once again rather than the general, let’s be specific. I believe it removes the issues of how much extra work would be created. Dr. Steig has said that he is willing to provide the data to legitimate researchers. My response is to simply post what he would provide. I still haven’t heard from Dr. Schmidt what the objection is to that concept.
“Also as to the specifics in Dr. Steig’s paper. I believe that there is probably sufficient information on AWS trends. However I don’t think there is sufficient information to reproduce the gridded AVHRR {satellite} temperature results. They are quite dependent on corrections for clouds, and manipulation to produce temperature values as I understand it.”
[Response: Joey Comiso is apparently working on making that available with appropriate documentation – patience. – gavin]
(9) Reply to comment #63 by Nicolas Nierenberg at RealClimate on February 9:
“… Other than a couple of people the conversation seems to be converging on the fact that providing the code and data is preferable. Dr. Schmidt has said that this will be done in the case of the Steig paper.”
[editor note: this was done in the case of Steig et al withrespect to code though perhaps not with as much hand-holding as you seem to want. some of these data are proprietary (NASA), but will be made available in the near future]
(10) Reply to comment #136at RealClimate on February 11:
“If possible, please provide more information about the ‘proprietary NASA data’ used in Steig 2009. There have been several (at least 4) clear statements from Dr. Steig, both at his website and RealClimate, that his study used only ‘publically available data sources.’”
[reply: the raw data are public; the processed data (i.e. cloud masking) are not yet, but will be in due course. so relax]
(11) Gavin Schmidt replying to Comment #225by captdallas2 on February 16 at RealClimate:
“The satellite data was IR not RSS or MSU so the replicators could not gather the data needed to complete the replication. Do you know if that data is available online?”
[Response: Joey Comiso is apparently working on the data preparation along with sufficient explanation to make it usable. I have no particular insight into the timetable. – gavin]
Conclusion: You have your orders. Relax. The authors will release this vital information when they are good and ready to do so. Trust them, and rely on the fawning articles in the mainstream media.
2. Reminder: some scientists were skeptical after reading Steig 2009
Note that initial media coverage included a few cautionary notes, ignored by true believers.
“This looks like a pretty good analysis, but I have to say I remain somewhat skeptical,” Kevin Trenberth, climate analysis chief at the NationalCenter for Atmospheric Research, said in an e-mail. “It is hard to make data where none exist. (AP; Trenberth has been a lead author for the UN’s IPCC)
“One must be very cautious with such results because they have no real way to be validated,” says atmospheric scientist John Christy of the University of Alabama-Huntsville, who was not part of the study. “In other words, we will never know what the temperature was over the very large missing areas that this technique attempts to fill in so that it can be tested back through time.” (USA Today)
Roger Pielke Sr also had some questions, posted at his website. He is emeritus professor of the Department of Atmospheric Science at Colorado State U. See his Wikipedia entry for more information. His concluding comment:
In terms of the significance of their paper, it overstates what they have obtained from their analysis. In the abstract they write, for example, “West Antarctic warming exceeds 0.1C per decade over the past 50 years”.
However, even a cursory view of Figure 2 shows that since the late 1990s, the region has been cooling in their analysis in this region. The paper would be more balanced if they presented this result, even if they cannot explain why.
3. Significance of the debate over Steig 2009
First: the smokescreen has not stopped replication of Steig 2009
The smokescreen to conceal key aspects of has not stopped efforts to reverse-engineer it. Outside experts have made great progress. The initial skirmishes were reported in Science in action, a confused and often nasty debate that produces real progress (5 February 2009). Much progress has been made since then, but that is a subject for another time. You can see their work at many websites, such as Climate Audit and The Air Vent.
Here is the first result: “Steig’s Antarctic Heartburn“, by Jeff Id and Jeff C, posted at Anthony Watt’s Watts Up with That, 28 February 2009. It’s long; you might find the 16 page PDF easier to read.
Second, there is an important and long-term dynamic at work here.
The printing press unlocked knowledge, perhaps most important for the 15th century it made words of the Bible available to everyone. That blasted away the foundation to the power of the medieval Catholic Church’s priests . Today the Internet is doing the same thing to the guild of scientists.
Cloistered in a small number of institutions, they controlled the process of pure science by dominating key institutions. The major journals, academia (the gateway to the guild) and the great quasi-government research organizations (e.g., NASA, IPCC). The relevant knowledge was also widely available in the private sector, but most of these people were isolated outsiders — impotent to affect the institutional dynamics of the science establishment. The Internet changed this balance — linking them together and gave them voices. Now they are slowly realizing that they are powerful.
Although lacking the magic juice of corporate and government funding, modern communications and computer technology allows them to pool their resources. Their numbers, properly combined, allow them to challenge even the most entrenched scientific elites. Today that means the climate scientists.
SSSailor comments on another post that this is ..
“What Glen Reynolds (Instapundit) has described as an “Army of Davids” is in real terms, a Distributed Intelligence. The rate of change of information acquisition and the velocity of its dispersion is accelerating via the internet and other bandwidth modes. The Global Warming issue represents a clear example as: the consensus vs. science debate, in the digital domain.”
The reference is to Reynolds’ book An Army of Davids: How Markets and Technology Empower Ordinary People to Beat Big Media, Big Government, and Other Goliaths (2006),
Watch this conflict. The fun is just beginning.
4. Nature’s policy about availability of data and materials
From their website. It seems quite clear and unambiguous.
An inherent principle of publication is that others should be able to replicate and build upon the authors’ published claims. Therefore, a condition of publication in a Nature journal is that authors are required to make materials, data and associated protocols promptly available to readers without preconditions. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript, including details of how readers can obtain materials and information. If materials are to be distributed by a for-profit company, this should be stated in the paper.
Supporting data must be made available to editors and peer-reviewers at the time of submission for the purposes of evaluating the manuscript. Peer-reviewers may be asked to comment on the terms of access to materials, methods and/or data sets; Nature journals reserve the right to refuse publication in cases where authors do not provide adequate assurances that they can comply with the journal’s requirements for sharing materials.
After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal (or the chief biology/chief physical sciences editors in the case of Nature). In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors’ funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.
Details about how to share some specific materials, data and methods can be found in the sections below. The preferred way to share large data sets is via public repositories. Some of these repositories offer authors the option to host data associated with a manuscript confidentially, and provide anonymous access to peer-reviewers before public release. These repositories coordinate public release of the data with the journal’s publication date (advance online publication (AOP) or, if the manuscript is not published AOP, print/online publication). This option should be used when possible, but it is the authors’ responsibility to communicate with the repository to ensure that public release is made promptly on the journal’s AOP (or print/online) publication date.
Any supporting data sets for which there is no public repository must be made available as Supplementary Information files that will be freely accessible on nature.com upon publication. In cases where it is technically impossible for such files to be provided to the journal, the authors must make the data available to editors and peer-reviewers at submission, and directly upon request to any reader on and after the publication date, the author providing a URL or other unique identifier in the manuscript.
4. Afterword
Please share your comments by posting below. Per the FM site’s Comment Policy, please make them brief (250 words max), civil, and relevant to this post. Or email me at fabmaximus at hotmail dot com (note the spam-protected spelling).
For information about this site see the About page, at the top of the right-side menu bar.
5. For more information from the FM site
To read other articles about these things, see the FM reference page on the right side menu bar. Of esp relevance to this topic:
Posts on the FM site about the debate over climate science:
- More forecasts of a global cooling cycle, 15 July 2008
- Two valuable perspectives on global warming, 4 August 2008
- Good news about global warming!, 21 October 2008 – More evidence of cooling.
- Watching the world change before our eyes, 29 November 2008
- This week’s report on the news in climate science, 7 December 2008
- The Senate Minority report is out: “More Than 650 International Scientists Dissent Over Man-Made Global Warming Claims”, 12 December 2008
- Weekend reading recommenations about climate change, 13 December 2008
- An important new article about climate change, 29 December 2008
- My “wish list” for the climate sciences in 2009, 2 January 2009
- Important new climate science articles, 11 January 2009
- Climate science articles which you might enjoy reading!, 18 January 2009
- How warm is the Earth? How do we measure it?, 28 January 2009
- Science in action, a confused and often nasty debate among scientists, 5 February 2009
- Richard Feynmann, one of the 20th centuries greatest scientists, talks to us about climate science, 12 February 2009
What we have here is the typical pattern of the nutter with a cause – Demands that someone else turns over their work. Coupled with the astonishing opinion that scientific fact is determined when all the nutters have finished auditing the data.
It’s obvious that FM doesn’t understand how science actually works. But it is beyond that. Science in the US is in retreat. FM’s attitude that they are just another unaccountable elite somehow screwing their great American Dream doesn’t exist in Europe and Asia and it accounts for the huge shift that is occurring in design work out of the US right now.
Americans kid themselves that they are innovative but the reality is those days are decades ago. What has fueled the last 20 years was the availability of cheap VC cash not scientific talent. Now the cash is drying up as the financial sector ponzi schemes that made it possible collapses. The only way out is to heavily invest in science at all levels, but the culture in the US is so anti-science that it is impossible. The US simply has an woefully uncompetitive culture globally.
By the time the shift has finished it will be too late but at least Americans can keep warm telling each other conspiracy stories about how the UFOs screwed them.
.
.
Fabius Maximus replies: I love comments like this, adding some humor to the site. Not a trace of evidence or logic to support their confident assertions, heavy on the sneering and insults. I’ll leave this comment up as example of irrationality, but repetitions will be deleted.
While this comment is to bizarre to respond to, it is a good place to re-iterate my recommendations. (see here for more detail):
(1) Climate science data should be made public in conformity with the existing regulations of universities, funding agencies (governmental and non-profit), and professional journals.
(2) We should substantially increased funding for the collection and review of climate data. For example, the surface temperature recording system is grossly inadequate to our needs. And both both the data and global climate models should be analyzed by multi-disciplinary teams.
(3) IMO the best available model here is testing of new drugs. While there may be an ample peer-reviewed literature for new treatment, the Food and Drug Administration has their own requirements. These include review by a committee of multi-discilpinary experts. Spending trillions of dollars to save the world should require equivalent standards.
Thanks for the article. The pattern played out behind the scenes as well too. I personally made several polite requests which were simply deleted prior to the public debate beginning. I was then told all of the data existed and was public but was not fooled. Several smart people told me –Jeff the data’s right here. I had to explain that the links were not pointing to actual code used but rather the public matlab RegEM function. We then spent days guessing at what the inputs were, back calculated several inputs and found that Steig et. al. didn’t actually use the data at all in the satellite reconstruction but rather a low information version.
The code and data for processing the satellite data are also still missing.
Steig even took the time to infer that I was an idiot and everything was available, prior to stating that it would be available soon. It will only be through extreme diligence that the code and data will ever be seen by the public.
Re: comment #1: This work is used as an excuse in the government cause to take our American/British/Canadian/European cash in the name of global warming. If it’s real I’ll support them, if not we should know. I have no idea how people like you get the idea that disagreement or ‘checking the math’ is anti-science. It may give you comfort but it doesn’t have any meaning.
Update
Steig 2009 was published in Nature. I have added to the post an except from Nature’s Editorial policy regarding data availablity (source). It’s quite clear.
The pattern of obfuscation is clearly presented, thanks. This does not mean that the paper is bogus, but it casts a heavy doubt over the honest intentions of the authors.
Nature’s willingness to bend its rules for particular hypotheses is likewise of great concern. Nice work.
.
.
Fabius Maximus replies: Agreed. As Steve McIntyre has said so many times, we cannot draw conclusions about climate change either way on the basis of flawed research processes. But recognition of these problems is a necessary if not sufficient step to get climate science back on track. The importance of these issues should make this a high priority for our government.
Re. Comment #1: I spent about twenty years (1985-2005) as a practicing bench scientist, and am an author on 19 peer-reviewed primary manuscripts and three reviews. In addition, I have served as a peer-reviewer for about 40 or 50 manuscripts during this time. My field, cell and molecular biology, is distant from climatology. However, many of the issues Fabius Maximus raises on the practice of science are the same–hardly a surprising observation. For better and worse, people are people, engaging in their professional lives with a mixed bag of traits: altruistic, scheming, co-operative, competitive, insightful, passionate, scholarly, detached, engaged…
In my opinion, the conduct of prominent AGW consensualists that Fabius Maximus has documented do not represent best scientific practices. This conduct does not meet the standard of acceptable scientific practice. The reasoning behind these assertions have been eloquently stated by Richard Feynmann (and quoted by Fabius Maximus). It has also been presented by Fabius Maxiumus in his own words, repeatedly.
Clearly, the AGW consensualists have potent reasons to deny the AGW skeptics access to full and complete accounts of their Materials and Methods–or else they would not continue to do so. As a guess, they fall into three broad categories.
* The first may be Career concerns, such as worries about being scooped on pending projects, fearing erosion of the competitive edge that comes with being first mover, and wanting to stay well-positioned for future rounds of grant funding.
* The second may be Emotional issues, namely feelings of disdain for unqualified and uncredentialed skeptics seeking to unjustly reap the fruits of their labors.
* The third may be Public policy concerns, in that the consensualists are certain that AGW poses an urgent threat to the ecosphere that demands priority action by politicians, supported by the voting public. “This is no time to dilly-dally with academic philosophical matters, especially when our opponents aren’t seeking Truth, but Delay and Confusion.”
This set of reasons may be valid, but they are not easily explained to the lay public. Hence, there is the temptation to re-frame the de facto policy of holding back information on data and modeling methodology as “usual scientific practice.” The skeptics’ quest to replicate and critique the consensualists’ work is presented variously as a fool’s errand and as a stunt.
This is not so. It is the skeptics (that is, those skeptics whose work Fabius Maximus highlights) who are playing by the accepted rules of science.
To the extent that I’ve made a pseudonymous claim to an expert opinion, readers should rightly view it with suspicion. To partly address that problem, I’ve emailed a list of my publications to our host.
More about replication of Steig 2009
Despite the stonewalling about providing the data required under Nature’s Editorial policy, work continues to understand Steig 2009. Here are two comments by Jeff C (source).
#1 About the satellite data (bold emphasis added)
#2 Prior research about temperature trends in Antarctica
“Variability and Trends in Antarctic Surface Temperatures from In Situ and Satellite Infrared Measurements“, Josefino C. Comiso (NASA Goddard Space Flight Center), Journal of Climate, 15 May 2000 — Abstract:
The comments of Prof. Frank Tipler, a renowned theoretical physicist, strike me as far more persuasive than anything I could say on the state of science:
“Increasingly, government grants are used to defend dogma, not discover new truth: 28 percent of the scientists supported by NIH admitted recently to cooking data to support establishment theory, and 66 percent admitted to cutting corners to achieve the same end. I myself no longer trust the data claims appearing in the leading science journals.”
http://pajamasmedia.com/blog/stimulating-scientists-into-proving-global-warming/
I find your documentation of these events very helpful and revealing. Thank you. Likewise, the effort put forth by the folks at Air Vent and Climate Audit should be saluted. They’ve done a great service for all of us. Certainly a testament to the Army of Davids concept.
From my own perspective, the science by AGW scientists seems to be of similar nature to other science research over the last few decades in a variety of fields. In the end, much of this “science” and the associated claims/predictions can’t hold up to the scrutiny and testing needed to validate research, theory and outcomes.
As a result of these failures, I’ve become much more cynical about studies and conclusions that scientists now provide, in general. I’m sure there are others that feel the same way and it’s a shame.
How can NASA have “proprietary data”? We US taxpayers paid for that data. It belongs to us. Every bit of it should be available at nasa.gov so that independent researchers can replicate the work, try other methodolgies, etc.
If your results can’t be independently confirmed, they don’t count for anything. Dismissing “illegitimate” “deniers” isn’t science; what’s the word I’m thinking of… ah, yes: Inquisition.
Seems to me ( UK ) ; First , global warming was the maverick idea , territory of tree-huggers , that was fiercely resisted by the Establishment .That made sense . Oil industry , nice modern life , nuclear power risks etc .
Then global warming became the mainstream idea . Huh ?
Politicians jumping on the the bandwagon of public opposition to ANY government behavior ? ( Gov having made a mash of most things recently .)
It would then seem logical that the Oil industry ( with the money ) and government ( with a dearth of vision as to how to successfully go Green ) , would bite back .
Goodness knows where the scientific truth lies . Always try to uncover who pays the scientist ..