The Impact of Pre-Publication Presentation of Results: Epi Inquiry Accolade 03/20/2006

[Epidemiologic Inquiry 2006, 1: 21]

When is it a good idea, and when is it a bad idea to announce scientific results before formal publication? Does the medical community and the public quickly adopt pre-publication findings even though the results are not yet peer-reviewed? Such were the questions address by the recent article in JNCI by Giordano et al. and its accompanying editorial by Woloshin and Schwartz.

Giordano et al. investigated whether a new taxane treatment for lymph-node positive breast cancer significantly increased after its efficacy results were reported at the May 1998 meeting of the American Society of Clinical Oncology. Even though the taxane was not yet FDA approved and the formal publication was not to appear until 5 years later, the authors found that use of the newly reported treatment significantly increased >400% overall after the conference report. Furthermore, even though the treatment was shown only efficacious in lymph-node positive breast cancers (use increased 800%), results also showed that its use in patients with lymph-node negative breast cancers also increased 300%, which is in fact quite clinically disturbing as the drug was not even shown efficacious for such a disease.

The accompanying editorial by Woloshin and Schwartz summarized the risk and benefits of early adoption of pre-publication results, as well as noted examples in which early adoption has been clinically detrimental. Notably, the drug gefitinib (Iressa) had once been granted early FDA approval for increasing survival of patients with non-small-cell lung cancer who failed chemotherapy, even though the trial was unpublished, was a single uncontrolled trial, and subsequent issues were raised regarding adverse pneumonia deaths. Though over 200,000 people used the drug by 2004, a subsequent placebo-controlled trial shows no increased survival benefit. Such is an example in which adoption of pre-publication results did not benefit the public, and may have in fact caused more harm.

Woloshin and Schwartz summarize the following set of guideline regarding when to potentially adopt any pre-publication results:
1) large difference in all-cause mortality
2) no adverse effects
3) results are from a large randomized trial with long duration
4) confirms prior trial results, or presents first randomized trial evidence
5) no alternative treatments exist

For highlighting such important issues, the editors select as the dual-Epidemiologic Inquiry Accolade: Investigation of the Week...

Giordano SH, Duan Z, Kuo YF, Hortobagyi GN, Freeman J, Goodwin JS. Impact of a scientific presentation on community treatment patterns for primary breast cancer. J Natl Cancer Inst. 2006 Mar 15;98(6):382-8.

Woloshin S, Schwartz LM. What's the rush? The dissemination and adoption of preliminary research results. J Natl Cancer Inst. 2006 Mar 15;98(6):372-3.

~The Editors

Trusting Early Results Before Publication? Transition from Meetings to Journals

[Epidemiologic Inquiry 2006, 1: 20]

While doing a literature search, it is relatively common to find that either the results of a RCT presented at a scientific meeting were never published in a journal; or the results in the journal publication are discordant from those presented at the scientific meeting. Should we use the results from the scientific meeting, even if the results were never published later? Or should be just use the results from the peer-reviewed journal article, which might ensure completeness and proper interpretation of the data? This week's paper by Toma et al. in JAMA attempts to address these questions by examining the proportion of RCTs presented at American College of Cardiology (ACC) scientific meetings which were subsequently published as full-length journal articles; and the consistency between the data presented in the meeting abstract and the subsequent full-length publication.

The authors selected all RCTs presented at the scientific meetings of ACC between 1999 and 2002 and then searched the literature to see if these were ever published as full-length journal articles till the time of literature search. A distinction was made between the late- breaking clinical trials and the trials presented at other sessions of the ACC (oral or poster sessions). Significant results included the findings that the late-breaking clinical trials were more likely to be published as a full-length journal article subsequently (92% vs 69%); it was more likely that a design paper had been published prior to presentation of results for the late-breaking clinical trials (31% vs 13%); the late-breaking trials were likely to be larger (Median n=725 vs n=196); and the late-breaking trials were less likely to report a favorable effect of the intervention (OR of 0.46; 95% CI = 0.24, 0.90). The late-breaking trials also had higher quality scores and were more likely to be published in a journal with higher impact factor as compared to the RCTs presented at other sessions.

The authors believe that the differences between the late-breaking clinical trials and others can be explained to a certain extent by the process of selection of these RCTs. The application for presenting at the late-breaking session needs to be made at least 3 months in advance of the meeting and needs to be supported by details of the purpose, design, and methods of the trial. Also, since the results of these RCTs are not known at the time of application, it is more likely that they would report both favorable as well as negative results for the tested interventions.

Most importantly though, 41% of the RCTs (both late-breaking trials as well as others) which were subsequently published had a different estimate of primary outcome in the journal article as compared to what was presented at the scientific meetings. This suggests that we should not be over-eager in embracing the results from scientific meetings into clinical practice or in meta-analyses and we should wait for a peer-reviewed publication first. As the authors say - "As Shakespeare may have said in the 21st century, there are many slips betwixt podium and page".

~The Editors

Reference: Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, Armstrong PW. Transition From Meeting Abstract to Full-length Journal Article for Randomized Controlled Trials. JAMA 2006;295:1281-1287.

Sex Differences of Sex Hormones in Type 2 Diabetes: Implications for Sex-based Medicine

[Epidemiologic Inquiry 2006, 1: 18]

Recently, the theme of sex/gender-based medicine has been emerging as an ever more prevalent theme in clinical medicine. Thus, this also forces researchers to also ponder and consider sex-based epidemiologic investigations more carefully.

In last week's issue of JAMA, Ding et al. investigated sex differences of how plasma sex hormones affect risk of type 2 diabetes. From a systematic review of cross-sectional and prospective studies and randomized trials, it was consistently found that testosterone increased the risk of type 2 diabetes in women, but testosterone had opposite effects in men by decreasing the risk of type 2 diabetes. Additionally, the binding protein, sex hormone-binding globulin, was found to be strongly protective in women, but only marginally protective against type 2 diabetes in men.

Given such innate sex differences in action of one the most fundamental sex hormones, there is now strong biologic plausibility of sex differences in a wide variety of other etiologic pathways for disease, not just related to diabetes and its associated morbidities. Thus, perhaps epidemiologists and other clinical investigators should more carefully examine future exposure-disease relationships stratifying on sex and assessing sex-specific associations, rather than just adjusting for sex and giving little notice to sex interactions.

~The Editors

Reference: Ding EL, Song Y, Malik VS, Liu S. Sex differences of endogenous sex hormones and risk of type 2 diabetes: a systematic review and meta-analysis. JAMA. 2006 Mar 15;295(11):1288-99.

Smokefree U.S. States: Still Progress to be Made

[Epidemiologic Inquiry 2006, 1: 18]

In the ever waging war against tobacco, the concept of "smokefree" states has gained momentum in several U.S. states as well as certain international countries. Below is a tabulation of U.S. states with laws ensuring smokefree offices, restaurants, bars, and casinos. However, the list is still short, and much progress is yet to be made.

~Editors

Source: www.smokefree.net

"Reproducibility Should Be Minimum Standard for Epidemiologic Research" - press release of JHSPH

[Epidemiologic Inquiry 2006, 1: 17]
(press release of Johns Hopkins Bloomberg School of Public Health)

Epidemiologic findings, especially those on which public policies are based, are strengthened when they can be replicated by others. However, full replication by independent investigators is not always possible, due to time constraints or a lack of funding. A commentary by researchers from the Johns Hopkins Bloomberg School of Public Health suggests that analytic research data should be made available so that reproducibility of epidemiologic studies can be the new minimum standard for which investigators strive. The commentary will be published in the April 15, 2006, print edition of the American Journal of Epidemiology and can currently be viewed on the journal’s webpage.

“All epidemiologic studies should be held to the standard of full replication. But in cases where this is not possible, study investigators should make it possible for others to reproduce their findings,” said Roger Peng, PhD, lead author of the commentary and an advocate for making research reproducible by others.

Peng and colleagues said the first requirement in reproducibility is that the analytical data set must be made available for others to view and use. The availability of data sets enables other investigators to verify previously published findings, conduct alternative analyses of the same data, eliminate uninformed criticisms and expedite the exchange of information among scientists. In addition to providing the computer code, or instructions for data analysis, authors must also explain how the computer code is linked to the data and which code sections apply to which data.

The Hopkins researchers acknowledged that making data available to others gives the original investigator little control over how the data will be used. They suggest a system by which partial rights are licensed to interested investigators according to how the data will be used.

“Providing others with partial rights to the data benefits both the original investigator and those interested in the data. The recipients obtain access to the data and the donor meets data disclosure obligations and maintains some control over others’ use of the data,” said Peng.

As an example of how epidemiologic studies can meet the reproducibility minimum, Peng and his coauthors applied the standard to a study on quantification of air pollution risk, called the National Morbidity, Mortality and Air Pollution Study (NMMAPS). They created the Internet Health and Air Pollution Surveillance System—at www.ihapss.jhsph.edu—to disseminate the entire database and software used for the study. Other scientists are now able to fully reproduce the study results, apply the study’s methodology to their own data or apply their methodology to the NMMAPS data.

A handful of scholarly journals, such as Science and Nature, already require authors to place biologic data in public databases, and the National Institutes of Health requires grantees to have a data-sharing policy. Biologists have already made great strides toward integrating research databases, sharing software and making their analyses reproducible. “

Reproducibility is feasible now. Journals can play an important role in ensuring that their published work is reproducible,” said Scott Zeger, PhD, professor and chair of the Bloomberg School’s Department of Biostatistics.

The compendium, which is a full study linked with the data and code, for Peng and his colleagues’ commentary can be found at www.biostat.jhsph.edu/~rpeng/reproducible/.“Reproducible Epidemiologic Research” was authored by Roger D. Peng, PhD, Francesca Dominici, PhD, and Scott L. Zeger, PhD, all with the Johns Hopkins Bloomberg School of Public Health.

Mendelian Randomization: A Perfect Causal Epidemiologic Approach to Simulate a Randomized Trial?

[Epidemiologic Inquiry 2006, 1: 16]

In epidemiology, we always seek to find the perfect approach to assess causation, with the aim to simulate the randomized controlled trial in observational research. One special example in which perfect randomization can be observed is the case of Mendelian randomization in genetic epidemiology.

Recently, there have been great interest in Mendelian randomization approaches for estimating causal effects of gene products. Briefly, because genetic polymorphisms are randomly assorted at conception, all covariates between individuals should theorectically be perfectly balanced between those with different polymorphisms--thus theorectically eliminating all confounding for any association between the genetic polymorphism and disease. Furthermore, if one assumes that the measured genetic polymorphisms directly affects the gene product (e.g. a biologic hormone), then one can also estimate the unconfounded association between the gene product and disease by implementing a instrumental variable analysis approach. (for more info, see Greenland, IJE 2000;29:722-729).

While such a method theorectically estimates an unconfounded Mendelian-randomized causal association between the gene product and disease, a recent article in AJE summarized the many limitations of such a method. Beside general problems in genetic studies such as population stratification and linkage disequilibrium, one subtle but important issue that Mendelian-randomization's analytic approach cannot account for is the problem of "canalization", or the post-genetic adaptation for the genetic effects by other uncontrolled factors. Examples of such a phenomenon include differential nutrient intakes to compensate for genetic deficiency, different lifestyle behaviors to adapt to obesity, or differential use of exogenous hormone agents to compensate for genetically-predisposed low levels of endogenous hormones. (FYI: causally, this can also be considered a type of time-dependent confounding for those familiar with structural DAGs).

Thus, while Mendelian randomization appears to be the perfect epidemiologic approach to directly estimate causal effects, it still has limitations and assumptions in its application, just as there are limitations of all study designs including randomized controlled trials.

For highlighting such important issues, the editors select as the Epidemiologic Inquiry Accolade: Investigation of the Week...

Limits to Causal Inference based on Mendelian Randomization: A Comparison with Randomized Controlled Trials
Nitsch et al.
Am. J. Epidemiol. 2006 163: 397-403.

WHI Low-Fat Dietary Trial: What a Billion-Dollar Trial Showed that Epidemiology Already Knew

[Epidemiologic Inquiry 2006, 1: 15]

As they say in baseball and criminal felonies- "3 strikes and you're out" - however, is this adage necessary true? In a recent blockbuster issue of JAMA, investigators from the decade-long Women's Health Initiative low-fat dietary trial simultaneously reported the results for breast cancer, colorectal cancer, and cardiovascular disease... in essense, results indicated no overall benefit of a low-fat dietary pattern.

This opens lots of questions: Was the trial worth the multi-million dollar investment? Was there any gain from the trial? What were major limitations in this trial? Should this trial have been conducted in the first place?

Over a decade ago, when scientists first began proposing and planning this low-fat trial, FAT was the hottest craze in medical science. Total fat, not giving regard to types of fat, was deemed the key culprit to chronic diseases. However, around the same time, many large prospective epidemiologic studies had consistently found that total fat was NOT associated with breast cancer, total fat was NOT associated with colorectal cancer, and total fat was NOT associated with cardiovascular disease.

Nevertheless, several vocal advocates pushed such a low-fat trial through the NIH by lobbying Congress, despite mounting scientific evidence that it was not total fat per se, but rather different types of fats that differentially influenced the risk of such diseases. Notably, trans-fat was already known in the early 1990s that it increased the risk of CVD, while other unsaturated fats were more beneficial for CVD. Differences in types of fat was also being recognized for cancer risks. Thus, 10 years late, the trial ultimately found what epidemiologists already knew all along about total fat.

As for the WHI low-fat trial itself- it had many limitations. Its study protocol specified that women reduce their fat intake to <20% size="2">Low-Fat Dietary Pattern and Risk of Invasive Breast Cancer: The Women's Health Initiative Randomized Controlled Dietary Modification Trial
Prentice et al.
JAMA. 2006; 295:629-642.

Low-Fat Dietary Pattern and Risk of Colorectal Cancer: The Women's Health Initiative Randomized Controlled Dietary Modification Trial
Beresford et al.
JAMA. 2006; 295:643-654.

Low-Fat Dietary Pattern and Risk of Cardiovascular Disease: The Women's Health Initiative Randomized Controlled Dietary Modification Trial
Howard et al.
JAMA. 2006; 295:655-666.