Adaptive Clinical Trials Using Bayesian Inference and Decision Theory

I recently returned from Phoenix where I summarized a consensus conference on Comparative Effectiveness Research (CER). The conference was sponsored by the American College of Sports Medicine (ACSM), one of my favorite clients.  Among other things, the conference focused on the idea that unlike randomized control trials (RCTs), long considered the gold standard in clinical research, factorial designs allow you to construct adaptive interventions. As I understand it, at the heart of these study designs are a form of game theory and bayesian modeling. I know little on either topic so I asked my colleagues from the American Medical Writers Association what they knew.

One of my AMWA colleagues let me know that Bayesian methods have the potential to significantly reduce sample sizes, and therefore research costs. Game theory isn’t used all that much for adaptive clinical trials except in the form of statistical decision theory. A list of references on Bayesian inference and decision theory is provided at the bottom of this blog post (thanks to my AMWA colleague Robert Ryley).

Another AMWA member stated that within the medical device community people turn to Don and Scott Berry for information on adaptive clinical trial design. Check out Berry Consultants website. Another trusted AMWA colleague pointed me toward a recent NPR article, in which Stuart Kauffman states, “…when RCTs work, they do really work, often well. But they often fail in complex biological-medical situations where causality is multifactorial, as it typically is. In place of RCT, our group has found a better alternative in these cases which we call ‘Team Learning.’” Finally an AMWA member said that while she couldn’t comment on adaptive clinical trial design, she did know that game theory, particularly a form of “crowd sourcing”, is being used in the design of diagnostic algorithms.

The Patient-Centered Outcomes Research Institute (PCORI) will be awarding grants in CER through 2019. What are your thoughts on applications of Bayesian principles and decision theory to medical research?

Bayesian Inference and Decision Theory References-

Provided by Robert Ryley (rryley1976@GMAIL.COM)

Here is a list of decision theory references with a numerical ranking of the mathematical competence required to understand the text:

1. High School Level — elementary algebra or geometry at most
2. Undergraduate Level text — Multivariable calculus helpful. Basic calculus a must.
3. Advanced Undergraduate/Graduate Level Text —  Complex Analysis and Measure Theory presumed

Classic Texts
Title: Elementary Decision Theory
Initially Published: 1959
Authors:  Herman Chernoff, Lincoln Moses
Difficulty Rank: 1
Summary: A classic in decision theory that provides a very good introduction to basic frequentist statistics and their application to decision problems.   It has been kept in print by Dover publications and is very cheap compared to more modern texts.  One thing to keep in mind — the data analysis methods described here were for people who only had pencil and paper as an aid. Start here first if your understanding of basic undergraduate statistics is a bit rusty.

Title: Games and Decisions: Introduction and Critical Survey
Authors: R. Duncan Luce, Howard Raiffa
Initially Published: 1957
Difficulty Rank: 2
Summary: Provides more intuitive justifications for game theoretic reasoning.  The authors wrote the text to broaden the knowledge of these methods for social scientists.  Apparently, this text was used by John Nash when he taught an intro game theory course.  A more modern, and less mathematically oriented approach is also provided in the book Negotiation Analysis, which has Howard Raiffa as one of the co-authors.

Title: Theory of Games and Statistical Decisions
Authors: David Blackwell, M.A. Girshick
Initially Published: 1954
Difficulty Rank: 3
Summary: I believe this text was widely used in graduate-level statistics programs.  At the very least, it is widely cited in more modern statistics books.  It is probably overkill for most of us, but those who want to understand how statistical procedures are evaluated by experts will likely want to study this.  I have a copy lying around somewhere.  This has also been kept in print by Dover publications.

Modern Texts
Title: Statistical Decision Theory and Bayesian Analysis
Author: James O. Berger
Initially Published: 1985
Difficulty Rank: 3
Summary: A highly recommended graduate level text in statistical decision theory. Although it continues to be used in graduate-level statistics programs as far as I know, it could be followed by someone with college-level algebra and calculus who has persistence to work through the examples and look up what is unfamiliar.

Title: Bayesian Data Analysis: A Tutorial
Authors: D.S. Silva, John Skilling
Initially Published: 2006
Difficulty Rank: 3
Summary: This book was designed for undergraduates in science and engineering.  It encourages thinking in probabilistic terms and shows how to apply mathematical methods commonly used in engineering toward statistical problems.  Physicist and prominent Bayesian protagonist E.T. Jaynes recommended this book as a complement to his more theoretical book Probability Theory: The Logic of Science.

Title: Introduction to Applied Bayesian Statistics and Estimation for Social Scientists
Author: Scott M Lynch
Initially Published: 2007
Difficulty Rank: 2
Summary:  A very good intro for social scientists — psychology, sociology, etc.  The author provides some basic methods from calculus and matrix algebra for those who are lacking in this area.  It will certainly help you bridge the gap from conventional frequentist methods toward a Bayesian way of thinking about problems.

Title: Bayesian Adaptive Methods for Clinical Trials
Author: Scott M Berry
Initially Published: 2010
Difficulty Rank: N/A
Summary: I have not purchased this book, but it is certainly on my wish list.

How I Transitioned From Bench Science To Medical Writing

I get emails weekly from people who want to know how to switch from bench science to medical writing. I don’t have a good answer, but I can share how it happened for me, which was a combination of luck, persistence, and New York-style chutzpah.

Soon after I began my post-doc at NINDS, a postcard arrived in our lab saying, “Congratulations Dr. Baker, you have been chosen from among hundreds of applicants to be one of a dozen post-doctoral fellows to participate in a prestigious writing seminar.” The workshop was to be taught by Dr. Ruth Levy Guyer, herself a bench scientist-turned-renowned writer of books, essays, articles, reviews, and commentaries. As luck would have it, Dr. Baker had departed for a job at Genentech the week before. Naturally I went to the workshop, and was permitted to stay because Dr. Guyer admired my moxy. I enjoyed her course thoroughly and remember many of her lessons to this day.

I love to write. Invariably the feedback on my dissertation, grant applications, and journal articles began with praise for the writing. So when a fellow post-doc asked me to edit an article for her because she was bogged down with work, I readily accepted. It was not a research article—she volunteered as an editor for a journal called Women In Science. I enjoyed editing the article so much that I too began to volunteer. Soon thereafter the editor, Pam Hines (a senior editor at Science magazine), gave me a column as well.

One day an article arrived for me to edit, and the author was none other than Dr. Guyer. The idea of editing the work of my writing teacher filled me with dread. I must have proven myself a marginally acceptable editor, because afterward Ruth graciously informed me that her husband worked for NHGRI and that soon the institute would be advertising for a staff writer. Was I interested?

I was not on the job market and had never considered leaving my post-doc after just two years. But off I went to see Leslie Fink, the head of communications at NHGRI. I had no writing samples but Leslie took a chance on me: Had I seen a movie lately? Could I write a review?

So I landed my first medical writing position with a “review” of the wonderful Muhammad Ali documentary “When We Were Kings.” Leslie took a risk, but so did I: I agreed to work at my post-doc salary, which was half what they paid writers; but worse, I agreed to work for six months with no obligation for them to hire me. The conventional wisdom was that if you left the bench for six months, you were finished in medical research. So I was burning a bridge with no promise of a future in writing.

My lab chief at NINDS thought I had lost my mind. My department chair at Mt. Sinai Medical School, where I had earned my Ph.D., called to express his dismay, telling me, “You are one of the ones who would have made it.” (My dissertation had been published as a two-author paper in a prestigious journal.) My research colleagues almost without exception told me frankly that I was making a huge mistake and would live to regret it.

I didn’t. I set to work at NHGRI, happily clocking the insane hours to which I had grown accustomed as a post-doc. Those working in government administration are not necessarily known for working long hours, so I quickly impressed my supervisors with my hard work, if not my dazzling writing skills. Shortly after arriving they advertised for a writer (“Seeking a medical writer with curly brown hair and blue eyes, who plays French horn and piano and trained as a dancer…”—the ad fell just short of this level of specificity) and I became an inside hire, as often is the case at NIH.

I learned a great deal while working in the Office of Press, Policy, and Communications, as the NHGRI budget at the time allowed for only one staff writer position. I drafted press releases, fact sheets, policy documents, appropriations testimony, budget justifications to Congress, meeting summaries, and annual reports. I helped with fact checking, press prep, lecture prep, speech writing, and book chapters. I even got to work on a Shattuck lecture for the New England Journal of Medicine. Leslie Fink was patience personified in teaching me the basics. I learned about policy from Dr. Kathy Hudson, our hard-nosed and extremely talented policy wonk who was herself a former bench scientist (Kathy is now in a top leadership position at NIH.) I had the extreme good fortune to work under Dr. Francis Collins, then NHGRI Director and head of the Human Genome Project (now NIH Director.) Those who read my blog already know of my high regard for Dr. Collins.

Take-home messages for those looking to transition from bench science to medical writing:

  • Be persistent. Expect that most avenues you pursue will lead nowhere. When I started my own medical writing company, I must have shaken a thousand hands at hundreds of onerous networking functions before I got a break. When I was considering a career change, I joined National Association of Science Writers (NASW), D.C. Science Writers Association (DCSWA; lyrically pronounced “duck-swa”), and later the American Medical Writers Association (AMWA). I participated in any and all forums they offered, both electronically and otherwise.
  • Be pushy. Joe Palca, science correspondent for NPR (and married to Kathy Hudson), gave a talk on how to break into science writing. His advice was, “Shamelessly exploit every contact you have,” or something to that effect. When considering a career change, I told every person I came across, whether they were in research, writing, or they happened to be sitting next to me on the metro. Hey, you never know. When that postcard arrived in the lab announcing Dr. Baker’s acceptance into a writing workshop, there was no question that I would go in her stead.
  • Take risks. I accepted the position at NHGRI knowing they had no obligation to hire me at the end of my contract, and knowing that I could not return to bench science. Had I not done so, I would probably still be plating cells and running gels.
  • Expect to work your arse off. I succeeded at my first writing job in large part because of the hours I put in. Later, I succeeded at launching a medical writing company in the middle of a recession while living in the middle of nowhere, in large part because I worked seven days a week. My company has been extremely successful, and I still work seven days a week. If you love what you do, you won’t mind, or even notice, usually. (Please note however that your family will notice. And mind.)
  • Don’t be bothered by the naysayers. I took an enormous amount of flack from my research colleagues when I made the switch. This is only troublesome if you fail afterward. When you succeed, they will all suffer collective amnesia concerning their doubts and criticism, and will be full of praise for your successes.
  • Do it for free. Perhaps the best piece of career advice I have ever received. If you want to transition into something new, do it for free for a while, at least part time (time and finances permitting.) I volunteered to write and edit at a free journal. It led to my first job offer, and put me in contact with an editor at Science magazine. I also accepted my first writing position at a greatly reduced salary (short-term). Few people will turn down free (or greatly reduced cost) work, and it will help you gain experience, contacts, writing samples, and references in your field.

I hope this story is helpful to someone, or at least was mildly entertaining. Best of luck to anyone looking to make the switch!