Bayesian Methods for Hackers

Using Python and PyMC

The Bayesian method is the natural approach to inference, yet it is hidden from readers behind chapters of slow, mathematical analysis. The typical text on Bayesian inference involves two to three chapters on probability theory, then enters what Bayesian inference is. Unfortunately, due to mathematical intractability of most Bayesian models, the reader is only shown simple, artificial examples. This can leave the user with a so-what feeling about Bayesian inference. In fact, this was the author's own prior opinion.

After some recent success of Bayesian methods in machine-learning competitions, I decided to investigate the subject again. Even with my mathematical background, it took me three straight-days of reading examples and trying to put the pieces together to understand the methods. There was simply not enough literature bridging theory to practice. The problem with my misunderstanding was the disconnect between Bayesian mathematics and probabilistic programming. That being said, I suffered then so the reader would not have to now. This book attempts to bridge the gap.

If Bayesian inference is the destination, then mathematical analysis is a particular path towards it. On the other hand, computing power is cheap enough that we can afford to take an alternate route via probabilistic programming. The latter path is much more useful, as it denies the necessity of mathematical intervention at each step, that is, we remove often-intractable mathematical analysis as a prerequisite to Bayesian inference. Simply put, this latter computational path proceeds via small intermediate jumps from beginning to end, where as the first path proceeds by enormous leaps, often landing far away from our target. Furthermore, without a strong mathematical background, the analysis required by the first path cannot even take place.

Bayesian Methods for Hackers is designed as an introduction to Bayesian inference from a computational/understanding-first, and mathematics-second, point of view. Of course as an introductory book, we can only leave it at that: an introductory book. For the mathematically trained, they may cure the curiosity this text generates with other texts designed with mathematical analysis in mind. For the enthusiast with less mathematical background, or one who is not interested in the mathematics but simply the practice of Bayesian methods, this text should be sufficient and entertaining.

The choice of PyMC as the probabilistic programming language is two-fold. As of this writing, there is currently no central resource for examples and explanations in the PyMC universe. The official documentation assumes prior knowledge of Bayesian inference and probabilistic programming. We hope this book encourages users at every level to look at PyMC. Secondly, with recent core developments and popularity of the scientific stack in Python, PyMC is likely to become a core component soon enough.

PyMC does have dependencies to run, namely NumPy and (optionally) SciPy. To not limit the user, the examples in this book will rely only on PyMC, NumPy, SciPy and Matplotlib.

Printed Version by Addison-Wesley

<div style="float: right; margin-left: 30px;"><img title="Bayesian Methods for Hackersg"style="float: right;margin-left: 30px;" src="http://www-fp.pearsonhighered.com/assets/hip/images/bigcovers/0133902838.jpg" align=right height = 200 /></div>

Bayesian Methods for Hackers is now available as a printed book! You can pick up a copy on Amazon. What are the differences between the online version and the printed version?

• Additional Chapter on Bayesian A/B testing
• Updated examples
• Answers to the end of chapter questions
• Additional explanation, and rewritten sections to aid the reader.

Contents

See the project homepage here for examples, too.

The below chapters are rendered via the nbviewer at nbviewer.jupyter.org/, and is read-only and rendered in real-time. Interactive notebooks + examples can be downloaded by cloning!

PyMC2

• Prologue: Why we do it.

• Chapter 1: Introduction to Bayesian Methods Introduction to the philosophy and practice of Bayesian methods and answering the question, "What is probabilistic programming?" Examples include:

  • Inferring human behaviour changes from text message rates

• Chapter 2: A little more on PyMC We explore modeling Bayesian problems using Python's PyMC library through examples. How do we create Bayesian models? Examples include:

  • Detecting the frequency of cheating students, while avoiding liars
  • Calculating probabilities of the Challenger space-shuttle disaster

• Chapter 3: Opening the Black Box of MCMC We discuss how MCMC operates and diagnostic tools. Examples include:

  • Bayesian clustering with mixture models

• Chapter 4: The Greatest Theorem Never Told We explore an incredibly useful, and dangerous, theorem: The Law of Large Numbers. Examples include:

  • Exploring a Kaggle dataset and the pitfalls of naive analysis
  • How to sort Reddit comments from best to worst (not as easy as you think)

• Chapter 5: Would you rather lose an arm or a leg? The introduction of loss functions and their (awesome) use in Bayesian methods. Examples include:

  • Solving the Price is Right's Showdown
  • Optimizing financial predictions
  • Winning solution to the Kaggle Dark World's competition

• Chapter 6: Getting our prior-ities straight Probably the most important chapter. We draw on expert opinions to answer questions. Examples include:

  • Multi-Armed Bandits and the Bayesian Bandit solution.
  • What is the relationship between data sample size and prior?
  • Estimating financial unknowns using expert priors

  We explore useful tips to be objective in analysis as well as common pitfalls of priors.

PyMC3

• Prologue: Why we do it.

• Chapter 1: Introduction to Bayesian Methods Introduction to the philosophy and practice of Bayesian methods and answering the question, "What is probabilistic programming?" Examples include:

  • Inferring human behaviour changes from text message rates

• Chapter 2: A little more on PyMC We explore modeling Bayesian problems using Python's PyMC library through examples. How do we create Bayesian models? Examples include:

  • Detecting the frequency of cheating students, while avoiding liars
  • Calculating probabilities of the Challenger space-shuttle disaster

• Chapter 3: Opening the Black Box of MCMC We discuss how MCMC operates and diagnostic tools. Examples include:

  • Bayesian clustering with mixture models

• Chapter 4: The Greatest Theorem Never Told We explore an incredibly useful, and dangerous, theorem: The Law of Large Numbers. Examples include:

  • Exploring a Kaggle dataset and the pitfalls of naive analysis
  • How to sort Reddit comments from best to worst (not as easy as you think)

• Chapter 5: Would you rather lose an arm or a leg? The introduction of loss functions and their (awesome) use in Bayesian methods. Examples include:

  • Solving the Price is Right's Showdown
  • Optimizing financial predictions
  • Winnin