Since at least the time when it was understood that the circumference of a circle is pi multiplied by its diameter, the applications of mathematics have raced on far ahead of the foundations of the subject itself. By considering a variety of examples, principally from the 19th century, we will explore the tension between mathematics and its applications, and reasons why it remains a valuable and rewarding occupation to develop the necessary framework for existing and “well understood” theories. If time permits, I will give a short discussion of recent work I have done on the mathematical foundations for diffusion models in population genetics.