In this paper, we introduce Physical Bongard Problems (PBPs) as a novel and potentially rich approach to study the impact the constraints of a physical world have on mechanisms of concept learning and scene categorization. Each PBP consists of a set of 2D physical scenes which are positive or negative examples of a concept that must be identified. We discuss the properties that make PBPs challenging, analyze computational and representational requirements for a computational solver, and describe a first implementation of such a system. It can solve a subset of non-trivial PBPs using a version space approach for achieving its scene categorizations. The key element is a physics engine that is used both for the construction of information-rich physical features and for the prediction of how a given situation might evolve.