Business analytics use advanced techniques that can analyze and process large and diverse data sets in order to generate valuable insights and lead to better business decisions. Of the three types of business analytics – descriptive, predictive, and prescriptive – only the latter focus on decision making. This paper aims to address two limitations of existing approaches in prescriptive analytics: (i) the lack of a transparent integration between predictive and prescriptive analytics and (ii) the incorporation of human knowledge and experience within the decision-making process. In order to address these points, the paper develops a framework that integrates data-driven predictions and the decision-making process by taking account human experience. The framework adopts interactive reinforcement learning algorithms and provides a concrete approach for data-driven human-AI collaboration. The main challenges and limitations of the approach are also discussed.