Artificial Intelligence (AI) systems that can provide clear explanations of their behaviors have been suggested in many studies as a critical feature for human users to develop reliance and trust when using such systems. Medical Experts (ME) in particular while using an AI assistant system must understand how the system generates disease diagnoses before making patient care decisions based on the AI’s output. In this paper, we report our work in progress and preliminary findings toward the development of a human-centered explainable AI (XAI) specifically for the diagnosis of Coronary Artery Disease (CAD). We applied syllogistic inference rules based on CAD Clinical Practice Guidelines (CPGs) to interpret the data mining results using a Support Vector Machine (i.e., SVM) classification technique—which forms an early model for a knowledge base (KB). The SVM’s inference rules are then explained through a voice system to the MEs. Based on our initial findings, we discovered that MEs trusted the system’s diagnoses when the XAI described the chain of reasoning behind the diagnosis process in a more interpretable form—suggesting an enhanced level of trust. Using syllogistic rules alone, however, to interpret the classification of the SVM algorithm lacked sufficient contextual information—which required augmentation with more descriptive explanations provided by a medical expert.