X-ray computed tomography (XCT) is an established non-destructive testing (NDT) method that, in combination with automatic evaluation routines, can be successfully used to establish a reliable 100% inline inspection system for defect detection of cast parts. While these systems are robust in automatically localizing suspected defects, human know-how in a secondary assessment and decision-making step remains indispensable to avoid an excess of rejected parts. Rather than changing the existing defect detection system and risking difficult to anticipate changes to a solid evaluation process, we propose the integration of human know-how in a subsequent support system through end-to-end learning. Using XCT data and the corresponding decisions performed by the XCT operator, we aim to support and possibly automate the secondary quality assessment process. In our paper we present a Convolutional Neural Network (CNN) architecture to predict both, the final decision of the XCT operator and a defect class indication, for cast parts rejected by the defect detection system based on XCT slice images. On a dataset of 19,459 defect records categorized in 7 classes, we achieved an accuracy of 92% for the decision and 93% for the defect class indication on the testing split. We further show that, by binding decisions to the reliability of the predicted defect class, our model has the potential to enhance also a production process with a near-faultless condition. Based on production-line data, we estimate that our model can reliably relabel 11% of defects reported during production and provide a defect class indication for another 57%.