Machine learning techniques are increasingly applied in Decision Support Systems. The selection processes underlying a conclusion often become black-boxed. Thus, the decision flow is not always comprehensible by developers or end users. It is unclear what the priorities are and whether all of the relevant information is used. In order to achieve human interpretability of the created algorithms, it is recommended to include domain experts in the modelling phase. Their knowledge is elicited through a combination of machine learning and social science techniques. The idea is not new, but it remains a challenge to extract and apply the experts’ experience without overburdening them. The current paper describes a methodology set to unravel, define and categorize the implicit and explicit domain knowledge in a less intense way by making use of co-creation to design human-centered algorithms, when little data is available. The methodology is applied to a case in the health domain, targeting a rheumatology triage problem. The domain knowledge is obtained through dialogue, by alternating workshops and data science exercises.