Benchmarks, wikis, and open-source causal discovery
Patrik Hoyer, University of Helsinki, Finland
The discovery of causal relationships from non-experimental data is a challenging (and somewhat controversial) problem. Although a great number of methods for this 'causal discovery' problem have been developed, the empirical question of how well they typically perform, both in absolute and in relative terms, is not yet adequately understood. This is, at least to some extent, due to the dearth of standardized test problems on which to compare the methods. Echoing the ideas of the causality 'pot-luck' challenge, I will argue that there is a need for well-defined causal discovery benchmark tests. Further echoing the challenge, I argue that collecting the benchmarks should be a collaborative, bottom-up process. Emphasis needs to be placed on the openness and transparency of the system, the stability and future availability of all contributed materials, and the fostering of collaborative efforts. Such properties might be easiest achieved in an open wiki with full open-sourcing (including proper licensing) of all contributed data, tasks, and solutions.