“No, You Shouldn’t Use Machine Learning”

Why not? Or more precisely, why do some scientists think that?

Machine learning methods are opaque, with varying levels of obscurity surrounding their implementation. In the name of this opacity, sceptics time and again utter the line “no, you shouldn’t use machine learning, because it’s a black box”. Through these words, reluctant researchers attempt to dissuade others from using machine learning methods or at least urge caution (while a small minority calls for a total prohibition). While a critical approach is obviously vital, the argument rests on an assumption: model simplicity and transparency are needed for understanding phenomena. It seems that this assumption is misguided, or so argues philosopher of science E. Sullivan.

Last year, I attended a conference in London on machine learning and computational fluid dynamics. After one of the keynote speeches, many in the audience were eager to pose their questions. Eventually, someone asked the innocent question, “Do you think ML can be as intelligent as us humans?”. The speaker was quick and replied with “Well, what do you mean by intelligent?” leading to a short burst of laughter that filled the lecture hall. Slowly, the questions ascended into the abstract, and the hall found itself knee-deep in epistemology¹. Witnessing that exchange, I later found myself puzzled: What does it mean to understand things through a machine learning model? Is it capable of mediating our understanding of phenomena, and if so, what determines the quality of that mediation? Is the black box objection legitimate, or does it point at something else?

Understanding from Machine Learning Models

Minimal models are not unique to machine learning and are being used in many fields to explain phenomena by abstracting the low-level details. A major example in the field of computational fluid dynamics, which is used as an illustrative case (in Batterman and Rice, 2014²), is the Lattice Gas Automaton method, which looks nothing like a fluid at the microscale yet successfully explains continuum fluid behaviour. Similarly, black-box models abstract away low-level details, so why is machine learning under so much scrutiny?

According to her 2022 publication, titled “Understanding from Machine Learning Models”, E. Sullivan argues for a re-framing of the black box objection³. The work opens by drawing distinctions between different levels of explanation and their relation to understanding. Different explanatory questions that we ask of a model will lead to different levels of understanding: why (why a phenomenon occurs), how-actually (how is the phenomenon produced in practice), how-possibly (how the phenomenon could be produced)… etc. For example, a machine learning model predicting the weather could be asked: why does it rain? (why); how does a storm develop over a city? (how-actually); or could certain atmospheric conditions lead to rainfall? (how-possibly). Explanatory questions of the implementation of the method do not allow us to directly access a better understanding of the observed phenomenon, which is why it is argued that the “black-boxedness” of the implementation is not really the problem at hand. In fact, questions on the explanation of the implementation can obscure the locus of the lack of understanding.

The goal is not to dismiss black boxing as a problem altogether. Rather, it is to point out that it only limits our understanding of the phenomenon once the method is black-boxed at the highest level (only the inputs and outputs are accessible), and even then, with sufficient knowledge linking the model to the phenomenon, we can obtain answers to how-possibly questions. It is the lack of such knowledge that Sullivan identifies as the source of limitations in understanding, and this lack is known as link uncertainty.

Link Uncertainty

Link uncertainty is the lack of scientific or empirical evidence connecting the model to the target phenomenon. Link uncertainty exists independently of the opacity of the model; a transparent model can have high link uncertainty. Sullivan provides three examples that demonstrate the effect of link uncertainty in applications that use deep neural networks (a machine learning method).

The first is a model that is used to identify melanoma through skin imaging. This is an example of a low link uncertainty method, where there is strong scientific evidence that has already established the relation between the model and the phenomenon. Doctors already know how to diagnose melanoma through imaging, which allows the model to answer why and how-actually questions. The second is “the deep patient model”, developed by Mt. Sinai Hospital, which is a model that can predict certain conditions and illnesses based on the medical history of a patient. There is a medium level of link uncertainty, allowing for the model to answer some why and how-actually questions where there is an established scientific or empirical link. The third example, highly controversial, is a model trained to predict the sexual orientation of a person through photographs. The link uncertainty is high, since there is no scientific evidence that links facial features and sexual orientation. Therefore, the model is only capable of answering how-possibly questions, suggesting traits that warrant scientific investigation.

It is worth noting here that opacity can obscure evidence for linking; a high-level black-box implementation makes establishing such link more difficult. As the three examples above demonstrate, our epistemic predicament – our general problem of understanding – comes from link uncertainty, not implementation opacity or model illegibility (opacity of model structure). Yet not all hope is lost. A model does not lose its epistemic value due to link uncertainty; it can still answer how-possibly questions.

Where Does This Leave Model Explainability, Then?

To overcome model illegibility, a model can be made more explainable. Explainability is the ability to understand why a model outputs a given prediction (e.g. through visualising the behaviour, component analysis, etc.). Maximising model explainability enables the extraction of potential understanding of a phenomenon subject to link uncertainty. I plan to share an example of increasing the explainability of a reinforcement learning agent from my PhD project in my next blog. Stay tuned!

Learn more about my project here: DC-18 – Aufrande

1. Epistemology is a branch of philosophy concerned with knowledge. According to Merriam-Webster, it is defined as “the study or a theory of the nature and grounds of knowledge especially with reference to its limits and validity”. Epistemology. 2026. In Merriam-Webster.com. Retrieved May 13, 2026, from https://www.merriam-webster.com/dictionary/epistemology ↩︎
2. Batterman, R. W., & Rice, C. C. (2014). Minimal model explanations. Philosophy of Science, 81(3), 349-376. ↩︎
3. Sullivan, E. (2022). Understanding from machine learning models. The British Journal for the Philosophy of Science. ↩︎