MPI CBS Open Science, Leipzig, hosted a paper discussion of “Theory before the test”, that I co-authored with Giosuè Baggio. The meeting was part of a ReproducibiliTea journal club series organized by Gisela Govaart, Katarzyna Gugnowska, Greta Häberle and Mariella Paul and hosted by Katarzyna Gugnowska and Lieneke Janssen.

Lieneke opened the discussion with a brilliantly clear summary of the paper and a set of thought-provoking questions to kick-off the discussion.

What do we do now in cognitive science? Are we all building on “air castles”? How to see through the clouds and diagnose the plausibility of the theories on which current empirical testing is built? Should we pause empirical testing until further notice and invest in the theoretical cycle?

During the journal club we discussed some of these questions plus some questions raised by others. But I also feel some questions were left hanging. Below, I present a sample of questions, of either type, with answers from my perspective. Some of these questions may make more sense if you read the paper first (or for a TL;DR version, see this twitter thread).

If you want to add more questions to this list, feel free to let me know!

1. What should cognitive science now do? Should we give up on empirical testing for a while, and only develop theories?

I do not think we need to stop empirical research altogether. I do think we need a better balance. The last decades psychology has come to dominate cognitive science more and more (Gentner, 2010), leading to an almost exclusive focus on experiments. I think this disbalance is not doing cognitive science a service. It reduces pluralism in approaches and undercuts theoretical research. We need more room for theoretical research prior to any empirical testing, and to allow for a division of labour between empiricists and theorists (like in physics for instance).

2. What is needed to enact change?

We need a change in culture. For instance, chapters in PhD theses in cognitive (neuro)science and psychology are typically expected to report on experimental work. Alleviating this constraint would give PhD students opportunity to seek more theoretical depth in their research projects [¹]. And we should also allow for PhD theses that are fully theoretical in nature [²].

We need a change in education. Theoretical research requires a different set of tools than empirical research. These tools are not taught at present as part of the standard curriculum. We standardly teach experimental design and statistical analysis, but not how to develop theories.

3. Does your proposal imply that we need more slow science?

Yes. We need to allow more time for thinking. Less emphasis on results. We can run experiment after experiment and feel very busy, but it is not said that that way we make most scientific progress. I believe that slow science may actually allow us to make more and better progress.

4. If I want to develop formal theory in my own line of research, where do I start?

One possible place to start is the tutorial “Formalizing verbal theories: A tutorial by dialogue”, that I wrote with Mark Blokpoel. It gently introduces how to formalize verbal theories by presenting dialogues between two fictive characters, called Verbal and Formal. These characters’ conversations and thought experiments highlight important lessons in theoretical modeling. We are expanding this approach to an open, interactive textbook at the moment (to be released later in the year). I also recommend checking out other tutorials and approaches (for instance, Smaldino (2020), Robinaugh et. al (2021) and Borsboom et al. (2021)). Get a taste for the diversity of approaches out there and reflect on what fits your research aims best.

5. Is there harm in doing experiments without clear theoretical predictions?

I think there can be great value in that. For instance, I am involved in a team science project where we study empirically how people can communicate about things for which they do not yet have a shared vocabulary. We observe that people can do this quite well, remarkably so. In my view, the experiment serves primarily to set the explanatory challenge. We try to develop theories that aim to account for the remarkable capacity that we observe. This turns out much harder than expected (see van de Braak et al., 2021). Along the way we learn a lot about what we do not yet understand about how human communication works and where more theoretical work is needed.

6. In your paper you speak of ‘constraints’ on theory. What do you mean by that?

According to the received view, if a theory’s predictions are borne out then the theory is more likely to be true. But this ignores whether or not the theory was a priori possible at all (I explain this in this video clip). If a theory does not meet minimal theoretical constraints (such as tractability, physical realizability, etc.), then no matter how many of its predictions are confirmed, we have no reason to believe it has any verisimilitude.

7. What is meant by ‘verisimilitude’?

This is a technical term from philosophy of science, which means something like resemblance to truth or truth-likeness. One may wonder if that is the same as ‘likely to be true’, but I think it isn’t. Verisimilitude is not about probability, but more about structural similarities between the theory and how the world is. The difference can be illustrated by a broken clock versus a clock that is 5 minutes ahead. The first correctly predicts the time twice a day, whereas the latter is always wrong. Still we’d say the latter is ‘closer’ to the true time.

If you have more questions, possibly this blogpost about an earlier ReproducibiliTea journal club, answers them. If it doesn’t, please feel free to send me a message on twitter.

Further Reading

If you liked the Theory before the test paper, you may also like the Theory development requires an epistemological sea change commentary that I co-authored with Giosuè Baggio as well.

Acknowledgements

Image is a cropped version of a picture on Pixabay by Comfreak.

Footnotes