Around meta-analysis (12): decision trees

​by Losia Lagisz
 
Get right tools for the right jobs. Surprisingly, decision trees are seldom mentioned in the context of systematic reviews / meta-analyses (I will use the latter term thereafter). An obvious application is to use decision trees to streamline screening of hundreds and thousands of publications found in online databases of academic literature.

 What are the decision trees?

​It is easiest to imagine them as a list of interlinked questions, where each answers leads you to the next appropriate questions. More formally, they are a structural representation (usually visual) of a series of decisions that have to be made. When screening literature, we need to assess each paper against all our pre-defined inclusion criteria. These criteria, expressed as a series of questions, naturally form a backbone of a linear decision tree. It is easy to build a decision tree if we break our inclusion criteria according to PICO (or similar) components of our research question. Thus, a generic decision tree for screening literature would look like this:

How to make a screening decision tree

1. Frame your meta-analytic question in a PICO framework (or other, as relevant), then express each PICO component as a question with a negative answer leading to exclusion (e.g. “is it a study on Drosophila flies?” If answer is “yes” (or “probably”, se below) – continue to next question, if “no” – exclude and move on to a next study)
2. You can add more general questions before the PICO-related questions. These should be questions you can usually answered very quickly, e.g. just by looking at the study title or first sentence of the abstract (e.g., “Is it an empirical study?” or “Is it a study in English?”)
3. You can add more specific questions after the PICO-related questions. These should be questions you can usually answer after reading abstract / full text (e.g., “Is the study performed in a lab?”
4. You will actually need two decision trees – first for screening bibliometric records (titles + abstracts + keywords), and second for screening full texts of publications (with SI, data etc., if relevant). In the first tree the positive answers do not need to be definitive (so “probably” or “maybe” is as good as “yes” – this is because there is often not sufficient information in the abstract to be sure, but if you feel that the study might be relevant, then you should be inclusive until you can be sure that the study does not meet given criterion, so just keep it until full text screening).
5. Customise, add or remove tree elements, to make it relevant to your specific meta-analytic question and stage of screening.
6. Even at full text screening stage, if data or details are missing, you can try to contact the study authors asking for these, before making final inclusion/exclusion decision.

​Template decision trees for two stages of screening:

Other practical tips for creating decision trees

1. Pilot your decision trees on a sub-sample of bibliometric records AND full texts.
2. If you screen in parallel with another person, do the piloting independently, compare the results and discuss any difficulties that arose when making decisions.
3. You can add explanatory notes to the nodes of the decision trees, as needed, especially for tricky questions.
4. Present your final decision trees in Supplementary Materials / Appendices of your resulting publication.

Advantages of using decision trees for screening

1. Faster screening
2. Less mistakes
3. Fewer decision conflicts (if more than one person screening)
4. Better transparency