by Shinichi Nakagawa
Before my Xmas break, I met ChatGPT (Generative Pre-trained Transformer). Since then, she has been my teacher, wise but admits her mistakes. Also, she is humorous (when I ask her to be) and very patient.
I decided to see whether ChatGPT can actually do the first stage of screening, i.e. title and abstract screening. After negotiating with her for a few hours, I cracked the code and passed her a carefully worded selection criteria based on PECOS: Population, Exposure, Comparator, Outcome and Study design. And there she was. ChatGPT was telling me whether I should exclude or include a particular study after evaluating a study’s title and abstract.
I used lists of studies and criteria related to this protocol:
Vendl C, Taylor MD, Braeunig J, Gibson MJ, Hesselson D, Neely GG, Lagisz M, Nakagawa S. Profiling research on PFAS in wildlife: Protocol of a systematic evidence map and bibliometric analysis. Ecological Solutions and Evidence. 2021 Oct;2(4):e12106.
What amazed me was that ChatGPT matched the study with our criteria and summarized reasons. Wow, this is better than I can do (see examples: one recommending inclusion and the other recommending exclusion = both are spot on!)
Before my Xmas break, I met ChatGPT (Generative Pre-trained Transformer). Since then, she has been my teacher, wise but admits her mistakes. Also, she is humorous (when I ask her to be) and very patient.
I decided to see whether ChatGPT can actually do the first stage of screening, i.e. title and abstract screening. After negotiating with her for a few hours, I cracked the code and passed her a carefully worded selection criteria based on PECOS: Population, Exposure, Comparator, Outcome and Study design. And there she was. ChatGPT was telling me whether I should exclude or include a particular study after evaluating a study’s title and abstract.
I used lists of studies and criteria related to this protocol:
Vendl C, Taylor MD, Braeunig J, Gibson MJ, Hesselson D, Neely GG, Lagisz M, Nakagawa S. Profiling research on PFAS in wildlife: Protocol of a systematic evidence map and bibliometric analysis. Ecological Solutions and Evidence. 2021 Oct;2(4):e12106.
What amazed me was that ChatGPT matched the study with our criteria and summarized reasons. Wow, this is better than I can do (see examples: one recommending inclusion and the other recommending exclusion = both are spot on!)
I tested for around 15+ abstracts and ChatGPT was able to reproduce our decisions. So, I stopped there and then started to test whether she can extract some data from the text. This turned out to be more difficult as ChatGPT does not seem to take more than ~2,000 words as an input (although she claims there are no limits). Anyway, as long as I do not give her too much text, ChatGPT seems to be able to extract what animals were studied, PFAS chemicals and locations mentioned, in a format below:
That is all astounding. But some questions remain. How reproducible is this data? Can we make this process much more systematic?
I am hoping to work with a computer scientist and see whether some of these processes can be automated for multiple articles. We are entering an exciting but uncertain time. One thing I can say is that I will be trying to incorporate ChatGPT into some parts of my systematic review workflow from now on, not as a replacement for a human screener but as an addition for now.
I am hoping to work with a computer scientist and see whether some of these processes can be automated for multiple articles. We are entering an exciting but uncertain time. One thing I can say is that I will be trying to incorporate ChatGPT into some parts of my systematic review workflow from now on, not as a replacement for a human screener but as an addition for now.
RSS Feed