The Replication Crisis and Psychotherapy Efficacy Studies
Kutlu Kağan TürkarslanIn the last ten years, the replication crisis, which has manifested itself and shaken the field of psychology profoundly, has led researchers in every subfield of psychology to reconsider research findings and general assumptions. Though late to this trend, clinical psychology researchers have started to participate in these discussions. Psychotherapy efficacy studies, carried out over many years, constitute a major experimental area of clinical psychology. Psychotherapy efficacy studies are conducted to evaluate the effects of psychological treatments on various mental disorders. The findings of the replication crisis indicate that randomized controlled trials, which are frequently used to examine the efficacy of psychotherapies and meta-analyses that help evaluate the results of randomized controlled trials, have several methodological problems and biases. Randomized controlled trials have problems and biases such as low sample size, allegiance bias, selective outcome reporting, strictness of inclusion or exclusion criteria, using wait-list as a control group, problems of randomization and blinding, exclusion of drop-out subjects from the analyses, and ignoring psychotherapists effects. The main methodological issues of meta-analyses are related to publication bias and the inclusion of low-quality studies in meta-analyses. Research over the years has demonstrated that psychotherapies are effective tools for treating various mental disorders. However, there are two critical points to note. First, the studies showing the efficacy of psychotherapies barely meet current scientific methodological standards. Second, the real-life effectiveness of psychotherapies may be lower than those found in clinical trials. This review article aims to increase awareness regarding replication crisis within the field of clinical psychology and strengthen the methodological approach of future studies by addressing the methodological problems of and suggesting solutions for psychotherapy efficacy studies/meta-analyses.
Tekrarlama Krizi ve Psikoterapi Etkililik Çalışmaları
Kutlu Kağan TürkarslanSon 10 yıldır psikoloji alanında kendini gösteren ve alanı derinden sarsan tekrarlama krizi, psikolojinin her alt alanını, araştırma bulgularını ve genel kabullerini yeniden değerlendirmesi için harekete geçirmiştir. Klinik psikoloji alanının geç de olsa tekrarlama krizi tartışmalarına katılmaya başladığı görülmektedir. Uzun yıllardır gerçekleştirilen psikoterapi etkililik (efficacy) çalışmaları, klinik psikoloji içindeki başlıca deneysel alanlardan birini oluşturmaktadır. Psikoterapi etkililik çalışmaları çeşitli psikolojik tedavilerin psikolojik rahatsızlıklar üzerindeki etkilerini değerlendirmek için gerçekleştirilmektedir. Tekrarlama krizi bağlamında ortaya çıkan bulgular, psikoterapilerin etkililiklerini değerlendirmek için sıkça kullanılan seçkisiz kontrollü çalışmaların ve bu çalışmaların sonuçlarının toplu olarak değerlendirilmesine yardımcı olan meta-analizlerin pek çok yöntemsel problem ve yanlılık içerdiğini ortaya çıkarmıştır. Seçkisiz kontrollü çalışmalar düşük örneklem sayısı, bağlılık yanlılığı, seçici sonuç raporlama, örnekleme dahil etme ya da dışarıda bırakma kriterlerinin katılığı, kontrol grubu olarak bekleme listesi kullanımı, seçkisizleştirme ve körleştirme sorunları, çalışmayı bırakan katılımcıların analizlere dahil edilmemesi ve psikoterapist etkilerinin ihmal edilmesi gibi problemler ve yanlılıklar içermektedir. Meta-analizler ile ilişkili başlıca sorunlar ise yayın yanlılığı ve düşük kaliteli çalışmaların metaanalizlere dahil edilmesidir. Tüm bu durumlar psikoterapilerin olumlu etkilerini olduğundan daha yüksek gösterirken, edinilen bulguların sağlıklı bir şekilde değerlendirilmesini engellemektedir. Uzun yıllardır gerçekleştirilen çalışmalar, psikoterapilerin çeşitli psikolojik rahatsızlıkların tedavisinde tercih edilebilecek etkili araçlar olduğunu göstermektedir. Ancak dikkat edilmesi gereken iki önemli husus vardır. Bunlardan ilki psikoterapilerin etkililiğini gösteren çalışmaların yöntemsel anlamda güncel bilimsel standartları yakalamakta zorlanmasıdır. İkinci önemli husus ise psikoterapilerin gerçek etkilerinin çalışmalarda bulunan etkilerden daha az olabileceğidir. Bu derleme makalesinin amacı, psikoterapi etkililik çalışmalarını ve meta-analizleri etkileyen yöntemsel problemlere ve onların çözümlerine değinerek klinik psikoloji alanında bu konuda ortaya çıkan farkındalığın arttırılmasına ve gerçekleştirilecek çalışmaların yöntemsel olarak güçlendirilmesine yardımcı olmaktır.
Recent discussions in the field of psychology confirm the existence of the replication crisis and reveal that psychology has been facing a methodological crisis for several years (Maxwell et al., 2015; Open Science Collaboration, 2015). A major attempt (Open Science Collaboration, 2015) to replicate well-known cognitive and social psychology studies published in high-quality journals resulted in only a quarter of the studies being replicated. These unexpected results led researchers to question and review the research methods and practices used in the field of psychology (Nosek et al., 2015). Compared to other fields of psychology, clinical psychology was late to respond to these methodological discussions (Leichsenring et al., 2017; Sakaluk et al., 2019; Tackett et al., 2017). Clinical psychology research mainly includes studies on psychological evaluation, diagnosis of mental disorders, investigation of individual differences in psychopathologies, and treatment of mental disorders, meaning that the research utilizes both experimental and observational methods (Tackett et al., 2017). Psychotherapy efficacy studies are commonly conducted experimental studies in clinical psychology. Researchers use randomized controlled trials (RCTs) to examine the efficacy of a psychotherapy intervention for a specific mental disorder (Mulder et al., 2018; Wampold et al., 2011). Furthermore, the findings of RCTs are evaluated with meta-analyses, which can quantitatively summarize the results of numerous studies (Berkeljon & Baldwin, 2009; Leichsenring et al., 2017; Smith & Glass, 1977). With the replication crisis, a need to review psychotherapy efficacy studies and meta-analyses in terms of methodological problems has arisen.
RCTs may have a number of methodological problems and biases, including (1) small sample size (Button et al., 2013; Sakaluk et al., 2019), (2) allegiance bias (Leichsenring et al., 2017; Leykin & DeRubeis, 2009), (3) selective outcome reporting (Bradley et al., 2017; Miguel et al. 2021; Shinohara et al., 2015), (4) the strictness of inclusion or exclusion criteria (Hoertel et al., 2014, 2015; Westen et al., 2004; von Wolff et al., 2014), (5) using wait-list as a control group (Cuijpers et al., 2016; Furukawa et al., 2014; Munder et al., 2019), (6) randomization and blinding problems (Cuijpers et al., 2010; Cuijpers & Cristea, 2016), (7) the exclusion of drop-out participants from the analyses (Cooper & Conklin, 2015; Cuijpers & Cristea, 2016; Fernandez et al., 2015), and (8) ignoring psychotherapists’ effects (Del Re et al., 2012; Johns et al., 2019; Leichsenring et al., 2017; Owen et al., 2015). These problems can be prevented by applying a rigorous methodology. In terms of meta-analyses, publication bias (Driessen et al., 2015; Flint et al., 2015; Fonagy et al., 2017; Linardon et al., 2019; Niemeyer et al., 2012) and the inclusion of low-quality studies in meta-analyses (Coyne & Kok, 2014; Cuijpers & Cristea, 2016; Hengartner, 2018) deflate the validity of the results of meta-anaylses. Therefore, a new publishing paradigm is needed to overcome publication bias (Bradley et al., 2017). Moreover, eliminating methodological problems and biases of RCTs by itself increases the quality of the studies analyzed in meta-analyses. Considering the problems related to psychotherapy efficacy studies and meta-analyses, it could be argued that methodological crisis in clinical psychology is responsible for the Dodo Bird Verdict (Lambert, 2013). Besides the problems related to RCTs and meta-analyses, the interpretation of the psychotherapy efficacy studies must consider four important issues: meanings of the effects size, the duration of the observed effect, response rate of the treatment, and transfer of the effect to real-world settings.
Discussion
Goldfried (2020) asserted that the field of psychotherapy is still an infant field, and it cannot be regarded as a mature science, which has a solid and established core as well as new developments. In contrast, the field of psychotherapy is a conflicting and competitive field in which different theoretical approaches strive to maintain their existence and prove the effectiveness of their methods (Budd & Hughes, 2009). Based on the literature, it could be asserted that psychotherapy is an effective treatment for various mental disorders (Lambert, 2013; Rubio-Aparicio et al., 2018). However, all these methodological problems and biases of the RCTs and meta-analyses indicate two important issues. First, the studies conducted to test the effects of psychotherapies barely meet current scientific standards (Coyne & Kok, 2014; Hengartner, 2018; Leichsenring et al., 2017; Sakaluk et al., 2019). Second, the real-life effectiveness of psychotherapies is less than that found in previous studies (Schäfer & Schwarz, 2019; Wampold, 2019). For example, when specific biases were corrected, the efficacy of psychotherapies for depression was slightly above the accepted threshold (Cuijpers et al., 2018). The findings of RCTs and meta-analyses play a critical role in the selection of treatment options for mental disorders (Shean, 2016). This issue perhaps adds even more ethical responsibilities for clinical psychologists (Tackett et al., 2017). In conclusion, the findings suggest that clinical psychology researchers and clinicians must be aware of the effects of the replication crisis on the field and strive to bring their research closer to up-to-date scientific standards (Tackett et al., 2019).
