"Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 De" by Brooke Levis, Parash Mani Bhandari et al.
 

Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Parash Mani Bhandari, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Dipika Neupane, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Suiqiong Fan, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Ying Sun, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Chen He, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Yin Wu, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Ankur Krishnan, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Zelalem Negeri, Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Ontario, Canada.
Mahrukh Imran, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Danielle B. Rice, Department of Psychiatry and Behavioural Neurosciences, McMaster University, Hamilton, Ontario, Canada.
Kira E. Riehm, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Marleine Azar, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Alexander W. Levis, Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada.
Jill Boruff, Schulich Library of Physical Sciences, Life Sciences, and Engineering, McGill University, Montréal, Québec, Canada.
Pim Cuijpers, Department of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.
Simon Gilbody, Hull York Medical School and the Department of Health Sciences, University of York, Heslington, York, UK.
John P. Ioannidis, Department of Medicine, Stanford University, Stanford, California.
Lorie A. Kloda, McGill University Libraries, Montréal, Québec, Canada.
Scott B. Patten, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada.
Roy C. Ziegelstein, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland.
Daphna Harel, Department of Applied Statistics, Social Science, and Humanities, New York University, New York.
Yemisi Takwoingi, Department of Applied Health Sciences, School of Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK.
Sarah Markham, Department of Biostatistics and Health Informatics, King's College London, London, UK.
Sultan H. Alamri, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia.
Dagmar Amtmann, Department of Rehabilitation Medicine, University of Washington, Seattle.
Bruce Arroll, Department of General Practice and Primary Health Care, University of Auckland, Auckland, New Zealand.
Liat Ayalon, Louis and Gabi Weisfeld School of Social Work, Bar Ilan University, Ramat Gan, Israel.
Hamid R. Baradaran, Endocrine Research Center, Institute of Endocrinology and Metabolism, Iran University of Medical Sciences, Tehran, Iran.
Anna Beraldi, Kbo-Lech-Mangfall-Klinik Garmisch-Partenkirchen, Klinik für Psychiatrie, Psychotherapie and Psychosomatik, Lehrkrankenhaus der Technischen Universität München, Munich, Germany.
Charles N. Bernstein, University of Manitoba IBD Clinical and Research Centre, Winnipeg, Manitoba, Canada.
Arvin Bhana, Centre for Rural Health, School of Nursing and Public Health, College of Health Sciences, University of KwaZulu-Natal, Durban, KwaZulu-Natal, South Africa.

Document Type

Journal Article

Publication Date

11-4-2024

Journal

JAMA network open

Volume

7

Issue

11

DOI

10.1001/jamanetworkopen.2024.29630

Abstract

IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

Department

Psychiatry and Behavioral Sciences

Link to Full Text

Share

COinS