TY - JOUR TI - Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. AU - Soboczenski, Frank AU - Trikalinos, Thomas A. AU - Kuiper, Joël AU - Bias, Randolph G. AU - Wallace, Byron C. AU - Marshall, Iain J. T2 - BMC Medical Informatics & Decision Making AB - Objective: Assessing risks of bias in randomized controlled trials (RCTs) is an important but laborious task when conducting systematic reviews. RobotReviewer (RR), an open-source machine learning (ML) system, semi-automates bias assessments. We conducted a user study of RobotReviewer, evaluating time saved and usability of the tool.Materials and Methods: Systematic reviewers applied the Cochrane Risk of Bias tool to four randomly selected RCT articles. Reviewers judged: whether an RCT was at low, or high/unclear risk of bias for each bias domain in the Cochrane tool (Version 1); and highlighted article text justifying their decision. For a random two of the four articles, the process was semi-automated: users were provided with ML-suggested bias judgments and text highlights. Participants could amend the suggestions if necessary. We measured time taken for the task, ML suggestions, usability via the System Usability Scale (SUS) and collected qualitative feedback.Results: For 41 volunteers, semi-automation was quicker than manual assessment (mean 755 vs. 824 s; relative time 0.75, 95% CI 0.62-0.92). Reviewers accepted 301/328 (91%) of the ML Risk of Bias (RoB) judgments, and 202/328 (62%) of text highlights without change. Overall, ML suggested text highlights had a recall of 0.90 (SD 0.14) and precision of 0.87 (SD 0.21) with respect to the users' final versions. Reviewers assigned the system a mean 77.7 SUS score, corresponding to a rating between "good" and "excellent".Conclusions: Semi-automation (where humans validate machine learning suggestions) can improve the efficiency of evidence synthesis. Our system was rated highly usable, and expedited bias assessment of RCTs. DA - 2019/05/08/ PY - 2019 DO - 10.1186/s12911-019-0814-z DP - EBSCOhost VL - 19 IS - 1 SP - N.PAG EP - N.PAG J2 - BMC Medical Informatics & Decision Making SN - 1472-6947 UR - http://ezproxy.lib.utexas.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=rzh&AN=136337278&site=ehost-live DB - rzh KW - Clinical Trials KW - Comparative Studies KW - Evaluation Research KW - Feedback KW - Human KW - Multicenter Studies KW - Prospective Studies KW - Risk Assessment KW - Scales KW - Validation Studies ER -