Trends in Adaptive Design Methods in Dialysis Clinical Trials: A Systematic Review

Rationale & Objective Adaptive design methods are intended to improve the efficiency of clinical trials and are relevant to evaluating interventions in dialysis populations. We sought to determine the use of adaptive designs in dialysis clinical trials and quantify trends in their use over time. Study Design We completed a novel full-text systematic review that used a machine learning classifier (RobotSearch) for filtering randomized controlled trials and adhered to the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) guidelines. Setting & Study Populations We searched MEDLINE (PubMed) and ClinicalTrials.gov using sensitive dialysis search terms. Selection Criteria for Studies We included all randomized clinical trials with patients receiving dialysis or clinical trials with dialysis as a primary or secondary outcome. There was no restriction of disease type or intervention type. Data Extraction & Analytical Approach We performed a detailed data extraction of trial characteristics and a completed a narrative synthesis of the data. Results 57 studies, available as 68 articles and 7 ClinicalTrials.gov summaries, were included after full-text review (initial search, 209,033 PubMed abstracts and 6,002 ClinicalTrials.gov summaries). 31 studies were conducted in a dialysis population and 26 studies included dialysis as a primary or secondary outcome. Although the absolute number of adaptive design methods is increasing over time, the relative use of adaptive design methods in dialysis trials is decreasing over time (6.12% in 2009 to 0.43% in 2019, with a mean of 1.82%). Group sequential designs were the most common type of adaptive design method used. Adaptive design methods affected the conduct of 50.9% of trials, most commonly resulting in stopping early for futility (41.2%) and early stopping for safety (23.5%). Acute kidney injury was studied in 32 trials (56.1%), kidney failure requiring dialysis was studied in 24 trials (42.1%), and chronic kidney disease was studied in 1 trial (1.75%). 27 studies (47.4%) were supported by public funding. 44 studies (77.2%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review. Limitations We limited our search to 2 databases (PubMed and ClinicalTrials.gov) due to the scale of studies sourced (209,033 and 6,002 results, respectively). Conclusions Adaptive design methods are used in dialysis trials but there has been a decline in their relative use over time.

Rationale & Objective: Adaptive design methods are intended to improve the efficiency of clinical trials and are relevant to evaluating interventions in dialysis populations. We sought to determine the use of adaptive designs in dialysis clinical trials and quantify trends in their use over time.
Study Design: We completed a novel full-text systematic review that used a machine learning classifier (RobotSearch) for filtering randomized controlled trials and adhered to the Preferred Reporting Items for Systematic Review and Metaanalysis (PRISMA) guidelines.
Setting & Study Populations: We searched MEDLINE (PubMed) and ClinicalTrials.gov using sensitive dialysis search terms.
Selection Criteria for Studies: We included all randomized clinical trials with patients receiving dialysis or clinical trials with dialysis as a primary or secondary outcome. There was no restriction of disease type or intervention type.

Data Extraction & Analytical Approach:
We performed a detailed data extraction of trial characteristics and a completed a narrative synthesis of the data.
Results: 57 studies, available as 68 articles and 7 ClinicalTrials.gov summaries, were included after full-text review (initial search, 209,033 PubMed abstracts and 6,002 ClinicalTrials.gov summaries). 31 studies were conducted in a dialysis population and 26 studies included dialysis as a primary or secondary outcome. Although the absolute number of adaptive design methods is increasing over time, the relative use of adaptive design methods in dialysis trials is decreasing over time (6.12% in 2009 to 0.43% in 2019, with a mean of 1.82%). Group sequential designs were the most common type of adaptive design method used. Adaptive design methods affected the conduct of 50.9% of trials, most commonly resulting in stopping early for futility (41.2%) and early stopping for safety (23.5%). Acute kidney injury was studied in 32 trials (56.1%), kidney failure requiring dialysis was studied in 24 trials (42.1%), and chronic kidney disease was studied in 1 trial (1.75%). 27 studies (47.4%) were supported by public funding. 44 studies (77.2%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review.
Limitations: We limited our search to 2 databases (PubMed and ClinicalTrials.gov) due to the scale of studies sourced (209,033 and 6,002 results, respectively).
Conclusions: Adaptive design methods are used in dialysis trials but there has been a decline in their relative use over time.
R andomized clinical trials (RCTs) are the gold standard for evaluating the efficacy, futility, or harm of new therapies. 2 Compared with similar medical specialties, nephrology has traditionally had a low number of RCTs, particularly evident for patients with kidney failure requiring dialysis. 3 The comparatively low number of trials are postulated to be due to difficult recruitment, previous history of underpowered trials, and lack of funding. 4,5 Although the number of trials is increasing, nephrology continues to lag behind other specialties such as cardiology, hematology/oncology, and gastroenterology. 6,7, * Adaptive clinical trials use interim data analyses to modify the trial design or duration in a predefined way 8 without undermining the integrity or validity of the trial, thereby preserving the type 1 error (false-positive) rate. The most common type of adaptive design is the group sequential design, in which planned interim analyses permit stopping of trials for efficacy or futility. Other designs include sample size re-estimation, multiarm multistage trials, adaptive randomization, biomarker adaptive, and seamless phase 2/3 trials 9 (Box 1).
Adaptive clinical trials appear particularly suitable for the evaluation of novel interventions in dialysis by reducing resource requirements, decreasing time to study completion, and increasing the likelihood of study success, that is, power to answer hypothesis. 10 Previous trials in dialysis have overly relied on observational data to inform trial design, including assumptions of expected effect size and variance, 11 rather than estimates from early-phase clinical trials. If incorrect, trials may be underpowered with an insufficient sample size to answer the underlying research question. 11 Adaptive sample size re-estimation is a potential solution, as commonly used in cardiology trials, 12 such as planned blinded sample size re-estimation, which identifies inaccurate assumptions, thereby triggering altered recruitment targets midtrial to ensure adequate power.
Adaptive design may also be relevant when evaluating more established interventions. For example, the Deutsche Diabetes Dialyse Studie (4D) 13 reported that atorvastatin, 20 mg per day, did not reduce cardiovascular events in kidney failure requiring dialysis despite evidence of a 20% to 30% reduction in other populations. 14 This trial included a single dose of statin; it is hypothesized that alternative or multiple doses may have been more beneficial in a dialysis population given the significantly altered pharmacokinetics and pharmacodynamics. 11,15 An adaptive multiarm multistage trial design may have been more appropriate with 1 interim analysis at the end of stage I to identify an optimum dose to take forward into stage II. For example, the Telmisartan and Insulin Resistance in HIV (TAILoR) trial used a multiarm multistage design with 1 interim analysis to identify the most appropriate dose among 3 telmisartan doses (20,40, and 80 mg daily). All 3 doses were tested in stage I and telmisartan, 80 mg, was taken forward into stage II. 16 This systematic review aims to: (1) summarize the use of adaptive design methodology in RCTs in dialysis populations and populations at risk for requiring dialysis; (2) describe the characteristics of the trials that use adaptive designs, including dialysis modality, funding, and geographical location; (3) describe the characteristics of adaptive trial designs in dialysis trials; (4) estimate the percentage of adaptive clinical trials in dialysis among all dialysis RCT; and (5) outline temporal trends in all of the above.

METHODS
We performed a systematic review, reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines. 17 The protocol was registered with PROSPERO (CRD42020163946) and published separately. 18 There were no age or English language restrictions. After testing our predefined search strategy, 18 we found a small number (n = 16) of dialysis RCTs that reported an adaptive design method. We discovered that the adaptive design methods are often not reported in the title and abstract of articles and would not be detected in a traditional systematic search. To overcome this, we developed a novel "full-text systematic review" protocol and to our knowledge, this is the first use of this methodology.

Electronic Search: Dialysis Studies
We performed an electronic search on MEDLINE (PubMed) and ClinicalTrials.gov from database inception until June 1, 2020. Zotero was used as our reference manager. The dialysis search terms were adapted from Beaubien-Souligny et al, 19 2019 (and included dialysis, peritoneal dialysis, hemodialysis, hemodiafiltration, hemodiafiltration, hemofiltration, haemofiltration, extracorporeal blood cleansing, haemodialysis, renal dialysis, renal replacement, end stage kidney, end stage renal, stage 5 kidney, and stage 5 renal (Table S1). The output was stored in the Research Information Systems file format for PubMed and XML files for ClinicalTrials. gov.

Box 1. Adaptive Trial Designs
Seamless phase 2-3 design: Combines a traditional phase 2 with a phase 3 trial. Referred to as the "learning" phase and "confirmatory" phase. This design can reduce sample size and time to market for a positive treatment.
Sample-size re-estimation design: Allows for samplesize adjustment or re-estimation based on the results of interim analysis. Particularly useful if there is uncertainty about the treatment effect and variability and when inaccurate estimates could lead to overpowered or underpowered trials.
GSD: Allows a trial to stop early based on the results of interim analysis. GSD is the most common type of adaptive design. GSD can take 3 forms: early efficacy stopping, early futility stopping, and early efficacy or futility stopping design.
Multiarm multistage: A multistage design with several treatment arms. At interim analysis, inferior treatment arms are dropped based on prespecified criteria. Ultimately the best arms and the control group are retained. Some examples are pick-the-winners or drop-the-loser designs.
Biomarker-adaptive design: Allows for adaptations using information obtained from biomarkers. Often used in drug trials to target very selective populations for whom the drug likely works well. The biomarker response at interim analysis can be used to determine the target population.
Adaptive dose-escalation design: The dose level used to treat the next patient is based on the toxicity in the previous patients and escalation rules.

PLAIN-LANGUAGE SUMMARY
Adaptive designs make clinical trials more efficient and are one part of the solution for optimizing the design of clinical trials in dialysis. We performed a systematic review by searching 2 large databases for dialysis trials with adaptive designs and found 57 examples. They are used mostly in trials of acute kidney injury, affected (changed a trial) half the studies they were used in, and are usually not reported in titles or abstracts of articles. We also found that the relative use of adaptive designs in nephrology is decreasing over time. Greater knowledge of adaptive design examples in dialysis will further improve uptake in dialysis randomized clinical trials.

Machine Learning Classifier: RCTs
We used the high-sensitivity machine learning classifier (RobotSearch) to identify RCTs from the PubMed dialysis search output. 15 RobotSearch is a machine learning classification algorithm combining an ensemble of support vector machines and convolutional neural networks with a reported area under the curve of 0.987 (95% CI, 0.984-0.989) for RCT classification. We adjusted the parameters of RobotSearch to perform a sensitive search to increase the proportion of RCTs that are correctly identified. 15 Studies classified as likely to be RCTs were sourced for the full-text systematic review.

Full-Text Systematic Review: Adaptive Design Methods
We used Recoll for Windows to perform a full-text systematic review on our dialysis randomized clinical trial search results from PubMed and ClinicalTrials.gov. Recoll is based on the Xapian search engine library and provides a powerful text extraction layer and a graphical interface. The adaptive design search terms were adapted from Bothwell et al, 20 2018, and included phase 2/3, treatment switching, biomarker adaptive, biomarker adaptive design, biomarker adjusted, adaptive hypothesis, adaptive dose finding, pick the winner, drop the loser, sample size reestimation, re-estimations, adaptive randomization, group sequential, adaptive seamless, adaptive design, interim monitoring, Bayesian adaptive, flexible design, adaptive trial, play the winner, adaptive method, adaptive and dose and adjusting, response adaptive, adaptive allocation, adaptive signature design, treatment adaptive, covariate adaptive, and sample size adjustment (Table S2).

Manual Full-Text Review
We then performed manual full-text review to confirm studies that were included in the final systematic review. This process is summarized in a PRISMA flowchart (Fig 1). Full-text review was performed by C.J., R.M., and C.R. Disagreements were resolved by consensus and when a resolution was not reached by discussion, a consensus was reached through a third reviewer (M.J.O.).

Type of Study Design and Participants
RCTs of interventions in patients with kidney failure requiring dialysis and acute kidney injury (AKI) undergoing kidney replacement therapy including hemodialysis, peritoneal dialysis, hemodiafiltration, and hemofiltration. We did not limit our population to any specific disease. Additionally, we included studies that included dialysis as either a primary or secondary outcome.

Type of Intervention and Outcome
We did not place a restriction on the intervention type and included trials that studied medications during dialysis,  medical devices, dialysis parameters, and dialysis modality. Dialysis parameter is any specification of the dialysis treatment that can be changed at each session, for example, duration, ultrafiltration rate, and sodium profiling. We included all outcomes including surrogate markers, patient-centered outcomes, and hard clinical outcomes.

Selection and Analysis of Trials
C.J., R.M., and C.R. extracted the study characteristics independently and in parallel. Data collected included type of the adaptive design, stopping rule, impact of adaptive design (ie, stopping for futility or efficacy and sample size changes), trial population, intervention, dialysis modality, the country of the lead investigator, and the funder of the study (adapted from Hatfield et al, 21 2016; Table S3).

Assessment of the Quality of the Studies: Risk of Bias
We used the Cochrane Risk of Bias 2 Tool 22 to assess methodological quality of eligible trials, including random sequence generation, allocation concealment, blinding of participants and health care personnel, blinded outcome assessment, completeness of outcome data, evidence of selective reporting, and other biases. Risk-of-bias assessments were performed independently by C.J., R.M., C.R., and S.C. and disagreements were resolved by consensus. If 1 or more domains was rated as high, the study was considered at high risk of bias. We summarized our findings in a risk-of-bias table using the revised Cochrane risk-of-bias tool for randomized trials 23 (Table S4).

Data Synthesis
A descriptive synthesis of the data was performed. We reported overall outcomes and outcomes by: (1) frequency and type of adaptive design; (2) adaptive designs as a proportion of studies classified as dialysis RCTs by RobotSearch; (3) population, intervention, and outcome, including dialysis modality (hemodialysis, peritoneal dialysis, hemodiafiltration, and hemofiltration); (4) publication in high-impact journals; (5) geographic location and funding; (6) reporting of adaptive design methods in title and abstract; and (7) a risk-of-bias assessment.

RESULTS
The systematic search of articles on MEDLINE (PubMed) with dialysis keywords published before June 1, 2020, identified 209,033 results. A total of 5,452 articles were classified as probable RCTs by the machine learning classifier RobotSearch. 15 Full-text articles were sourced (n = 5,022) and we performed a full-text systematic review using adaptive design keywords that identified 358 studies for manual screening. A total of 50 studies, available as 66 articles, were included after full-text review (Fig 1). The systematic search of ClinicalTrials.gov with dialysis keywords published before June 1, 2020, identified 6,002 registered studies. A systematic search of ClinicalTrials.gov summary files using adaptive design keywords identified 54 studies for full review and 9 studies were included. In total, 57 studies, available as 68 articles and 7 ClinicalTrials.gov summaries, were included in the final analysis. A total of 31 studies were conducted in dialysis populations and 26 studies included dialysis as a primary or secondary outcome.

Study Characteristics
Frequency and Type of Adaptive Design Group sequential designs were the most common type of adaptive design method used; 35 (61.4%) trials (22 [71%] in dialysis populations and 13 [50%] in dialysis outcome trials; Table 1   ). The O'Brien-Fleming stopping boundary was the most common stopping rule, used in 9 trials (25.7%), followed by Lan DeMets, used in 8 trials (22.9%). A total of 29 trials (50.9%) were affected by the use of group sequential adaptive design, including 7 trials (41.2%) that stopped early for futility, 3 trials (17.6%) that stopped early for efficacy, and 4 trials (23.5%) that stopped early for safety.
Population, Intervention, and Outcome Studied AKI was studied in 32 trials (56.1%), kidney failure requiring dialysis was studied in 24 trials (42.1%), and chronic kidney disease (CKD) was studied in 1 trial (1.75%). Figure 3 reports the number of each population under study per year and shows a larger increase in adaptive design methods in AKI populations compared with kidney failure requiring dialysis populations. Medications were the most common intervention type, evaluated in 35 trials (61.4%), followed by dialysis modality in 7 trials (12.3%) and dialysis parameter in 4 trials (7%). Hemodialysis was the most common dialysis modality studied in 32 trials (56.1%), followed by hemodialysis and hemodiafiltration in 8 trials (14%); hemodialysis, hemodiafiltration, and hemofiltration in 7 trials (12.3%); and peritoneal dialysis in 4 trials (7%). Hard clinical outcomes were selected in 34 trials (59.6%), followed by surrogate outcomes in 20 trials (35.1%) and mixed in 3 trials (5.3%). The outcome measure was continuous in 15 trials (26.3%) and dichotomous in 42 trials (73.7%). Phase 3 studies were the most common study phase, studied in 41 trials (71.9%; Tables 1-3).

Publication in High-Impact Journals
A total of 32 studies (56.1%) were published in a highimpact journal (impact factor > 9). Fourteen studies (24.6%) were published in the New England Journal of Medicine, 6 studies (10.5%) were published in the Journal of the American Medical Association, 4 studies (7%) were published in Trials, and 2 studies (3.5%) were published in the Journal of the American Society of Nephrology.

Geographic Location and Funding
The most common country of the lead author was the United States in 24 studies (42.1%), followed by Germany in 7 studies (12.3%), France in 4 studies (7%), the Netherlands in 4 studies (7%), Australia in 3 studies (5.3%), and the United Kingdom in 3 studies (6% ;  Tables 1-3). Forty-nine studies (86%) were multicenter trials. Twenty-seven studies (47.4%) were supported by public funding, 21 studies (36.8%) were supported by private funding, 7 studies (12.3%) were supported by both public and private funding, and 2 studies (3.5%) did not report the source of funding.

Reporting of Adaptive Design Method in Title and Abstract
A total of 44 studies (77.2%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review search.

Risk of Bias
Risk of bias was assessed for 40 trials (protocols and clinicaltrials.gov were excluded; Fig S1; Table S4). Overall risk of bias was deemed to be "low" in 17 trials (42.5%), "some concerns" in 13 trials (32.5%), and "high risk" in 10 trials (25%). The randomization process led to some concerns for 10 studies (25%). Deviations from intended interventions led to some concerns for 4 studies (10%) and high risk for 6 studies (15%). Missing outcome data were deemed to be some concerns for 2 studies (5%) trials and high risk of bias for 2 studies (5%). Measurement of    outcome measures was deemed to be some concerns for 2 studies (5%) trials and high risk of bias for 1 study (2.5%). Selection of the reported result was deemed to be some concerns for 6 studies (15%) trials and high risk of bias for 1 study (2.5%).

DISCUSSION
In this systematic review, we report that adaptive design methods were used in 57 dialysis RCTs over a 20-year period. Although the absolute number has increased over time, the relative use of adaptive design methods in trials in dialysis populations and trials with dialysis as an end point has decreased. First, we report that the relative proportion of adaptive design methods in dialysis trials has decreased over time. The absolute number of dialysis trials using adaptive designs has increased each year, but this has not matched the overall increase in dialysis trials and therefore resulted in a relative decrease. We were unable to compare this result with other specialties because recent systematic reviews have not reported the relative use of adaptive designs. 21,90 Second, we report that group sequential designs are the most used type of adaptive design in dialysis trials. This is similar to previous systematic reviews in cardiology 91 and oncology 90 and in a review of registered clinical trials covering multiple specialties on clinicaltrials.gov. 21 Third, we report that adaptive designs were more common in AKI (56.1% of trials) than kidney failure requiring dialysis (42.1% of trials). This may reflect increasing use of adaptive design methodology in critical care 92 and sepsis-related trials, 93 in which AKI is most common. There were very few trials of CKD with a dialysis outcome (2%) that used an adaptive design. Many reasons for the paucity of CKD trials have been previously suggested, including the use of treatments in CKD despite a lack of evidence, difficulty recruiting to CKD trials due to stringent eligibility criteria, and underpowered subgroup analysis. 4,94 The infrequent use of adaptive designs in CKD trials may become a self-perpetuating barrier to using adaptive designs in future trials. 21 Fourth, we report that adaptive design methods affected the conduct of the randomized trial in most studies (50.9%). For example, 17 (48.6%) trials were affected by the use of group sequential adaptive design, including 7 trials (41.2%) stopped early for futility, 3 trials (17.6%) stopped early for efficacy, and 4 trials (23.5%) stopped early for safety. This finding is similar to a systematic review of published and publicly available trials in which the most common reason for stopping group sequential trials was futility. 20 Fifth, we found that the most common country of the lead author was the United States, 24 studies (42.1%), and the most common funding source was public, 27 studies (47.4%). This finding was different from a systematic review of published and publicly available trials in which 65% of trials reported industry funding. 20 Funding for    kidney research reached an all-time low in 2013 5 but this has recently changed in the United States with advocacy from scientific societies such as the American Society of Nephrology, whereby an executive order was signed in 2020 to reform the US end-stage kidney disease treatment industry. 95 Adaptive designs are one part of the solution for optimizing the design of clinical trials in dialysis and nephrology and will benefit from the improvement in the funding landscape. 94 Our study has several limitations. First, we limited our search to 2 databases (PubMed and ClinicalTrials.gov) due to the scale of studies sourced (209,033 and 6,002 results). This was a deviation from our protocol but necessary to make this full-text review feasible. Second, we decided to include RCTs with dialysis outcomes in addition to patients currently receiving dialysis. This permitted a more comprehensive review of the full landscape of AKI, kidney failure requiring dialysis, and CKD trials, but was a deviation from our original protocol. Third, the denominator for calculating the proportion of adaptive designs in all dialysis RCTs will include some false positives, that is, either not RCTs or not dialysis. We modified the parameters of the machine learning classifier to perform a sensitive search to include as many true positives as possible. We expect this misclassification bias to be independent of time and bias every year equally and therefore not affect the trend. Fourth, publication bias, in which negative studies are not published, will bias out results toward the null, for example, our estimate of the impact of adaptive design (50.9%) would be higher if unpublished studies stopped for futility and not published were included.
In summary, we developed a novel full-text systematic review search strategy. Forty-four studies (77.2%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review search methodology. This could introduce a reporting bias in which adaptive design methods are reported in the main article but not in the abstract. Our novel strategy combined classical systematic review, machine learning classifiers, and a novel full-text systematic review. This new method has broad applications in medical evidence synthesis and evidence synthesis in general.
Adaptive design methods improve the efficiency of RCTs in dialysis but their relative use in dialysis is decreasing over time. Greater knowledge of adaptive design examples in dialysis will further improve uptake in dialysis RCTs.

SUPPLEMENTARY MATERIAL
Supplementary File (PDF) Figure S1: Risk of Bias Assessment of Dialysis Randomized Clinical Trials With Adaptive Designs.