TLDR;
This webinar introduces the concept of randomized control trials (RCTs), providing both theoretical background and practical examples. It emphasizes the iterative nature of evaluation and situates RCTs as one method in the broader evaluation landscape. Key findings from case studies illustrate both effective and ineffective interventions, showcasing the power of RCTs in establishing causal relationships.
- RCTs are a tool for causal inference in evaluations.
- The session included historical context, and practical examples like summer school projects.
Introduction to Randomized Control Trials [0:00]
Helen Lawson introduces the webinar, mentioning that Rayne Sherlock will discuss the theory and background of RCTs for 30-35 minutes. The approach follows an evaluation cycle consisting of four phases: diagnosing, planning, measuring, and reflecting. RCTs are introduced as a methodology relevant during the planning phase of evaluations. Rayne notes the importance of situating RCTs within the evidence landscape, specifically referring to the Office for Students (OfS) standards of evidence.
The History and Methodology of RCTs [4:40]
Rayne discusses the historical context of RCTs, citing early experiments from the 18th century, such as James Lind's trials on scurvy. Significant developments include the introduction of placebo-controlled trials and the establishment of double-blinded studies in 1943. The chapter emphasizes the use of RCTs in various fields, including education, healthcare, and social sciences, where they have proven beneficial in drawing causal relationships.
Understanding Evidence and RCTs [11:34]
The discussion centers on the importance of causal evidence within the post-secondary education sector. Rayne explains the types of evidence available, with RCTs being particularly useful for establishing causality. The significance of combining RCTs with other methodologies is highlighted, including qualitative research and implementation evaluations that can provide context and depth.
Analysis of RCT Outcomes [18:41]
Rayne outlines the fundamental concept of RCTs, using population samples randomized into treatment and control groups. The discussion explains how RCTs allow researchers to observe differences in outcomes and determine causality. The significance of statistical power in RCTs is also noted, emphasizing that adequate sample size and proper statistical considerations are critical in avoiding errors in findings.
Practical Examples of RCTs [24:19]
Rayne shares two notable examples of RCTs applying in educational settings. The first involves a U.S. study targeting high-achieving, low-income students and demonstrating improvements in college application submissions following personalized outreach. The second example details a tutoring program in India that led to improved educational outcomes, showcasing practical implications of effectively designed RCTs.
Limitations of RCTs [29:41]
Rayne covers the advantages and limitations of RCTs, including ethical considerations like equipoise and the implications of failing to give interventions to control groups. The Hawthorne effect is also discussed, along with challenges in randomization and issues relating to internal and external validity. The necessity for multiple methodologies to understand the effectiveness of programs is reiterated.
Introduction to Summer School Projects [34:14]
Sarah Chapel introduces her segment on practical applications of RCTs through a summer school project. The presentation details the design of a summer program and the evaluation framework being utilized, linking it to the previously discussed RCT methodology. She emphasizes the goal of assessing true causal outcomes concerning higher education enrollment.
Study Design and Outcomes [40:53]
The summer school project framework is explained, detailing selection criteria and the random allocation of places. Data collection methods include surveys measuring students' attitudes towards higher education, and an intention to apply after participating in the summer program. The implementation process involves monitoring and collecting behavioral data to assess the intervention's effectiveness.
Ethical Considerations in RCTs [48:21]
Sarah addresses ethical considerations regarding the conduct of RCTs. The discussion includes the importance of equipoise, informed consent, and the handling of vulnerable populations, ensuring students are aware of their participation in the study. Random allocation is presented as a fair method of selecting participants for oversubscribed summer schools.
Emerging Findings and Future Directions [55:00]
Sarah presents preliminary findings from early analyses of survey outcomes from the summer school project, highlighting that while treatment and control groups exhibit similar attitudes towards applying to higher education, there are indications of higher confidence and perceived barriers among those who attended the summer school. Future reports will encompass further data, including behavioral outcomes related to enrollment in higher education.
