[PDF]What is effect size? - Rackcdn.com9d6445541cd52d10cbc6-1a34f99a934bffaa17579f57a98dc9a2.r62.cf1.rackcdn.com/...
17 downloads
155 Views
874KB Size
Calculating and Interpreting Effect Size for your CME activities Jason J. Olivieri, MPH Director, Outcome Services, Imedex, LLC* Ben Collins Manager, Outcome Services, Imedex, LLC*
*Imedex LLC is a CME-accredited MEC and outcome services provider
Target Audience • Those with little or no experience with effect size • Those who’ve been working with effect size and have questions about certain conditions
Outline • What can we do with effect size measures? • How do we calculate effect size? • Q & A (20 minutes)
What Can You Do with Effect Size? • Summarize the educational effectiveness of a CME activity • Conduct aggregate analysis of CME effectiveness by format, topic, etc • Determine benchmarks for CME effectiveness
EIS for live CME activities by type: June 2010 – Sept 2013
EIS for live CME activities by type: June 2010 – Sept 2013 large*
medium*
small*
*Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
EIS for live CME activities by type: June 2010 – Sept 2013 competence1
knowledge2
1. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15. .
EIS for live CME activities by type: June 2010 – Sept 2013
Expected magnitude of effect on knowledge outcome (.43 .65)
Using effect size allows CME providers to be more empirical with their CME • What’s a good CME effect size? • Why is my effect size lower than benchmarks? • How do my outcome methods / questions affect my effect size? • What affect does adding / subtracting certain CME elements have on my effect size? • Why is one format more / less effective than another?
How does it work?
CME activity
EOM plan
Data
Didactic presentation followed by case-based discussion
Pre- vs. post-activity assessment via ARS or paper survey
8 case vignette or clinical practice strategy questions
Example ARS question (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)
Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC
CME activity
Didactic presentation followed by case-based discussion
How do we summarize this data?
EOM plan
Data
Pre- vs. post-activity assessment via ARS or paper survey
8 case vignette questions or clinical practice strategies
And how do we then compare this result to results of other activities?
What is effect size? •Quantifies the magnitude of effect (maximum expected range: -3 to +3) •Difference in means (e.g., pre-test and post-test) divided by the square root of the pooled-group variances (Cohen’s d) •Enables the comparison of CME effects across activities on a common dimensionless scale •Calculated from comparison data (e.g., pre/post, post/control) linked directly to CME content −Knowledge questions −Case vignettes −Self-reported frequency of use of key clinical practice strategies
How is it calculated?
Calculating effect size • Can be done using only MS Excel® and free, online resources • Approach dependent upon variable type: – ordinal (e.g., clinical practice strategy) – categorical (e.g., case vignette)
Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC
Calculating effect size for an ordinal variable (e.g., clinical practice strategy) • Calculate average and standard deviation for each group (e.g., pre- and post-activity) – Pre-activity: mean (SD) = 2.5 (1.3) – Post-activity: mean (SD) = 3.1 (1.2) • Plug these values into an online calculator (http://www.uccs.edu/~lbecker/)
How is effect size interpreted? Ordinal variable example Clinical practice strategy (CPS) use rating (1= never, 5 = always) Pre-test
Post-test
Effect size
CPS #1
Standard deviation
CPS #1
Standard deviation
Cohen’s d
2.5
œ1.3
3.1
œ1.2
.48
*d = .2 (small effect), d = .5 (medium effect), d = .8 (large effect)
How big was the educational effect? How does the effect compare to other activities? Cohen (1988): .2 = small, .5 = medium, .8 = large Wolf (1986): .25 = educationally significant, .50 = clinically significant
Expressed in standard deviation units: The average score of a post-test respondent was .48 standard deviations above the average score of a pre-test respondent Effect sizes are proportional (.48 is twice as much effect as .24)
Example ARS question (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)
Calculating effect size for a categorical variable (e.g., case vignette) • Determine % correct for each group (e.g., pre- and postactivity) – Pre-activity: .51*65 = 33 correct / 32 incorrect – Post-activity: .75*65 = 49 correct / 26 incorrect • Plug these values into online calculator (www.vassarstats.net) – Click “frequency data” – Click “Chi-Square, Cramer’s V and Lambda”
Calculating effect size for a categorical variable, continued • Visit www.lyonsmorris.com/ma1/index.cfm • Select “Correlation coefficient (r) to Effect Size” • Enter Cramer’s V (.1474)
Determining overall effect size for a CME activity • Take average of all effect sizes (Cohen’s d) at each outcome level – Report overall effect size if you have at least four questions – Only combine effect sizes for the same question type • Keep a database of effect sizes to develop benchmarks
How effective was your live CME in 2013? 10 live CME activities
8 ARS questions per activity
80 slides
10 effect sizes
One summary effect size for live CME
Overall EIS (by format) June 2010 – Sept 2013 1Cohen
J 1988. Small effect = .2, Medium effect = .5, Large effect = .8
2Mansouri
& Lockyer 2007. Knowledge effect size = .6 4Casebeer 3Drexel
et al 2011. Competence effect size = .85
et al 2010. Knowledge effect size (eLearning) = .82
n=9 1. Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. 2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15. 3. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 4. Casebeer et al. BMC Med Educ 2010;10: 42.
Effect Size Summary •Summarizes effectiveness of individual CME activities and allows for aggregate comparisons •Easy to calculate •Uses:
• Evaluate CME Mission • Make improvement plans and track progress • Establish effectiveness benchmarks • Communicate effectiveness to sponsors & ACCME
Questions? Resource: www.assessCME.wordpress.com