Program Evaluation
Key Points:
- Program evaluation is a systematic method
- There is no reason to collect and analyze information if the results are not going to be used to make changes.
- Be mindful of creating and sustaining an effective feedback loop
- Program evaluation can help to determine the value, effectiveness and impact of educational programs and identify program areas for improvement, replacement or further development
- Formative evaluation, also known as process evaluation provides information on how well an individual or program is progressing toward a goal. This information is then "fed back" to improve/enhance progress and can allow for just in time corrections
- Summative evaluation is used to determine if pre-defined outcomes have been met. This information is often used to make a "final decision".
Objectives:
- Include evaluation processes in the initial design of any program
Recommended Resources:
Brief Review
Watch
What is Program Evaluation? A Brief Introduction. Arbor Educational & Clinical Consulting Inc. 5 minutes.
http://youtu.be/xZUg9rJ0pQ4
Robert Fischer Discusses Program Evaluation. FC Cleveland. 7 minutes.
http://youtu.be/ccFEy8YMSgQ
Read
♥Cook DA. Twelve tips for evaluating educational programs. Med Teach. 2010;32(4):296-301. doi: 10.3109/01421590903480121. PubMed PMID: 20353325.
http://www.ncbi.nlm.nih.gov/pubmed/20353325
Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783-97. doi: 10.3109/0142159X.2011.611022. Review. PubMed PMID: 21942477.
http://www.ncbi.nlm.nih.gov/pubmed/21942477
Musick DW. A conceptual model for program evaluation in graduate medical education. Acad Med. 2006 Aug;81(8):759-65. Review. PubMed PMID: 16868435.
http://www.ncbi.nlm.nih.gov/pubmed/16868435
Simpson D, Lypson M. The Year is Over, Now What? The Annual Program Evaluation. J Grad Med Educ. 2011 Sep;3(3):435-7. doi: 10.4300/JGME-D-11-00150.1. PubMed PMID: 22942982; PubMed Central PMCID: PMC3179219.
http://www.ncbi.nlm.nih.gov/pubmed/22942982
Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34(5):e288-99. doi: 10.3109/0142159X.2012.668637. Review. PubMed PMID: 22515309.
http://www.tandfonline.com/doi/abs/10.3109/0142159X.2012.668637
Vassar M, Wheeler DL, Davison M, Franklin J. Program Evaluation in Medical Education: An Overview of the Utilization-focused Approach. J Educ Eval Health Prof. 2010 Jun 15;7:1. doi: 10.3352/jeehp.2010.7.1. PubMed PMID: 20559515; PubMed Central PMCID: PMC2886237.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2886237/
Do
Sleight D. Evaluating Educational Programs: A Planning Tool. MedEdPORTAL Publications; 2007. This is an introductory level web-based tutorial that guides users through the six steps to developing an evaluation plan for a program or curriculum.
https://www.mededportal.org/publication/749
Enhancing Program Performance with Logic Models. UW Extension. This free online course provides a holistic approach to planning and evaluating education and outreach programs. It helps program practitioners use and apply logic models - a framework and way of thinking to help us improve our work and be accountable for results. You will learn what a logic model is and how to use one for planning, implementation, evaluation or communicating about your program.
http://www.uwex.edu/ces/lmcourse/#
In-Depth Review
Watch
Program Evaluation. UC Irvine OCW. 3 hours.
http://youtu.be/XD-FVzeQ6NM
Case Studies in Program Monitoring & Evaluation. Vanderbilt University. 1 hour and 14 minutes.
http://youtu.be/A5oFDGV2kCo
Read
Fink AG. Evaluation Fundamentals: Insights into Program Effectiveness, Quality, and Value. Third Edition. 2014. 304 pages. $52.74
http://www.amazon.com/Evaluation-Fundamentals-Insights-Program-Effectiveness/dp/1452282005/ref=sr_1_1?s=books&ie=UTF8&qid=1450123794&sr=1-1&keywords=Fundamentals+of+Program+Evaluation
Fitzpatrick JL. Program Evaluation: Alternative Approaches and Practical Guidelines, 2nd Edition. 558 pages. $44.92
http://www.amazon.com/Program-Evaluation-Alternative-Approaches-Guidelines/dp/0205579353/ref=mt_hardcover?_encoding=UTF8&me=
Barrett NF. Program Evaluation: A Step-by-Step Guide. 84 page Kindle book. $1.99
Do
♥Evaluating Educational Programs: A Planning Tool. Office of Medical Education Research and Development, Michigan State University. Web-based tutorial.
http://omerad.msu.edu/meded/progeval/index.html
Fundamentals of Program Evaluation. Johns Hopkins Bloomberg School of Public Health. Open Education Consortium.
http://ocw.jhsph.edu/index.cfm/go/viewCourse/course/FundamentalsProgramEvaluation/coursePage/index/
LCME Standards
ED-46: A medical school must collect and use a variety of outcome data, including national norms of accomplishment, to demonstrate the extent to which its educational program objectives are being met.
ED-47: In assessing program quality, schools must consider student evaluations of their courses and teachers, as well as a variety of other measures.