Program Evaluation

 

Key Points:

  • Program evaluation is a systematic method
  • There is no reason to collect and analyze information if the results are not going to be used to make changes.
    • Be mindful of creating and sustaining an effective feedback loop
  • Program evaluation can help to determine the value, effectiveness and impact of educational programs and identify program areas for improvement, replacement or further development
  • Formative evaluation, also known as process evaluation provides information on how well an individual or program is progressing toward a goal. This information is then "fed back" to improve/enhance progress and can allow for just in time corrections
  • Summative evaluation is used to determine if pre-defined outcomes have been met. This information is often used to make a "final decision".

 

Objectives:

  • Include evaluation processes in the initial design of any program

 

Recommended Resources: 

Brief Review

Watch

What is Program Evaluation? A Brief Introduction. Arbor Educational & Clinical Consulting Inc. 5 minutes.
http://youtu.be/xZUg9rJ0pQ4

Robert Fischer Discusses Program Evaluation. FC Cleveland. 7 minutes.
http://youtu.be/ccFEy8YMSgQ

Read

♥Cook DA. Twelve tips for evaluating educational programs. Med Teach. 2010;32(4):296-301. doi: 10.3109/01421590903480121. PubMed PMID: 20353325.
http://www.ncbi.nlm.nih.gov/pubmed/20353325

Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011;33(10):783-97. doi: 10.3109/0142159X.2011.611022. Review. PubMed PMID: 21942477.
http://www.ncbi.nlm.nih.gov/pubmed/21942477

Musick DW. A conceptual model for program evaluation in graduate medical education. Acad Med. 2006 Aug;81(8):759-65. Review. PubMed PMID: 16868435.
http://www.ncbi.nlm.nih.gov/pubmed/16868435

Simpson D, Lypson M. The Year is Over, Now What? The Annual Program Evaluation. J Grad Med Educ. 2011 Sep;3(3):435-7. doi: 10.4300/JGME-D-11-00150.1. PubMed PMID: 22942982; PubMed Central PMCID: PMC3179219.
http://www.ncbi.nlm.nih.gov/pubmed/22942982 

Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34(5):e288-99. doi: 10.3109/0142159X.2012.668637. Review. PubMed PMID: 22515309.
http://www.tandfonline.com/doi/abs/10.3109/0142159X.2012.668637

Vassar M, Wheeler DL, Davison M, Franklin J. Program Evaluation in Medical Education: An Overview of the Utilization-focused Approach. J Educ Eval Health Prof. 2010 Jun 15;7:1. doi: 10.3352/jeehp.2010.7.1. PubMed PMID: 20559515; PubMed Central PMCID: PMC2886237.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2886237/

Do

Sleight D. Evaluating Educational Programs: A Planning Tool. MedEdPORTAL Publications; 2007. This is an introductory level web-based tutorial that guides users through the six steps to developing an evaluation plan for a program or curriculum.
https://www.mededportal.org/publication/749

Enhancing Program Performance with Logic Models. UW Extension. This free online course provides a holistic approach to planning and evaluating education and outreach programs. It helps program practitioners use and apply logic models - a framework and way of thinking to help us improve our work and be accountable for results. You will learn what a logic model is and how to use one for planning, implementation, evaluation or communicating about your program.
http://www.uwex.edu/ces/lmcourse/#

 

In-Depth Review

Watch

Program Evaluation. UC Irvine OCW. 3 hours.
http://youtu.be/XD-FVzeQ6NM

Case Studies in Program Monitoring & Evaluation. Vanderbilt University. 1 hour and 14 minutes.
http://youtu.be/A5oFDGV2kCo

Read

Fink AG. Evaluation Fundamentals: Insights into Program Effectiveness, Quality, and Value. Third Edition. 2014. 304 pages. $52.74
http://www.amazon.com/Evaluation-Fundamentals-Insights-Program-Effectiveness/dp/1452282005/ref=sr_1_1?s=books&ie=UTF8&qid=1450123794&sr=1-1&keywords=Fundamentals+of+Program+Evaluation

Fitzpatrick JL. Program Evaluation: Alternative Approaches and Practical Guidelines, 2nd Edition. 558 pages. $44.92
http://www.amazon.com/Program-Evaluation-Alternative-Approaches-Guidelines/dp/0205579353/ref=mt_hardcover?_encoding=UTF8&me=

Barrett NF. Program Evaluation: A Step-by-Step Guide. 84 page Kindle book. $1.99

Do

♥Evaluating Educational Programs: A Planning Tool. Office of Medical Education Research and Development, Michigan State University. Web-based tutorial.
http://omerad.msu.edu/meded/progeval/index.html

Fundamentals of Program Evaluation. Johns Hopkins Bloomberg School of Public Health. Open Education Consortium.
http://ocw.jhsph.edu/index.cfm/go/viewCourse/course/FundamentalsProgramEvaluation/coursePage/index/

 

LCME Standards

ED-46: A medical school must collect and use a variety of outcome data, including national norms of accomplishment, to demonstrate the extent to which its educational program objectives are being met.

ED-47: In assessing program quality, schools must consider student evaluations of their courses and teachers, as well as a variety of other measures.