Program Evaluation
While program evaluation may not be the first item on your list as you create your online learning program, it is important that it be considered during the initial development of the online learning program. Creating an evaluation plan as part of the initial planning process helps to clarify your program’s purpose and audience as well as its mission and goals. Another important reason for having an evaluation plan from the onset is that an evaluation plan will require data collection processes which are much easier to design and implement as the program is being created. Finally, when program leadership communicates the value and importance of program evaluation, a culture of continuous improvement is established.
Program evaluation utilizes terms that may not be familiar to people who are not evaluation experts. This glossary of common evaluation terms,15 while written specifically for evaluation of magnet school programs, serves as a useful resource for understanding evaluation terminology for all types of program evaluation. Two terms that are part of this evaluation language are formative evaluation and summative evaluation. The Research Methods Knowledge Base16 provides a clear explanation of the differences between these types of evaluations.
Program evaluations fall into two categories: internal (conducted by the program staff) or external (conducted by someone outside the organization).17
Internal Evaluation
An internal evaluation is conducted by the program’s leadership or staff. While it can be comprehensive in nature, more typically it is targeted to one or more specific areas within a program and often is part of a program’s continual improvement process. An internal evaluation is almost always a formative evaluation (see the above highlighted resource for an explanation of a formative evaluation).
One of the main advantages of an internal evaluation is flexibility. The timing of the evaluation can be whatever is convenient for the internal staff that will complete the evaluation. The scope of the evaluation can easily be set as narrow or broad as desired and adjusted as needs change. Finally there is no monetary cost for the evaluation work. However it is important to realize that an internal evaluation has an opportunity cost, as the staff working on the evaluation have less time to work on other projects.
One of the main disadvantages of an internal evaluation is that the program staff may not have expertise in evaluation design and methods. One solution to this problem is to utilize the second type of program evaluation, an external evaluation, to design evaluation processes and instruments that can then be utilized in future internal evaluation work.
External Evaluation
As the name implies, an external evaluation is conducted by one or more people that are not part of online learning program. Typically the people conducting an external evaluation have specific expertise in program evaluation in addition to expertise in online learning. External evaluations may be either formative or summative in nature (see the above highlighted resource for an explanation of formative and summative evaluation).
Quality online learning programs will have a formative external evaluation completed on a regular schedule, typically at least every 3-5 years.
There are several important advantages of external evaluations. First of all, they are viewed with greater validity since the person conducting the evaluation is not part of the online learning program. Secondly, an external person brings a new perspective and can identify issues that internal program staff miss because they are too close to the situation.
The main disadvantages of an external evaluation are the cost of the evaluation and the time required to make sure the evaluator has sufficient understanding of the online learning program to conduct an effective evaluation.
These excerpts from two publications provide additional insight and information regarding online learning program evaluation. The first excerpt is from Promising Practices in Online Learning: Management and Operations of Online Programs18 and the second excerpt is from the 2008 edition of Keeping Pace with K-12 Online Learning19.
References for footnotes can be found on the resources page or click on the footnote number for a direct link.