Developing a Course Evaluation Plan

As you design and deliver your course, you will be using new techniques and facilitating learning activities that may be different from those students are used to. These two factors may cause you to have lower end-of-semester course evaluations temporarily. A part of blended course design should include a plan to collect data on the positive outcomes that are being facilitated through your new design that can be used alongside the traditional course evaluation process. The following are some suggestions to consider in identifying focus areas, choosing types of evidence, and developing your evaluation plan.

• Talk with your department chair about your plans to blend your course. Ask what evidence they would want to see about the performance of your course.
• If you are pre-tenured, engage in conversations with your chair and tenure committee about what evidence they would want to see in addition to formal end-of-semester evaluation results.
• Review your course outcomes, unit objectives, and student assessments. Think about the kinds of data you can collect to show process or improvement in achieving these designated levels.
• Think about the kinds of qualitative data you can collect to measure the softer outcomes of your course such as student engagement, quality of discourse, preparation for class, and/or class attendance.

Evaluation plan example:  MUS STU 352

The following example is from MUS STU 352 – Visitor Experience and Design in Museums, taken from the pre-class case studies. Before the instructor teaches her course, she identifies her goals as well as those of other stakeholders. Based on conversations with her chair and others, she identifies the following areas to focus on as measurements of success for her course design and builds the following plan to address these issues:

• Student professionalism in dealing with museum staff;
• Quality of semester projects (using a rubric she develops);
• Student compliance with learning pathway;
• Quality and level of student contributions in class; and
• Level of preparation for future Museum Studies courses.

Evaluation Plan

OUTCOMES	UNIT	UNIT OBJECTIVES
Program Outcomes in bold indicate supported program outcomes PO1: Develop a high degree of professionalism and responsibility, skills in problem-solving, critical thinking and evaluation, writing, oral presentation, decision-making, and teamwork. PO2: Apply professional museum standards and ethics and proper care and interpretation of collections. PO3: Explore the role of museums in our complex and ever-changing society. PO4: Develop skills around administration and organizational problems in modern museums, finances, and budgets, legal-administrative problems, relations with support groups and volunteers, record keeping, and management of museum projects. PO5: Develop skills around the collection, organization, storage, care, and scientific use of museum collections. PO6: Develop and design museum visitor experiences. Course Outcomes CO1: Analyze the elements of the visitors’ meaning-making: framing, resonating, channeling, and broadening. CO2: Design visitor experiences that build connections between content and viewers in ways that foster specific kinds of encounters. CO3: Design exhibit narratives for different levels of museum visitors. CO4: Develop an engaging user experience to support the narrative of the exhibit. CO5: Value the importance and impact of a well-designed museum experience. CO6: Use museum spaces to encourage reflection and dialogue. CO7: Design museum exhibits that develop and demonstrate cultural understanding. CO8: Develop competence, depth, and expertise in the design of museum visitor experiences. CO9: Develop a high degree of professionalism and responsibility, skills in problem-solving, critical thinking and evaluation, writing, oral presentation, decision-making, and teamwork. CO10: Develop research and interviewing skills to gain feedback on exhibits and further the field of Museum Studies.	Envisaging the Discipline of Museum Design	UO1: Situate exhibition design within the field of Museum Studies. UO2: Recognize the historic shifts in the discursive formation of exhibits. UO3: Recognize the historic changes in the purposes of exhibition design.
	User Experiences in Museums	UO1: Analyze the multiple ways in which experience has been conceptualized in museum studies. UO2: Hypothesize on future trends in visitor experience design. UO3: Develop strategies to respond to new trends in practical terms.
	Deconstructing Visitor Experiences	UO1: Build a conceptual decision-making framework for designing visitor experiences. UO2: Identify the ways an exhibit conceptualizes the subject and how they impact what visitors see through an exhibition. UO3: Explore the ways an exhibit facilitates the investigation of the subject. UO4: Develop a process for gathering feedback from visitors on their experiences with an exhibition.
	Framing	UO1: Consider how the visitors' conceptualization of ‘museum’ and ‘exhibit’ guide their participation in and evaluation of their museum experience UO2: Situate the exhibition in ways that help visitors classify what they are experiencing. UO3: Organize the exhibition into an appropriate framework to guide comprehension.
	Resonating	UO1: Analyze the way visitors meet each item within the collection. UO2: Organize items within the collection that connect with one another to build understanding. UO3: Analyze how the qualities of the visitor and the qualities of the museum display mesh in a transactional exchange.
	Channeling	UO1: Identify the intended wayfinding of the user within the exhibit. UO2: Design for conceptual, attentional, and perceptual elements of wayfinding. UO3: Analyze the collection to identify the major elements of the collection. UO4: Design wayfinding pathways (formal and informal) to highlight the collection.
	Broadening	UO1: Gauge the various content-related meanings the visitor may take away from the exhibit. UO2: Identify the meanings to be supported through the exhibit design. UO3: Design visitor experiences that help visitors find themselves in relation to the interpretive content of the exhibit.
	Design for Exhibition Ecologies	UO1: Describe how the four processes (framing, resonating, channeling, and broadening) form a mutually influencing system. UO2: Appreciate how design choices catalyze the visitor-in-exhibition experience.

Evidence to be collected

The instructor identifies the following kinds of qualitative and quantitative evidence to be collected. This data will help her make evidence-based changes to improve the effectiveness of her course design.

Qualitative evidence 

Review of museum design projects

In addition to the grade students are given, projects are analyzed for their quality based on course outcomes and project rubrics. Improvements in these qualities can be qualitative data to determine whether course design has yielded the desired impact.

Instructor reflection

After each class session, the quality of student engagement and contributions is recorded. After a full semester, it can be hard to recreate these impressions in meaningful ways. This approach is a quick and easy way to flag situations in which expectations were or were not met. She will look at some specific items in her reflection.

• PRE-CLASS SELF-ASSESSMENTS – Level of participation and quality of reflection.
• IN-CLASS GROUP DISCUSSIONS – Level of preparation to contribute to the discussion, level of contribution, and kinds of gaps in learning displayed through student comments.
• WORKLOAD – Amount of time to facilitate activity and provide feedback.
• FEEDBACK – How quickly did it take to give feedback to students?

Stakeholder meeting

The instructor plans on meeting with the instructors who teach classes students take later in the course sequence. She will ask them whether they see any changes in students' preparation for their course.

Quantitative evidence

Mid-semester survey

After the midterm, students will complete a survey to measure specific course questions.

• How well were unit objectives facilitated?
• How well did pre-class activities prepare you for in-class activities?
• How useful was the feedback you received so far?

Formal end-of-semester survey

The Museum Studies program has redesigned its end-of-semester surveys using AEFIS to measure specific course and departmental questions.

• How well did the instructor facilitate learning?
• How clearly were assignments presented and graded?
• Was the amount of feedback provided by the instructor sufficient?

Supplemental end-of-semester survey

• How well did pre-class activities prepare you for in-class activities?
• What problems (if any) did you experience with the semester project?
• What suggestions do you have for future improvements in the course?

Elements of a good evaluation plan

DoIT Academic Technology Evaluation Services has developed some principles of evaluation that can be helpful as you plan to evaluate your course. Good evaluation plans:

• Are not an afterthought;
• Have a defined and meaningful purpose;
• Include asking the right people for feedback;
• Are iterative and support your measurements of success;
• Include multiple types and sources of feedback;
• Are flexible and enable responsiveness;
• Take advantage of existing evaluation tools; and
• Include a process for acting on feedback.