There are a whole spectrum of feedback
options available to us with new tools being created all the time.
There is a danger that with the enthusiasm for gathering informative
data will result in an incoherent mass of data or, worse, an
alienated group of respondents who wilfully avoid giving feedback
with the tenacity generally reserved for avoiding pollsters and
’charity muggers' in public spaces. The key is to organise
delivery feedback systems is to be strategic, to work according to a
plan which balances the interests of everyone involved.
While
evaluation also refers to a component of the overall process, the
weighing and analysis of feedback data, we also refer to evaluation
in a holistic way in the planning and administration the system as a
whole, whether it be across a unit of study, a course or a
university. While some previous academic debates have drawn a
distinction between evaluation and research, this is arbitrary and
harms the credibility with which we should want to attach to the
evaluation process.
Evaluation,
as part of the big picture of course delivery is often unplanned and
is an area which all too frequently falls outside of anyone's direct
responsibility. Most universities have central feedback mechanisms,
gathering student data and administering surveys, but these
mechanisms are not directly planned around individual course needs
and questions. External audits and course accreditation share some
of the ground of evaluation but again lack the focus on the course's
individual context and needs.
Taking
charge of evaluation means building a system which is connected to
the institutional and discipline-based review mechanisms but which is
tailored to the individual course and is owned by the staff and
students in that course. Typically it will be the responsibility of
a course director or head of school to administer the process but the
design and implementation is most powerful if it is part of the
collective action and responsibility of the course community.
This
is a dynamic design process that requires us to be proactive in
looking for gaps in our knowledge base, designing feedback
innovations that fill those gaps and focusing on the future
direction. A well designed evaluation system activates the course
community and engages them, rather than ticking boxes in a
compliance-driven activity. Evaluation becomes a juncture at which
the course evolves in a changing knowledge ecology.
Some
members of the community, both students and staff may be reluctant to
participate and may have been burnt by previous experiences. Many
are frustrated by their environment and the slow rate of change. An
evaluation system must:
- Foster a culture of trust and respect
- Be evidence based and have mechanisms for testing assumptions
- All participants must be actively heard, even if their requests are not actioned
- Move beyond mere compliance, it must set the agenda for change
- NOT become an exercise in blaming or punishing, it must be oriented toward creating positive solutions for the future.
- Create a collaborative environment which is open and transparent
- Be oriented toward problem solving. Tap into the frustration where change is needed and channel that frustration into positive constructive activity.
- Be appropriately resourced and participation must be rewarded and acknowledged as a core activity.
Some
existing evaluation systems have grown out of external government
auditing and suffer some of the hallmarks of the process. They have
a ’performance review' orientation towards metrics and
micromanagement and individuals are understandably reluctant to fully
engage and share. This a hurdle which evaluation designers will have
to face at some point of the process to transcend the limitations of
a managerial and performative mindset. Part of the solution is being
careful about the language you use, constructing evaluation not as an
administrative chore but something in the realm of teaching and
learning research.
Evaluation
can be usefully connected to the learning research agenda. The
burden of received wisdom can be a hurdle to innovation and
evaluation systems must be conscious of the need to test assumptions,
whether they be traditional ways of conducting learning or myths
about the current generation of learners. In this way the evaluation
process is a
research process by which we establish and test our baseline
knowledge.
Choosing
the correct measures of
improvement is also vital.
Feedback measures such as surveys are often chosen because of the
ease with which you can collect and tabulate results, but if these
are not presented in a meaningful way they can derail the evaluation
process. Where staff promotion opportunities are tied to ’high
scores' on student surveys this can encourage teaching to the survey
rather than real reflection and innovation. We will return to
surveys in section 5 and look at some of the problems of
over-reliance on survey metrics.
Planning
the Evaluation Review Cycle
Gathering
and evaluating feedback works through regular careful increments and
like any other similar activity, such as a fitness program, careful
planning pays off dividends. This should involve developing a
regular review cycle and planning across multiple cycles so that the
work is evenly distributed and strategically managed so that the
various components complement each other rather than clash.
Most
courses face regular review cycles from the institution and from
external accreditation bodies. A good evaluation plan will intersect
with these cycles so that all the review processes share information
and do not create a set of competing priorities. If these are on,
say a three year cycle, it is worth designing your evaluation plan
over a longer period, perhaps five years. The furthest reaches of
this plan can be tentative, flexibility is very important in this
sort of plan, but it should nevertheless make sense of individual
tasks and activities over a broader canvas.
This
manual has a sample
evaluation plan in appendix
1 which demonstrates how a
designer can target important issues, effectively distribute a
spectrum of feedback mechanisms and improvement measures and keep
track of the innovation agenda. You are of course free to design
your own and we would be very interested in finding out more about
the different planning models that are used so please email the
author.
The
sample plan is divided into three sections: Core
Questions which are
relevant to all courses, Context
of the Course which
focussed on specific questions relevant to the discipline and
professional context and Impact
of Innovation which
evaluates the innovations designed in the previous cycle. The design
of this plan is oriented toward developing the critical questions
first, before deciding what feedback method is to be used and how
improvement will be measured. It asks evaluators to close the loop
and to consider timing carefully to avoid evaluation fatigue.
It
is important to cycle different feedback activities to attain a
spectrum of different feedback sources. While it may seem orderly to
deploy every feedback mechanism whenever a course or unit of study is
offered, it is far more effective to rotate and vary the methods so
that respondents do not grow weary of them and the larger picture can
emerge.
Remember
that the evaluation plan itself is also open to feedback and review.
We are constantly learning more about education, developing new
innovations for managing feedback and creating new technologies to
manage the process. For example the University of New England uses a
student word cloud called ’the Vibe' and an emoticon system as a
barometer of student mood. Alternatively, the UK Department for Work
and Pensions has developed ’Idea Street' an online environment
which gamifies the traditional workplace idea box and turns it into a
virtual stock market for innovation. These sorts of fresh new ideas
can reap benefits in other contexts too.
Good
information management is the backbone of any evaluation process,
making sure you are operating on current and credible data.
Effective curriculum mapping
is the foundation of this and maintaining and sharing a portfolio of
course data ensures that everyone has access to need to know
information.
It
is also important to consider the role of reflection in presenting a
balance to data-driven information and as a method of making sense
out of experiences. Section 12 examines reflective methods in more
depth but it is essential to allow time and space for reflection in
planning an evaluation cycle.
Methods:
- Course accreditation & Audit
- Curriculum Mapping
- Discussion Paper
- Mood Barometers
- Idea Street
Key
Points
- Evaluation is an overt strategy which combines collection of feedback evidence, reflection on that feedback and exercise of judgment in designing a response. Evaluation systems evolve over time and must envisage space for change.
- Effective evaluation draws from a spectrum of feedback methods and asks a variety of questions, cycled over time. You do not have to evaluate every aspect every time, make priorities and plan for a longer cycle.
- Evaluation is about building a holistic picture from evidential fragments.
- You need to plan how you will report and share the results of the evaluation process. Publishing in the field of learning research is a way of accessing an audience outside of your institution.
Course accreditation and audit
In a nutshell:
External accountability procedures require a cycle of auditing and review generally at an interval of several years. These processes are formal and reporting must comply with standards set by university review policies, government and funding body requirements or the parameters set by external accreditation bodies.
Example Questions:
Do the course outcomes and content comply with accreditation standards?
Is the course performing in a way that is economically sustainable?
Does course processes conform to policy standards?
Reporting:
Reports are prepared for consultation with faculty, students and stakeholders and then presented to external auditors or accreditation bodies. Generally these reports are also publicly available.
Pros:
Procedure has a high status and participation is required.
Process can be central to the identity of the course and the course community.
The urgency of the process might be used to leverage other evaluation methods and conducted at the same time under the same umbrella.
Cons:
High stakes may lead to a focus on surface compliance and a reluctance to fully explore issues that might reflect badly on the course.
Caution:
Participants may be numbed by the state of constant audit and disengage from the process. The urgency of the process might also distract from other evaluation work that may lead to more productive outcomes.
Curriculum mapping
In a nutshell:
All courses and units are documented, but over the years may evolve in ways that drift away from the original text. With large numbers of teachers and turn-over of staff it may not always be possible to ensure that these documents are kept up to date so be useful to undergo a process of curriculum mapping, to make sure everything is up to date and that new policies are properly implemented across the board.
Example Questions:
How well is the course structured as a coherent whole?
Is there horizontal articulation across the units in a year level and vertical articulation from semester to semester?
Are learning outcomes mapped to all units and consistent with course outcomes?
Is there a diversity of assessment options that are balanced across the curriculum?
Reporting:
A curriculum map is generally developed as a report but new technologies might mean that a course database could be a more effective model. Curriculum mapping is often annexed to other processes, such as the auditing or accreditation process.
Pros:
Knowledge is power and it can often be surprising how different a course is in application than to its appearance on the books.
Making sure policies are rolled out in an even and logical manner avoids hot spots later on where specific reforms have been missed.
Cons:
Curriculum mapping is time consuming
The focus on documentation may lead to further distance between the paperwork and the reality, especially if done under a compliance driven culture.
Caution:
Academics working 'at the coal face' may need to be carefully convinced of the importance of curriculum mapping. It can be all too easy to focus on what is happening within a single unit of study without seeing the impact on the course holistically.
Discussion paper
In a nutshell:
Public sector innovation is conducted under a formal policy making process that consists of discussion papers (sometimes called issues papers), public consultation and making of recommendations in final reports. This model may be used in the evaluation process either for general whole of course reviews or for investigation into specific issues or fields of innovation.
Example Questions:
What platform should be used for the provision of online resources?
How can diversity issues be reflected in the curriculum?
Are students being equipped for 21st century professional skills?
Reporting:
The policy process contains its own feedback loop. The discussion paper can be an effective way to alert stakeholders to issues in a way that opens up further conversation. The strength of a policy report is that while recommendations are not binding, they have a status which comes from the credibility of the process.
Pros:
A formal and inclusive process with well documented stages.
Discussion paper topics can focus attention on specific issues and draw in experts and researchers from various places.
Cons:
A time consuming process which requires careful management of consultation periods and plenty of pushing out content and trying to secure participation.
Caution:
Discussion papers should not implicitly promise more than will likely be delivered and can create false expectations for change.
Mood barometers
In a nutshell:
Social media has accustomed us to quick and immediate units of micro-feedback, a 'like' or a reblog or a tag. Universities have used this approach to gain quick, rough and ready indicators of student mood or feedback, through emoticons or word clouds.
Example Questions:
What is on your mind today? :) or }:( ?
Reporting:
Feedback loop is immediate and is automated, can readily be shared with respondents
Pros:
Process is informal but immediate, can be an alert for further investigation.
Respondents may find this an easy way to express feelings without much effort, particularly if access is provided via the LMS or home page.
Cons:
Data is very sketchy and is unlikely to be representative.
May be cathartic in the short term but students may not follow up and seek more substantial ways to have their voice heard.
Caution:
Pilot projects have been popular but we must be careful that we are not just creating a gimmick. Some more complexity in different moods and connection to specific issues might improve the usefulness.
Idea Street
In a nutshell:
An invention of the UK public sector, Idea Street supercharges the online feedback box by creating a virtual stock market for innovation. Stakeholders enter suggestions for improvement and through a gamified interface can comment on and strategically invest a virtual currency in different ideas. If innovations are implemented, the investors reap virtual currency rewards so they become strategic about supporting excellent and practical ideas.
Example Questions:
How could we improve our internal procedures to improve sustainability? (aimed at staff)
How could the timetabling system be improved? (aimed at students)
Reporting:
The online network manages most of the process, when successful ideas are chosen they are reported back to Idea Street (along with dividends) and projects then become part of broader organisational reporting systems as they become enacted.
Pros:
A clever way to crowdsource innovation and use gamification concepts to encourage participation.
Cons:
Requires investment in setting up the system and really needs the support and engagement of stakeholders if it is to work properly.
Caution:
Idea Street has been very successful and has saved the UK a good deal of money, but since it is very new there has been limited testing of the concept in other contexts.
No comments:
Post a Comment