One
of the reasons why anonymous student surveys are so widely used in
learning evaluation is the appearance of objectivity – the
anonymous process, the quantification of data and the presentation of
neat metrics all has the patina of a credible scientific process.
Expressing the perspective of teaching staff on the other creates the
anxiety the much qualitative and experiential data faces, that it is
subjective, unscientific and difficult to analyse.
In
other areas of research this rivalry between quantitative and
qualitative data has reached equilibrium with respect for the
strengths of both forms. Joseph Raelin's overview of the history of
management education describes the struggle that practice-based
experiential learning has had to achieve credibility in a discipline
dominated by metrics.1
However in the field of course evaluation we all too often revert to
old prejudices about hard data versus experiential knowledge.
Of
course student survey data is profoundly subjective, although it is
expressed through Likert scales and graphs, and staff experiential
data is a crucial part of the feedback spectrum. Helpfully, we can
draw on decades of advancement in reflective learning innovation for
tools and approaches that help us gather and use experiential data in
the feedback and evaluation process. Using these methods the
experiences of teaching staff can be accessed, recorded, compared
with other feedback data and, where necessary, challenged.
This
is another arena in which the spectrum of feedback channels allows
for different points of view and ways of knowledge to be included in
the evaluation process. Beyond that, reflection is a key
professional ethical capability as demonstrated in the work of Donald
Schön, one that can be cultivated as part of the evaluation process.
Second order reflection,
or reflecting on reflection can be even more helpful in understanding
the ways in which we respond to feedback and building a more complex
picture of experiential knowledge over time.
There
are strong connections to be made between reflective evaluation
processes and professional development in building the capacity of
teaching staff to be stronger reflective practitioners and to use
this process to self and peer assess. These capacities ought to be
central to the promotions process and evidenced in teaching
portfolios and potentially in staff badging systems.
Reflection
is generally considered to be an individual activity but it is very
powerful as shared collective action. With both individual and
collective reflection it is extremely helpful to create a structure
to guide reflection, to focus activity and to keep the methodology
rigorous. Whether reflection occurs in a personal blog or through
a staff retreat, having a clear agenda keeps respondents on point and
builds confidence in the process.
Reflection
works well if you can make a habit of it and can be embedded within
the regular seasonal shifts of university work. Scheduling a
reflective session at the end of a semester can help to give closure
to work cycle and renew focus for the start of the next. Activities
such as examiners' boards are more concerned with administrative
process and may not be the best sites for open reflection, but
parallel reflective activities (which could be online) would provide
a forum for many of the concerns that emerge at these meetings. At a
course level, regular review cycles should also include spaces for
reflection and sharing of experience.
As
with all other forms of open feedback and evaluation, effective
reflection depends on the creation of a safe and respectful
environment in which it is to occur. It is a fundamental aspect of
leadership
to create and maintain this reflective culture and to avoid
self-censorship out of fear of ridicule or prejudice. Individuals
need to be able to reflect on their own practice as well as the
collective practice of their organisation if we are to reap the
benefits of self-evaluation.
Strong
evaluation systems are built on a diversity of feedback options, not
just on metrics and surveys. Teachers have always informally used
experiential feedback, sometimes in ways that are counterproductive
and which undermine other data such as the knee-jerk reflexive
responses we can have to negative survey data.2
Having an open reflective method allows us to process these
experiences and identify areas that require further investigation and
validation. It takes notice of feedback that might be missed through
more metrics-focussed feedback but also provides a way of identifying
and challenging preconceptions that might otherwise go unspoken.
Methods
- Chief examiners/school board
- Teaching portfolio as an evaluative tool
- Brainstorm/gamestorm
- Collective reflection and the learning B\log
- Retreat
- Media/publication folio
- Narratives/digital storytelling
Key
Points
- The increasing status of reflective learning methodology provides additional opportunities to gather unit and course feedback from teaching staff and to provide structure for self-evaluation.
- Reflection need not be an individual activity, collective and shared reflection is powerful.
- Structure helps keep reflection on track and builds confidence in the outcomes.
- Reflection can only occur in an environment that is safe and respectful. This is the responsibility of leaders and all other participants.
1Raelin
JA (2007)’Toward an Epistemology of Practice', Academy
of Management Learning and Education, 6(4),
495-519
2See
Arthur's typology of staff responses to student surveys. Arthur L
(2009)’From performativity to professionalism: lecturer's responses to student feedback',
Teaching in Higher Education, 14:4, 441-454
Chief examiners' meeting and School Board
In a nutshell:
Traditionally, the chief examiner's meeting is one of the most significant points at which colleagues gather together and discuss student results and the implications this might have for course development. This committee usually reports to the school or faculty board which provides another site where discussion can take place, particularly over hot spots in the results. This is one of the primary governance oversight mechanisms.
Example Questions:
Why are there large failure rates in this unit of study?
Where are the bottlenecks limiting students' ability to graduate?
Reporting:
Through formal oversight channels, although the evaluative component of proceedings is often off the record.
Pros:
A formal part of university governance that all must participate in.
A gathering of all key unit coordinators and course directors so that they can reach a consensus
Cons:
Process can be focussed on formalities rather than substance, there may be little time available to explore issues in much depth. However this might be an opportunity to direct conversations to other channels.
May be too late to be effective, everyone is keen to finalise grades at the end of semester and this might not be the best environment to ask open ended questions.
Caution:
Regular problems often come to light at chief examiner's meetings and it is one crucial point at which to build the culture of trust and respect that evaluation depends on. If participants fear being judged and blamed they will not be encouraged to make full and open disclosure.
Brainstorm/Gamestorm Problem Solving Forum
In a nutshell:
Frequently we share common problems across a course, even across an institution and it can be useful to create specific forums to address these issues. Various problem solving strategies might be used and the idea of 'gamestorming' uses gamified strategies to make the process more engaging (see gamestorming.com).
Example Questions:
What can we do about mid semester attrition?
How do we effectively assess on-line units?
Reporting:
The process should set is own outcomes and reporting process. As usual, closing the loop keeps participants engaged and feeling respected.
Pros:
Draws on a wide range of experience and participants have ownership of the process.
Cons:
Requires resources to organise, run and report on outcomes
May be difficult to arrange at a time when full participation is possible. Nevertheless, there may be ways of carrying on the conversation in an online forum.
Caution:
Because many of these problems are perennial, there can be a sense of cynicism around the process, or worse a closed-minded acceptance of old ineffectual solutions. Running one of these forums requires careful consideration of the culture of the group and finding ways to engage with the future facing nature of the process.
Collective Reflection and the Learning Blog
In a nutshell:
Reflection is a powerful method of understanding experiences and setting the agenda for change. Much of the literature focuses on individual reflection but collective reflection can also be an incubator for action. The blog has become a key tool for reflective practice and the ability to aggregate RSS feeds means that a collective staff blog can be drawn together from the individual strands, even if they are on different blogging services.
Example Questions:
How are we going so far this year?
Is everyone sharing this particular experience, or is it confined to my students?
How is the learning landscape changing?
Reporting:
Blogs fill a neat informal niche and can be a way of testing out ideas before deciding on a research or evaluation process.
Pros:
Informal, easy to publish and share
Allows the author's voice to be expressed more clearly than more formal publishing channels.
Interactive and engages the audience in further discussion.
Cons:
May not be taken seriously by some teachers
Some may fear that others will steal the best ideas and publish first in a journal, however the date management of blogs make it very clear who placed and idea in the public domain first.
Caution:
Blogging can seem very one-sided if others do not comment on your work. An evaluation manager and team leaders need to be proactive in posting and responding to the posts of others in order to build dialogues and invite others to also participate.
Retreat
In a nutshell:
Another established tool in the evaluation repertoire, the retreat takes the team away from the distractions of the normal work environment to focus on course issues in an intensive way.
Example Questions:
What is working, what isn't working?
What new policies do we need to implement and how are we going to go about it?
What is our collective vision for the future of this course?
Reporting:
Retreats ought to have a reporting process and can distribute responsibility among the participants for the preparation of different elements. In practice this is not often done well which can make retreats seem a little like Groundhog Day.
Pros:
The intensive focus builds concentration and brings everyone together.
The isolation from everyday work life can build camaraderie and a sense of shared purpose.
Cons:
Retreats are very expensive
It is increasingly difficult to secure everyone's attendance.
Caution:
Today we need to have very strict rules about answering emails and using mobile devices at retreats as many staff will claim to have urgent matters that require part of their attention. There needs to be clear agreement from the start to take the retreat process seriously and to commit to focus on the business at hand, no exceptions.
Good conflict management skills are necessary and sometimes an external facilitator will be appropriate.
Teaching Portfolio as an Evaluative Tool
In a nutshell:
Teaching portfolios are an excellent self-assessment and reflection tool for individual professional development but they might also create dividends for the evaluation process. Where teachers identify problems, create a change agenda and reflect on implementation, this can connect to course/unit evaluation processes in a number of ways. Teachers may get in the habit of sharing these items when they think they are relevant for the course team, and a course portfolio might usefully be connected to teacher portfolios. The use of ePortfolios, in particular the ability to share some items publicly and not others, creates new ways of assembling this sort of feedback.
Example Questions:
How are the key problems in the course being addressed?
What innovations are being developed?
Reporting:
The process is initiated by teachers making the choice to share folio items with the team.
Pros
Recognises the important role of teachers in developing strategies handling everyday problems.
Draws on the strength of local networks and helps to develop course culture
Further enhances the value of teaching portfolios
Cons:
Will not be of much use of teaching portfolios are not used by most of the course teams
May be some anxiety around sharing if the evaluation environment is not open and supportive.
Caution:
There may be some need for setting the right tone, so that material shared is not interpreted as an attack on colleagues or a way of applying blame for underlying problems. Evaluation administrator may need to take a leadership role in moderating and helping some to reframe their reflections.
Media & Publication Folio
In a nutshell:
Throughout the year there are times when items concerning the course, staff achievements and articles of relevance to the discipline/profession will appear in the news media, in journals and in online publications. We tend to circulate these informally, perhaps via email, but we do not often keep a central record. Collecting this evidence via a portfolio is a way to gain an overall picture of the way the course is perceived and provides a historical record for the university.
Example Questions:
How do we appear to the community?
Where are our areas of expertise as a student may see them? Where are our teachers publishing and what does this mean for the course that we deliver?
What is happening in the discipline/profession that we should respond to?
Reporting:
The folio makes a useful record that should be easily accessible.
Pros:
A useful way of paying attention to the context in which the course operates
Can provide reflective opportunities that do not emerge from the normal evaluation processes.
Can provide an insight into the professional/employment context that students are destined for.
Cons:
May only deal with issues in a surface way
Caution:
This sort of activity is often reserved for public relations departments and it is important to distinguish this from the evaluative process which needs to balance the good and the bad. While an 'in the news' link on a department website may include only the most positive material a media folio will have a more limited audience and will not necessarily be directed at students or outsiders.
Narratives/Digital Storytelling
In a nutshell:
Personal narratives are a way of making sense of experiences that is powerful and taps into issues that are difficult to address in more conventional data. It is easy to presume that your own experience of education is somehow universal and the sharing of narratives can give a broader perspective on different points of view. Digital storytelling is an established method of creating personal narratives and it is not often used by educators but has the potential for creating interesting conversations about what it means to be a learner and a teacher. Storytelling is the natural way in which teachers share experiences (few of us are natural report writers) and much of the information gathered for this manual came from listening to stories.
Example Questions:
What things influenced me to become a teacher?
What is the worst learning experience I had, and what did I learn from it?
Reporting:
Digital narratives can be shared via services such as youtube and connected into blogs and other publications.
Pros:
A very personal focus on issues that are difficult to express through other avenues.
Provides a venue for voices which may not otherwise be heard
Can serve a historical function in creating a record of how experiences are changing over time, particularly as older staff near retirement
Cons:
There is a learning curve for training how to create these narratives, but this is fortunately not too steep and digital storytelling in particular has been used throughout the community with people from a range of backgrounds and familiarity with technology.
May not produce direct action as a result, but it will cultivate understanding
Caution:
Some may see this process as self-indulgent and fail to see the value so expectations around the project would have to be carefully managed.
Chief examiners' meeting and School Board
In a nutshell:
Traditionally, the chief examiner's meeting is one of the most significant points at which colleagues gather together and discuss student results and the implications this might have for course development. This committee usually reports to the school or faculty board which provides another site where discussion can take place, particularly over hot spots in the results. This is one of the primary governance oversight mechanisms.
Example Questions:
Why are there large failure rates in this unit of study?
Where are the bottlenecks limiting students' ability to graduate?
Reporting:
Through formal oversight channels, although the evaluative component of proceedings is often off the record.
Pros:
A formal part of university governance that all must participate in.
A gathering of all key unit coordinators and course directors so that they can reach a consensus
Cons:
Process can be focussed on formalities rather than substance, there may be little time available to explore issues in much depth. However this might be an opportunity to direct conversations to other channels.
May be too late to be effective, everyone is keen to finalise grades at the end of semester and this might not be the best environment to ask open ended questions.
Caution:
Regular problems often come to light at chief examiner's meetings and it is one crucial point at which to build the culture of trust and respect that evaluation depends on. If participants fear being judged and blamed they will not be encouraged to make full and open disclosure.
Brainstorm/Gamestorm Problem Solving Forum
In a nutshell:
Frequently we share common problems across a course, even across an institution and it can be useful to create specific forums to address these issues. Various problem solving strategies might be used and the idea of 'gamestorming' uses gamified strategies to make the process more engaging (see gamestorming.com).
Example Questions:
What can we do about mid semester attrition?
How do we effectively assess on-line units?
Reporting:
The process should set is own outcomes and reporting process. As usual, closing the loop keeps participants engaged and feeling respected.
Pros:
Draws on a wide range of experience and participants have ownership of the process.
Cons:
Requires resources to organise, run and report on outcomes
May be difficult to arrange at a time when full participation is possible. Nevertheless, there may be ways of carrying on the conversation in an online forum.
Caution:
Because many of these problems are perennial, there can be a sense of cynicism around the process, or worse a closed-minded acceptance of old ineffectual solutions. Running one of these forums requires careful consideration of the culture of the group and finding ways to engage with the future facing nature of the process.
Collective Reflection and the Learning Blog
In a nutshell:
Reflection is a powerful method of understanding experiences and setting the agenda for change. Much of the literature focuses on individual reflection but collective reflection can also be an incubator for action. The blog has become a key tool for reflective practice and the ability to aggregate RSS feeds means that a collective staff blog can be drawn together from the individual strands, even if they are on different blogging services.
Example Questions:
How are we going so far this year?
Is everyone sharing this particular experience, or is it confined to my students?
How is the learning landscape changing?
Reporting:
Blogs fill a neat informal niche and can be a way of testing out ideas before deciding on a research or evaluation process.
Pros:
Informal, easy to publish and share
Allows the author's voice to be expressed more clearly than more formal publishing channels.
Interactive and engages the audience in further discussion.
Cons:
May not be taken seriously by some teachers
Some may fear that others will steal the best ideas and publish first in a journal, however the date management of blogs make it very clear who placed and idea in the public domain first.
Caution:
Blogging can seem very one-sided if others do not comment on your work. An evaluation manager and team leaders need to be proactive in posting and responding to the posts of others in order to build dialogues and invite others to also participate.
Retreat
In a nutshell:
Another established tool in the evaluation repertoire, the retreat takes the team away from the distractions of the normal work environment to focus on course issues in an intensive way.
Example Questions:
What is working, what isn't working?
What new policies do we need to implement and how are we going to go about it?
What is our collective vision for the future of this course?
Reporting:
Retreats ought to have a reporting process and can distribute responsibility among the participants for the preparation of different elements. In practice this is not often done well which can make retreats seem a little like Groundhog Day.
Pros:
The intensive focus builds concentration and brings everyone together.
The isolation from everyday work life can build camaraderie and a sense of shared purpose.
Cons:
Retreats are very expensive
It is increasingly difficult to secure everyone's attendance.
Caution:
Today we need to have very strict rules about answering emails and using mobile devices at retreats as many staff will claim to have urgent matters that require part of their attention. There needs to be clear agreement from the start to take the retreat process seriously and to commit to focus on the business at hand, no exceptions.
Good conflict management skills are necessary and sometimes an external facilitator will be appropriate.
Teaching Portfolio as an Evaluative Tool
In a nutshell:
Teaching portfolios are an excellent self-assessment and reflection tool for individual professional development but they might also create dividends for the evaluation process. Where teachers identify problems, create a change agenda and reflect on implementation, this can connect to course/unit evaluation processes in a number of ways. Teachers may get in the habit of sharing these items when they think they are relevant for the course team, and a course portfolio might usefully be connected to teacher portfolios. The use of ePortfolios, in particular the ability to share some items publicly and not others, creates new ways of assembling this sort of feedback.
Example Questions:
How are the key problems in the course being addressed?
What innovations are being developed?
Reporting:
The process is initiated by teachers making the choice to share folio items with the team.
Pros
Recognises the important role of teachers in developing strategies handling everyday problems.
Draws on the strength of local networks and helps to develop course culture
Further enhances the value of teaching portfolios
Cons:
Will not be of much use of teaching portfolios are not used by most of the course teams
May be some anxiety around sharing if the evaluation environment is not open and supportive.
Caution:
There may be some need for setting the right tone, so that material shared is not interpreted as an attack on colleagues or a way of applying blame for underlying problems. Evaluation administrator may need to take a leadership role in moderating and helping some to reframe their reflections.
Media & Publication Folio
In a nutshell:
Throughout the year there are times when items concerning the course, staff achievements and articles of relevance to the discipline/profession will appear in the news media, in journals and in online publications. We tend to circulate these informally, perhaps via email, but we do not often keep a central record. Collecting this evidence via a portfolio is a way to gain an overall picture of the way the course is perceived and provides a historical record for the university.
Example Questions:
How do we appear to the community?
Where are our areas of expertise as a student may see them? Where are our teachers publishing and what does this mean for the course that we deliver?
What is happening in the discipline/profession that we should respond to?
Reporting:
The folio makes a useful record that should be easily accessible.
Pros:
A useful way of paying attention to the context in which the course operates
Can provide reflective opportunities that do not emerge from the normal evaluation processes.
Can provide an insight into the professional/employment context that students are destined for.
Cons:
May only deal with issues in a surface way
Caution:
This sort of activity is often reserved for public relations departments and it is important to distinguish this from the evaluative process which needs to balance the good and the bad. While an 'in the news' link on a department website may include only the most positive material a media folio will have a more limited audience and will not necessarily be directed at students or outsiders.
Narratives/Digital Storytelling
In a nutshell:
Personal narratives are a way of making sense of experiences that is powerful and taps into issues that are difficult to address in more conventional data. It is easy to presume that your own experience of education is somehow universal and the sharing of narratives can give a broader perspective on different points of view. Digital storytelling is an established method of creating personal narratives and it is not often used by educators but has the potential for creating interesting conversations about what it means to be a learner and a teacher. Storytelling is the natural way in which teachers share experiences (few of us are natural report writers) and much of the information gathered for this manual came from listening to stories.
Example Questions:
What things influenced me to become a teacher?
What is the worst learning experience I had, and what did I learn from it?
Reporting:
Digital narratives can be shared via services such as youtube and connected into blogs and other publications.
Pros:
A very personal focus on issues that are difficult to express through other avenues.
Provides a venue for voices which may not otherwise be heard
Can serve a historical function in creating a record of how experiences are changing over time, particularly as older staff near retirement
Cons:
There is a learning curve for training how to create these narratives, but this is fortunately not too steep and digital storytelling in particular has been used throughout the community with people from a range of backgrounds and familiarity with technology.
May not produce direct action as a result, but it will cultivate understanding
Caution:
Some may see this process as self-indulgent and fail to see the value so expectations around the project would have to be carefully managed.
No comments:
Post a Comment