There
is a curious distinction argued in some of the literature on course
evaluation where the concepts of evaluation and research are treated
as distinct and disparate activities. It may be because evaluation
has often been the function of a separate university unit or a
distinction as a result of the teaching/research tension. Whatever
the cause, it has meant that the range of analytical research tools
brought to bear on evaluation has been somewhat limited. This
section opens up the social science researcher's toolkit for
inspiration in the evaluation process and argues that evaluation is
a form of research, into
learning and into the ways in which disciplinary knowledge is
constructed and transmitted. Now that academic teachers are more
often involved in course evaluation, it would be a shame not to draw
on their collective research experience in this context.
One
feature shared by both types of inquiry is the often elusive nature
of the subject which Bruno Latour might suggest takes place in a
black box.
Student learning, as we have seen, is typically measured by the
assessment process and quantified in student results. It is very
difficult to find hard scientifically testable extrinsic evidence of
the actual process of learning, although some scientists have used
MRI and other scanning technologies to try to gain a better
understanding. Typically learning research is done by observing the
effects of learning on individuals and groups, the environments in
which they learn and the attitudes of participants. Much like other
social science research, patterns sought and hypothesis are advanced
and tested to explain those patterns.
Much
of the data about learning comes from discussion with the learners
themselves. A variety of methods, from interviews, to focus groups
and surveys are utilised to gather discursive data from which we can
infer data about the learning which is happening ‘under the hood‘.
Good instrument design is crucial, asking questions that do not lead
by suggesting answers and that dig beneath surface knowledge; keeping
respondents informed as to the rationale beneath the question.
Poorly designed instruments gather little more than surface
impressions, a flaw which some anonymous student surveys also suffer.
Other
methods break the fourth wall between researcher and the learning
environment. Participant observation, action research and
auto-ethnography are established social science research methods that
can yield deep contextual knowledge that cannot be obtained by ‘white
coats and clipboards‘. There is a wealth of literature and online
information on the effective design and uses of these research
methods but once again it can be much easier just to tap into the
expertise already on staff.
Once
data is gathered it is essential that you are able to examine and
analyse it. Quantitative data requires a particular skillset in
statistical analysis and validation, qualitative data has its own
frameworks and analysis methods. Here again the evaluation process
can useful draw on experts in these fields for guidance in data
analysis. One common challenge is ensuring that a sample size is
large enough to make valid observations about learning, as students
can be reluctant to participate for various reasons. Given a class
or course size, a statistician will be able to tell you how large a
sample you need to be credible.
Another
intersection of research and evaluation is the learning and teaching
grant, where recipients are required to evaluate the success of their
project on completion. The methods described here, and throughout
the rest of this manual should be of value in preparing the
evaluation aspect of grant application documents and thinking beyond
the simple online survey which is relied on very frequently.
Ethical
evaluation and the use of social science methods
Social
science researchers have experience in obtaining ethical clearance
for research projects involving confidential interviews and other
interactions with human subjects. One unfortunate consequence of the
artificial evaluation/research distinction is that there can be a
lack of experience on evaluation teams and even a lack of awareness
regarding the need for ethics approval.
Some
evaluation units have been closely tied to the administration of
universities and see evaluation as an internal matter that does not
require oversight. Whether or not this was true in the past, the new
evaluation culture requires a keen awareness of the legal and ethical
responsibilities of data gathering particularly where confidential
and controversial matters may be under discussion. Students, and
other participants, need the protection of ethical oversight and we
can draw on the experience of academic researchers, particularly
those who have used the same methods in their research previously.
Action
research and closing the loop
While
social scientists once tried to distance themselves from research
subjects in order to don the lab coat of objectivity, action research
oriented practice finds objectivity in robust methodological design
and emphasises the fully informed participation of subjects.
Researchers have acknowledged a responsibility to subjects and to the
community at large to share the results and even sometimes to allow
subjects to collaborate on the future direction that research takes.
In the evaluation context this responsibility also manifests itself
through closing the feedback loop.
Effective
representation and visualisation of data is an important issue here
as learners are unlikely to have the same time to focus on the
evaluative aspects of their course, being somewhat focussed on the
primary business of learning. Section 11 discusses the effective use
of learning analytics and reiterates the importance of the
information visualisation process, an increasingly central aspect of
the research process.
While
learners do not need to be swamped with statistics, tables and
charts, the feedback process is a useful juncture for them to learn
more about their own learning and the premises upon which their
learning environments are being constructed. Students who are better
informed are more able to enhance their own learning and to
participate in the construction of the learning community. Some of
these students will be also be the academics and evaluation managers
of the future who will one day be designing evaluation systems of
their own.
Methods
- Participant Observation
- Structured and semi-structured interviews
- Focus groups
- Ethnography and learning portfolios
Key
Points
- Course evaluation does not exist in a vacuum and it should draw on the expertise and innovation of researchers in other fields, especially social scientists who are accustomed to gathering data about people and social environments.
- Closing the loop is an essential expression of respect for the students and others who take the time to engage with the evaluation process.
- Look within your organisation for people skilled in qualitative and quantitative data methods, in the analysis and presentation of data and in obtaining ethical clearance.
Participant Observation
In a nutshell:
Rather than seeking to be an impartial outsider, observation is conducted from the inside of a learning activity. This is not so effective in a traditional hierarchical classroom but can be very useful where educational activities are co-produced and teachers are active collaborators in learning.
Example Questions:
How was I, a teacher, involved in the learning of others?
How effectively do I model learning strategies?
Reporting:
This is would generally be a reflective self-assessment process but it may be shared with others as research.
Pros:
Can be a unique point of view to observe learning actually happening.
There is a body of social science research theory on the method, particularly concerned with objectivity and bias.
Cons:
A time consuming process which may be difficult to write up in a way that can be shared and published.
Caution:
As with social research, careful objectivity and impartiality protocols need to be set up and maintained. Particular care must be taken in transitioning to the assessor role so that students do not feel betrayed by someone who they had assumed to have a friendly rather than professional relationship.
Structured/Semi Structured Interviews
In a nutshell:
Interviews conducted on the basis of a set of structured or semi-structured questions, determined in advance and cleared through ethics channels.
Example Questions:
What aspects of this course helped you develop your information technology skills?
Did the new learning app help you to think differently about this area of knowledge?
Who is the most influential person when it comes to the way in which you manage teams?
Reporting:
Generally published as research, but can be part of other reporting mechanisms.
Pros:
Interviews can seek out focussed information that may not be accessible through more formal instruments like surveys.
For semi-structured interviews, the ability to ask further questions and to allow respondents to elaborate.
Plenty written about the method.
Cons:
Time consuming and costly, particularly if interview recordings need to be transcribed (which is not always necessary)
Survey fatigue also applies here.
Caution:
Recruitment can be difficult to do and to provide appropriate incentives for. Incentives need not be financial, but participants should see some direct benefit from being involved.
Focus Groups
In a nutshell:
Focus groups are similar to interview but allow group dynamics to be analysed.
Example Questions:
If we asked you to solve this problem, how would you go about it?
Reporting:
Generally published as research, but can be part of other reporting mechanisms.
Pros:
Ability to examine social aspects of learning.
May be more time effective than interviewing individuals separately.
May enhance collective ownership of the process.
Plenty written about the method.
Cons:
Time and resources
Caution:
This method has been developed by advertising and it if not carefully managed can quickly devolve into surface discussion of personal likes and dislikes rather than the planned questions.
Ethnography & Learning Portfolios
In a nutshell:
The ethnographic methods involves focussed investigation of life stories and narrative in a particular context. In social sciences this method seeks a deep understanding of the personal circumstances of and individual or a community. For learning research ethnography places students in a variety of contextual networks, not simply as a learner in a classroom. Auto-ethnography is an established method which is gaining prominence as we realise that researchers are also located in context with their field of inquiry.
Example Questions:
How does learning interface with other aspects of students' lives?
What does career mean for students in this course?
How do childhood learning experiences impact on adult learning styles?
Reporting:
Generally published as research, but can be part of other reporting mechanisms.
Pros:
An in-depth understand of learning as it relates to specific individuals
Investigates how individuals create meaning out of context
Can bring out structural issues underlying an environment
There is a body of literature on methods.
Cons:
A very labour intensive method which requires extensive resources and demands on participants.
Must be careful in making generalisations based on the experiences of individuals.
Caution:
Student ePortfolios might provide a source for ethnographic research, so long as proper consent and ethical clearance is obtained.
No comments:
Post a Comment