5 Using the anonymous student survey effectively

The anonymous student survey has a deeply entrenched, yet highly contested place in the evaluation of education systems. It is one of the easiest and cheapest methods (although running costs remain significant) and it is perceived as highly objective and resistant to tampering. Yet it also has become the site of hotly contested argument about the governance of universities and the role of student feedback in performance management of staff. It is an issue which has fractured departments and more journal ink has been spilled on the topic than perhaps any other learning research issue.

Beginning with the positive aspects:

• Student surveys can give a quick ’snapshot' of student satisfaction across a course with trends across time. This can be useful to locate recurrent hotspots which require further investigation and carefully considered evidence gathering.
• As anonymous instruments, these surveys can provide a voice for students who feel too intimidated to express themselves in any other way. We do have to be careful with stereotyping here as many contemporary students are quite confident in expressing their opinions in a more open forum such as social networks
• As a centrally organised process, surveys can seem more accountably and tamper-proof than initiatives organised closer to the interface. Again there are assumptions here that would need to be tested, but certainly in the context of staff promotions a staff member may feel more confident in presenting survey data that they have not gathered themselves.
• If a survey is focused on the students own learning needs, then the student is often in a unique position to describe their individual needs and experiences.

There have been many concerns about over-reliance on student surveys over the years that they have been used, which have intensified under recent trends toward managerialism and performance review in universities, where survey scores can be a lazy and misleading metric to rely on.¹ This is another strong reason for embracing a spectrum of feedback methods.

The challenges directed toward anonymous surveys include:

• Survey results are reductive, especially when they are quantified and presented on a graph. Even Likert scales add little in the way of texture to the data.
• Survey data tells only part of the story (like any data) and can become very divisive if presented in an adversarial manner. If not managed carefully this quickly leads to a blaming and punitive use of metrics which itself will make everyone reluctant to participate.
• Survey questions are difficult to write and poor questions can ask students for judgement on a topic that they are not equipped to make, for instance on whether the teaching approach was good or bad. It is much more appropriate to ask a student-centred question such as whether the teaching approach helped them, but even then care must be taken in interpreting the data.
• Whether or not a student enjoyed or liked a course or unit is very subjective and can be influenced by many factors outside of academic control
• While most surveys distinguish between an evaluation of a course/unit or evaluation of a teacher, in practice this distinction means little to many students. If asked the wrong questions the process can quickly become very personal and focussed on the individual personality of the teacher rather than student learning needs.
• Surveys are often conducted too late in the semester, when some students are no longer attending class regularly. Of course the engagement with classroom activities for students IS a very important area for course evaluation in itself.
• Surveys are usually conducted too late in the semester to make any real changes
• Survey fatigue. Students are over-surveyed and may be reluctant to engage with the process, particularly where the feedback loop is not closed and students cannot see the benefit of participation in the process.
• Surveys styled in the ’customer service' or ’complaint box' mode reinforce the hotly contested idea that education is a commodity product and students are passive consumers. Many academics, managers and students do not agree with the politics of this.

There is such a long history of vague or overt dissatisfaction with the student survey that we even respond to survey data in stereotypical ways. Linet Arthur has described a ’typology of responses to feedback’ where we face unwelcome feedback by reflexively shaming, blaming and taming the students or by reframing negatives as something positive.² This means that the data can have little impact as we tend to explain it away with well-worn rationalisations rather than investigate further.

Given these criticisms it is important that we put effort into making the most of student surveys. They are likely to remain a key feature of evaluation system and many of the criticisms can be avoided by being more careful in the questions that we ask and more open and collaborative in the process itself, as well as providing other forms of evidence.

Timing the Survey

Surveys can reveal different information at different stages of a course or unit and it can be difficult to decide when is the best time. Too early and students may not have experienced enough of a program, too late and they may have already disengaged or fail to see how the survey will benefit them in any real way. As with any feedback method, the answer may be to tailor the survey to different points of progression to allow for flexibility in the timing as much as possible to avoid fatigue.

Surveys are typically conducted at the end of a program but earlier surveys can be useful in understanding expectations, in asking students what their goals are and to build on previous feedback activities. Once again if we are clear that we are focussing on student needs and perspectives, not on a judgement of course quality, it will be easier to tease out the best questions for the particular point of progress. If students are already active participants in the evaluation system (see section 10) they will be primed to engage with surveys in a positive way which is attuned to the sort of data that you seek.

The Australian Graduate Survey has been used to canvas students one year out from graduation. While this is itself an interesting point to seek feedback, the survey has been oriented toward course quality assessment and becomes all too easily muddled by the career prospects of students whose expectations may or may not have been satisfied. It is also difficult to obtain a significant survey sample at this point in time as reply rates tend to be very low. Even so, publications such as The Good Universities Guide rely on this data without exploring the methodological weaknesses.

The new University Experience Survey aims to understand the students' perspective at first and third year levels, which again provide distinctive junctures at which to understand the learning experience and should gain a better response. While these changes are occurring students also have access to a range of unofficial online surveys, of varying credibility, in which to channel feedback. Websites such as ’rate my college' have been very popular in the United States and Australian versions are set to be increasingly significant.

Using Surveys Well

Surveys therefore have a continuing place on the evaluation landscape, but we must use them wisely and modify them to suit the context. Assumptions need to weighed carefully, for instance a survey does not NEED to be anonymous – respondents may be happy to provide their contact details particularly where there is a chance for a follow up.

We should carefully consider the feedback nexus with surveys and design our instruments accordingly. There should be significant differences between a survey which asks students whether a course or unit meets their learning needs and one where they provide individual feedback on teaching staff. The latter is a tool which can be used in professional development and career advancement but students can easily blur distinctions between the individual and the course, especially where poorly designed surveys confuse the issue.

Finally, a plea on behalf of students everywhere to think carefully about the amount of surveying that you do, especially when the entire learning context is considered. Students are sometimes asked to complete four different unit surveys, four (or more) teacher surveys as well as various university surveys, all within a few weeks. If you close the loop they will see more value in the exercise but still, spare a thought for them and how irritable they will become if further bombarded by surveys even with a chance to win a consumer gadget. This caution also applies to survey length which often gets out of hand especially with online surveys which can grow like weeds. Be focussed on what you need to know and ask for it in the simplest, most concise way possible.

Methods

• Summative anonymous student surveys
• Formative surveys
• Graduate surveys
• Employer surveys
• Pre-emptive surveys

Key Points

• Be wary of simplistic and reductionist use of survey data, such a single satisfaction metric or a simple graph.
• Triangulate data, use surveys to locate hotspots but always check and follow up with further investigation
• Think about timing, the different points in progress that you can survey and the different perspectives these can give.
• Be very careful when making claims about survey data to ensure that the methodology was sound and that there was is statistically significant sample in the responses. A statistician will be able to tell you on a given survey group, how many responses you need to have a significant sample.
• Remember that with very small samples, particularly on contentious issues, students may be identifiable from their answers even where surveys are conducted anonymously
• Surveys do not have to be anonymous. Sometimes respondents will welcome the opportunity for further feedback and personal engagement.
• Be wary of survey fatigue in students and always close the loop so they can see the value of participation. Limit the number of questions you ask to the essentials. Don't presume that a chance to win a prize will be an incentive.

1See Blackmore, Jill (2009)’Academic pedagogies, quality logics and performative universities: evaluating teaching and what students want', Studies in Higher Education, 34(8), 857-872

2 Arthur L (2009)’From performativity to professionalism: lecturer's responses to student feedback', Teaching in Higher Education, 14:4, 441-454







Summative Anonymous Student Survey
In a nutshell:

Students are surveyed on their experience at the end of a unit or course. This is the most frequently used student feedback mechanism.

Example Questions:

Did the learning activities in this unit help students meet their learning objectives?

Did students understand what was required of them?

Reporting:

Generally these surveys are collected centrally and distributed back to unit coordinators. Different universities have different protocols for who else can access the data, but generally unit/course information is reported and personal feedback goes to the academic who can choose how to share it.

Pros:

It is a formal and transparent mechanism.

Students may feel more comfortable being anonymous.

Cons:

Can be reductive and focus on a one figure result.

Timing is difficult as the survey may come too late for students who have disengaged.

May be used to answer questions that were not asked in the survey and be used as a blunt quality indicator.

Students may be disillusioned with the process, particularly if they cannot see results.

Poorly designed questions can ask students for answers that require expertise that they do not have.

Caution:

Rates of feedback have to be carefully monitored to ensure that results are actually representative of the wider student body.



Formative Surveys
In a nutshell:

Students are surveyed throughout the semester about their needs and expectations. This can be combined with formative tests of students' knowledge and skills that can be used to match up student progress with feedback on specific activities and resources.

Example Questions:

Did this video help you understand the concepts for the activity in class?

Which type of resources do you find the most helpful in learning a new skill?

What was the key to understanding how to complete this task?

Which of these areas would you like to focus on this semester?

Reporting:

Generally informal, but can benefit from sharing among academics particularly those teaching in the same course.

Pros:

Can be more informal and flexible than the summative feedback instruments.

Cons:



Students may be reluctant to participate if they feel responses may prejudice their results later, especially if they are not used to an environment of continuous feedback. If overdone this can contribute to survey fatigue.

Caution:

This method can be anonymous if there are concerns that students are not participating for fear of identification, but this is better remedied with addressing the underlying fears.






Graduate Surveys
In a nutshell:

Students are surveyed after they have graduate and had a chance to apply what they have learnt in a professional context. The Australian Graduate Survey is one example.

Example Questions:

Are the graduates in full time employment?

What are the graduates' opinions of the course now that they have graduated?

Reporting:

These surveys are often conducted by government and publicly reported and then picked up on in the Good Universities Guide and other publications.

Pros:

Provides an overall of course experiences once a graduate has completed.

Cons:

It may be too early for a graduate to report on how a course has influenced their career opportunities

Has the same problems as other summative student surveys

Caution:

Rates of response are generally quite low but are nevertheless re-published without analysis of the statistical significance.

Results may be more concerned with graduate employment opportunities, particularly if they graduated with unrealistic expectations of the job market – although this in itself is significant where course managers need to engage with these expectations during a course.