2 Designing Feedback

Universities tend to implement feedback at one end of the spectrum or the other. Most have formal survey collection systems where students remain anonymous, their identities protected against the possibility of backlash from criticism. On the other hand all teachers have their own informal feedback systems, where they judge the progress of a learning session through various cues given by students, often on a non-verbal intuitive level. In between these modes there is a whole spectrum of data and gathering methods that we can use, each of which gives a different perspective on the problems at hand.

In our context, feedback is of research into learning and learning design. Like any other research we need to begin with the questions to be asked and then develop a methodology which best allows us to collect evidence. All too often the discussion of academic feedback is led by the methods, or even the specific technological tool used, rather than the fundamental questions which drive improvement. We start by asking the best questions we have, and developing these over time as cycles of feedback are evaluated and integrated into innovation.

In section 3 we will consider the design of an evaluation system that integrates different forms of feedback but before this can be done we need to focus on the methods themselves. Once we have the questions we want to ask, we then need to choose the best method to obtain evidence, whether it be qualitative or quantitative in nature. Multiple methods can be used to triangulate data, to ensure reliability or to provide context. For example statistics might provide a quick method of identifying problems, and this can be followed up more qualitative and individualised methods to provide the texture required in order to explain and understand.

Are you ready for feedback?

The evaluative process, particularly the act of receiving feedback can be fraught with anxiety and it is often difficult to separate the professional/institutional critique from the personal. It is therefore essential that individuals and organisations are ready for feedback, for taking critique in a positive and constructive spirit and for resiliently managing the impact of negative comments and criticism.

There is a certain thin-skinned-ness in the current academic environment, where individuals fear that critical feedback will impact on them personally and they will suffer for circumstances that are often outside their control. This anxiety can cripple the evaluation process because those who are in fear for their jobs and career prospects will be overly cautious about sharing information or engaging in curriculum development that might be construed as condemnation of their past and existing practices. We have to acknowledge the way that this environment constrains the evaluative process and find ways of moving forward with a positive attitude. Likewise, this overly defensive posture can function as an excuse not to understand critique and make responsible use of it.

There is an important organisational aspect to this. For individuals to fully participate in the evaluation process it must be a safe space for them to engage with critique, weigh the merits of it and construct strategies for change. If individuals fear that they will be individually and personally persecuted or disadvantaged by the evaluation process, of course they will be fearful of participating. It is up to the evaluation process managers and departmental heads to ensure that participants are prepared for the process, have a positive mindset, are protected from negative consequences and are able to talk freely about both the feedback and the process itself.

Asking the Right Question

Clarity is a key concern for any research or feedback design question, it should focus on obtaining the best evidence so we can then design a methodology by which we can capture it. Questions that are unclear might act as ’fishing expeditions' where you can collect unexpected results, but these are more likely to waste time and resources.

Where we gather data and analytics we are limited only by the resources needed to process information, but with any method that relies on respondents contributing their time and attention we must be wary of survey fatigue. Students in particular feel over-surveyed and where they do not see tangible results of previous feedback can become disinclined to be involved. Clear focused questions are therefore better than lots of general and vague ones.

There is also a question of research integrity here, where data is gathered under one pretext (or a vague pretext) and then used for other purposes. Research ethics rules prohibit us (generally) from deceiving respondents and our National Privacy Principle rules (under the Privacy Act 1988 (Cth)) prevent us from using private information for a purpose other than that for which it was obtained. Good information management means that clear questions and clear purposes are essential.

Where we put attention into building a feedback culture, where students and other stakeholders feel that their voices are heard and that their efforts in providing feedback contribute in a meaningful way, it is much easier to retain enthusiasm and commitment. If feedback is built into learning design in a way which is integrated and regular, than a feedback culture can be built on shared values and respect. And this whole system depends on us being careful about the drain on time and asking questions that are relevant and actionable.

Some fundamental questions:

• Who are our students?
• What are their individual characteristics that shape the way they learn?
• What did the students learn?
• Did students achieve the outcomes promised?
• What do students need to support their learning?
• What factors impeded student learning?
• What was the impact of delivery modes and other procedures on student learning?
• What is the impact of innovative measures?

Seeking Effective Feedback

We have already emphasised the importance of asking concise questions, but there are other factors that help ensure that feedback is effective and treated seriously.

Feedback should be embedded in everyday practice so that students (and other stakeholders) are accustomed to providing feedback and do not see it as a distraction from core business. Feedback then becomes core business, a part of the relationship which defines and monitors the learning space. Feedback becomes the mechanism through which learning is made interactive and learners can keep track of progress.

Feedback should be meaningful with tangible results. This means ’closing the loop' and communicating your response to feedback. It means setting up a selection of options which are valid and respecting the choices that students make. Where choices cannot be implemented you need to be clear as to why. Certainly feedback is useful for long term subject development but if students are being asked to provide feedback that will only benefit future students, they will feel exploited by the process.

Feedback should ask questions which the respondent is in a position to answer. Some of the worst implementations of surveys are those which ask students to make expert judgements on overall course quality, design or content. It is much better to ask questions that students can answer and encourage them to answer from their point of view, so instead of asking ‘what was the quality of resources provided?‘ it is better to ask ‘what resources did you need and were those provided suitable?‘ Instead of ‘was there too much reading‘, ask ‘did you manage to keep up with the reading required? ‘ The difference can be subtle but it can be crucial in defining the feedback relationship that you want, to focus on key questions of needs and motivation rather than generic ideas of value or whether a course was ‘interesting‘.

Feedback occurs in an environment of trust and this must be built over time. Learners need to be clear that they will not be punished for providing disagreeable feedback, either through punitive marking or even from disapproval from their teacher. It helps if you ask questions that do not invite responses that are adversarial, invite personal criticism or open the door for ill-directed complaints. Organisations need to be very clear about the differences between feedback mechanisms and the complaints/grievance procedures and students should be informed about the processes attached to each. Even when this does occur, it is vitally important that the person seeking the feedback does not become defensive or respond in a way that amplifies the conflict, rather it is better to reframe the statements in a less inflammatory way. Remember that in receiving feedback you are also modelling the kind of behaviour you want to see in students when they receive feedback on their own work. This is a skill that we have to learn and must practice to develop.

Consider the kind of evidence the feedback will produce. Just as any other kind of research data, try to obtain feedback in a format that you can record, share, weigh up, publish and use. Some questions require empirical data, others will need more practical and experiential data. As with the question of clarity, this will help to avoid asking broad, meandering questions and focus the feedback on issues where the feedback will have a real impact.

When framing a feedback question refer to the competing ideas and theories which are brought into play. You should not be afraid to be overt about theory and should respect respondents enough to clue them in on the context of the questions you asking. As with other forms of action research, you are more successful when you invite respondents to be active participants rather than experimental subjects.

Evidence of Feedback

One of the more difficult aspects of making the transition from informal feedback to evidence-based feedback is the task of logging and recording feedback results. It is relatively easy to set up an in-class survey, either via paper or electronically, but what do you do with the results? How do you record these in a way that will be accessible in a year's time the next time a subject is delivered? How do you share your findings with others, compare results or publish using the data? When a new academic takes over responsibility for a unit, how do they gain access to the feedback obtained in the past?

When designing feedback systems we need to consider the recording and keeping feedback as a credible form of evidence. One reason for the popularity of surveys is that it is reasonably easy to tabulate the results of a survey and keep it as a form of evidence. Where this is done as a central institutionally activity there is a reasonably stable warehouse of data that can be drawn on, but what about all the other forms of feedback, at different levels of formality? How do you gather experiential data in a way that is reliable, credible and accessible to others?

If you are overseeing an evaluation system you will need to be sure that there are accepted methods of gathering and sharing feedback data and that all participants are aware of the protocols. This may be something as simple as a set of folders on a shared drive or take the form of a more dynamic online database, perhaps built using a secure open source platform such as Drupal. If you are collecting and perhaps collating data from a large number of sources, such as including the results of formative assessment tasks, storage and access can be

Individuals may also want to keep their own feedback log, particularly where feedback may include personal feedback on teaching. As long as this is done with due consideration of any security or privacy issues, this would form an excellent opportunity for reflection in a teaching portfolio, or other kind of e-portfolio. If you are managing an evaluation process you should make participants aware of these opportunities and of the risks they will have to manage.

Remember that evidence also has to be weighed. Make sure you store your evidence with a description of your methodology so that others, particularly those looking at the data after you have left the organisation, will be able to understand how the feedback was gathered and evaluate its relevance on the basis of the methods used.

Suggestions for managing feedback data include:

• Keep a feedback log for each unit of study, as text document or perhaps even a database which can be shared with others. This can even be shared online, for instance a cloud-based document that is shared through your learning blog.
• Record feedback in your teaching portfolio
• Course portfolios have become more common for the purposes of benchmarking and auditing, these already include some feedback data and could perhaps be created to include other forms of feedback or at least make connections to other repositories.
• An innovative web2.0 solution would be an institution-level online database designed for the secure logging and sharing of feedback.

Presenting Feedback: Data Visualisation and Information Design

The impact of feedback data, like any other form of information, depends on how it is presented. Dull, impenetrable tables and lifeless graphs may hamper the evaluative process by hiding the crucial data behind mountains of irrelevant data. Therefore information design is a crucial part of the feedback and evaluation process, particularly in clearly and ethically presenting data to readers who may not have a lot of time to make sense of it by themselves.

In an ideal world we would have regular access to help from an information designer, but you may well have someone with the correct skills in your department or networks. You should consider the different ways in which the same data can be presented using different graph formats, wordles, infomatics or other data visualisations. IBM runs an open data visualisation service through ManyEyes at http://www-958.ibm.com which should give plenty of inspiration, but be careful that data you upload there is visible to everyone online and should not include confidential information.

As with emerging feedback technologies, the presentation of data is a rapidly changing field where creative designers are embracing new ways in which we can communicate large sets of data in a way that aids understanding and helps us to ask the next set of questions. Feedback and evaluation systems are a dynamic feature of evolving teaching practice and innovation.

Ethics and ethics approval

Traditionally there has been some ambiguity in the difference between feedback and research. Sometimes feedback has been considered part of the business operation of a university and has been treated outside of the ethics approval framework, but this is not a distinction which holds up under much scrutiny. While students are in a very different position to many research subjects as they are already ’inside' the institution and have a stake in the outcomes, the feedback process should not have an exception from ethical oversight. Rather, it may be more appropriate to provide various types of ongoing ethical approval for feedback by category or short-form low handling approval mechanisms. The evaluation engine would quickly grind to a halt if teachers needed approval every time they asked students for any kind of informal feedback at all, so it is crucial that universities find the right balance between oversight and flexibility.

Any data that we want to use in academic publication will require the approval of an ethics committee before the data is gathered, so it is important that you consider the likelihood of publishing before you obtain feedback. Further, the evaluation process should always be conducted ethically and transparently, even where you do not require formal ethics committee approval.

Formal approval can delay the process and it may not be needed where the actual data set is not published, but might potential form the foundation of reflection or other analysis. For example you would not need ethics committee approval to ask students in class, by show of hands, to indicate whether they needed more resources on a particular topic. There are substantial grey areas here and it may be worth considering what kinds of blanket, pre-emptive ethical clearance might be obtained for a range of different feedback activities by the evaluation manager rather than each individual academic involved.

In any case the design of feedback process must consider the ethical, and often legal, interests of the students:

• Confidentiality
• Privacy, protection of private data
• Potential harm and vulnerability of students, especially fears of retribution after critical feedback. Some sub-groups may be particularly vulnerable in the use of particular methods.
• Ownership of ideas and intellectual property
• Security of data collected. Deletion of data when no longer required to be kept.
• De-identification of individuals
• Respecting the contribution of respondents by closing the loop and informing individuals of results and impact

Some general dos and don’ts for Feedback

Here are some quick pointers for effective and embedded feedback:

• Get into the regular habit of seeking and gathering feedback. Set aside time for feedback so that it is embedded in your workflow. For instance, set aside seminar time for feedback activities and let students know it is an important regular activity.
• When it is regular habit, make sure you vary the questions and the methods to keep it fresh. Don't let it become a chore.
• Make yourself open to feedback by receiving it well and closing the loop by letting participants know what impact the feedback had made.
• Don't be defensive about feedback, solicited or unsolicited. If someone provides it in an inappropriate manner, address the issues of the etiquette or format, not the content itself.
• Focus on building constructive criticism skills in others, as them to reframe the feedback or reframe yourself by taking the ’heat' out (‘what I think you are saying is.... ‘)
• Focus on what it is that motivates and helps learners. Don't ask for value judgements such as whether it was good or interesting.
• Present feedback opportunities dialectically – not as binary alternatives but as different choices out of which new innovations can be synthesised.
• When you close the loop, you can also ask further questions. How are we going to implement this? How are we going to test if we are successful?
• Good feedback is derived from a strong respectful relationship and an environment in which feedback is valued and acted on.

Key points

• Feedback is research data and should abide by the same principles of open and ethical collection
• Feedback is evidence and must be collected, reported and shared in a way which is open and credible. The issue of how and where feedback is recorded has become increasingly important.
• You need to prepare to accept feedback and manage anxieties about the personal consequences of critical feedback.
• It is more important to ask the right questions than to ask the same questions over and again
• Good feedback is derived from a strong respectful relationship and an environment in which feedback is valued and acted on.
• Beware of survey fatigue, be respectful of the time and energy you expect respondents to put into providing feedback and consider the stake they have (or don't have) in the outcomes.