13 Evaluation and Learning Research

The often arbitrary division between teaching and research continues to trouble universities and has serious funding consequences which impact on priorities. The evaluation process risks being sidelined if the activity is seen as an administrative chore rather than a research activity in itself, investigation into the general field of learning but also into specific disciplinary fields concerning how discipline knowledge is produced and transmitted. Building publishing opportunities into the evaluation process is one way that institutions can demonstrate that it is an activity which is valued and taken seriously.

There have always been opportunities for academics to present the results of evaluation at conferences and to publish in journals but there remains some elitist distinctions between discipline-based research and learning research or ‘the scholarship of teaching and learning‘, as it is sometimes known. This culture is changing, evidenced by the fact that OLT learning and teaching grants are now recognised in the top category of academic grants.

Presenting evaluation research in a public forum and publishing it should also be recognised as part of the feedback loop of the evaluation process. Sharing learning research with the wider academic community is a way of seeking and receiving feedback from peers and presenting the results of evaluation in a transparent and credible way. Course portfolios should include these outcomes in their repertoire of evidence.

Because of importance of academic reflection and practical experience, much of the knowledge around learning is anecdotal and distributed throughout informal networks. One of the benefits of the research process is the gathering of these various strands of knowledge, weighing their validity, and presenting it in a format that is credible and engages in wider debate.

This debate is not limited to the academic field, in the new publishing environment academic journals are a often cumbersome option with a significant publication delays and diminishing audience. Many of the critical conversations are occurring in dynamic publishing spaces, mediated by blogs and social networks. When managing an evaluation process it is important to consider the balance between these more effective publishing routes and the more traditional options that still remain necessary in order place the patina of respectability on research.

Interdisciplinary knowledge is one of the daunting features of learning research but it is also one of its pleasures. A researcher is required to stay connected to their own disciplinary knowledge base but also to educational research and today to innovations in the media and technologies that shape the learning environment. This is not to be mistaken for techno-fetishism, a learning researcher does not necessarily have to present a hipster's knowledge of breaking social media trends, but they do need to stay informed.

These knowledge burdens provide specific challenges for evaluation managers in motivating researchers to contribute to the evaluative learning research project. Fortunately the answer lies in collegial networks – in your department, in central support specialists, in discipline networks – all of which can spread the load and contribute to the community knowledge base. The evaluation manager's challenge is to activate these connections and support the cultivation of healthy knowledge sharing networks.

Partnership is another feature of the evolving knowledge environment and we are much more likely to be involved in research teams and have partners across different institutions or in other locations such as industry, the community or the public sector. This presents its own challenges for the evaluation process as these relationships need careful management, especially where there may be some institutional rivalry or where confidential data may need to be shared. As with any such relationship, obtaining clear agreement at the foundation stage about boundaries, protocols and etiquette can build a stronger working partnership.

Accounting for research in the evaluation process

At the end of the day it may still prove difficult to properly connect the learning research and evaluation processes because of the traditional conceptual divide and because the administrative functions are often distributed in different locations. However these might be some useful strategies to sustain the connection:

• Course evaluation managers should be included in the learning grant process, not necessarily as primary investigators but they can take on other roles such as membership of the project reference group
• Learning grant applications, both successful and unsuccessful should be included in the course portfolio along with reflection on it significance in the big picture of evaluation
• The evaluation process can act as a hub in the learning research network, in particular for gathering and distributing useful publications and
• Evaluation managers have a role in learning research networks. If your department does not already have one, the evaluation manager could lead the network
• The evaluation aspect should be emphasised in any internal research funding or conference funding applications
• Make connections to share the load, look for partners who face the same evaluation issues who might be encouraged to collaborate on comparative evaluation and benchmarking on a quid pro quo basis. Make connections with central learning support divisions, the library, IT services, facilities and any other limbs of the university that have something to offer for both the evaluation and the learning research process.

Methods

• Conference Paper
• Literature review
• Learning Grant Application
• Learning yearbook

Key Points

• Evaluation is a critical site for developing the teaching and research nexus and for developing the evidence-base of learning.
• It is important to be proactive in documenting relevant learning research activities and accounting for them in evaluation portfolios.
• Academic workloads are a key consideration in designing evaluation systems and where possible enable academics to score multiple goals with the same work, in their teaching, in contribution to the evaluation process and in evidencing research quantum.

Conference Paper
In a nutshell:

Not all feedback comes from inside a program and there is a tradition of academic presentation and publishing of learning research which can also function as a useful part of the evaluation system. The process of writing and presenting a paper is an opportunity to reflect on practice, conduct further research and crystallise ideas. Responses to that paper, at conferences, in further publication and informally are part of the feedback loop.

Example Questions:

How does this practice fit into an established theoretical framework?

Is the data that has been gathered validated and applicable to other situations.

Reporting:

The academic publishing process.

Pros:

Exposes innovation to a wider audience outside of your institution.

The ability to get publication recognition for evaluation work

Cons:

Process is slow, and it can take a long while

Publishing in appropriate journals that are recognised for research quantum may be difficult.

Participation in conferences can be costly, especially where travel is required.

Caution:

The tension between learning research and discipline-based research can undermine the ability to effectively publish evaluation data. Where academics are pressured to publish in their disciplinary knowledge, where they have to prioritise travel funding, learning research inevitably suffers.









Literature Review
In a nutshell:

In some disciplines a literature review is a commonplace genre of academic publishing but this is not true for everyone. For evaluators, the ability to access a succinct overview of research and publishing in a field can be a vital tool for responding to other feedback and designing innovations.

Example Questions:

What is the body of research on social media as tool for reflective learning?

What are the key debates on the use of student surveys as a feedback mechanism?

Reporting:

The review forms a report which can be shared among the evaluation team and potentially published.

Pros:

A useful tool to get an overview of an area of research and to target key areas that require further reading.

Cons:

It is time consuming to produce, although it may be a task for a research assistant.

Caution:

There are usually gaps in literature reviews and these may be the foundation of useful further research.







Learning Grant Application
In a nutshell:

Learning grants are one way to fund innovation and to extend the research connected to course evaluation. Funded projects might include gathering of further research on learning, applying and evaluation an innovative program, designing a good practice guide on a particular topic and creating how-to guides for implementing practice. The Office of Learning and Teach administers national grants but it is also worth considering other funding bodies, particularly where community and education grants overlap and can demonstrate benefits across society.

Example Questions:

What is the best way to use mobile devices to design formative assessment opportunities?

How well does our student retention strategy work?

Reporting:

Granting bodies provide feedback on unsuccessful applications.

A successful grant will contain its own reporting an evaluation measures.

Pros:

An excellent way to crystallise practice and to assemble a team of interested people, even if the application is ultimately unsuccessful.

Funding allows you to temporarily enlarge the team and focus on an aspect of evaluation which you do not otherwise have the resources to conduct.

Cons:

Difficult to plan for as you do not know whether the application will be successful.

Caution:

The funded project will itself have to be evaluated on completion. The techniques described in this manual are just as applicable to project evaluation as they are to course evaluation and should hopefully be a source of ideas for designing learning research grant applications.
















Learning Yearbook
In a Nutshell:

The genre of a collected papers book is well established but generally connected to a conference or research group. Universities could support learning research, and with it the evaluation process, by producing an annual peer reviewed yearbook featuring papers by innovators in their institution. This would make the process of publication easier and it published under a creative commons licence, allow authors to seek publishing options in open journals as well.

Example Questions:

What have we, as a university, been focussing on this year?

How have developments in other courses influenced the development in this course?

Reporting:

Published as an eBook, although a Print On Demand option would be pleasing, especially if as a year to year collection could look handsome on a shelf.

Pros:

An efficient method of generating research without the long delays of journal publishing.

Cons:

The institution would have to resource the time for editors and reviewers as well as the costs of proofreading and typesetting

Caution:

Care would need to be taken to emphasise the objectivity of the peer review process, so that it is clear that these papers have been through a rigorous approval process.