11 Evaluation through Learning Analytics

New information technologies contain barely explored potential for connecting knowledge networks and accessing information in ways that are both more immediate and more complex. The engines that drive these systems can tell us more about the ways in which knowledge is constructed and can expose some of the hidden mechanisms of learning. In section 4 we looked at how collection of data such as resource use profiles and grading could build better pictures of learning and give feedback on the way that students access our learning spaces. With the power of aggregation and cloud-based data sharing this capability is going global.

There have been many promises made about the potential of learning analytics to revolutionise education and some academics are rightly cautious about the technological determinist flavour of some of these claims. This is particularly true where commercial vendors are selling a product, particularly commercial LMS vendors who are trying to recoup a market that is now dominated by open source LMS solutions. A chief flaw in many of their claims is that a single institutional LMS is a closed system and lacks the capacity to embrace the globalised potential of learning analytics. Other vendors' claims require that courses use their proprietary learning resources and assessments for the data to make any sense.

Nevertheless, beyond the commercial hyperbole there is a dedicated community of innovators looking at solutions consistent with free knowledge that embrace ideas of connectivism in learning as expressed by George Siemens and others. Organisations like Educause and SoLAR (the Society for Learning Analytics Research) provide measured and objective overviews of current trends and technologies.

For the present purpose of course and unit evaluation, learning analytics provide opportunities to create feedback that does not require any additional effort on behalf of the student. Their patterns of use can form a clearer picture of how course resources are used than asking them to respond to a survey. Where analytics are open for students to see their own data as well (and they should be!), they may be surprised at the difference between what they assume they do and what they actually do.

Beyond the data that is collected passively, web2.0 technologies have accustomed learners to tagging behaviours and folksonomies, to rating and ranking systems, to likes and reblogging and the personalised customer interface of Amazon and other corporations. This low effort feedback has the potential to transform the way that learning resources are evaluated and shared. Where students can rank resources and recommend additional resources or activities, the ’wisdom of crowds' can build complex and sophisticated ’push' systems that take into account individual learner preferences and similarities to other learners in the network. This works best at a scale larger than the individual unit or even course, it opens the potential of global knowledge networks as curated and guided by teaching staff and by peers.

Learning analytics forms part of current debates of Big Data and it is worth maintaining an awareness of debates occurring in that field. The ability to measure many micro-transactions and to store and cross-reference that data promises to change the way our cities work and our societies function but it carries with it risks. This kind of power is open to abuse and these intensive surveillance systems threaten to open individuals to monitoring, fraud and political oppression.

Ethics and Learner Data Rights

The one thing that unites all successful web2.0 social networks is their voluntariness. People engage and tag and share and build because it is a voluntary activity. Learning networks need to be carefully shepherded, they need to be safe spaces for students to engage where they feel confident that their rights will be protected. Certainly students DO already participate in commercial social networks where their rights are not protected, but the public learnings spaces we build as educators must adhere to higher standards.

There are a large number of legal and ethical issues pertaining to learner data rights, as listed in the callout box. This is not the place to explore these in depth but merely to note that we cannot allow the enormous potential of learning analytic feedback to blind us to the risks we might expose students too. Dalhousie University has recently stopped using the Turnitin anti-plagiarism service, one which provides very useful data, because using the service exposes students to the risk of monitoring by foreign government intelligence services, a matter which could influence their future liberty.¹

No university data system is impenetrable and we have to be responsible about the longevity of the data we collect and the potential to identify individuals, even from aggregated data, and prejudicing them in the future. One partial solution is for universities to agree to a Charter of Learner Data Rights that promises to safeguard the liberties and academic freedoms of students.

There are other concerns about using metrics mentioned in section 4, that they are reductive and risk reinforcing stereotypes if not used carefully. Just as the idea of ethnic profiling in criminal justice has proved contentious, there is a risk that analytic data can be used in simplistic and irresponsible ways, especially where commercial vendors are promising just this kind of ’magic bullet' solution. As with any other data feedback systems, learning analytics can identify issues for further investigation, testing and triangulation of data from other sources.

Learning Analytics and Future Learning Spaces

Learning Analytics have the potential to teach use much about the learning process as a socially distributed activity and to give feedback on whether our learning systems are meeting learner needs. There is a challenge in managing and understanding these massive amounts of big data, the same concerns arise as discussed in section 2 on presenting feedback and effective data visualisation. Along with the increasing demand of learning analytic systems experts we are also experiencing a need for skilled specialists in information design. Luminaries such as Edward Tufte provide some guidance and demonstrate the importance of effective data visualisation and data presentation.

Of all the sections in this manual this is the one most likely to evolve in the short term as new innovations are presented, tested, adopted and debated. The future of learning analytics depend largely on their ability to harness student engagement, to move beyond the surveillance of the digital truant officer and to evolve into a learning companion that is personal to the individual and exists independent of institutional priorities. These future systems will have more in common with open social bookmarking systems like Learni.st than the will have of the ’creepy treehouse' of the LMS. If this can occur, then these systems might also become central to our feedback and evaluation systems as well.

Methods

• Learning Data capture
• New knowledge network

Key Points

• Big Data, the ability to measure, record and online micro-transactions, has the potential to help us understand learning and gather feedback on learning environments with no or little effort from students.
• The new generation of learning analytics will be student centred and will be based on personalised social media networking technologies including tagging, ranking and sharing.
• Learner data rights are crucial to these evolving technologies, not just because of our responsibility as educators but also because students vote with their feet and fail to embrace systems that expose them to risk. Compliance can be compelled but engagement cannot.

1http://www.cbc.ca/news/canada/nova-scotia/story/2011/08/31/ns-dalhousie-anti-plagiarism-site.html



Learning Data Capture
In a nutshell:

LMSs and other student information systems have the potential to capture a large amount of data about student learning from individuals and to aggregate trends across student populations. Unlike the small data of the past, this can be potentially be combined on a massive scale to reap the benefits of big data.

Example Questions:

Which resources did successful students utilise most heavily? For students who struggled early and later succeeded, what resources were most commonly accessed?

Which quizzes are strongly connected to final grade performance?

What are the critical time points in the semester where students begin to disengage, at least from accessing online resources?

Reporting:

Data is generally gathered within an information system and is extracted on request of the system manager.

Pros:

Provides a broad picture of what students do, at least from the point of view of resource use.

Aggregates large amounts of data automatically and allows for pattern seeking.

Cons:

Does not give a full picture of learning

There is a risk of data not being actually useful in solving problems and learning analytics becoming a gimmick

Caution:

The collection and storage of student learning data faces significant ethical and legal challenges.




New Knowledge Networks
In a nutshell:

Student facing learning analytics look to the way in which students use wider knowledge networks rather than simply examining internal LMS resources. Built on emerging web technologies, these networks will allow students to draw on the 'wisdom of the crowds' and provide recommendations for them based on others with similar profiles and learning needs.

Example Questions:

What resources did other learners find useful when answering this question?

What are the different connections that can be drawn between the key areas of knowledge in this field?

Reporting:

Networks would be de-centred but the data could be drawn on in units and courses to understand how students learn in a particular field and the ways in which knowledge is connected

Pros:

Could transform learning analytics from the 'digital truant officer' mode of surveillance to a 'learning companion' student centred mode.

These would be de-centred systems which exist independent of any one institution and would draw on the distributed labour of the networks.

Creates a new mode of benchmarking.

Cons:

New and evolving technology which is untested.

Recommendations may not be useful where the field of knowledge is small or is constructed very differently in various institutions and international contexts.

Caution:

The same caution about learner data rights raised in 'Learning Data Capture' also applies here.