The fundamental question asked by any evaluation system is this:
Does this enhance student learning?
This question is notoriously difficult to answer, partially because learning is an individual, multifaceted and complex experience with multiple goals of learning are but also because learning is difficult to measure. We DO quantify learning through assessment, that is we condense a variety of learning measurement techniques into a single percentage grade, but this is a representation of learning not a real metric measure of what a student has learnt. Learning, from the institutional point of view, is always filtered through the assessment process.
Like student grades, there are a number of different facts and figures produced and collected by university and this section considers the impact these have as a form of feedback in the evaluation process. The recurrent distinction between qualitative and quantitative data is relevant here as it is in most research. Quantitative data is easier to produce and represent, it is amenable to graphs and other forms of visualisation, but it can also have a dangerously seductive character that can be misleading and seem more objective and impartial than its foundations deserve.
Any evaluation system will draw on both qualitative and quantitative data as it triangulates knowledge and uses different feedback methods to support or challenge the information that is gathered. This section is primarily concerned with the traditional sorts of data that universities gather – grades, completions, demographics, rather than the ’big data' promised by learning analytics which is discussed in section 11.
Grades as Feedback
Grades give feedback to students but an overall analysis of grade patterns can also give a picture of how well a course or unit is performing and can map trends over time, identifying hot spots which require attention and changes which need to be carefully examined.
Many forms of data sit uncomfortably on the qualitative/quantitative divide. Surveys for example are often presented as quantitative data when they are really a representation of qualitative data (student opinion), a matter which is further explored in section 5. Similarly, grades have their origins in a qualitative assessment of learning obtained through the assessment, a form of academic judgement, yet they are usually expressed as a simply numerical value.
Grades are a quantification of qualitative data. If this distinction seems too pedantic, remember that there is a danger of reification of such values, mistaking a representation of a concept for the true concept itself. If we forget that grades are a symbolic representation of learning, derived through human-designed measurement activities, then we can end up ignoring the role of the learning environment in producing those grades and therefore miss the systemic factors which contribute to it.
One example of this problem is that a simple percentage grade can reinforce the idea of learning as content transferral, that student learning can be read like a fuel gauge, how ’full' it is being determined by how much course content a student retained and can recall through examination. Recently the focus on student learning outcomes rather than content delivery, goes some way to break up the monolithic nature of grades providing different scales of success beyond a simple numeric metric. These have the potential to break the reductive ’tyranny of the GPA' and provide some texture to understanding student success. They will also require innovative means to measure and visualise.
While the grading process involves the representation of learning, once grades have been allocated they become a fact in themselves. A student's progress will be influenced by these grades, as will their career prospects. When connected to completion, progress and retention we can see grades as quantitative facts with direct bearing on how well a system is functioning.
Student Progress Data
Student progress is an essential measure of learning and it includes a number of different factors such as:
Attrition – students who do not complete a unit or course
Success – students who pass a unit or complete enough units to finish a course
Degree of success – students who earn higher grades or course accolades
Persistence – students' ability to complete over time, even if they need to take units multiple times
We have to be careful when using progress data that we are clear as to what kind of data we are actually describing. We also must make sure that data is weighed appropriately as student progress can be influenced by a range of personal and financial factors that may be distinct from the educational environment itself. In some situations a failure to progress may be a successful outcome for a student who has selected the wrong course or for whom it is not the right time to study. Focus on the institutional perspective on progress as success (and the budgetary consequences) may create tension with the student perspective.
Progress data can identify bottlenecks within a course, units that have high failure rates and which cause further problems upstream as students are forced to depart from the model course plans. Where units are only offered in some semesters this can have a cascading effect as students are compelled to take units out of the sequence which the course plan would otherwise help us understand. The data can help in the scheduling of units, particularly in the organisation of pre-requisites, the offering of electives or provision of ’catch-up intensives' where some units have high failure rates.
Demographic Data
Demographic data can provide a deeper understanding of student learning, particularly where it is indexed against other forms of data collected. This does come at a risk as an over-reliance on demographic data can lead to a simplistic form of profiling where students are stereotyped according to a personal attribute that may not influence their learning. Corridor discussions about ’most international students' or ’most distance students' have no place in an objective evaluation process, nor for that matter in the corridors (but we know they occur).
Demographic data is very useful in challenging assumptions and received wisdom. For instance prejudices that students following TAFE pathways or from a low SES background will not be as successful may be challenged by demonstrating that these students perform equally to others. As with any form of quantitative research, we have to be very careful and make sure we check what we presume to be facts.
Methods
- Grades analysis
- Student ability testing
- Retention & progress analysis
- Demographic data analysis
- Course patterns
- Resource usage data
Key Points
- Define your questions carefully and make sure you pick the right data set for each question
- Look out for common data errors, for biases within the methods of collection. Always consider the methodology carefully and do not presume impartiality.
- Not all statistics are quantitative, although some qualitative data can be quantified, and represented by metrics.
- Regard debates about ’grade inflation' with healthy scepticism.
Grades Analysis
In a nutshell:
Innovation ought to show results in the patterns of grades that students achieve. We should not expect continual overall growth, but we should see gains in areas that have been targeted for improvement such as a reduction in fails or growth in students eligible for honours programs.
Example Questions:
Have the large number of fails in this unit been reduced?
In a course, which units act as bottlenecks on progression?
Which majors produce the most honours candidates?
Reporting:
Each semester results are overseen by a chief examiner's board (or similarly named committee) which may have further input in identifying areas of concern. Academics differ in opinion as to whether overall grades statistics ought to be published further.
Pros:
The primary measure of success for students. Important for transparency and comparison of outcomes across a course.
Cons:
Can focus too much on a single bottom-line and standardisation, some units will have different profiles than others.
Caution:
Analysis is strongly dependent on criterion-referenced assessment and effective moderation techniques. See the main text for concerns about grade inflation which may be misplaced.
Student Ability Testing
In a nutshell:
Students are sometimes required to undertake skills testing independent of the program of assessment, for instance to demonstrate minimum language skills or competency with a piece of equipment or policy framework. These tests have formative value and are sometimes connected to summative assessment. Testing patterns also have feedback value for a course where these skills are used and developed.
Example Questions:
Are students able to express themselves according to external standards testing measures?
Are students qualified to use the lab equipment they need for this unit?
Can students earn certificate qualifications parallel to their unit of study?
Reporting:
Often tests are administered or at least specified by external bodies. They can be of internal design as well.
Pros:
Provides information on the broader set of skills students are developing in parallel to academic skills.
Cons:
Places more obligations on students.
Caution:
Parallel certification can be very useful for students, especially if they do not complete a full degree program, but you would need to be careful that certification is recognised and desired.
Retention and Progress Analysis
In a nutshell:
Analysis of the patterns in student progress, particularly course and unit completion. Retention can be calculated in a number of different ways, looking at course completion, withdrawals and transfers.
Example Questions:
What percentage of students completed each course?
Where are the key points in a course that students withdraw?
Where students withdraw from a course, do they then transfer to another course?
Reporting:
Records are kept centrally and are reported to government.
Pros:
A central figure of student success.
Cons:
The figures can be trickier to compare than they first seem. Full time and part time completions may not compare directly, study interruptions and non-traditional enrolment can be difficult to incorporate.
Caution:
A large number of withdrawals are the result of poor course selection, so effective and timely course transfer mechanism may aid a student in moving to a more suitable program.
Demographic Data Analysis
In a nutshell:
Personal details of students including age, gender, ethnicity, socio-economic status (derived from post code and ABS tables), previous study, employment, distance travelled to campus, study mode,
Example Questions:
Are the course results unevenly distributed over one section of students?
Does the distance that students travel or their work status impact on completion?
Does this course have atypical enrolments given the demographics of the students?
Reporting:
Generally these statistics are kept by central university units but may not be readily accessible.
Pros:
Of themselves not necessarily very useful, but can be useful in comparison to other data such as grades or completion rates where the common problems that classes of students face can be investigated.
Cons:
Can be very reductive and requires further feedback methods to understand the reasons for unusual patterns. Publishing demographics may tend to reinforce pre-conceptions, particularly around gender stereotypes.
Caution:
Too much reliance on data alone can lead to stereotype profiling and treating students as members of a class rather than as individuals when designing intervention strategies.
Course Patterns
In a nutshell:
Rather than focussing on completions, analysis of course patterns looks at the ways in which students pass through a course including choice of enrolments, repeats of units failed, paths taken, analysis of electives and majors/minors. Simple completion rates may be misleading because of the diverse ways in which student’s progress through a course.
Example Questions:
How often do students interrupt their study in this course, does that impact on chances of final completion?
Where do pre-requisites create bottlenecks or cut students off from elective choices?
What are the most popular electives in this course, does this popularity depend on which semester the elective is offered?
Reporting:
Course pattern analysis is generally ad hoc and much of the knowledge is informally gathered and transmitted. This does not mean that more formal and open methods of analysis cannot be developed and reported back to academics and administrators.
Pros:
Important for a thorough understanding of how a course functions in reality rather than how it looks on paper.
Cons:
Can be very complex and difficult to map. No real accepted standard method of analysis.
Caution:
A great course design can be undermined by problems in timetabling or other circumstances that may not even be visible to course managers and academics.
Resource Usage Data
In a nutshell:
Basic analytics provide data on rates of access to university resources such as the LMS, library and to a lesser extent of physical access to rooms and laboratories. These provide basic patterns of engagement over time.
Example Questions:
Which are the most utilised resources? Which are the least?
Which usage statistics correlate to student progress and success?
At what points in the semester do students tend to reduce their use of learning resources? Does this relate to other events, such as the due dates for assignments?
Reporting:
This data is generally kept internally to the resource system, so the unit coordinator generally has access to LMS data and administrators control other resource systems. There is generally a process of reporting up but this can be selective and difficult to access.
Pros:
Can provide a rough picture of engagement and disengagement, particularly looking for time based triggers. Can be effective in isolating resources that students do not use and then evaluating the reasons why and possible need for an alternative.
Cons:
Data can be very misleading, rates of access to a reading resource may not measure how many students have actually read it nor how many have understood it.
Caution:
Universities are moving toward use of big data learning analytics (see chapter 11) which have the potential to build on the 'small data' of this sort of data by looking for patterns in much larger data sets.
Quality of data can be improved by including web2.0 style rating systems where students give feedback on the usefulness of resources.
No comments:
Post a Comment