TTN - 21st Century Learner


Limerick Case Study

on Monday, February 15th, 2010


Our work with Limerick University is a great example of how Comparative Judgement assessments can be used for teachers to assess each other.

Design and Technology student teachers were given an open ended project to create design projects which they might use in their practice once qualified. The projects were competed over a number of weeks with each teacher encourgaed to create a number of learner journal entries along the way. These posts began as brain storms of initial ideas or photographs of early inspiration alongside their reflections on the same and as their work developed evidenced their progress. This might include photographs of their projects in development, video of the manufacture process of audio reflections of their progress to date.

Teachers were also able to use mobile phones to capture their progress without needing internet access or installing any special software. The MAPS system allows the teachers to register a mobile phone number against their eportfolio. When an opportunity arose to video themselves completing a task or perhaps reflecting in audio on their progress they were able to use their mobile phone devices to digitally capture that evidence and then MMS or SMS the evidence to their portfolio. This means that evidence capture can be 'live', as it happens, rather than in retrospect as would traditionally be the case with eportfolio building.

Once the diary entries were created MAPS then allows the entries to be tagged against the assessment activity. By doing this the teachers automatically created story board portfolios of their learning process.

34382Having created the story board portfolios assessment was then completed by the teachers themselves using the Comparative Judgement method. In this approach all of the portfolios are entered into the Comparative Judgement engine. When teachers then log into the system they are shown pairs of portfolios that are chosen from those available using an algorithm derived from the Law of Comparative Judgement, a branch of mathematics created by Thurstone in the 1920s.

As the teachers make their judgements this in turn determines which pair of portfolios are shown to the next judge. In this way the system is able to refine the pairs that are shown to judges to improve the scaled rank which the approach produces.

At the end of the process the system produces a scaled rank order of all their work. This scaling has an internal reliability upwards from 0.95 which in assessment terms is unheard of. By exposing each other to the comparative approach and without reference to any criteria the teachers are exposed to the creative value in the process illustrated by each of the portfolios they judge.

Posting by: TTN - 21st Century Learner, at 2:45 PM