Manage

Can my portfolio space be increased as I have run out?

By default the Digital Assess portfolio space is set to 180Mb. This can be increased by Digital Assess for specific user demand. Please always ensure that all files have been suitably optimised before uploading.

Latest research

Thurstone’s Law of Comparative Judgment

The science of comparative judgement began with Louis Leon Thurstone of the University of
Chicago. A pioneer of psychophysics, he proposed several ways to construct scales for
measuring sensation and other psychological properties. One of these was the Law of
comparative judgment (Thurstone, 1927a, 1927b), which defined a mathematical way of
modeling the chance that one object will ‘beat’ another in a comparison, given values for the
‘quality’ of each. This is all that is needed to construct a complete measurement system.
For more about this history of the use of this Law of Comparative Judgement in education,
please read more here.

Latest research

The national and the local: Professor Kimbell Keynote presentation at the Technology Education Research Conference (TERC2014), Sydney, Australia.

The national and the local: Conflicting requirements in the assessment of learners’
performance. Keynote presentation at the Technology Education Research Conference
(TERC2014), Sydney, Australia. Nov 2014, Professor Richard Kimbell, Goldsmiths University
of London.
“The formal assessment of performance in schools is typically undertaken by recognized authorities in assessment. In the UK this is sometimes private Awarding Organisations like
AQA or Cambridge Assessment (eg for GCSE and A level qualifications) and sometimes it isGovernment bodies like the Standards and Testing Agency (eg for National Curriculum Assessments). In either event the priorities informing these assessments will be national.The tests must be deliverable and manageable nationally and the data must produce an articulated and reliable national standard. But the vast majority of assessments in schools are done by teachers, and they typically have other, more local, classroom concerns. Of course they are interested in how they measure up nationally, but they principally want to know how they can help their individual students to improve their performance. What might they do differently in their own classroom to enhance their learners’ performance? The assessment of performance is one of those fields where technical national requirements (for reliability and standards) meet local cultural practices (of pedagogy and individuality).And the meeting is frequently uncomfortable and unsatisfactory. In this paper I will outline an approach to assessment that reconciles local cultural practices with national requirements.” –Professor Richard Kimbell Goldsmiths,University of London.

The approach Kimbell focuses on in this paper is Adaptive Comparative Judgement which provides a much more reliable means of assessing performance, allowing for holistic judgement and sorting networks than conventional marking.“Do not underestimate the significance of this. If this democratised model of construct assessment were to be adopted nationally and internationally, it would dramatically empower classroom teachers – enabling them to develop and share their constructs of quality in learners’ work. And at the same time it would equally dramatically improve the reliability of national assessments.”

Latest research

The method of Adaptive Comparative Judgement, Assessment in Education: Principles, Policy & Practice, Alastair Pollitt, 2012

Abstract
Adaptive Comparative Judgement (ACJ)is a modification of Thurstone’s method of comparative judgement that exploits the power of adaptivity, but in scoring rather than testing. Professional judgement by teachers replaces the marking of tests; a judge is asked to compare the work of two students and simply to decide which of them is the better.
From many such comparisons a measurement scale is created showing the relative quality of students’ work; this can then be referenced in familiar ways to generate test results. The judges are asked only to make a valid decision about quality, yet ACJ achieves extremely high levels of reliability, often considerably higher than practicable operational marking can achieve.

It therefore offers a radical alternative to the pursuit of reliability through detailed marking schemes. ACJ is clearly appropriate for performances like writing or art, and for complex portfolios or reports, but may be useful in other contexts too. ACJ offers a new way to involve all teachers in summative as well as formative assessment. The model provides strong statistical control to ensure quality assessment for individual students. This paper describes the theoretical basis of ACJ, and illustrates it with outcomes from some of our trials.

Alastair Pollitt(2012): The method of Adaptive Comparative Judgement, Assessment in Education: Principles, Policy & Practice,DOI:10.1080/0969594X.2012.665354

Latest research

Using Adaptive Comparative Judgement to Obtain a Highly Reliable Rank Order in Summative Assessment

Using Adaptive Comparative Judgement to Obtain a Highly Reliable Rank Order in Summative Assessment by Claire Whitehouse andAlastair Pollitt, 2012.

Abstract
Adaptive Comparative Judgment (ACJ)offers an alternative to marking,especially for performance assessments for which achievement can be difficult to describe in mark schemes. The ACJ system uses a web-browser to deliver pairs of candidates’ work to judges. The judges apply their subject expertise to select which of the two sets of work is better in terms of meeting appropriate educational objectives.

A quality parameter is estimated for each piece of candidates’ work using the Rasch logistic model to analyse the judges’ decisions. Placing the pieces of work on a measurement scale gives them a rank order. The adaptive nature of ACJ lies in the pairing of essays for each judge. The selection of essays is based on a maximum distance between the quality parameter estimates,allowing useful information to be gained from each paired comparison.

This paper reports a study in which ACJ was used to rank order a random sample of 564 essays on a topic in physical geography based on the judgements of a group of 23 teachers and examiners. The reliability of the rank order was 0.97. Evidence is presented for the need for judges to be teaching at the same qualification level as they are judging at. There is a discussion of the factors that need to be addressed before implementing ACJ in summative
assessment.

To see the full article, please click here

Latest research

Single Level Tests of KS2 Writing: the method of paired Comparative Judgement.

Pollitt A, Derrick K,
Lynch D,
(2010) Single Level Tests of KS2 Writing: the method of paired Comparative Judgement.

Latest research

e-Scape Portfolio Assessment: Phase 3 Report, 2009

Abstract

The e-scape project has progressed through phases one and two and this is the report
at the conclusion of phase 3. The e-scape concept is built on a series of innovations in
the world of performance portfolios and their assessment. Phase 1 (2004-5) was a
‘proof the concept’ in which we explored a range of hand-held tools and systems to
establish that it was possible to link them directly to web-portfolios to capture live
‘performance’. In phase 2 (2005-7) we built a prototype system for portfolio
assessment in design & technology based on learners’ performance in a 6 hour design
task. The prototype worked very effectively and in phase 3 (2007-9) we were presented
with two additional challenges:

• to establish the transferability of the system for assessments in a range of subjects other
than design & technology
• to establish the scalability of the system and its applicability to coping with and managing
national assessments.

There are two principal innovations in the e-scape system.First, we have created a system in which school le arners use hand-held digital tools in the classroom to create real-time web-portfolios. The hand-held tools are linked dynamically to their teachers’ laptop – operating as a local server. This sends a series of tasks to the learners and ‘hoovers-up’ anything that they produce in response to them. The local server is enabled to upload – dynamically (in real time) – all the data from a class/group into a website where learners’ web-portfolios emerge. In the phase3 trials learners from 19 schools have created 350 design & technology, 60 science and 60 geography portfolios.Second, we have created a web-based assessment system founded on a ‘Thurstone pairs’ model of comparative assessment.

The web-based portfolios can readily be distributed anywhere at anytime – enabling multiple judges to scrutinise the portfolios simultaneously. The judging in phase 3 involved 28 design & technology judges, 6 science and 6 geography judges. All the judging was completed on-line in a short time – window and with extraordinarily high reliability (0.95). All these data can be downloaded into existing Awarding Body back-end systems for awarding and certification purposes.
Kimbell R,Wheeler T, Stables K, Shepard T, Pollitt A, Whitehouse G.(2009) E-scape portfolio assessment: Phase 3 report Technology Education Research Unit [TERU], Goldsmiths University of London, London UK. http://www.gold.ac.uk/media/e-scape_phase3_report.pdf

Latest research

Let’s stop marking exams

Pollitt,A.(2004, September). Let’s stop marking exams. Paper presented at the IAEA Conference, Philadelphia.

Abstract
In this paper I will propose an alternative method for carrying out summative
assessment, one that seems to be intrinsically more valid than the familiar
procedure of awarding marks to lots of questions, little or large, and adding
them up to get a student’s total mark or score.

Although the new approach may seem radical it is rooted in the psychophysics
of the 1920s; practical considerations have, however, made it impossible to apply the
procedures in this way until now. I shall argue that the method brings us much closer to the essential purpose of summative assessment and that, although many concerns still need to
be addressed, there are enormous advantages to be gained.

I begin by considering the quantitative theories that are supposed to underpin the business
of certifying educational achievement.

For the full paper, please click here: http://www.camexam.co.uk/

Latest research

Creative Performance Assessed with Creative Spectacles

Presentation by Professor Richard Kimbell, Goldsmiths, University of London, at the IAEA 2009 Assessment for Creative World conference, 13 – 18 September, Brisbane.
Abstract

Creative performance lies at the heart of every school subject. Innovative investigating in
science; imaginative spatial exploration in geography; creative product development in
design. Attempting to capture these creative performances for assessment purposes has
frequently destroyed them. Just as the illusive butterfly is damaged or crushed by crude
wafting of a catch net.

Designing activities that enable us to reveal and capture creative performance has led us to
explore a mass of digital technologies – and to the evolution of ‘real-time’ portfolios that illuminate collaborative as well as innovative performance. The activities move through a
series of sub-tasks enabling learners to leave behind them an evidence-trail of their route
through the task. The performance is all captured in real-time web-portfolios.
At the assessment end of the process, creative performance has typically been ripped
apart by the application of atomistic criteria. Yet teachers know which of their students
are the really imaginative scientists; the innovative designers; the eloquent authors. When
the bits don’t add up to the right answer – confident teachers change their ‘bit’ scores to
make sure it does.

Holistic judgement circumvents this time-wasting nonsense. Criteria are used to inform
the judgement – but the judgement itself is of the whole integrated performance. To be
taken seriously in high stakes assessment, such an approach has to be especially alert to
technical challenges. With Pollitt we have created a ‘pairs engine’ that automates a
comparative pairs (Thurstone) judging process. Not only is the reliability of the assessment
extraordinarily high (0.95) but the data on individual portfolios and judges
allows us to identify and deal with any technical problems.

To see the full paper, please click here:http://www.iaea.info/documents/paper_4d7284bf.pdf

Latest research

Coursework assessment in high stakes examinations: authenticity, creativity, reliability

Kimbell RAand Pollitt A (2008) “Coursework assessment in high stakes examinations: authenticity, creativity, reliability” Third international Rasch measurement conference. Perth: Western Australia: 22nd-24th Jan 2008.