Judge evaluation

[Jeff H. Snider]

Doug Cronkhite writes:
> That hypothesis assumes that all judges who score high are correct. I have
> often been the low score on a judging line and been the correct judge.

The "Computational" hypothesis assumes that a good judge tends to
pick the winners accurately.  He should score the winners high, and
the losers low.  Someone who regularly scores everyone high will
be flagged as a judge of lesser skill than someone who scores more
in line with what hundreds of other judges have done over the season.
Of course winners sometimes turn in poor flights, and losers sometimes
nail them.  That's why this is a year long statistical effort, and
it can't address the fact that we're comparing you to other judges,
who may be wronger than you.  The hope is that over the course of
a year the wrong judgements disappear in the bell curve.

However, this does lead to a problem I've seen in other sports which
require a judgement call on the part of the referee: Having a good
reputation earns you more points.  If a judge knows the pilot on the
line regularly wins contests, it's in his interest to give him good
scores.

That's one reason I don't actually believe the computational method
is the better one of the two I put forward.

 - Jeff Snider
 - jeff at snider.com
 - Northern VA, NSRCA D2
=====================================
# To be removed from this list, go to http://www.nsrca.org/discussionA.htm
and follow the instructions.