<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.6000.16546" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT face=Arial>Some more thoughts on the variation between judges.
</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>The NSRCA Judge Cert Program is an excellent base for
ensuring that everyone is on the same page regarding the rules as applied to
judging. The Program suggests practice judging sessions of actual flights with
subsequent discussion of the scoring. Unfortunately, only a few Cert Classes
actually do this. (The logistics of practice scoring sessions are difficult with
most classes occurring in the off season.) This is sort of analogous to studying
a sequence in the rule book, but not flying it until in competition. We are now
practicing "on the job" training in many cases and this isn't fair to the
competitor. </FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>We need to find a way to better train / calibrate judges
outside of competition. Flying sessions during Certification, where scores are
discussed by maneuver within a peer group would be a very good start. Several
flights are generally flown during WC judges practice and scores are discussed.
We've done this at our Team Selections in the past. Unfortunately, we haven't
incorporated this practice into the Nats. We fly judge warm-up flights before
the Nats finals, but these are not for judge calibration. (At major events any
such flying for judges practice requires flights by non-competitors which adds
to the logistics.) We do little to none of this at local meets. The idea of
pre-contest judging practice has merit. Often the sun precludes using the entire
box, but several will practice at "off-box" angles and parts of sequences. Why
not judge these flights and discuss the scores and reasons for downgrades?
Probably best to not make these scores / discussions available to the pilot
in competition - that's better left for training events.</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>It would be nice to have some form of judge ranking system
other than the limited ranking done for nomination of WC judges.
Unfortunately, this is difficult to define and operate. The experienced based
system used by the old USPJA was mostly without merit. When volunteers for Team
Selection judges numbered in the 30's, the program participants voted
for the judges that were used. That may work when a lot of
reasonably qualified folks are available. One thing is for sure, presently it's
hard to find enough warm bodies to fill the judging chairs.</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>TBL and other forms of massaging the scoring data are
fine, useful, and often necessary. However, they are post processing exercises
to mathematically minimize the effects of inaccuracies in the actual scoring.
It's much better to strive to ensure that the initial score is
correct.</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>Let's look at some common forms of judging variations (I
apologize in advance if I step on any toes). I suggest that there are two
categories, those that are wrong and intolerable vs. those that are differences
of opinion.</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>In the first category we find the judge who observes no
defects in a maneuver and scores it an 8 so as to have more room if something
else is more appealing. Or the judge who sees a defect and scores a 10 because
he gave the pilot the benefit of doubt. Or the judge who "overlooks" a major
error because the rest of the flight is great. Or the judge that overlooks
excessive distance because that's where he/she flies. Or the judge who fails to
watch the maneuver from start to finish - including the exit. Or the judge who
downgrades maneuvers for having different roll rates or radii than he/she
prefers. Or the judge who recognizes he/she is more strict / lenient in a group
of judges where scoring analysis will be applied and changes his/her practice.
Or the judge who simply "likes" one pilot more than another and ensures the
favorite scores best. There are other examples, but the best correction for this
may be a cattle prod! Anyone guilty of this needs to seriously consider their
behavior!</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>The second category is (thankfully) more prevalent. Two
judges observe a difference in radii - one deducts a point and the other two.
Likewise, line length before and after roll elements, or changes in roll rate,
or heading, or angle, or distance, or? Given the difficulty of
determining these criteria visually, there will always be some
difference in judgment of the error magnitude. One judge will look
tough (we rarely consider a judge easy - unless he/she's judging our
competition), but may actually be the most accurate. There will still be some
difference in the judges scores, scoring practice in training
sessions would go a long way toward minimizing these
differences.</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial>Earl</FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial></FONT> </DIV></BODY></HTML>