Judge evaluation
Jeff H. Snider
jeff at snider.com
Fri Oct 29 09:14:49 AKDT 2004
Several kinds of judging problems have been raised here in the past
day. I have too many thoughts on the various subjects to usefully
put in one message, and I've been unduly prolix this week, so I'll
limit myself as much as possible.
In my opinion, we have (or really ought) to have some method of
ranking judges. Two solutions present themselves to my overactive
mind, one computational and one personal.
Computationally: Collect all the scores from every round at every
contest and it's not difficult (using software written by math-minded
people) to see which judges consistently score the winners high
and losers low. It's fuzzy at first, but after a year you get
fair results. It's not a maneuver-by-maneuver comparison, but it
gives you an idea of which Sportsmen can judge well and which
Masters can't.
Collecting all the numbers, processing them, and turning the results
into some kind of meaningful individual rank as a judge is the part
that takes a strong corporate will. Think we could get every CD
all year to email in the complete round-by-round results? Make it
a requirement for event sanctioning, and it will happen (most of
the time).
I am not advocating that, just pointing out it's feasible if the
NSRCA really wanted to do it.
The more individual, personal method: Create a judge ranking system
and allow high ranked judges to move low ranked judges up the
hierarchy. The best judges are classified as 1 (one), the next as
2 (two), etc., until the lowest judges, who can pass a basic written
test on the rules, are 10 (ten). The present NSRCA judge ranking
system will pick the top guys, the 1s, 2s, and 3s, etc., and give
a rank to everyone in that system at Nats.
At contests, pair up a high judge and a low judge in each round.
After comparing the two sets of scores for an entire round, the
high judge can recommend advancing the low judge up a notch in the
scale, and talk to the lower judge about the scores, etc., in that
round. After a certain number of recommendations have accumulated,
the lower judge's classification improves. Some rules to govern
the system would be necessary, like a judge can't recommend someone's
advancement to his own level, and a high ranked judge's recommendation
counts for more than a low ranked one. Also, everyone in the country
who doesn't have a rank assigned at Nats has their rank go down by
one after it's over, to keep the ever-improving ranks in check.
It's really just about having a good judge talk to a less good judge
after each round and help him improve, and tracking who is doing
well and who needs more practice.
I am hoping I can work on my own judging skills next season. Maybe
over the winter we can swap RealFlight recordings of ourselves
flying our pattern, judge each other and talk about why we gave the
scores we did.
- Jeff Snider
- jeff at snider.com
- Northern VA, NSRCA D2
Dean Pappas writes:
> Hi Jeff and All,
> There is no end of what can be said on the subject! Yes, some of
> the best judges I have known were either not competitors, and in
> some cases not pilots, either. Your assessment is right.
>
> If you look to Ron Van Putte's comments in the next (or one of
> the next) e-mails, he correctly points out the real problem in judge
> evaluation. The following statement is made with careful consideration
> of the context ... When it comes to individual issues, the consensus
> of a large panel of judges has often wrong! All of the judge
> evaluation techniques I have seen or heard of, suffer from this
> problem. That's right, not just sometimes, but often. By and large,
> the right guy wins. We can thank the meshing (convulution) of many
> bell-shaped curves for this good fortune.
>
> closing on an overly philosophical note ... If the judges ever
> do get perfect, then the whole reason for Pattern competition will
> dissappear: since the pilots will all be flying perfectly, too.
> What would the point be, in a contest? The event would have to be
> replaced with something even more bizarre and inexplicable.
>
> Judging, and thinking about the process will make you a better flyer.
>
>
> Dean Pappas
> Sr. Design Engineer
> Kodeos Communications
> 111 Corporate Blvd.
> South Plainfield, N.J. 07080
> (908) 222-7817 phone
> (908) 222-2392 fax
> d.pappas at kodeos.com
>
>
> -----Original Message-----
> From: discussion-request at nsrca.org
> [mailto:discussion-request at nsrca.org]On Behalf Of Jeff H. Snider
> Sent: Thursday, October 28, 2004 5:58 PM
> To: discussion at nsrca.org
> Subject: Re: CD's
>
>
> Coming from outside, I'm amazed we don't have a process for ranking
> judges in this sport. It may be that a Sportsman has keen eyes and
> a good understanding of the rules, and can capably judge Masters,
> or even FAI, even though he doesn't yet have the skill to compete
> in Intermediate. It may also be that a Masters pilot lacks the
> proper temperment or attention span and can't consistently judge
> an entire round of any class.
>
> Unfortunately we don't have any idea how accurate or consistent any
> one of us is. I wonder if this is being or ever has been addressed.
>
> Lacking that information, it seems like the only thing we can do
> is to assume a pilot's class also gives a fair representation of
> his judging ability. In situations where you have too few pilots
> in neighboring classes you have to reach further afield. It seems
> obvious you can rely on some Sportsmen, but not all of them, to
> judge Intermediate pretty well. But make sure you don't have two
> untested judges working the same round.
>
> In the end, the CD's judgement is all you have to work with. If
> he looked at the sportsment present and knew none of them were ready
> to judge, perhaps the decision was for the better.
>
> -Jeff
>
> Mark Grabowski writes:
> > Well, I'm a complete pattern nobody BUT I know that as a novice or even
> > sportsman flyer I would feel very unqualified to fairly and adequately judge
> > an upper-level flyer. I'd probably want to scribe, listen and learn from the
> > "advanced" judges but certainly not want my unqualfied .02 cents worth of
> > scoring counted....FWIW
> > Mark Grabowski
> >
> > -----Original Message-----
> > From: Nat Penton [mailto:natpenton at centurytel.net]
> > Sent: Thursday, October 28, 2004 4:08 PM
> > To: discussion at nsrca.org
> > Subject: CD's
> >
> >
> > The Sulphur , La. contest was just completed with six rounds flown. There
> > were 23 fliers with 8 sportsman and 7 intermediate.
> >
> > The judge assignments were approached with the following (mis) conceptions:
> > Sportsman no judging
> > Intermediate Sports and Adv only
> >
> This placed quite a burden on the upper class judges.
> >
> > Would appreciate comments regarding criteria. TIA
> >
> >
> >
> >
> > -----------------------------------------
> > Confidentiality Note: The contents of this e-mail message and its
> > attachments, if any, are intended solely for the addressee(s) hereof. If
> > you are not the named addressee, or if you believe this message has been
> > addressed to you in error, you are requested not to read, disclose,
> > reproduce, distribute, disseminate or otherwise use this information.
> > Delivery of this message to any person other than the intended recipient(s)
> > is not intended in any way to waive confidentiality. If you have received
> > this transmission in error, please alert the sender by reply e-mail. It is
> > also requested that you immediately delete this message and its
> > attachments, if any. Thank You.
> >
> > ------_=_NextPart_001_01C4BD33.B895E6F0
> > Content-Type: text/html;
> > charset="iso-8859-1"
> >
> > <HTML>
> > <BODY>
> > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
> > <HTML><HEAD>
> > <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
> >
> >
> > <META content="MSHTML 6.00.2800.1458" name=GENERATOR>
> > <STYLE></STYLE>
> > </HEAD>
> > <BODY bgColor=#ffffff>
> > <DIV><SPAN class=968372021-28102004><FONT face=Arial color=#0000ff size=2>Well,
> > I'm a complete pattern nobody BUT I know that as a novice or even sportsman
> > flyer I would feel very unqualified to fairly and adequately judge an
> > upper-level flyer. I'd probably want to scribe, listen and learn from the
> > "advanced" judges but certainly not want my unqualfied .02 cents worth of
> > scoring counted....FWIW</FONT></SPAN></DIV>
> > <DIV><SPAN class=968372021-28102004><FONT face=Arial color=#0000ff size=2>Mark
> > Grabowski</FONT></SPAN></DIV>
> > <BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
> > <DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
> > size=2>-----Original Message-----<BR><B>From:</B> Nat Penton
> > [mailto:natpenton at centurytel.net]<BR><B>Sent:</B> Thursday, October 28, 2004
> > 4:08 PM<BR><B>To:</B> discussion at nsrca.org<BR><B>Subject:</B>
> > CD's<BR><BR></FONT></DIV>
> > <DIV><FONT face=Arial size=2>The Sulphur , La. contest was just completed with
> > six rounds flown. There were 23 fliers with 8 sportsman and 7
> > intermediate.</FONT></DIV>
> > <DIV><FONT face=Arial size=2></FONT> </DIV>
> > <DIV><FONT face=Arial size=2>The judge assignments were approached with the
> > following (mis) conceptions:</FONT></DIV>
> > <DIV><FONT face=Arial size=2>
> > Sportsman no
> > judging</FONT></DIV>
> > <DIV><FONT face=Arial
> > size=2> Intermediate
> > Sports and Adv only</FONT></DIV>
> > <DIV><FONT face=Arial size=2></FONT> </DIV>
> > <DIV><FONT face=Arial size=2>This placed quite a burden on the upper class
> > judges.</FONT></DIV>
> > <DIV><FONT face=Arial size=2></FONT> </DIV>
> > <DIV><FONT face=Arial size=2>Would appreciate comments regarding criteria.
> > TIA</FONT></DIV></BLOCKQUOTE></BODY></HTML>
> >
> >
> > <P><hr size=1></P>
> > <P><STRONG>
> > Confidentiality Note: The contents of this e-mail message and its
> > attachments, if any, are intended solely for the addressee(s) hereof. If
> > you are not the named addressee, or if you believe this message has been
> > addressed to you in error, you are requested not to read, disclose,
> > reproduce, distribute, disseminate or otherwise use this information.
> > Delivery of this message to any person other than the intended recipient(s)
> > is not intended in any way to waive confidentiality. If you have received
> > this transmission in error, please alert the sender by reply e-mail. It is
> > also requested that you immediately delete this message and its
> > attachments, if any. Thank You.
> > </STRONG></P>
> > </BODY>
> > </HTML>
> >
> > ------_=_NextPart_001_01C4BD33.B895E6F0--
> >
> =====================================
> # To be removed from this list, go to http://www.nsrca.org/discussionA.htm
> and follow the instructions.
>
> # To be removed from this list, go to http://www.nsrca.org/discussionA.htm
> and follow the instructions.
>
=====================================
# To be removed from this list, go to http://www.nsrca.org/discussionA.htm
and follow the instructions.
More information about the NSRCA-discussion
mailing list