Performance Judging? (back to original discussion)

[Ed Deaver]

The interesting thing to this thought is what happened to me last year at the Nats. 
 
On my 1.5 snap, the nose didn't pitch(not quite sure why but the plane didn't stall as it were)  I was given a 0, 7, 8.
 
OK, so the zero was correct, and I concurr, however the average or initial assessment is the zero judge missed the call, when in  fact the 7/8 judges missed the call.
 
This is why, averaging judges to rate them, or any system is fallable because there is no reference as to who is correct.  If we had a scale that weighed to a micro gram 5 lbs, and could calibrate any scale to that, it would be different but we don't.
 
As stated, judges are human, and just because someone gives a low score doesn't make them incorrect.
 
ed

Rick Wallace <rickwallace45 at hotmail.com> wrote:
(Back to Jim's original line of discussion...) 
 
How does one compare judging results when (as in most local contests) there are only 2 judges per round/class? When there's a spread, which judge is right? In a world where the judges oare often the 'best available' rather than ones who meet a standard, is there any value in such a comparison? 
 
For that matter, when there are three judges and one's score is significantly different from another, do we assume (with or without knowing the identities of the judges concerned) that the one 'different' judge is wrong? If so, why?
 
Is it more or less significant when one judge's score is different than the others' in ONE maneuver rather than across the entire flight? 
 
Eric has made great points in his points on this topic - especially about the fact that two judges will focus on different aspects of a maneuver, but frequently give very similar scores - for overtly different reasons. 
 
FWIW, some personal info on scores at the recent Nats (from a first year Masters Guy who finished REAL close to the bottom of the pack - and deserved to)
- One of my rounds had a spread across the three judges of more than 110 points. (Maybe some non-zero 'mercy' scores here? who knows?) 
- As a first time FAI Finals judge, one of the fnalists volunteered the information that his score spread for the round I judged was 11 or 12 points across 5 judges... 


I'm not entirely comfortable with the idea of comparing the scores of a panel of judges -- without knowing their background, flying experience, judging experience, etc - and assuming that the 'low' guy is automatically wrong.  -- especially if this practice is institutionalized, and is recorded as 'facts' in the files of some or all of the judges concerned. 
 
It seems that recollections of my college statistics courses reminds me that this line of thinking will result not in recognition of the best absolute standard, but rather in creating a de facto standard of mediocrity... 
 
My gut says that the 'right' answer, unfortunately, is 
   - A REAL set of standards about what the rules say and mean - and Don seems well on the way to getting that done
   - An effective classroom instruction plan that makes those rules clear (and again, Don has done a great job in his 2005 Judge Certification packages for students and instructors) 
   - A REAL judge teaching program -that would involve practical exercise / live judging followed by discussion after each maneuver (lots of fuel burned there!). This is the part that seems to need the most progress. 
 
Understand the pain of driving hundreds of miles and not getting reasonable feedback on one's flights... especially in a world where we, the pilots, are judging each others' performance...   
 
just my $.02... 
Rick

>From: Jim_Woodward at beaerospace.com
>Reply-To: discussion at nsrca.org
>To: discussion at nsrca.org
>Subject: Performance Judging?  Trial Balloon
>Date: Tue, 2 Aug 2005 08:36:49 -0400
>
>Trial Balloon Email (beware) :)
>
>Hi All,
>
>I'm going to propose a different way of looking at judging.  Of course,
>the pilot is the "performer" that is on stage for all to see, however, at
>the same time the judge(s) are performing as well.  At the US Nationals,
>the judging committee is calculating persons judging marks, and continuing
>to rank the judges.  Thus, the judges are performing at the same time as
>the pilot.  However, we typically do not ever get to see how the
>performance of the judge ranks against the pilot, or peers in an open
>forum.  The current atmosphere is sort of  hush quiet, or a "don't tell"
>type in regards to what judge gave what score.  Almost as if, its
>considered good pilot etiquette not to ask judging questions.  Many judges
>do not like to be approached after a round to discuss scoring.  Instead of
>this, we could turn the tables completely 180 degree around.
>
>Judging could be made to be a completely open from start to finish of the
>contest.  Perhaps within the scoring system, after each round, judges
>scores for all pilots are posted (tear sheets essentially) at the same
>time as the round postings.  The posting of side-by-side scores, could
>become a POWERFUL training tool for younger pilots (judges).  Currently,
>there is no award for the "performance" of judging.    Flyers go to a
>contest to fly, yet the task of judging takes double or more of the
>combined numbers of people than contestants (at least in total effort).
>
>If we acknowledge that in the current local contest and Nationals setting,
>there is already volumes of discussion going on between pilots regarding
>scoring (is there any bigger topic with 90% of the pilots?), we can then
>foster an environment which turns what is currently ambivalent or negative
>judge critiquing, into one in which takes these volumes of discussion and
>focuses on "constructive" or "objective" results - results beyond simply
>complaining amongst each other (avoid the misery loves company syndrome).
>
>Doing something like this will address two important issues.  1.) The
>contestant should feel more "in-tune" with scoring, and perhaps use this
>feedback to better their own flying and judging.  2.)  The judge should
>feel satisfied in knowing at the end of a round, whether or not his/her
>"calibration" is more or less correct (I'm tending to think that within a
>few posted rounds of these scores, judges will "self-correct" any trends
>without asking if such a trend has become obvious).  Individually, I hope
>each judge feels compelled to "talk" to pilots about the round if asked.
>
>Finally, recognizing that judging is an important criteria, I recommend a
>"District Award" for the top 1,2,3 persons who performed the most amount
>of judging in the district (or most accurate if possible to calculate).
>Also possibly a District award for the "team" that performed the most
>contest scoring.
>
>Key thoughts:
>1.  Contestants are more in-tune with total system of contest
>running/scoring, etc.
>2.  Judges (or CD's???)  offered an avenue to "self-correct" if & when
>necessary (... could happen to anybody)
>3.  District awards for the massive judging/scoring effort already taking
>place.
>4.  Re-focus the INCREDIBLE VOLUMES of discussion already taking place
>regarding scoring into a beneficial training event
>5.  Possibility:  Upon the conclusion of Saturdays competition, the CD
>could offer an open-discussion or review of the days scoring and judging -
>read as, a quick pilots meeting that gives folks the open forum to discuss
>anything - which can also be viewed as mini-judge-training-event.
>
>Anyway - just thinking of ways to take the effort that is already going
>on, and refocusing into beneficial paths.  Definitely open to other ideas
>and suggestions.
>
>Jim W.
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.f3a.us/pipermail/nsrca-discussion/attachments/20050804/d09e03bc/attachment.html