[NSRCA-discussion] D3 Championship - Scoring System Overhaul

[Mark Atwood]

George and I spent some considerable email on this in the past, but aren¹t
we confusing the statistical relevance of saying ³Pilot A is better than
Pilot B² (which clearly there is not enough math to support) vs ³Pilot A was
scored better than Pilot B today in round 1²??

The quality of the score...is a completely independent issue from whether or
not the score itself is valid.  My ability to judge the same maneuver as an
7 every time is one component (and a challenging one to say the least), not
to mention whether or not the ³7² is the correct score.    But the math that
says that 7,7,7,7,7 loses to 7,7,8,7,7 is pretty sound.  Is the second guy a
better flyer?  Statistically no...they¹re equal.  But in that round he
scored better.  Period.  At the end of the day, this is a subjective sport.
Our objective guidelines are there to help with consistency, but bottom
line, presentation IS part of the program...so by definition, the guy I (the
judge) think won...won.

Maybe we need to do some google-ing on Olympic scoring...clearly they have
the same issue with a LOT more people watching....

That said...I¹m all in favor of reviewing everything Jim has brought to the
table.  

-Mark


On 10/18/07 10:43 AM, "vicenterc at comcast.net" <vicenterc at comcast.net> wrote:

> I agree with George.  If I remember right from statistics courses 20 years
> ago, this type of problem follows the normal distribution or bell shape curve.
> In order to have any significant precision in the scoring system we will need
> to have at least 33 judges per round.  We all know that  it is impossible for
> us to have 33 judges per round.  I also agree with George that at the end of
> the contest the winner is usually the best pilot and if you ask around very
> high percentage will agree with the results.
>  
> Well, we need a PHD in statistics to help us out.
>  
> --
> Vicente "Vince" Bortone
>  
>> -------------- Original message --------------
>> From: <glmiller3 at suddenlink.net>
>> 
>>> > I've said this before and most people don't "get it", but we are asking
>>> more 
>>> > precision from our scoring system than is mathematically possible.
>>> > 
>>> > In the case of FAI scores, the raw scores are rounded to one significant
>>> digit 
>>> > (whole numbers only from 1-10). This means that after any mathematical
>>> > manipulation, the result is only "Accurate" to one digit and the result
>>> should 
>>> > be rounded to that digit using some standardized "rounding algorithm"....
>>> We are 
>>> > manipulating a single digit of significance and basing outcomes on up to
>>> EIGHT 
>>> > digits (1234.5678 ). Mathematically speaking any "normalized score"
>>> between 
>>> > 950.0000 and 1000.0000 is the same score.....900 to 949; 850-900; etc. All
>>> > other problems of judgin! g incon sistency, bias, averaging, etc pale in
>>> > comparison. 
>>> > 
>>> > As I've said before, I'm astounded that a system that is so mathematically
>>> > flawed can provide results that are as good as they are....for my part, I
>>> > usually feel like I have been ranked pretty fairly compared to the other
>>> pilots 
>>> > in my class even though the ranking is based on statistically meaningless
>>> > numbers. 
>>> > 
>>> > George 
>>> > 
>>> > 
>>> > ---- "Woodward wrote:
>>>> > > Guys - 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 1. Why do we average judge's scores together?
>>>> > > 
>>>> > > a. The whole system is predicated on judges being
>>>> > > competent/consistent/correct/un-biased. If this first rule is violated
>>>> > > ranging from small to 100 point raw score difference, the idea of now
>>>> > > "averaging" the score also has no validity.
>>>> > > b. Why not just let each judge's score stand as! is, > > unaltered, and
>>>> produce two normalized scores per round?
>>>> > > c. Averaging the scores together may be doing a disservice
>>>> > > to everyone.
>>>> > > d. You would basically have two sets of scores per round.
>>>> > > IE you may end up with 1000 points on one card, and an 800 on the
>>>> other. 
>>>> > > 
>>>> > > e. This would identify immediately any cause for concern.
>>>> > > f. Would it provide an immediate training tool back to our
>>>> > > system of pilots, judges, and CD?
>>>> > > g. Attempting to "un-average" the scores to determine what
>>>> > > happened takes place anyway on the flight line, after the scores are
>>>> > > printed. 
>>>> > > 
>>>> > > 2. Are we asking too much from judges?
>>>> > > 
>>>> > > a. Is applying downgrades, then counting backwards from 10,
>>>> > > in the context of "turnaround" pattern where maneuvers can happen
>>>> > > back-to-back quickly, too d! ifficul t across the full spectrum of
>>>> > > competitor/judges?
>>>> > > 
>>>> > > 3. Dropping Rounds:
>>>> > > 
>>>> > > a. Is this still a good idea?
>>>> > > b. I wouldn't mind dropping one round, but it was explained
>>>> > > to me last night that this is an artifact from the days of when people
>>>> > > would break a prop on touch-n-goes, and in general lower equipment
>>>> > > reliability.
>>>> > > c. In the age of higher equipment reliability, is the
>>>> > > 'round-drop' scenarios still good, left as is?
>>>> > > 
>>>> > > 4. Dropping Rounds - Take 2:
>>>> > > 
>>>> > > a. In the context of point #1, maybe we should be allowed
>>>> > > to drop the lowest scored judge from each round, versus the entire
>>>> > > round. 
>>>> > > b. Why should the pilot drop the entire round, when one
>>>> > > judge may have scored him 1000 points, and the other 800?
>>>> > > c. If you end up with a! tie at the end, you just keep
>>>> > > counting "1000's" until the other pilot runs out - tie is now "untied."
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I hope some smart guys can chime in on potential over-hauling idea.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > ________________________________
>>>> > > 
>>>> > > From: nsrca-discussion-bounces at lists.nsrca.org
>>>> > > [mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Lisa &
>>>> > > Larry 
>>>> > > Sent: Wednesday, October 17, 2007 10:27 PM
>>>> > > To: 'NSRCA Mailing List'
>>>> > > Subject: Re: [NSRCA-discussion] D3 Championship
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > OK...I probably shouldn't start this, but I will...
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I haven't read all the threads, but I have read the ones below
>>>> > > in this series.
>>>> > > 
>>>> > > 
>>>> > >! ; 
>> gt; > The NSRCA has already set the standard and a method to determine
>>>> > > judging bias and has held a NSRCA member accountable to this standard
>>>> > > this year and the AMA sanctioned the individual. This is fact....Agree
>>>> > > with the method to determine bias (or not) it was used to impose an AMA
>>>> > > sanction on a member.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > IMHO this discussion suggests that bias has occurred in the D3
>>>> > > championship or possibly another at the same level FAI. If this is the
>>>> > > case the NSRCA must review this and apply the same discipline using the
>>>> > > same measurables to provide for the same sanctions.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > If the NSRCA is unwilling to investigate or isn't willing to use
>>>> > > the same method to determine bias, then clearly we (the NSRCA and AMA)
>>>> > > have disenfranchised a NSRCA member and should r! ethink his sanction.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > Our rules and penalties must check and balance. Then they must
>>>> > > be applied to all members equally regardless of status in membership.
>>>> > > This is the only way to reduce / eliminate bias. I'm also unwilling to
>>>> > > entertain the thought the District Championship is any less important
to 
>>>> > > the NATS. They are both sanctioned contests ran by a CD accountable to
>>>> > > the AMA. 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > Flame suit on...
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > Larry Diamond
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > ----- Original Message -----
>>>> > > 
>>>> > > From: Mike Hester
>>>> > > 
>>>> > > To: NSRCA Mailing List
>>>> > > 
>>>> > > 
>>>> > > Sent: Wednesday, October 17, 2007 2:24 PM
>>>> > > ! 
>>> > ; > Subject: Re: [NSRCA-discussion] D3 Championship
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > He's not alone. Although he probably should work on the delivery
>>>> > > ;) 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I would support any of the 5 proposals that Ryan listed. Judging
>>>> > > FAI can be frustrating enough, but to be told you're not getting it
>>>> > > right when you're already doing everything you know how to do, that's a
>>>> > > hard pill to swallow regardless of the statement's accuracy.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > You guys out there do need to realize these guys can fly...and
>>>> > > are very good...problem is they're flying against this Jason dude,
>>>> > > travels a lot, flys all the time, might even have a national title or 2
>>>> > > along the way, not sure. I'm sure you know the type. *ahem*
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > Because my wife ! general ly keeps scores in D3, we have some
>>>> > > pretty good access to each and every score entered. I can tell you guys
>>>> > > without a doubt at times there are some SERIOUS differences in scores
>>>> > > between judges on the same round. I don't mean a little, I mean like
>>>> 100 
>>>> > > points on the RAW score. Even if this Jason character was flying
>>>> > > straight 10s, the differences if you work them out mean the others are
>>>> > > barely flying a straight line....and that's not the case. I have no
>>>> > > doubt these guys don't think they should be beating Jason in a 6 round
>>>> > > contest where 2 of the rounds are "F" rounds, but I am sure most people
>>>> > > would agree the scoring could use some improvements.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > Being one of these evil incompetent D3 masters judges *ahem* I
>>>> > > would certainly support more of a cooperative effort than some kind of
>>> >! > p rotest. I have been very supportive of all the FAI guys and
>>> especially 
>>>> > > the scoring, and am usually the guys everybody throws something at
>>>> > > during a judging seminar because I'm trying to clarify something that
>>>> > > effects mainly FAI.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I think to identify the "problem" will take a willingness to
>>>> > > recognize that the situation is caused by a LOT of factors, not any one
>>>> > > or two. If anyone's interested, I'll outline the ones I see clearly.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I'm not sure if this will all have the intended effect that jim
>>>> > > was looking for in the end, but if nothing else it does draw some
>>>> > > attention to a situation and we should have a closer look.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > As for me, soon I'll be practicing, bracing for the onslought of
>>>> > > FAI pilots come to master! s to pu nish me =)
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > -Mike 
>>>> > > 
>>>> > > ----- Original Message -----
>>>> > > 
>>>> > > From: McLaughlin, Ryan (FRS.JAX)
>>>> > > 
>>>> > > 
>>>> > > To: NSRCA Mailing List
>>>> > > 
>>>> > > 
>>>> > > Sent: Wednesday, October 17, 2007 1:50 PM
>>>> > > 
>>>> > > Subject: Re: [NSRCA-discussion] D3 Championship
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > I didn't want you to stand alone in this...it's too
>>>> > > important.
>>>> > > 
>>>> > > -----Original Message-----
>>>> > > From: nsrca-discussion-bounces at lists.nsrca.org
>>>> > > [mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of
>>>> Woodward, 
>>>> > > Jim 
>>>> > > Sent: Wednesday, October 17, 2007 1:31 PM
>>>> > > To: NSRCA Mailing List
>>>> > > Subject: Re: [NSRCA-discussion] D3 Champ! ionship
>>>> > > 
>>>> > > Ryan M., 
>>>> > > 
>>>> > > I think this takes the cake as a first time
>>>> > > nsrca-list email. Thank you for the support.
>>>> > > 
>>>> > > Jim W. 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > CONFIDENTIALITY NOTICE: This e-mail message,
>>>> > > including any attachments, is for the sole use of the intended
>>>> > > recipient(s) and may contain confidential and proprietary information.
>>>> > > Any unauthorized review, use, disclosure or distribution is prohibited.
>>>> > > If you are not the intended recipient(s), please contact the sender by
>>>> > > reply e-mail and destroy all copies of the original message.
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > ________________________________
>>>> > > 
>>>> > > 
>>>> > > From: nsrca-discussion-bounces at lists.nsrca.org
>>>> > > [mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of
>>> >! ; > McLaughlin, Ryan (FRS.JAX)
>>>> > > Sent: Wednesday, October 17, 2007 1:19 PM
>>>> > > To: nsrca-discussion at lists.nsrca.org
>>>> > > Subject: [NSRCA-discussion] D3 Championship
>>>> > > 
>>>> > > This is my first post to the NSRCA list as I am
>>>> > > a bit 'internet shy', but I thought I might be able to add some value
to 
>>>> > > the FAI judging discussion Jim W started. Although I tend to err on the
>>>> > > side of diplomacy : ), I believe the feelings Jim expressed are
>>>> > > legitimate and shared by many FAI competitors throughout the country.
>>>> > > As a long time participant, I realize that bias is not a new problem
>>>> but 
>>>> > > I do not think we should accept this is as a "fact of life" and move
>>>> on. 
>>>> > > I think we have an excellent opportunity here and we should make the
>>>> > > most of it.
>>>> > > 
>>>> > > The primary issue to address in my opinion is
>>>> > > not disparity i! n judgi ng standards between judges, though as Earl
>>>> points 
>>>> > > out, this is important. Rather, it is the different standard applied to
>>>> > > pilots within one score set--i.e.. scoring a pilot lower or higher
>>>> based 
>>>> > > on who he is. Our penchant for creating "superstars" is the most
>>>> > > discouraging aspect of FAI competition. To remedy this, we must all
>>>> > > make a conscious decision to change a long established tradition in our
>>>> > > sport. Are we ready to take this on?
>>>> > > 
>>>> > > Complaining isn't the answer and neither is
>>>> > > staying quiet, a mistake that has made the FAI competitors as
>>>> > > responsible as anyone else for the situation. To this end, I submit for
>>>> > > your review the following ideas to specifically target the FAI bias
>>>> > > issue: 
>>>> > > 
>>>> > > 1. Sacrifice one FAI round per contest to serve
>>>> > > as an "open" round for all contestants e! xpected to judge FAI during
>>>> the 
>>>> > > event. Allow everyone to compare notes and use this as a coaching
>>>> > > opportunity.
>>>> > > 
>>>> > > 2. Drop one FAI pilot to Masters at each
>>>> > > contest to serve as a judge for all rounds and use volunteers from
>>>> other 
>>>> > > classes to serve as the others. This would have to be an agreement made
>>>> > > among FAI pilots.
>>>> > > 
>>>> > > 3. Extend the pilots meeting to go over
>>>> > > specific issues, maybe a new one or two every meet rather than just
>>>> > > pointing out the landing zone, etc. Make a "mini" judging seminar
>>>> > > mandatory each contest.
>>>> > > 
>>>> > > 4. Certify judges for FAI on a volunteer basis
>>>> > > and only use "certified" judges in the contest.
>>>> > > 
>>>> > > 5. Utilize peer judging, in other words, have
>>>> > > FAI pilots judge themselves. If a pilot is not flying, he is judging
>>>> > > his fel! low com petitors.
>>>> > > 
>>>> > > Some of this may seem radical, but I believe
>>>> > > there is room for a bit of this. Pattern belongs to us right? I
>>>> > > welcome any ideas or critique anyone can offer. I will clarify any of
>>>> > > the above upon request.
>>>> > > 
>>>> > > Thank you for your consideration.
>>>> > > 
>>>> > > Ryan McLaughlin
>>>> > > Eustis, Florida
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > ________________________________
>>>> > > 
>>>> > > 
>>>> > > This message w/attachments (message) may be
>>>> > > privileged, confidential or proprietary, and if you are not an intended
>>>> > > recipient, please notify the sender, do not use or share it and delete
>>>> > > it. Unless specifically indicated, this message is not an offer to sell
>>>> > > or a solicitation of any investment products or other financial product
>>>> > > or service, an official ! confirm ation of any transaction, or an
>>>> official 
>>>> > > statement of Merrill Lynch. Subject to applicable law, Merrill Lynch
>>>> may 
>>>> > > monitor, review and retain e-communications (EC) traveling through its
>>>> > > networks/systems. The laws of the country of each sender/recipient may
>>>> > > impact the handling of EC, and EC may be archived, supervised and
>>>> > > produced in countries other than the country in which you are located.
>>>> > > This message cannot be guaranteed to be secure or error-free. This
>>>> > > message is subject to terms available at the following link:
>>>> > > http://www.ml.com/e-communications_terms/. By messaging with Merrill
>>>> > > Lynch you consent to the foregoing.
>>>> > > 
>>>> > > 
>>>> > > ________________________________
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>>>> > > 
>> 
>> 
>>>> > > 
>>>> > > _______________________________________________
>>>> > > NSRCA-discussion mailing list
>>>> > > NSRCA-discussion at lists.nsrca.org
>>>> > > http://lists.nsrca.org/mailman/listinfo/nsrca-discussion
>>>> > > 
>>>> > > 
>>>> > > ________________________________
>>>> > > 
>>>> > > 
>>>> > > _______________________________________________
>>>> > > NSRCA-discussion mailing list
>>>> > > NSRCA-discussion at lists.nsrca.org
>>>> > > http://lists.nsrca.org/mailman/listinfo/nsrca-discussion
>>>> > > 
>>> > 
>>> > _______________________________________________
>>> > NSRCA-discussion mailing list
>>> > NSRCA-discussion at lists.nsrca.org
>>> > http://lists.nsrca.org/mailman/listinfo/nsrca-discussion
> 
> 
> _______________________________________________
> NSRCA-discussion mailing list
> NSRCA-discussion at lists.nsrca.org
> http://lists.nsrca.org/mailman/listinfo/nsrca-discussion


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.nsrca.org/pipermail/nsrca-discussion/attachments/20071018/86557e03/attachment-0001.html