[NSRCA-discussion] Scoring Process Question

Tue Jun 26 12:56:02 AKDT 2007

I don't think adding judges actually changes anything as far as the statistical significance of the final number is concerned.  It just makes the inter-observer variation less of a factor in the outcome.  

---- Mark Atwood <atwoodm at paragon-inc.com> wrote: 
> I have to sit down and look at the math, but how does adding judges impact
> this statistically?  I.e. More judges will technically improve the accuracy,
> and yet doesn't change the number of significant digits in the equation.
> 
> There is no question that we separate pilots by a whim at large contests.
> The arbitrary 7 vs 8 on one maneuver, by one judge with all else being
> identical is all that separates the winners at the nats in any of the
> classes usually.  
> 
> That said, this "style" of judging, assigning 0-10 scores multiplied by a
> difficulty factor and then normalized is used throughout almost every
> subjective sport from Diving to Figure skating...with great (albeit similar
> disputes) success. And there's a lot more money and attention paid to those
> activities to get the attention of people far more savvy at higher math than
> I...so I'm going to suggest that we must be missing something, and that it
> works. :):)
> 
> -M
> 
> 
> On 6/26/07 2:30 PM, "glmiller3 at suddenlink.net" <glmiller3 at suddenlink.net>
> wrote:
> 
> > Mike, 
> 
> Take some time and read it with a glass of wine tonight<G>...My point
> > is exactly that we are creating an ILLUSION of accuracy which is not
> > statistically present.  If my statistics are correct, scores are only accurate
> > to about 100 points of the 1000 point scale.  We are deciding most of our
> > contests on the statistical "noise".  
> 
> I haven't proposed any change, I'm
> > just asking for ideas......If I had a better solution, I'd offer it.  I think
> > that you are right in that expanding the judges score to more digits won't
> > help because it is an inherently subjective number that can't be quantified
> > more accurately than "about a half a point" on a ten point scale.
> 
> George
> > 
> 
> 
> ---- Michael Wickizer <mwickizer at msn.com> wrote: 
> > My head hurts after
> > trying to read and follow that.
> > 
> > However, it strikes me that you are
> > trying to attach mathmatical and 
> > statisical validation to something that
> > only has two numbers and that each 
> > contain a varying amount of
> > subjectivity.  I am not sure that using a 1000 
> > point per manuver system or
> > even greater, would make it more valid but only 
> > an illusion.
> > 
> > 
> > >From:
> > <glmiller3 at suddenlink.net>
> > >Reply-To: NSRCA Mailing List
> > <nsrca-discussion at lists.nsrca.org>
> > >To: NSRCA List
> > <nsrca-discussion at lists.nsrca.org>
> > >Subject: [NSRCA-discussion] Scoring
> > Process Question
> > >Date: Tue, 26 Jun 2007 12:50:48 -0500
> > >
> > >I'm going to
> > open a can of worms here in hopes of coming up with a better 
> > >system out of
> > the discussion.  Perhaps this has been discussed before and 
> > >I'm not aware
> > of it.  Let me preface this by saying I am not a 
> > >mathematician or
> > statistician, but I have some familiarity with both 
> > >subjects and the
> > following question has been growing in my mind for some 
> > >time.
> > >
> > >It
> > seems to me that we are judging our maneuvers with limited accuracy 
> >
> > >(within 1 point in FAI and X.5 points in AMA classes) we are then creating 
> >
> > >the ILLUSION of accuracy by multiplying that score by a K factor and then 
> >
> > >normalizing to a 1000 point scale.  Here is a fairly brief explanation of 
> >
> > >"Significant Digits" that I've copied from the web which will introduce you
> > 
> > >to this thought if you haven't seen it before:
> > >
> > >****"SIGNIFICANT
> > DIGITS
> > >
> > >The number of significant digits in an answer to a calculation
> > will depend 
> > >on the number of significant digits in the given data, as
> > discussed in the 
> > >rules below. Approximate calculations (order-of-magnitude
> > estimates) always 
> > >result in answers with only one or two significant
> > digits.
> > >
> > >When are Digits Significant?
> > >
> > >Non-zero digits are always
> > significant. Thus, 22 has two significant 
> > >digits, and 22.3 has three
> > significant digits.
> > >
> > >With zeroes, the situation is more complicated:
> >
> > >
> > >Zeroes placed before other digits are not significant; 0.046 has two 
> >
> > >significant digits.
> > >Zeroes placed between other digits are always
> > significant; 4009 kg has four 
> > >significant digits.
> > >Zeroes placed after
> > other digits but behind a decimal point are 
> > >significant; 7.90 has three
> > significant digits.
> > >Zeroes at the end of a number are significant only if
> > they are behind a 
> > >decimal point as in (c). Otherwise, it is impossible to
> > tell if they are 
> > >significant. For example, in the number 8200, it is not
> > clear if the zeroes 
> > >are significant or not. The number of significant
> > digits in 8200 is at 
> > >least two, but could be three or four. To avoid
> > uncertainty, use scientific 
> > >notation to place significant zeroes behind a
> > decimal point:
> > >8.200 Â´  has four significant digits
> > >8.20 Â´  has three
> > significant digits
> > >
> > >8.2 Â´  has two significant digits
> > >
> >
> > >Significant Digits in Multiplication, Division, Trig. functions, etc.
> > >
> >
> > >In a calculation involving multiplication, division, trigonometric 
> >
> > >functions, etc., the number of significant digits in an answer should equal
> > 
> > >the least number of significant digits in any one of the numbers being 
> >
> > >multiplied, divided etc.
> > >
> > >Thus in evaluating sin(kx), where k = 0.097
> > m-1 (two significant digits) 
> > >and x = 4.73 m (three significant digits),
> > the answer should have two 
> > >significant digits.
> > >
> > >Note that whole
> > numbers have essentially an unlimited number of significant 
> > >digits. As an
> > example, if a hair dryer uses 1.2 kW of power, then 2 
> > >identical hairdryers
> > use 2.4 kW:
> > >
> > >1.2 kW {2 sig. dig.} X 2 {unlimited sig. dig.} = 2.4 kW {2
> > sig. dig.} 
> > >"******
> > >
> > >My Point is this:
> > >
> > >I've seen many contests
> > decided by less than 10 points on a scale of 4000 
> > >which has been expanded
> > from (at most) 2 significant digits.  As a matter 
> > >of "statistics" I think
> > that any separation of less than 100 points (two 
> > >significant digits, ie,
> > 3X00 points) is "artificial accuracy".  
> > >Unfortunately, I don't have any
> > great ideas about how to improve upon the 
> > >current system, I'm just
> > pointing out what I think is a scientifically 
> > >valid problem with it.
> > >
> >
> > >I smile when I see round scores posted to ten thousanths of a point on a 
> >
> > >scale that has been expanded from two significant digit accuracy to a 1000 
> >
> > >point scale.  This turns a two significant digit answer into eight 
> >
> > >significant digits!  (ie, 1234.5678)    I think that scientifically, the 
> >
> > >scores would be more accurately posted as in scientific notation at   x.x  
> >
> > >* 10 to the second power.  Most of the contests that I've been to this year
> > 
> > >have been decided essentially by random statistical "noise" rather than >
> > >actual scoring decisions.
> > >
> > >
> > >Has anyone ever thought/talked about
> > this before ?
> > >
> > >Let me add, that despite what I think are statistically
> > invalid methods, in 
> > >most cases the system seems to work pretty well.  In
> > general the superior 
> > >pilots get enough better scores to overcome the
> > "noise" but it sure would 
> > >be nice to come up with a more mathematically
> > valid solution, IMO.
> > >
> > >George
> > >
> > >
> > >
> > >
> >
> > >_______________________________________________
> > >NSRCA-discussion mailing
> > list
> > >NSRCA-discussion at lists.nsrca.org
> >
> > >http://lists.nsrca.org/mailman/listinfo/nsrca-discussion
> > 
> >
> > 
> 
> _______________________________________________
> NSRCA-discussion mailing
> > list
> NSRCA-discussion at lists.nsrca.org
> http://lists.nsrca.org/mailman/listinfo/
> > nsrca-discussion
> 
> _______________________________________________
> NSRCA-discussion mailing list
> NSRCA-discussion at lists.nsrca.org
> http://lists.nsrca.org/mailman/listinfo/nsrca-discussion