[NSRCA-discussion] Scoring Process Question

[Mark Atwood]

I have to sit down and look at the math, but how does adding judges impact
this statistically?  I.e. More judges will technically improve the accuracy,
and yet doesn't change the number of significant digits in the equation.

There is no question that we separate pilots by a whim at large contests.
The arbitrary 7 vs 8 on one maneuver, by one judge with all else being
identical is all that separates the winners at the nats in any of the
classes usually.  

That said, this "style" of judging, assigning 0-10 scores multiplied by a
difficulty factor and then normalized is used throughout almost every
subjective sport from Diving to Figure skating...with great (albeit similar
disputes) success. And there's a lot more money and attention paid to those
activities to get the attention of people far more savvy at higher math than
I...so I'm going to suggest that we must be missing something, and that it
works. :):)

-M


On 6/26/07 2:30 PM, "glmiller3 at suddenlink.net" <glmiller3 at suddenlink.net>
wrote:

> Mike, 

Take some time and read it with a glass of wine tonight<G>...My point
> is exactly that we are creating an ILLUSION of accuracy which is not
> statistically present.  If my statistics are correct, scores are only accurate
> to about 100 points of the 1000 point scale.  We are deciding most of our
> contests on the statistical "noise".  

I haven't proposed any change, I'm
> just asking for ideas......If I had a better solution, I'd offer it.  I think
> that you are right in that expanding the judges score to more digits won't
> help because it is an inherently subjective number that can't be quantified
> more accurately than "about a half a point" on a ten point scale.

George
> 


---- Michael Wickizer <mwickizer at msn.com> wrote: 
> My head hurts after
> trying to read and follow that.
> 
> However, it strikes me that you are
> trying to attach mathmatical and 
> statisical validation to something that
> only has two numbers and that each 
> contain a varying amount of
> subjectivity.  I am not sure that using a 1000 
> point per manuver system or
> even greater, would make it more valid but only 
> an illusion.
> 
> 
> >From:
> <glmiller3 at suddenlink.net>
> >Reply-To: NSRCA Mailing List
> <nsrca-discussion at lists.nsrca.org>
> >To: NSRCA List
> <nsrca-discussion at lists.nsrca.org>
> >Subject: [NSRCA-discussion] Scoring
> Process Question
> >Date: Tue, 26 Jun 2007 12:50:48 -0500
> >
> >I'm going to
> open a can of worms here in hopes of coming up with a better 
> >system out of
> the discussion.  Perhaps this has been discussed before and 
> >I'm not aware
> of it.  Let me preface this by saying I am not a 
> >mathematician or
> statistician, but I have some familiarity with both 
> >subjects and the
> following question has been growing in my mind for some 
> >time.
> >
> >It
> seems to me that we are judging our maneuvers with limited accuracy 
>
> >(within 1 point in FAI and X.5 points in AMA classes) we are then creating 
>
> >the ILLUSION of accuracy by multiplying that score by a K factor and then 
>
> >normalizing to a 1000 point scale.  Here is a fairly brief explanation of 
>
> >"Significant Digits" that I've copied from the web which will introduce you
> 
> >to this thought if you haven't seen it before:
> >
> >****"SIGNIFICANT
> DIGITS
> >
> >The number of significant digits in an answer to a calculation
> will depend 
> >on the number of significant digits in the given data, as
> discussed in the 
> >rules below. Approximate calculations (order-of-magnitude
> estimates) always 
> >result in answers with only one or two significant
> digits.
> >
> >When are Digits Significant?
> >
> >Non-zero digits are always
> significant. Thus, 22 has two significant 
> >digits, and 22.3 has three
> significant digits.
> >
> >With zeroes, the situation is more complicated:
>
> >
> >Zeroes placed before other digits are not significant; 0.046 has two 
>
> >significant digits.
> >Zeroes placed between other digits are always
> significant; 4009 kg has four 
> >significant digits.
> >Zeroes placed after
> other digits but behind a decimal point are 
> >significant; 7.90 has three
> significant digits.
> >Zeroes at the end of a number are significant only if
> they are behind a 
> >decimal point as in (c). Otherwise, it is impossible to
> tell if they are 
> >significant. For example, in the number 8200, it is not
> clear if the zeroes 
> >are significant or not. The number of significant
> digits in 8200 is at 
> >least two, but could be three or four. To avoid
> uncertainty, use scientific 
> >notation to place significant zeroes behind a
> decimal point:
> >8.200 ´  has four significant digits
> >8.20 ´  has three
> significant digits
> >
> >8.2 ´  has two significant digits
> >
>
> >Significant Digits in Multiplication, Division, Trig. functions, etc.
> >
>
> >In a calculation involving multiplication, division, trigonometric 
>
> >functions, etc., the number of significant digits in an answer should equal
> 
> >the least number of significant digits in any one of the numbers being 
>
> >multiplied, divided etc.
> >
> >Thus in evaluating sin(kx), where k = 0.097
> m-1 (two significant digits) 
> >and x = 4.73 m (three significant digits),
> the answer should have two 
> >significant digits.
> >
> >Note that whole
> numbers have essentially an unlimited number of significant 
> >digits. As an
> example, if a hair dryer uses 1.2 kW of power, then 2 
> >identical hairdryers
> use 2.4 kW:
> >
> >1.2 kW {2 sig. dig.} X 2 {unlimited sig. dig.} = 2.4 kW {2
> sig. dig.} 
> >"******
> >
> >My Point is this:
> >
> >I've seen many contests
> decided by less than 10 points on a scale of 4000 
> >which has been expanded
> from (at most) 2 significant digits.  As a matter 
> >of "statistics" I think
> that any separation of less than 100 points (two 
> >significant digits, ie,
> 3X00 points) is "artificial accuracy".  
> >Unfortunately, I don't have any
> great ideas about how to improve upon the 
> >current system, I'm just
> pointing out what I think is a scientifically 
> >valid problem with it.
> >
>
> >I smile when I see round scores posted to ten thousanths of a point on a 
>
> >scale that has been expanded from two significant digit accuracy to a 1000 
>
> >point scale.  This turns a two significant digit answer into eight 
>
> >significant digits!  (ie, 1234.5678)    I think that scientifically, the 
>
> >scores would be more accurately posted as in scientific notation at   x.x  
>
> >* 10 to the second power.  Most of the contests that I've been to this year
> 
> >have been decided essentially by random statistical "noise" rather than >
> >actual scoring decisions.
> >
> >
> >Has anyone ever thought/talked about
> this before ?
> >
> >Let me add, that despite what I think are statistically
> invalid methods, in 
> >most cases the system seems to work pretty well.  In
> general the superior 
> >pilots get enough better scores to overcome the
> "noise" but it sure would 
> >be nice to come up with a more mathematically
> valid solution, IMO.
> >
> >George
> >
> >
> >
> >
>
> >_______________________________________________
> >NSRCA-discussion mailing
> list
> >NSRCA-discussion at lists.nsrca.org
>
> >http://lists.nsrca.org/mailman/listinfo/nsrca-discussion
> 
>
> 

_______________________________________________
NSRCA-discussion mailing
> list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/
> nsrca-discussion