[NSRCA-discussion] Matrix improvements

Mon Aug 3 12:00:53 AKDT 2009

Systems which are ideal (no bias) on paper can not always be implemented
because of logistical considerations (time, resources, labor, etc).
Additionally, systems that are ideal on paper are often more susceptible in
practice to incurred bias (increased judging fatigue, changes in weather,
etc).

When a threshold number of entrants in a given class is not exceeded, the
arrangement of two flight groups with rotating judges is preferred (ie,
Intermediate and Advanced on Site 4, and F3A 2009 at Sites 1 and 3).  For
F3A 2009, there was still a bias as all pilots were not judged at the same
flying site (and we all know flying sites 1 and 3 are not equal), but this
bias was accepted as the lesser of two evils (the 2nd being doubling the
needed flight time).

The current matrix system has bias and is only preferred when the entrants
(flight time) in Masters and FAI exceeds a threshold (and the Matrix system
would be used if Advanced / Intermediate exceeded a threshold as well).  The
benefits of the matrix system are that it is quite efficient and versatile
from the perspectives of time, resources, labor, judging fatigue, and
changing weather.  To realistically avoid using the matrix system (with a
large number of pilots) would require fewer flights and longer judging
stints.  

The seeding typically spreads the top 8 pilots equally among the 4 groups
(#1 in A, #2 in B, #3 in C, #4 in D).  Keep in mind that in years past the
frequency distribution also needed to be considered when forming the groups
(with 2.4, this is increasingly less of a concern), so ideal distribution of
seeded pilots was not always possible.  Assuming the pilots performed
consistently with ranking, the bias to pilots seeded 4+ results in the
following way - 

- Pilots in Group A must normalize all 6 prelim flights against the #1 seed
(the eventual contest winner).

- Pilots in Group B must normalize 2 prelim rounds against the #1 seed, and
the remaining 4 prelim rounds against the #2 seed.

- Pilots in the C and D groups must normalize 2 prelim rounds against the #1
seed, 2 prelim rounds against the #2 seed, and 2 rounds against the #3 seed.

Accurate seeding does NOT eliminate and has little effect on the inherent
bias of the matrix system.  The inherent bias in the matrix system will
exist unless the top 4 seeds (actually top 3 seeds because the 4 groups are
always flying in pairs) fly equally and score (RAW scores) equally every
round of the prelims.  The difference in flight quality between the top 3
seeds will produce bias which, if great enough (and it is, based on review
and analyses of historical NATs scores), will produce inaccuracies in the
results (of the prelims).  Inaccurate seeding can increase the bias, and the
greater the difference between the flying quality of the top 3 seeds, the
greater the bias imparted to the scores of pilots seeded 4+.

The mini-contest suggestion has the same flaw at the same basic level as the
matrix system - non equal exposure of pilots in the prelims, the bias of
which can be magnified if the seeding is not accurate (or the ultimate
performance of the seeded pilots is not as predicted due to no shows,
equipment failure, having a bad day, etc).

All systems will either include some inherent bias, and/or be more/less
susceptible to the introduction of bias during the process (from judging
float, changes in weather, differing flying sites, etc).  Only the inherent
bias can be eliminated (the others are random), and only when it is
logistically feasible (increasingly difficult with increasing numbers of
pilots).  For a National Championship, the goal (my opinion) of any
preliminary system should not be to determine an accurate ranking of the
pilots going to the finals, but accurate in choosing the top 8 pilots for
the finals.  Inaccuracies in the ranking of pilots 8 and 9 (or even 7 thru
10, etc) may happen and may keep a deserving pilot out of the finals, but,
it is not likely that the 7th or 10th ranked pilot is going to be the
ultimate winner.  At the end of the day, the system with the least amount of
inherent bias, is the least susceptible to incurred bias, and is
logistically feasible is the system that should be used.  Keep in mind it is
highly desireable to have a system that maintains equal exposure (or as
close to it as possible) at all points during the contest - so winners can
be announced if the contest is cut short by bad weather.

In the past, I've gone through huge amounts of historical data to arrive at
my conclusions/opinions.  Part of the flaw / source for bias of any system
using seeding is that the scores for each pilot are essentially determined
based on a single data point (the highest raw score for the round).  I think
the inherent bias in the existing matrix system could be reduced (but not
eliminated) if the flight rounds were normalized to the average raw score
for the flight round - but I've not run numbers (and don't currently have
time to do so) to verify or disprove this, and obviously allowances would
have to be made for incomplete rounds (flameouts, midairs, etc).

Oh, and for anyone interested in the matrix "bias" number I came up with for
non-seeded flyer "X" finishing outside the top 8 - if flyer "X" is in Group
A vs Group C or D, flyer "X" will average 14 normalized points lower per
round.  Take a look at historical NATs results, and see what 14 normalized
points does for finish order in the prelims.  And then try to find a system
that is logistically feasible to handle 30+ pilots with less bias than the
matrix system.

Regards,

Dave Lockhart

DaveL322 at comcast.net

  _____  

From: nsrca-discussion-bounces at lists.nsrca.org
[mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Mark Hunt
Sent: Monday, August 03, 2009 1:44 PM
To: General pattern discussion
Subject: Re: [NSRCA-discussion] Matrix improvements

I too would like to see an article on this.  No offense, but initially it is
unclear to me how this would give any better exposure of pilots to the same
judges/conditions than the current matrix system allows for.  Would seeding
not become even more critical in this scenario?

Mark

  _____  

From: Anthony Romano <anthonyr105 at hotmail.com>
To: nsrca-discussion at lists.nsrca.org
Sent: Monday, August 3, 2009 11:56:34 AM
Subject: [NSRCA-discussion] Matrix improvements

Maybe Jerry could detail this in a Kfactor article. Perhaps could be used at
locals to help with an oversized Masters group.

Anthony

  _____  

Date: Thu, 30 Jul 2009 15:45:19 -0700
From: derekkoopowitz at gmail.com
To: nsrca-discussion at lists.nsrca.org
Subject: Re: [NSRCA-discussion] Flash Poll - Relocate the Nats in 2010?

Jerry Budd had a good suggestion in running 4 mini-contests for 6 rounds
where each pilot would fly against their group for 6 straight rounds and
then the top 3 from each group would fly in the finals.  I'm leaning toward
this because the current format does not work.  We also need to do something
about FAI - because there isn't equal exposure there either.

On Thu, Jul 30, 2009 at 2:41 PM, John Fuqua <johnfuqua at embarqmail.com>
wrote:

Every Nats I went to that had all events in 2 weeks ended up screwing
Pattern out of time or space.  It never failed.  I am against it even
thought I liked to go see other events.  We cannot do a first rate job when
we compete for runway space and days to fly.

I thought AMA wanted to reduce Nats costs.  Moving them around does not do
that if you look at history and read up you will find a consistent comment
about reducing Nats costs.  NPAC was fully funded by the pilots and it cost
more than a typical Nats so factor that in.

No one has addressed the equal exposure to judges issues for the current
format yet and Mike's proposal does not correct that situation for Masters.
I had one person suggest to me that if we continue to use the Matrix system
that we take the top 3 pilots from each "Group" to a 3 round finals.  At
least then we have equal judging exposure and more or less equal weather
exposure per round.

John

-----Original Message-----
From: nsrca-discussion-bounces at lists.nsrca.org

[mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Tony
Sent: Thursday, July 30, 2009 2:56 PM
To: 'General pattern discussion'
Subject: Re: [NSRCA-discussion] Flash Poll - Relocate the Nats in 2010?

That is a possibility also.  AMA is thinking about returning the NATS to a 2
week all-events NATS like used to be done.  If all events are together, HQ
can put all costs into one effort.  If they are all split up, Each group
would have more costs to deal with due to the fact that HQ can't send a
group of people to every site.

Tony Stillman, President

Radio South, Inc.

139 Altama Connector, Box 322

Brunswick, GA  31525

1-800-962-7802

www.radiosouthrc.com <http://www.radiosouthrc.com/> 

________________________________

From: nsrca-discussion-bounces at lists.nsrca.org
[mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Tim Taylor
Sent: Thursday, July 30, 2009 3:39 PM
To: General pattern discussion
Subject: Re: [NSRCA-discussion] Flash Poll - Relocate the Nats in 2010?

It was done, It was called NPAC

Tim

--- On Thu, 7/30/09, Bill's Email <wemodels at cox.net> wrote:

       From: Bill's Email <wemodels at cox.net>
       Subject: Re: [NSRCA-discussion] Flash Poll - Relocate the Nats in
2010?
       To: "General pattern discussion" <nsrca-discussion at lists.nsrca.org>
       Date: Thursday, July 30, 2009, 2:06 PM

       Why not allow the SIGS to do their own thing? IMAC and NSRCA do not
need to share a site. The LSF can find a site for the soaring NATS. Pylon
knows what venues work for them and so on. FF can do their own thing as
well. Why tie soaring and FF together and so on??

       Tony wrote:

       Matt:
       Yes, it is just difficult to tell if it is actually feasible.  The
problem is that it requires a large site for Pattern/Pylon/IMAC and will
require another large site for Soaring/Outdoor FF.  It may be very difficult
to actually find places that can handle this group.

       Tony Stillman, President

       Radio South, Inc.

139 Altama Connector, Box 322

Brunswick, GA   31525

 1-800-962-7802

www.radiosouthrc.com <http://www.radiosouthrc.com/>
<http://www.radiosouthrc.com/>

________________________________

-----Inline Attachment Follows-----

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org

<http://us.mc623.mail.yahoo.com/mc/compose?to=NSRCA-discussion@lists.nsrca.o
<http://us.mc623.mail.yahoo.com/mc/compose?to=NSRCA-discussion@lists.nsrca.o
rg> 
rg>

http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

  _____  

Get back to school stuff for them and cashback for you. Try BingT now.
<http://www.bing.com/cashback?form=MSHYCB&publ=WLHMTAG&crea=TEXT_MSHYCB_Back
ToSchool_Cashback_BTSCashback_1x1>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nsrca.org/pipermail/nsrca-discussion/attachments/20090803/9a01ae54/attachment.html>