[NSRCA-discussion] Was Bakersfield contest - now discussion on normalization.

Wed Apr 30 06:46:31 AKDT 2014

+1

From: nsrca-discussion-bounces at lists.nsrca.org [mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Atwood, Mark
Sent: Wednesday, April 30, 2014 1:44 AM
To: General pattern discussion
Subject: [NSRCA-discussion] Was Bakersfield contest - now discussion on normalization.

The reality is that "most" of the time, all of these methods will get it right.  All of them how outliers that demonstrate their flaws.  

The question becomes what flaws are more egregious and lead to bad outcomes.  Raw scoring, and normalizing to the perfect score are virtually the same thing and will mathematically result in the same outcome.  It does not normalize the rounds to be equal in weight and allows for a "blowout" round to offset multiple close rounds.   There's a myriad of reasons we no longer use raw scores.  But even raw scores provided the right outcome "most" of the time. 

Normalizing to the highest score has been the accepted way of late. It has the significant advantage of making each round equal in value.  It suffers most notably when one or several pilots miss a round ( for whatever reason).  This very very rarely changes the outcome of the 1st place winner, but can significantly alter the lower standings.  This is most noticed at nationals when people are fighting for a position in the semi finals or finals. 

Normalizing to the average does the reverse.  Still maintains equal value in each round but Improves the accuracy of the middle outcomes at the expense of a possibly less accurate 1st place outcome.  This is appealing if your fighting for the finals cutoff but I think most would agree that at the end of the day we're trying to pick the winner.  

With that in mind, I think we currently have the correct normalization process.  

Mike'a concern of throwing away too much data is a very valid, albeit completely different issue.  That to me is less of a mathematical issue than a potentially contentious discussion about the subjectivity of our discipline.  The argument to throw out highs and lows (I disagree with doing this) is based fundamentally on the assumption that those scores are wrong.  The 10 was erroneous or the 0 was erroneous etc.  In practice this only serves to call into question the quality of the judge.  It was meant to counter the problem with regional "style" especially with regard to snaps and spins, where a spurious Zero could ruin a round.  We've done a LOT to reduce these problems over the years and I think our overall judging is the best it's ever been.  

Keeping rounds is slightly different in its intent.  Only part of that is meant to address equipment reliability.  Many many many sports reward consistency and still allow for a misstep.  Athletic sports typically play series... Best 3of5, 4 of 7, etc.   Subjective sport that don't do this (skating, gymnastics, etc) also don't have you perform more than once.    We are a hybrid of the two.  

My $0.02

Mark

Sent from my average intelligence  phone

On Apr 29, 2014, at 6:02 PM, "Jas S" <justanotherflyr at gmail.com> wrote:

Can the results be run normalized like normal to see if there are any changes? Would be interesting to see if anything changes. My guess is it probably won't, but might show any kind of difference between doing it this way and the other way.

On Tue, Apr 29, 2014 at 4:12 PM, Del R <drykert2 at rochester.rr.com> wrote:

Hi Mike.. While I concede it has rarely happened in the past but their have been repeated occurrences d/t possible judge overload, Miss understanding of maneuver being flown etc. some have earned positive scores when a zero was deserved. My gripe with the current system especially at the NATS settings is when the lowest scoring judge is look at with suspicion when they are they only one who catches a zero maneuver. On the surface of the data it appears to be the bad judge. In actuality the other judges missed the zero. Again Compromise. I accept it is never done intentionally by anyone but when judges work harder than pilots it makes me question the system that is at times rewarding the wrong apple. Now I'm fairly confident in the overall scheme of average flying over the season the consistently best pilot is still truly the one who wins most often. My issue is the lack of acknowledgment by those that control the direction of P.A. at times are willing to throw out the baby with the bath water. The drive to constantly make maneuvers more complex to challenge the pilots is rarely evaluated to what it does to the people that are required to become familiar with the constantly changing schedule depending on which class they are to be called on to judge. Some have had to judge 3 and in rare cases 4 other classes besides the one they are flying in. That isn't ideal either. With everything being purely voluntary and some have to leave part way through without judging skews the load. In the past some would even show up late to register to attempt to avoid judging at all. If judging has become that difficult then the community should be taking a hard look at how fledgling pilots in the lower classes handle and accept those challenges. While some can accept it quite readily others are not so adept. Do we slam the door on their desire to compete and attend more contests. I have never seen the community reverse any of it major changes when it has hurt attendance. The priority is always for the betterment of FAI pilots in the WORLDS. While those are admiral goals of the organization my contention is if they want to encourage participation they need to change some of their priorities a little. 

    Del

----- Original Message ----- 

From: Dr. Michael Harrison, DDS <mailto:drmikedds at sbcglobal.net>  

To: 'General pattern discussion' <mailto:nsrca-discussion at lists.nsrca.org>  

Sent: Tuesday, April 29, 2014 3:14 PM

Subject: Re: [NSRCA-discussion] Bakersfield Bashscores areon the NSRCA website...

and  data we are picking may not be the right data or we are throwing away good data and possibly picking the wrong competitor,,,

From: nsrca-discussion-bounces at lists.nsrca.org [mailto:nsrca-discussion-bounces at lists.nsrca.org] On Behalf Of Del R
Sent: Tuesday, April 29, 2014 12:47 PM
To: General pattern discussion
Subject: Re: [NSRCA-discussion] Bakersfield Bash scores areon the NSRCA website...

Mike.. Over the years many of the negative down sides have been repeatedly pointed out regarding using the current system and while I truly concur with your opinion the ruling part sees nothing wrong with throwing away some rounds which is against the very attribute they contend to support. If the overall best average pilot shows up each round and does their best then they would secure their place. Belittling any judges work manning the chair by throwing away their work is counter-productive to the hard work they put in with their calibrated eyeballs. It is all about what is an acceptable compromise. In a perfect world all judges would be focused and catch every zero but when the maneuvers are as complex as they have become it is a daunting task for new pilots to master let alone feel comfortable about advancing to the point they have to man the chair to compete.  The sport has become so complex one needs professional caller and when you have maintaince issues to tend to while having to go man the chair makes it less than desirable. Maybe that is why many are glad to have the choice of throwing one of their flights away. 

    Del

----- Original Message ----- 

From: Dr. Michael Harrison, DDS <mailto:drmikedds at sbcglobal.net>  

To: 'General pattern discussion' <mailto:nsrca-discussion at lists.nsrca.org>  

Sent: Tuesday, April 29, 2014 11:51 AM

Subject: Re: [NSRCA-discussion] Bakersfield Bash scores areon the NSRCA website...

My 2-1=1 cents,  There is no perfect scoring system.  The system used at bakersfield regresses back to winning with a " BarnBurner score" on the last flight of the contest and this suffers the dilemma of becoming the popularity contest we used to deal with.  Normalizing normally rewards the consistently best pilot, however there are always exceptions and extreme examples to the contrary.  This holds true in all competitive events of any type.  

I believe where we fall short today is that we throw away too much data.  Examples would be dropping the high and low score of each flight, dropping the worst flight at a contest, counting only 4 flights of contests with more than 4 flights, throwing away previous flights at major events, ie.  eliminating the prelims scores at major events, etc.

With the reliability of all our equipment and the technology present today, we should do more to retain more of the judging we have done.

More and more is being done via various methods(TBL, dropping high and low scores, dropping worst flights, etc) to create an "average score" which is erroneously considered the best most fair score.  I believe this is a mistake.  I believe we should keep the high and low scores, the worst flights, whatever.  We don't really have the statistical data to do TBL properly, in my opinion, it should be eliminated.  It definitely has an adverse affect on the judges ability to do his job.  

I believe we should work to keep much more data.  I believe we can create systems to do that.  

On another note, I believe this is one really good contest, would love to have been there.  Probably the premier "local" contest of the nation.

Mike 

On Apr 29, 2014, at 8:19 AM, ronlock at comcast.net wrote

Ron Lockhart

  _____  

From: "Mark Atwood" <atwoodm at paragon-inc.com>
To: "General pattern discussion" <nsrca-discussion at lists.nsrca.org>
Sent: Tuesday, April 29, 2014 2:01:04 AM
Subject: Re: [NSRCA-discussion] Bakersfield Bash scores are on the        NSRCA        website...

It does not normalize the rounds.  One round can be worth more than another.  Easy vs hard judges, calm vs windy conditions, etc.  Especially with throw aways it's critical that all the round have the same "value".  

That said, I can see an interest in normalizing to the average score, rather than to the top score.  It minimizes some of the variance when a key flyer doesn't fly the round either due to a problem or simply choosing not to fly after winning the first 4 rounds.  Particularly in that last scenario, it makes the remaining rounds much more valuable to the 2nd and 3rd place flyers.  

Sent from my average intelligence  phone

On Apr 28, 2014, at 8:27 PM, "Peter Vogel" <vogel.peter at gmail.com> wrote:

Well, it's normalization of a sort, in that it's a percentage of perfect within each class, so a 984 in Intermediate and a 984 in Sportsman mean the same thing, you flew 98.4% of the perfect score (no, I haven't seen a score that high! :-)  

Overall though, I don't like the dynamic of it.  It's very clear when you have a highly critical set of judges, everyone pretty much drops that round.  Normalization to best is definitely a superior way for people to see how they are doing relative to the best pilot in the round for a particular set of judges, regardless of how critical (or not) they happen to be.  An interesting experiment, but I wouldn't recommend it for future contests.

Peter+

On Mon, Apr 28, 2014 at 7:08 PM, John Gayer <jgghome at comcast.net> wrote:

It is a misnomer to call what was used normalization. It is actually a reversion to the pre-normalization scoring scheme of yesteryear. The only difference from the old days is that by "normalizing" to a perfect score, you can immediately see the average raw score per maneuver.
John

On 4/28/2014 6:15 PM, Anthony Romano wrote:

So why the change in normalization?

Anthony

  _____  

Date: Mon, 28 Apr 2014 17:04:28 -0700
From: vogel.peter at gmail.com
To: nsrca-discussion at lists.nsrca.org
Subject: Re: [NSRCA-discussion] Bakersfield Bash scores are on the NSRCA website...

Final data from the contest is also available here:

http://www.patternscoring.com/560, the per-contestant report is considerably richer than what you get on nsrca.us <http://nsrca.us/> 

I also just completed an analysis of what would have happened if we'd normalized to best instead of to perfect, the only class where it would have made a difference in final placement was FAI Silver (MP15) where Jon's win in one round and 2nd place rounds were closer to Sean's scores than Dale's 2nd place rounds would have put him about 30 points over Dale.

Peter+

On Mon, Apr 28, 2014 at 11:17 AM, Peter Vogel <vogel.peter at gmail.com> wrote:

That's not the scoring program, the XML says "Prelim", "1", "2" for the flight number, but the web server's transformation is showing it as "1", "2" and "0".

Peter+

On Mon, Apr 28, 2014 at 11:07 AM, Derek Koopowitz<derekkoopowitz at gmail.com> wrote:

Everyone,

The NSRCA website is back up and running again... our host was doing some maintenance on the server.  The scores for the Bakersfield contest have been posted and just to let everyone know...

1.  Every score is normalized to PERFECT, not to best pilot

2.  Every AMA class flew 2 final rounds on Sunday using unknowns

3.  Best 3 of 4 prelim rounds were AVERAGED to produce a SINGLE round score to carry over into the finals.

4.  Best 2 of 3 (prelim carry over + 2 final rounds = 3) count for the final score.

In FAI F, Prelim carry over is shown as Round 1, Round 2 is first finals round, and round 3 is the 2nd final round -- for some reason the scoring program is showing that as round 0.

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

-- 

Director, Fixed Wing Flight Training

Santa Clara County Model Aircraft Skypark

  <http://farm8.staticflickr.com/7163/6513778381_5569cc985d_m.jpg> 

-- 

Director, Fixed Wing Flight Training

Santa Clara County Model Aircraft Skypark

  <http://farm8.staticflickr.com/7163/6513778381_5569cc985d_m.jpg> 

_______________________________________________ NSRCA-discussion mailing list NSRCA-discussion at lists.nsrca.orghttp://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

-- 

Director, Fixed Wing Flight Training

Santa Clara County Model Aircraft Skypark

  <http://farm8.staticflickr.com/7163/6513778381_5569cc985d_m.jpg> 
_______________________________________________

NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

  _____  

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

  _____  

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

-- 

Jason
http://jasonshangar.weebly.com/

_______________________________________________
NSRCA-discussion mailing list
NSRCA-discussion at lists.nsrca.org
http://lists.nsrca.org/mailman/listinfo/nsrca-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nsrca.org/pipermail/nsrca-discussion/attachments/20140430/75331483/attachment.html>