Hi Ken, Thanks for providing such a clear and concise response. I would venture to suggest that the fundamental question we should now be asking ourselves is whether the system, as currently proposed, would allow the GNSO Council to decide that projects should be stricken from the to-do list because they are rated too low in priority? A secondary question may be, should the GNSO Council determine that there is a finite number of projects that can be undertaken at the same time, considering the available resources (# of Councillors, # of support Staff, etc.)? I am copying the Council because this may be a point we want to discuss during our teleconference tonight. I guess what I am saying is that whilst prioritization is a good first step, to be truly efficient, the GNSO may want to decide that it can only work on 15 projects in one given time period (e.g. a year) and that more projects can only be added when one of the 15 is finished (the "15" is arbitrary here, I only use it to illustrate my idea, I am not suggesting that should be the number)? I freely admit to being inspired by Staff's batching idea for new gTLDs in this regard ;) Stéphane Le 9 juin 2010 à 22:26, Ken Bour a écrit :
Stéphane:
I will try…
You wrote: “I wonder if the reason we are not getting any projects with a lower score than 2 is that Councilors are not sufficiently aware that they can strike projects all together should they wish to when they rate them?”
Shorter response: 1) Councilors are rating projects lower than 2 in Value; lots of “1s” have been registered. 2) My original comment about “>2” dealt with the Range statistic, not the individual Councilor project ratings.
I apologize for using your inquiry to springboard into a broader discussion of central tendency measures, but I wanted to convey those thoughts to the WPM-DT while they were fresh on my mind. Sorry, if my explanation was overly confusing…
Ken
From: Stéphane Van Gelder [mailto:stephane.vangelder@indom.com] Sent: Wednesday, June 09, 2010 3:56 PM To: Ken Bour Cc: gnso-wpm-dt@icann.org Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2
Hi Ken,
Well if I was confused before, I am now completely flummoxed after trying to read your latest email :)
If there's any way you can explain it to me in a couple of sentences, I would be grateful. If not, please just ignore my previous comment.
Stéphane
Le 9 juin 2010 à 15:18, Ken Bour a écrit :
Stephane:
I think you might have misunderstood my comment. Every project’s Range statistic is > 2, which means that, when I subtract the highest rating from the lowest rating, that difference is larger than 2 for every project. On the surface, that statistic indicates a wide spread among certain Councilors as to their perception of the each project’s relative Value. It also means that there are no projects that can be automatically removed from the Brussels discussion based upon the individual ratings step, which required a Range of 2 or less.
One of the possible improvement steps, going forward, may be to change the way that central tendency is measured after the individual ratings round. When we only had 5 testers, the Range looked like the best indicator; it is easy to calculate and understand. Now that I am seeing over a dozen values, the Standard Deviation might be a more useful statistic for determining agreement since the population size is so much larger. For example, out of 12 ratings for one particular project, we have the following ratings: 5, 3, 4, 3, 6, 4, 4, 4, 5, 4, 4, 4. Even though the Range is 3 (6-3), the Mode, the Median, and the Mean are all equal to 4 and the Standard Deviation is less than 1.0 (actually .8), which indicates a very tight spread not only statistically, but visually as well. For our current exercise, we will not be excluding this project from discussion; but, in the future, it could be a candidate for pre-determining agreement (in this case: Rating=4) on the basis of its low Standard Deviation.
Here is one more example that is interesting. The 12 ratings thus far are: 6, 5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7. The Range=3, the Mode=7, Median=7, Mean=6, and Std Dev=1.04. I think a strong case could be made for accepting 7 as the group Value Rating although, if the rule were written such that Std Dev had to be < 1.00, it would fail the test.
I expect to have a more fully thought-out recommendation once this part of the process has completed and I have more time to analyze the results.
Lastly, I just want to be clear there are tons of individual project ratings that are “1” and, for that matter, “7”. From what I can discern in examining each Councilor’s spreadsheet, it doesn’t appear that anyone misunderstood the directions. Any value from 1-7 could be selected for any cell and, while some Councilors used the entire range and others did not, I don’t think there is anything to be concluded other than that is how they perceived Value across the range of Eligible Projects.
Ken
From: Stéphane Van Gelder [mailto:stephane.vangelder@indom.com] Sent: Wednesday, June 09, 2010 6:30 AM To: Ken Bour Cc: gnso-wpm-dt@icann.org Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2
Thanks Ken for that update. I wonder if the reason we are not getting any projects with a lower score than 2 is that Councillors are note sufficiently aware that they can strike projects all together should they wish to when they rate them?
Stéphane
Le 8 juin 2010 à 21:08, Ken Bour a écrit :
WPM-DT Members:
I thought you might appreciate receiving a brief status report concerning Step 2-Individual Councilor Ratings…
As of this afternoon, 8 June, I have received 12 Councilor ratings spreadsheets. The deadline, as you may know, has been extended to 9 June (tomorrow). Happily, other than a few names/dates being left off (I am saving the emails and renaming the attachments so that I can positively ID each one), the data aggregation process is going as planned and tested. No one, thus far, has failed to provide a 1-7 rating for each of the Eligible Projects.
You may be interested, if not surprised, to learn that not a single project can be excluded from discussion after the individual rating step. Every project’s Range is already > 2 and, of course, it cannot get any tighter as more results are received. Of the 15 Eligible Projects: · 11 or 73% have a Range >= 5 · 7 or 46% have a Range = 6 (max)
I have developed a consolidation spreadsheet, which is automatically color-coded to reveal the top/bottom ratings and the most prevalent answer (or Mode). Fortunately, several projects have pretty stable Mode/Median/Mean results meaning that, while we might have a couple of 7’s and 1’s (thus Range=6), most participants rated the project similarly. In a few cases, the Mode, Median, and Mean are the identical value indicating strong central tendency (so far)! In those instances, at least theoretically, it should be possible to influence the small number of outliers to move closer to the group’s most common rating. Even if that is not possible, after discussion, it will be somewhat comforting to know that there was reasonably strong agreement statistically.
For Brussels, I estimate that we will have about 105 minutes net (if we can hold preliminaries to 15), which leaves an average of 7 minutes per project for discussion and polling.
I am currently drafting a letter that I plan to send out early next week (14th or 15th) addressing as many preliminaries as possible so that the Brussels meeting (on Saturday morning) can be quickly focused on the group ratings discussions. This letter will cover such topics as: Councilor Preparation, Meeting Setup, Guiding Principles, and Process Flow (briefly). I will be encouraging participants to arrive a few minutes early so that we can speed up the routine process of settling in…
If WPM-DT members would like to preview the letter before it goes out, please let me know. Although I recognize that you are all very busy, I would appreciate another set of eyes on this next communication...
Regards,
Ken Bour