Critique of INTA survey
Hi folks, I'm not sure how many have had a chance to read the INTA materials for tomorrow's call yet, or have any background in statistics, but the survey has truly deep and fatal flaws, making any conclusions drawn from it entirely unreliable and non-robust. I could write 50 pages on this (I've read the report three times now, in horror), but I'll keep it relatively brief (and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut). The entire basis of statistical inference is that one can make statements about an entire population with a certain level of confidence using only data from a subset of that population (i.e. the sample in question). Prerequisites are that (a) the sample be random, and (b) the sample be of sufficient size. INTA's study fails on both counts (self-selected and unrepresentative sample, and a mere 33 responses). INTA claims to represent 7,000 organizations as members: https://www.inta.org/About/Pages/Overview.aspx While they acknowledge on page 5 of the slides the small sample size and suggest "some caution", alarm bells should be ringing regarding that small sample size. Page 6 then demonstrates how unrepresentative and non-random that sample is, with 52% of the 33 respondents having total revenue exceeding $5 billion/year, and a whopping 77% (27%+52%) having revenues exceeding $1 billion. This is hardly representative of typical TM owners. Similarly, 39% of this sample had 25,000 or more employees, and 78% (39%+39%) had 5,000 or more employees. All throughout the report, the slides say "INTA members" (i.e. wrongly attempting to extrapolate and assert a truth about the entire population, rather than limiting the statements to be applicable only to the sample of 33 respondents). Basic sanity checks were not done with those extrapolations/inferences. On page 25, the report asserts that "more than 4 in 10 members have applied to operate a new TLD"? 45% of 7000 members implies 3,150 INTA members applied for new gTLDs. That's not correct. The total applications by everyone was 1930 -- see https://newgtlds.icann.org/en/program-status/statistics, and the number by brand owners is a subset of that total (664 according to https://icannwiki.org/Brand_TLD and that will be a bit high, due to multiple applications). If one extrapolated that to the entire universe of trademark holders (i.e. including non-INTA members), millions of TM owners, it would be even more obvious how unrepresentative and non-random the data in this sample is relative to a "typical" TM holder. This sample is highly skewed to the largest of the large organizations who happened to self-select a response to this survey. All throughout the report, important data on confidence intervals is missing, obscuring the fact that the level of confidence is extremely low (and the margin of error is high) due to the small sample size. [confidence intervals are statements like "+/- 5%, 19 times out of 20] There are actually calculators that let one know how big a sample should be, in order to have a certain level of confidence and/or a margin of error. e.g. see: https://www.surveymonkey.com/mp/sample-size-calculator/ For a population size of 7000 members (INTA's total membership) and a 95% confidence level, with a huge 10% margin of error, you'd still need 95 survey responses. Yet, there were only 33 responses. This is particularly important to be kept in mind for charts with percentages (pp. 17 and beyond), where the margin of error, even if sampled properly, would be enormous. Furthermore, those would have had to have been RANDOMLY sampled responses to be proper, which we know isn't the case. If you wanted smaller margins of error, say +/- 5%, you need an even larger sample size (in this case, 365). Another useful calculator is at: https://www.surveymonkey.com/mp/margin-of-error-calculator/ ICANN has done surveys, by Nielsen even, that didn't suffer from these deficiencies, e.g. see: https://newgtlds.icann.org/en/reviews/cct/registrant-survey-faqs-25sep15-en A key takeaway from that work was "Due to a low response rate to emailed invitations to complete the survey, ICANN then worked with Domain Tools to procure a larger sample of WHOIS records." They took greater care in that study to have *randomized* samples, too, along with the larger sample size. While it is somewhat interesting to have a glimpse into brand protection of some of the largest companies, ultimately this study is not robust. In summary, any conclusions from this INTA study really need to be taken with a grain of salt, due to the small sample size, combined with the non-random and unrepresentative sample itself. Indeed, many of the conclusions need to be read as the *opposite* of what the study suggests (i.e. if defensive costs are $150K/year for companies with $5 billion+ in revenues, that's a drop in the bucket, and would be much, much smaller for a "typical" TM owner). To correct these deficiencies, future surveys need to be random (easily done, e.g. random sample the USPTO database or other national registries) and have a much larger sample size. Understandably, that costs money, but that's what it takes to do things properly. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Mon, Aug 28, 2017 at 4:02 PM, Mary Wong <mary.wong@icann.org> wrote:
Dear all,
The proposed agenda for our next Working Group call, scheduled for 0300 UTC on Thursday 31 August, is as follows:
Roll call (via Adobe Connect and phone bridge only); updates to Statements of Interest Review and discuss results of INTA Cost Impact Survey Next steps/next meeting
For Agenda Item #2, please review the survey results here: https://community.icann.org/download/attachments/61606864/INTA%20Cost%20Impa...
Lori Schulman of INTA, and a member of this Working Group, also did a presentation of the results to the Competition, Consumer Protection & Consumer Trust (CCT) Review Team recently that may be helpful to review: https://community.icann.org/download/attachments/61606864/ICANN%20New%20gTLD.... We are hopeful that Lori will be able to join us for this call, to facilitate our review and discussion.
Thanks and cheers
Mary
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi again, On Tue, Aug 29, 2017 at 5:05 PM, George Kirikos <icann@leap.com> wrote:
total revenue exceeding $5 billion/year, and a whopping 77% (27%+52%) having revenues exceeding $1 billion.
Apologies for the typo. That's obviously 79% (not 77%). Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
Hi Everyone: With the exception of the hyperbole at the outset and some slight garbling on confidence level definitions, I largely agree with George. The extremely low response rate and (more importantly) lack of randomness in the sample of responders essentially prohibits one from drawing any conclusions about the population as a whole. IF the selection was random, then the survey margin of error would be about +/- 18% with a 95% confidence level. This would mean we are 95% sure the margin of error is 18% or less and there is a 5% chance the margin of error is greater. However, “random" means that we picked the 33 members that responded essentially out of a hat filled with the names of the 6600 members. But that is not what happened. The survey probably received responses from their most DNS-savvy members - those that found the purpose of the survey interesting or where the questions seemed more straight-forward. This significantly skews the results. Georges email demonstrates this in more detail. I don’t think the survey deals with the skewed data set fairly or honestly. The survey characterizes the findings as traits of the entire membership rather than as traits of the population that responded. This can’t be defended. For instance, I don’t think it is correct to say: "Vast majority (97%) of members registered domain names in past 24 months, with 9 in 10 registering new TLDs,“ and "9 in 10 members have registered new TLD domains in the past two years in the Sunrise Period." I think new TLD owners would be very surprised (and happy) to hear this. It would be accurate to say that "97% of the respondents registered….” (See slides 9 and 11) More harmful to the credibility of the study are statements such as: 3 in 4 members (76%) have incurred costs for internet monitoring of trademarks in the past 2 years, with more than half (57%) of the members spending $10k or more. (see slide 12) On average, INTA members spend $150,000 per year on defensive actions (see slides 10 & 27) These are the types of quotes that find themselves into print and become believed. (“INTA members spend $150K each in defensive efforts, a ~$1 billion cost to industry!”) As George noted, as a rule, larger companies responded and so it can not established by this survey that each of the remaining (smaller) INTA members average $150,000 per year in defensive spend. There is another interesting facet to the asserted $150K / year spend rate. One company spent $5.2MM. Assuming this $5.2MM spend was over a two-year period, that means that the other 32 respondents averaged (33 x $292K — $5.2MM) / (32 x 2) = $69,000 / year. So except for one outlier, the per year spend by the brand owners that chose to answer the study is half of what the study states. Why didn’t the study make this clear? (see slide 10). I am not sure of the purpose of the study but there are uses that can be made of it: There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51) Also, the study makes one fact clear that we have already supposed: that business are not aware of new gTLDs and domain utility in general. There are several data sets that point to this. Rather than education efforts that identify costs and target abuse prevention and mitigation only, Brand education could also describe the benefits of domains as strategic tools, that provide greater access to products and indicia of reliability to brands’ customers. I know this was way pedantic. Sorry. I can’t be on the call as it is at 4AM my time but I’d be pleased to respond to comments or questions. Best regards, Kurt
On Aug 29, 2017, at 10:05 PM, George Kirikos <icann@leap.com> wrote:
Hi folks,
I'm not sure how many have had a chance to read the INTA materials for tomorrow's call yet, or have any background in statistics, but the survey has truly deep and fatal flaws, making any conclusions drawn from it entirely unreliable and non-robust.
I could write 50 pages on this (I've read the report three times now, in horror), but I'll keep it relatively brief (and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut).
The entire basis of statistical inference is that one can make statements about an entire population with a certain level of confidence using only data from a subset of that population (i.e. the sample in question). Prerequisites are that (a) the sample be random, and (b) the sample be of sufficient size. INTA's study fails on both counts (self-selected and unrepresentative sample, and a mere 33 responses).
INTA claims to represent 7,000 organizations as members:
https://www.inta.org/About/Pages/Overview.aspx
While they acknowledge on page 5 of the slides the small sample size and suggest "some caution", alarm bells should be ringing regarding that small sample size. Page 6 then demonstrates how unrepresentative and non-random that sample is, with 52% of the 33 respondents having total revenue exceeding $5 billion/year, and a whopping 77% (27%+52%) having revenues exceeding $1 billion. This is hardly representative of typical TM owners. Similarly, 39% of this sample had 25,000 or more employees, and 78% (39%+39%) had 5,000 or more employees.
All throughout the report, the slides say "INTA members" (i.e. wrongly attempting to extrapolate and assert a truth about the entire population, rather than limiting the statements to be applicable only to the sample of 33 respondents).
Basic sanity checks were not done with those extrapolations/inferences. On page 25, the report asserts that "more than 4 in 10 members have applied to operate a new TLD"?
45% of 7000 members implies 3,150 INTA members applied for new gTLDs. That's not correct. The total applications by everyone was 1930 -- see https://newgtlds.icann.org/en/program-status/statistics, and the number by brand owners is a subset of that total (664 according to https://icannwiki.org/Brand_TLD and that will be a bit high, due to multiple applications). If one extrapolated that to the entire universe of trademark holders (i.e. including non-INTA members), millions of TM owners, it would be even more obvious how unrepresentative and non-random the data in this sample is relative to a "typical" TM holder. This sample is highly skewed to the largest of the large organizations who happened to self-select a response to this survey.
All throughout the report, important data on confidence intervals is missing, obscuring the fact that the level of confidence is extremely low (and the margin of error is high) due to the small sample size. [confidence intervals are statements like "+/- 5%, 19 times out of 20]
There are actually calculators that let one know how big a sample should be, in order to have a certain level of confidence and/or a margin of error.
e.g. see: https://www.surveymonkey.com/mp/sample-size-calculator/
For a population size of 7000 members (INTA's total membership) and a 95% confidence level, with a huge 10% margin of error, you'd still need 95 survey responses. Yet, there were only 33 responses. This is particularly important to be kept in mind for charts with percentages (pp. 17 and beyond), where the margin of error, even if sampled properly, would be enormous. Furthermore, those would have had to have been RANDOMLY sampled responses to be proper, which we know isn't the case. If you wanted smaller margins of error, say +/- 5%, you need an even larger sample size (in this case, 365). Another useful calculator is at:
https://www.surveymonkey.com/mp/margin-of-error-calculator/
ICANN has done surveys, by Nielsen even, that didn't suffer from these deficiencies, e.g. see:
https://newgtlds.icann.org/en/reviews/cct/registrant-survey-faqs-25sep15-en
A key takeaway from that work was "Due to a low response rate to emailed invitations to complete the survey, ICANN then worked with Domain Tools to procure a larger sample of WHOIS records." They took greater care in that study to have *randomized* samples, too, along with the larger sample size.
While it is somewhat interesting to have a glimpse into brand protection of some of the largest companies, ultimately this study is not robust.
In summary, any conclusions from this INTA study really need to be taken with a grain of salt, due to the small sample size, combined with the non-random and unrepresentative sample itself. Indeed, many of the conclusions need to be read as the *opposite* of what the study suggests (i.e. if defensive costs are $150K/year for companies with $5 billion+ in revenues, that's a drop in the bucket, and would be much, much smaller for a "typical" TM owner). To correct these deficiencies, future surveys need to be random (easily done, e.g. random sample the USPTO database or other national registries) and have a much larger sample size. Understandably, that costs money, but that's what it takes to do things properly.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Mon, Aug 28, 2017 at 4:02 PM, Mary Wong <mary.wong@icann.org> wrote:
Dear all,
The proposed agenda for our next Working Group call, scheduled for 0300 UTC on Thursday 31 August, is as follows:
Roll call (via Adobe Connect and phone bridge only); updates to Statements of Interest Review and discuss results of INTA Cost Impact Survey Next steps/next meeting
For Agenda Item #2, please review the survey results here: https://community.icann.org/download/attachments/61606864/INTA%20Cost%20Impa...
Lori Schulman of INTA, and a member of this Working Group, also did a presentation of the results to the Competition, Consumer Protection & Consumer Trust (CCT) Review Team recently that may be helpful to review: https://community.icann.org/download/attachments/61606864/ICANN%20New%20gTLD.... We are hopeful that Lori will be able to join us for this call, to facilitate our review and discussion.
Thanks and cheers
Mary
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi Kurt, Thanks for mostly agreeing with my analysis. However: On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits. Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics. To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that: https://www.inta.org/Membership/Pages/Membership.aspx "63% of our member organizations are outside of North America." This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness). This is a classic case of "If you torture the data long enough, it will confess to anything." Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible. It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard. The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
Dear All, This working group chairs requested that I present INTA's survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening's call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience. Lori Lori S. Schulman Senior Director, Internet Policy International Trademark Association (INTA) +1-202-704-0408, Skype: lsschulman From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible. It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard. The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com><mailto:icann@leap.com%3e> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com><mailto:kurt@kjpritz.com%3e> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg ________________________________
Lori, It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with. I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hello everyone, It may be helpful to recall that the INTA survey was expressly referenced in the Draft Report of the Competition, Consumer Protection & Consumer Trust (CCT) Review Team, published in March this year, in the section of the report that deals with Rights Protection Mechanisms. You may recall also that several of the CCT Review Team’s draft recommendations were also addressed to our Working Group. In addition, our Working Group Charter expressly requires us to track the work of the CCT Review Team, whose final recommendations are expected later this year. As such, and in a similar fashion to the Analysis Group’s report on the Trademark Clearinghouse (TMCH), the INTA survey results can be seen as data-based input into the PDP deliberations. As others have noted, what the Working Group then decides is the utility of all such input is a matter for the Working Group to discuss and determine. Cheers Mary On 8/30/17, 15:29, "gnso-rpm-wg-bounces@icann.org on behalf of George Kirikos" <gnso-rpm-wg-bounces@icann.org on behalf of icann@leap.com> wrote: Lori, It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with. I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so. Sincerely, George Kirikos 416-588-0269 https://urldefense.proofpoint.com/v2/url?u=http-3A__www.leap.com_&d=DwIGaQ&c... On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote: > Dear All, > > > > This working group chairs requested that I present INTA’s survey results and > that I what I intend to do. I am here to present existing data. It is up > to the group to decide if there is any value here. George comments show > immediate bias and hostility toward the work before we have even started a > discussion. I have stated all along that the study was intended for another > purpose and that we had challenges with conducting it. If the PDP WG > wishes to exclude the findings that is for the group to decide. Everything > we do is a learning. George, if you feel that this evening’s call will > have little or no value to your participation, you have the option of not > dialing in and listening to the recording at your convenience. > > > > Lori > > > > Lori S. Schulman > > Senior Director, Internet Policy > > International Trademark Association (INTA) > > +1-202-704-0408, Skype: lsschulman > > > > From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] > On Behalf Of George Kirikos > Sent: Wednesday, August 30, 2017 3:05 PM > To: gnso-rpm-wg <gnso-rpm-wg@icann.org> > Subject: Re: [gnso-rpm-wg] Critique of INTA survey > > > > P.S. There are roughly 8 hours to go until our scheduled call. I would > invite Lori and/or INTA to simply withdraw the paper from this PDP > (and the other ICANN group to which it was presented), since > ultimately it is not a scientifically valid study. Any conclusions > from it are indefensible. > > It would bring more credibility to INTA to withdraw it, in my opinion, > recognizing it as deeply flawed now, rather than to attempt to defend > it for 90 minutes tonight, and ultimately see it abandoned/ignored by > the PDP. As a group, we're always seeking efficiencies --- withdrawing > this paper and giving everyone back their Wednesday night appears to > me to be "low hanging fruit" in that regard. > > The sooner it's withdrawn, the more time folks will have to make > arrangements to enjoy their Wednesday evening. > > Sincerely, > > George Kirikos > 416-588-0269 > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.leap.com_&d=DwIGaQ&c... > > > On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote: >> Hi Kurt, >> >> Thanks for mostly agreeing with my analysis. However: >> >> On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote: >>> There was one conclusion I could draw. It states that UDRP and Sunrise >>> were >>> the favored rights protection mechanisms, used to a major or moderate >>> extent >>> by 67% and 64% of the respondents respectively. The next most utilized >>> RPMs >>> were Trademark Claims and URS (by 36% and 27% respectively). To me this >>> says >>> that, to those who are in-the-know, Sunrise is a highly-valued RPM and, >>> therefore, should be continued. (Sorry, George) (see slides 15 and 51) >> >> The first part of your conclusion is correct (obviously anyone who >> personally benefits from "front of the line" privileges is going to >> value it), but the second part (therefore, that it should be >> continued) is NOT correct. As a PDP, our job is to weigh the benefits >> against the costs of policy choices amongst ALL stakeholders, not just >> ones receiving benefits. >> >> Thus, if that was "the one conclusion (you) could draw", and it's now >> debunked, then we're left with the truth, that no conclusions can be >> drawn from it --- it's for entertainment value only, i.e. it's an >> advocacy piece, marketing fluff, not a scientifically-valid survey >> that would endure any serious peer review from those in the field of >> statistics. >> >> To be clear, I tried to keep yesterday's email as short as possible >> (remember, it was a response to a very long document), and didn't >> point out every flaw with the survey. To point out another, note that >> on page 6 it notes that 67% of responses were from USA and Canada. >> However, INTA's own website states that: >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.inta.org_Membership... >> >> "63% of our member organizations are outside of North America." >> >> This further reinforces my point that it was an unrepresentative >> sample. As we know from election polling, the survey companies make >> adjustments in weighting to attempt to compensate for the >> unrepresentative samples (e.g. if too many men were sampled relative >> to the known proportion, they'd lower the weights accordingly, etc.). >> No attempts were made to do this (nor could they credibly have done >> so, given the small sample size, and lack of randomness). >> >> This is a classic case of "If you torture the data long enough, it >> will confess to anything." >> >> Sincerely, >> >> George Kirikos >> 416-588-0269 >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.leap.com_&d=DwIGaQ&c... > _______________________________________________ > gnso-rpm-wg mailing list > gnso-rpm-wg@icann.org > https://mm.icann.org/mailman/listinfo/gnso-rpm-wg > _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Georges, I am going to ask that all tough questions be put in writing so I can review and discuss with the survey’s administrators if I can’t answer them myself. As you noted, there is a low response rate and the analysis is complex and may not apply to this group’s work. If you have your questions already formulated, please submit them to the list. Lori Lori S. Schulman Senior Director, Internet Policy International Trademark Association (INTA) +1-202-704-0408, Skype: lsschulman From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:29 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey Lori, It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with. I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org><mailto:lschulman@inta.org%3e> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org> [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org><mailto:gnso-rpm-wg@icann.org%3e> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com><mailto:icann@leap.com%3e> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com><mailto:kurt@kjpritz.com%3e> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg ________________________________
Hi Lori, [It's "George", by the way. There's another person named "Georges" on this list] I already submitted my comments, nearly 24 hours ago, and already gave ample notice. The 2nd paragraph of my email even said: "and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut" The questions I ask will flow from that analysis, today's followup, and also Kurt's email (since he won't be there to ask them himself). There are no surprises here. No one's trying to ambush you. Here's the very first question I will ask: "In order to draw valid conclusions from a survey, the sample must be both randomly selected, and the sample size be of sufficient size. Isn't it correct that the INTA survey is (a) a self-selected, non-random and unrepresentative sample, and (b) far too small, in order to make any statistically reliable conclusions about the larger population it purports to represent?" Any answer but "Yes, you're correct George" will be difficult to defend. The emails (from both myself and Kurt) give clear examples demonstrating how unrepresentative it is, and even links to relevant calculators for determining how large a sample size needs to be for various error margins, etc. After that, assuming you answer "Yes", a natural followup would be "Do you withdraw the study?" The best answer to that would be "Yes". Note, that the invitation (very politely offered -- to save everyone time) was given to withdraw it 8 hours before the conference call. It's now 7 hours before the conference call. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 3:45 PM, Lori Schulman <lschulman@inta.org> wrote:
Georges,
I am going to ask that all tough questions be put in writing so I can review and discuss with the survey’s administrators if I can’t answer them myself. As you noted, there is a low response rate and the analysis is complex and may not apply to this group’s work. If you have your questions already formulated, please submit them to the list.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:29 PM
To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Lori,
It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with.
I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi all, Rather interesting what George dug up here. But how do we judge this? As a Registrar, and we are not brand focussed at all, as we are a Wholesale Registrar, I do note that that sunrise period does seem to work for resellers who want to use it for their customers. UDRP, besides some personal issues it seems to work for most folks? From my perspective, there is this survey that George questions due to the low amount of participants. I am looking at what I see in the real world, and I do not see many issues currently. Sure there could be improvements, but again how do we judge this? I am just looking at the reality of things, and so far things are somewhat okay. How do we connect this and gap the divide? Thanks, Theo Geurts Compliance & Policy Officer | Realtime Register B.V. Ceintuurbaan 32A 8024 AA - ZWOLLE - The Netherlands T: +31.384530759 F: +31.384524734 U: www.realtimeregister.com E: legal@realtimeregister.com Skype: geurts.theo On 30-8-2017 22:02, George Kirikos wrote:
Hi Lori,
[It's "George", by the way. There's another person named "Georges" on this list]
I already submitted my comments, nearly 24 hours ago, and already gave ample notice. The 2nd paragraph of my email even said:
"and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut"
The questions I ask will flow from that analysis, today's followup, and also Kurt's email (since he won't be there to ask them himself). There are no surprises here. No one's trying to ambush you.
Here's the very first question I will ask:
"In order to draw valid conclusions from a survey, the sample must be both randomly selected, and the sample size be of sufficient size. Isn't it correct that the INTA survey is (a) a self-selected, non-random and unrepresentative sample, and (b) far too small, in order to make any statistically reliable conclusions about the larger population it purports to represent?"
Any answer but "Yes, you're correct George" will be difficult to defend. The emails (from both myself and Kurt) give clear examples demonstrating how unrepresentative it is, and even links to relevant calculators for determining how large a sample size needs to be for various error margins, etc.
After that, assuming you answer "Yes", a natural followup would be "Do you withdraw the study?" The best answer to that would be "Yes". Note, that the invitation (very politely offered -- to save everyone time) was given to withdraw it 8 hours before the conference call. It's now 7 hours before the conference call.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:45 PM, Lori Schulman <lschulman@inta.org> wrote:
Georges,
I am going to ask that all tough questions be put in writing so I can review and discuss with the survey’s administrators if I can’t answer them myself. As you noted, there is a low response rate and the analysis is complex and may not apply to this group’s work. If you have your questions already formulated, please submit them to the list.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:29 PM
To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Lori,
It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with.
I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51) The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi Theo, On Wed, Aug 30, 2017 at 4:23 PM, theo geurts <gtheo@xs4all.nl> wrote:
From my perspective, there is this survey that George questions due to the low amount of participants.
It wasn't just the small sample size. It's also the fact that it's a non-random, self-selected and entirely unrepresentative sample. For example, if you wanted to determine the average height of adults in the USA, and survey 3000 current or former NBA players, that would be a large sample size. However, it would fail the other part, namely being non-random, entirely unrepresentative of the entire population of adults, etc. The combination of both defects in the INTA study (small sample size, and also non-random) makes the study essentially worthless for saying anything about the larger population it purports to represent. Lori talked about "Everything we do is a learning." The only learning that will happen tonight, unfortunately, is a lesson in how not to a proper survey. That might be of some value, in helping to design the future surveys (e.g. making them much shorter, to encourage greater number of responses; doing proper randomization, like the other study I mentioned in my first email of this thread, investing more money, etc.). Nielsen obviously knows how to do things right -- they're a highly respectable organization. Unfortunately in this instance, what they produced can't withstand scrutiny. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
Hi George, As I recall, no one has offered this survey into evidence as conclusive proof of anything. Both subteams (sunrise and claims) called for the WG to review as many data sources as possible, including mere anecdotes. Both subteams were well aware of the limited data available for scrutiny. I think your (and Kurt's) observations are meaningful, in that they will spark a relevant discussion, but that does not mean the WG should ignore the user stories. Thirty-three companies took significant time and resources to provide their feedback and it's incumbent upon us to at least listen. This WG, though the co-chairs, have asked Lori to present the survey. I think our responsibility here is to listen respectfully and determine what we may be able to use or build on. Thanks, Kristine -----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 1:35 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey Hi Theo, On Wed, Aug 30, 2017 at 4:23 PM, theo geurts <gtheo@xs4all.nl> wrote:
From my perspective, there is this survey that George questions due to the low amount of participants.
It wasn't just the small sample size. It's also the fact that it's a non-random, self-selected and entirely unrepresentative sample. For example, if you wanted to determine the average height of adults in the USA, and survey 3000 current or former NBA players, that would be a large sample size. However, it would fail the other part, namely being non-random, entirely unrepresentative of the entire population of adults, etc. The combination of both defects in the INTA study (small sample size, and also non-random) makes the study essentially worthless for saying anything about the larger population it purports to represent. Lori talked about "Everything we do is a learning." The only learning that will happen tonight, unfortunately, is a lesson in how not to a proper survey. That might be of some value, in helping to design the future surveys (e.g. making them much shorter, to encourage greater number of responses; doing proper randomization, like the other study I mentioned in my first email of this thread, investing more money, etc.). Nielsen obviously knows how to do things right -- they're a highly respectable organization. Unfortunately in this instance, what they produced can't withstand scrutiny. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
I agree with Kristine. George, Are you aware of other surveys commissioned on the costs imposed by the new gTLD program that we can review? Personally, I believe the responsibility of conducting this falls on ICANN, since ICANN is the organization that has responsibility over coordinating the DNS, introducing new gTLDs, and for the aftermath that follows. Has the Board directed that these studies be commissioned? If not, why not? The organization has known about these issues for a long time, in fact, the call for economic study was one of the main issues identified during implementation of the program. In my view, INTA and those who expended the time and energy to complete this work, should be commended for taking on this substantial and complex task. Mary's note described the context of this work, and Lori stated that feedback and constructive criticism is both welcome and encouraged. As a result, I am confident we are following the right path. Best, Claudio On Wed, Aug 30, 2017 at 5:23 PM Dorrain, Kristine via gnso-rpm-wg < gnso-rpm-wg@icann.org> wrote:
Hi George,
As I recall, no one has offered this survey into evidence as conclusive proof of anything. Both subteams (sunrise and claims) called for the WG to review as many data sources as possible, including mere anecdotes. Both subteams were well aware of the limited data available for scrutiny. I think your (and Kurt's) observations are meaningful, in that they will spark a relevant discussion, but that does not mean the WG should ignore the user stories. Thirty-three companies took significant time and resources to provide their feedback and it's incumbent upon us to at least listen.
This WG, though the co-chairs, have asked Lori to present the survey. I think our responsibility here is to listen respectfully and determine what we may be able to use or build on.
Thanks,
Kristine
-----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 1:35 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Hi Theo,
On Wed, Aug 30, 2017 at 4:23 PM, theo geurts <gtheo@xs4all.nl> wrote:
From my perspective, there is this survey that George questions due to the low amount of participants.
It wasn't just the small sample size. It's also the fact that it's a non-random, self-selected and entirely unrepresentative sample.
For example, if you wanted to determine the average height of adults in the USA, and survey 3000 current or former NBA players, that would be a large sample size. However, it would fail the other part, namely being non-random, entirely unrepresentative of the entire population of adults, etc.
The combination of both defects in the INTA study (small sample size, and also non-random) makes the study essentially worthless for saying anything about the larger population it purports to represent.
Lori talked about "Everything we do is a learning." The only learning that will happen tonight, unfortunately, is a lesson in how not to a proper survey. That might be of some value, in helping to design the future surveys (e.g. making them much shorter, to encourage greater number of responses; doing proper randomization, like the other study I mentioned in my first email of this thread, investing more money, etc.). Nielsen obviously knows how to do things right -- they're a highly respectable organization. Unfortunately in this instance, what they produced can't withstand scrutiny.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/ _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi George & Lori, I think the survey may be useful as long as we recognise those filling out the survey are mostly from the largest members of the INTA. I think NTIA dill well getting such substantive replies from these participants. Theses replies may be helpful as the benefits and problems they have with new gTLDs aree likely to be very different from medium sized and smaller enterprises. It may be when considering just the largest members the statistical signifance of the survey changes? With this in mind I think it would be helpful if an extra column could be added to each of the tables on page 6. And in that column show the Total NTIA members percentages i.e. the percentages of the 6,600 next to the each of the %ages from the 33 that replied. I will not be on the call this evening as it will be 4.00am local time here in Europe, but I would be grateful Lori if you have time to talk on slide 22 as I can review the transcript. On slide 22 can you explain a bit more about the differences between the two columns and why the second column isn't simply the reverse of column 1? Best regards, Paul. On Wed, Aug 30, 2017 at 9:02 PM, George Kirikos <icann@leap.com> wrote:
Hi Lori,
[It's "George", by the way. There's another person named "Georges" on this list]
I already submitted my comments, nearly 24 hours ago, and already gave ample notice. The 2nd paragraph of my email even said:
"and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut"
The questions I ask will flow from that analysis, today's followup, and also Kurt's email (since he won't be there to ask them himself). There are no surprises here. No one's trying to ambush you.
Here's the very first question I will ask:
"In order to draw valid conclusions from a survey, the sample must be both randomly selected, and the sample size be of sufficient size. Isn't it correct that the INTA survey is (a) a self-selected, non-random and unrepresentative sample, and (b) far too small, in order to make any statistically reliable conclusions about the larger population it purports to represent?"
Any answer but "Yes, you're correct George" will be difficult to defend. The emails (from both myself and Kurt) give clear examples demonstrating how unrepresentative it is, and even links to relevant calculators for determining how large a sample size needs to be for various error margins, etc.
After that, assuming you answer "Yes", a natural followup would be "Do you withdraw the study?" The best answer to that would be "Yes". Note, that the invitation (very politely offered -- to save everyone time) was given to withdraw it 8 hours before the conference call. It's now 7 hours before the conference call.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:45 PM, Lori Schulman <lschulman@inta.org> wrote:
Georges,
I am going to ask that all tough questions be put in writing so I can review and discuss with the survey’s administrators if I can’t answer them myself. As you noted, there is a low response rate and the analysis is complex and may not apply to this group’s work. If you have your questions already formulated, please submit them to the list.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@ icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:29 PM
To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Lori,
It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with.
I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@ icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Dear Paul, Thank you for your feedback. I will look at Slide 22 and see if I can answer. I will explain on the call why the 1000 were chosen out of the 6600. 1000 are essentially “brand owners”. The others are organizations that support brand owners which could include law firms, trademark agents, search services, brand protection services, docket management services and even registrars and registries. The rationale for exclusion goes to the practicalities of discerning the service providers from brand owners and the desire to avoid duplicative answers from the brand owners and those who provide services to them and could be answering for them. Also, in terms of our membership the brand owners are mostly from the US and Canada and the service providers from outside of the US. The geographical range in the survey respondents represents the geographical range of our brand owners with about 70% coming from Canada and USA. 30% from other regions. Lori S. Schulman Senior Director, Internet Policy International Trademark Association (INTA) +1-202-704-0408, Skype: lsschulman From: Paul Tattersfield [mailto:gpmgroup@gmail.com] Sent: Wednesday, August 30, 2017 6:15 PM To: George Kirikos <icann@leap.com> Cc: Lori Schulman <lschulman@inta.org>; gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey Hi George & Lori, I think the survey may be useful as long as we recognise those filling out the survey are mostly from the largest members of the INTA. I think NTIA dill well getting such substantive replies from these participants. Theses replies may be helpful as the benefits and problems they have with new gTLDs aree likely to be very different from medium sized and smaller enterprises. It may be when considering just the largest members the statistical signifance of the survey changes? With this in mind I think it would be helpful if an extra column could be added to each of the tables on page 6. And in that column show the Total NTIA members percentages i.e. the percentages of the 6,600 next to the each of the %ages from the 33 that replied. I will not be on the call this evening as it will be 4.00am local time here in Europe, but I would be grateful Lori if you have time to talk on slide 22 as I can review the transcript. On slide 22 can you explain a bit more about the differences between the two columns and why the second column isn't simply the reverse of column 1? Best regards, Paul. On Wed, Aug 30, 2017 at 9:02 PM, George Kirikos <icann@leap.com<mailto:icann@leap.com>> wrote: Hi Lori, [It's "George", by the way. There's another person named "Georges" on this list] I already submitted my comments, nearly 24 hours ago, and already gave ample notice. The 2nd paragraph of my email even said: "and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut" The questions I ask will flow from that analysis, today's followup, and also Kurt's email (since he won't be there to ask them himself). There are no surprises here. No one's trying to ambush you. Here's the very first question I will ask: "In order to draw valid conclusions from a survey, the sample must be both randomly selected, and the sample size be of sufficient size. Isn't it correct that the INTA survey is (a) a self-selected, non-random and unrepresentative sample, and (b) far too small, in order to make any statistically reliable conclusions about the larger population it purports to represent?" Any answer but "Yes, you're correct George" will be difficult to defend. The emails (from both myself and Kurt) give clear examples demonstrating how unrepresentative it is, and even links to relevant calculators for determining how large a sample size needs to be for various error margins, etc. After that, assuming you answer "Yes", a natural followup would be "Do you withdraw the study?" The best answer to that would be "Yes". Note, that the invitation (very politely offered -- to save everyone time) was given to withdraw it 8 hours before the conference call. It's now 7 hours before the conference call. Sincerely, George Kirikos 416-588-0269<tel:416-588-0269> http://www.leap.com/ On Wed, Aug 30, 2017 at 3:45 PM, Lori Schulman <lschulman@inta.org<mailto:lschulman@inta.org>> wrote:
Georges,
I am going to ask that all tough questions be put in writing so I can review and discuss with the survey’s administrators if I can’t answer them myself. As you noted, there is a low response rate and the analysis is complex and may not apply to this group’s work. If you have your questions already formulated, please submit them to the list.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408<tel:%2B1-202-704-0408>, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org> [mailto:gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org>] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:29 PM
To: gnso-rpm-wg <gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org>> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Lori,
It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with.
I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so.
Sincerely,
George Kirikos 416-588-0269<tel:416-588-0269> http://www.leap.com/
On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org<mailto:lschulman@inta.org>> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408<tel:%2B1-202-704-0408>, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org> [mailto:gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org>] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org>> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269<tel:416-588-0269> http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com<mailto:icann@leap.com>> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com<mailto:kurt@kjpritz.com>> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269<tel:416-588-0269> http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg ________________________________
Hi George, I've read with some amusement your various critiques of the INTA survey. However, I could not find even imperfect surveys that you have submitted that tend to support your various positions taken on this list and on calls. I hope you will be as open minded about what we can learn from what you view as a flawed survey as we have all been about your positions taken based on non-surveys. However, if you have submitted surveys that tend to back your various positions -especially scientifically perfect ones - and I simply overlooked them, could you send them round again now? Thanks. Best, Paul -----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 2:29 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey Lori, It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with. I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg ________________________________ The contents of this message may be privileged and confidential. If this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author. Any tax advice contained in this email was not intended to be used, and cannot be used, by you (or any other taxpayer) to avoid penalties under applicable tax laws and regulations.
Hi Paul, On Wed, Aug 30, 2017 at 10:07 PM, icannlists <icannlists@winston.com> wrote:
I've read with some amusement your various critiques of the INTA survey. However, I could not find even imperfect surveys that you have submitted that tend to support your various positions taken on this list and on calls. I hope you will be as open minded about what we can learn from what you view as a flawed survey as we have all been about your positions taken based on non-surveys. However, if you have submitted surveys that tend to back your various positions -especially scientifically perfect ones - and I simply overlooked them, could you send them round again now? Thanks.
If you saw a specific fault in the analysis I provided (or Kurt's), please be specific/precise about what you feel is incorrect. If you accept the analysis as correct, and your only point is "nobody's perfect", that's a weak argument. Bad data is often worse than no data at all, because bad data can lead one towards making bad policy choices, and indeed even embolden that decision-making (because there's a pretense that it was supported by data). For instance, if there was an automotive transportation policy survey, and 80% of the non-random responses (all 33 of them) were from Rolls Royce owners (an unrepresentative small sample), it should be obvious that the survey would likely lead policymakers to make incorrect decisions for the population of 100 million+ drivers. An invalid survey (due to a non-random sample combined with a far too small sample size) is just that --- there's little to learn from it (except perhaps how to design better surveys). I've not submitted any survey data. I don't have the budget of ICANN or INTA supporting my participation in this PDP. My positions are backed by sound and logical reasoning, with deep analysis of the data that this group has had access to (e.g. answers from the TMCH operator, information from The Analysis Group, etc.). Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
Hi George, Thanks for confirming that you have submitted no surveys, flawed or otherwise, and that your various positions are merely supported by what you believe to be, in self-validating fashion, "backed by sound and logical reasoning." Best, Paul -----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 9:49 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey Hi Paul, On Wed, Aug 30, 2017 at 10:07 PM, icannlists <icannlists@winston.com> wrote:
I've read with some amusement your various critiques of the INTA survey. However, I could not find even imperfect surveys that you have submitted that tend to support your various positions taken on this list and on calls. I hope you will be as open minded about what we can learn from what you view as a flawed survey as we have all been about your positions taken based on non-surveys. However, if you have submitted surveys that tend to back your various positions -especially scientifically perfect ones - and I simply overlooked them, could you send them round again now? Thanks.
If you saw a specific fault in the analysis I provided (or Kurt's), please be specific/precise about what you feel is incorrect. If you accept the analysis as correct, and your only point is "nobody's perfect", that's a weak argument. Bad data is often worse than no data at all, because bad data can lead one towards making bad policy choices, and indeed even embolden that decision-making (because there's a pretense that it was supported by data). For instance, if there was an automotive transportation policy survey, and 80% of the non-random responses (all 33 of them) were from Rolls Royce owners (an unrepresentative small sample), it should be obvious that the survey would likely lead policymakers to make incorrect decisions for the population of 100 million+ drivers. An invalid survey (due to a non-random sample combined with a far too small sample size) is just that --- there's little to learn from it (except perhaps how to design better surveys). I've not submitted any survey data. I don't have the budget of ICANN or INTA supporting my participation in this PDP. My positions are backed by sound and logical reasoning, with deep analysis of the data that this group has had access to (e.g. answers from the TMCH operator, information from The Analysis Group, etc.). Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg ________________________________ The contents of this message may be privileged and confidential. If this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author. Any tax advice contained in this email was not intended to be used, and cannot be used, by you (or any other taxpayer) to avoid penalties under applicable tax laws and regulations.
Hi Paul, That's an obvious misstatement of what I wrote. I looked at the top 10 strings (data) in The Analysis Group report just like everyone else -- they were all common dictionary words. If they had been famous fanciful marks, my conclusions would have been different. I looked at the Deloitte responses to the set of examples that Rebecca submitted for analysis. And so on. I don't know why you keep trying to misstating what I actually wrote. I wrote what I wrote, and the group doesn't need a slanted reinterpretation of it. Kurt and I made a detailed analysis of the INTA survey. If you disagree with any of its points, be specific. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Wed, Aug 30, 2017 at 11:04 PM, icannlists <icannlists@winston.com> wrote:
Hi George,
Thanks for confirming that you have submitted no surveys, flawed or otherwise, and that your various positions are merely supported by what you believe to be, in self-validating fashion, "backed by sound and logical reasoning."
Best, Paul
-----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 9:49 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Hi Paul,
On Wed, Aug 30, 2017 at 10:07 PM, icannlists <icannlists@winston.com> wrote:
I've read with some amusement your various critiques of the INTA survey. However, I could not find even imperfect surveys that you have submitted that tend to support your various positions taken on this list and on calls. I hope you will be as open minded about what we can learn from what you view as a flawed survey as we have all been about your positions taken based on non-surveys. However, if you have submitted surveys that tend to back your various positions -especially scientifically perfect ones - and I simply overlooked them, could you send them round again now? Thanks.
If you saw a specific fault in the analysis I provided (or Kurt's), please be specific/precise about what you feel is incorrect. If you accept the analysis as correct, and your only point is "nobody's perfect", that's a weak argument. Bad data is often worse than no data at all, because bad data can lead one towards making bad policy choices, and indeed even embolden that decision-making (because there's a pretense that it was supported by data).
For instance, if there was an automotive transportation policy survey, and 80% of the non-random responses (all 33 of them) were from Rolls Royce owners (an unrepresentative small sample), it should be obvious that the survey would likely lead policymakers to make incorrect decisions for the population of 100 million+ drivers.
An invalid survey (due to a non-random sample combined with a far too small sample size) is just that --- there's little to learn from it (except perhaps how to design better surveys).
I've not submitted any survey data. I don't have the budget of ICANN or INTA supporting my participation in this PDP. My positions are backed by sound and logical reasoning, with deep analysis of the data that this group has had access to (e.g. answers from the TMCH operator, information from The Analysis Group, etc.).
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/ _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
________________________________ The contents of this message may be privileged and confidential. If this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author. Any tax advice contained in this email was not intended to be used, and cannot be used, by you (or any other taxpayer) to avoid penalties under applicable tax laws and regulations.
Here's a specific fault for you, George. It should be noted that George's challenges start with a basic fallacy -- that "the sample must be ... randomly selected." Not that a random sample is preferable, or that a random sample is better, etc. No -- it's absolute a sample MUST be randomly selected, according to George. This is just plain wrong. Many surveys are done with samples that are not random samples (also known as probability samples). There are various survey methods that involve non-probability samples, which are respected and valid. The idea that a survey is worthless unless it is based on a random sample is hogwash. Random samples are preferable as a starting point, but even there there are various types of biases. One is "non-response bias" -- that members of the selected sample don't respond to the survey. [George seems to ignore the distinction between the initial sample and the responses, but that wouldn't serve his purposes....] This is an important point -- the initial sample is really the 1000+ members of INTA, which is in turn a subset of all trademark owners. The group of respondents is a subset of that sample. I'm not going to go on, though I could. I could even dust off my old stats books from college. In any event, the results exist and there is no reason to "withdraw" this survey. In George's world, things that George doesn't like just go away. Thankfully, only one of us lives in George's world. Greg On Wed, Aug 30, 2017 at 10:07 PM, icannlists <icannlists@winston.com> wrote:
Hi George,
I've read with some amusement your various critiques of the INTA survey. However, I could not find even imperfect surveys that you have submitted that tend to support your various positions taken on this list and on calls. I hope you will be as open minded about what we can learn from what you view as a flawed survey as we have all been about your positions taken based on non-surveys. However, if you have submitted surveys that tend to back your various positions -especially scientifically perfect ones - and I simply overlooked them, could you send them round again now? Thanks.
Best, Paul
-----Original Message----- From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 2:29 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
Lori,
It is false to say the my comments have any bias or hostility. They are sound arguments. I was open to it being a scientifically valid survey, but then I read it, multiple times. So did Kurt (maybe not multiple times for him!?!?), who I have no affiliation with.
I don't know whether the working group chairs were aware of the study's deep flaws before they made the invitation to present it, or had even read it, but now they do. If they want to keep the schedule, I'll be there to ask the tough questions tonight, and let the PDP members that want to try to defend it do so.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 3:22 PM, Lori Schulman <lschulman@inta.org> wrote:
Dear All,
This working group chairs requested that I present INTA’s survey results and that I what I intend to do. I am here to present existing data. It is up to the group to decide if there is any value here. George comments show immediate bias and hostility toward the work before we have even started a discussion. I have stated all along that the study was intended for another purpose and that we had challenges with conducting it. If the PDP WG wishes to exclude the findings that is for the group to decide. Everything we do is a learning. George, if you feel that this evening’s call will have little or no value to your participation, you have the option of not dialing in and listening to the recording at your convenience.
Lori
Lori S. Schulman
Senior Director, Internet Policy
International Trademark Association (INTA)
+1-202-704-0408, Skype: lsschulman
From: gnso-rpm-wg-bounces@icann.org [mailto:gnso-rpm-wg-bounces@icann.org] On Behalf Of George Kirikos Sent: Wednesday, August 30, 2017 3:05 PM To: gnso-rpm-wg <gnso-rpm-wg@icann.org> Subject: Re: [gnso-rpm-wg] Critique of INTA survey
P.S. There are roughly 8 hours to go until our scheduled call. I would invite Lori and/or INTA to simply withdraw the paper from this PDP (and the other ICANN group to which it was presented), since ultimately it is not a scientifically valid study. Any conclusions from it are indefensible.
It would bring more credibility to INTA to withdraw it, in my opinion, recognizing it as deeply flawed now, rather than to attempt to defend it for 90 minutes tonight, and ultimately see it abandoned/ignored by the PDP. As a group, we're always seeking efficiencies --- withdrawing this paper and giving everyone back their Wednesday night appears to me to be "low hanging fruit" in that regard.
The sooner it's withdrawn, the more time folks will have to make arrangements to enjoy their Wednesday evening.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Wed, Aug 30, 2017 at 2:13 PM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
Thanks for mostly agreeing with my analysis. However:
On Wed, Aug 30, 2017 at 1:47 PM, Kurt Pritz <kurt@kjpritz.com> wrote:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
The first part of your conclusion is correct (obviously anyone who personally benefits from "front of the line" privileges is going to value it), but the second part (therefore, that it should be continued) is NOT correct. As a PDP, our job is to weigh the benefits against the costs of policy choices amongst ALL stakeholders, not just ones receiving benefits.
Thus, if that was "the one conclusion (you) could draw", and it's now debunked, then we're left with the truth, that no conclusions can be drawn from it --- it's for entertainment value only, i.e. it's an advocacy piece, marketing fluff, not a scientifically-valid survey that would endure any serious peer review from those in the field of statistics.
To be clear, I tried to keep yesterday's email as short as possible (remember, it was a response to a very long document), and didn't point out every flaw with the survey. To point out another, note that on page 6 it notes that 67% of responses were from USA and Canada. However, INTA's own website states that:
https://www.inta.org/Membership/Pages/Membership.aspx
"63% of our member organizations are outside of North America."
This further reinforces my point that it was an unrepresentative sample. As we know from election polling, the survey companies make adjustments in weighting to attempt to compensate for the unrepresentative samples (e.g. if too many men were sampled relative to the known proportion, they'd lower the weights accordingly, etc.). No attempts were made to do this (nor could they credibly have done so, given the small sample size, and lack of randomness).
This is a classic case of "If you torture the data long enough, it will confess to anything."
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
________________________________ The contents of this message may be privileged and confidential. If this message has been received in error, please delete it without reading it. Your receipt of this message is not intended to waive any applicable privilege. Please do not disseminate this message without the permission of the author. Any tax advice contained in this email was not intended to be used, and cannot be used, by you (or any other taxpayer) to avoid penalties under applicable tax laws and regulations. _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi Greg, On Wed, Aug 30, 2017 at 10:54 PM, Greg Shatan <gregshatanipc@gmail.com> wrote:
Random samples are preferable as a starting point, but even there there are various types of biases. One is "non-response bias" -- that members of the selected sample don't respond to the survey. [George seems to ignore the distinction between the initial sample and the responses, but that wouldn't serve his purposes....]
The report made not attempt to adjust for any of those biases, like self-selection bias, or that it was unrepresentative of the larger (1000+ members) group of INTA members. I didn't "ignore" anything. To anticipate every argument in the first email I sent, and make more in depth would have distracted from the key points. They'd have been lost in a 50 page email.
This is an important point -- the initial sample is really the 1000+ members of INTA, which is in turn a subset of all trademark owners. The group of respondents is a subset of that sample.
How is that an "important point"? We have 33 observations. Is that in dispute? We know those 33 are unrepresentative of the 1000+ members of INTA. And we know from the metrics of INTA that they're unrepresentative of all TM holders (i.e. INTA's membership is skewed large, and the 33 were skewed even larger). What attempts were made to "correct" (if it's even possible) for that? None. It's a lost cause, because there just wasn't enough data to begin with to make those assertions about the larger groups in a statistically robust manner. If ICA or EFF had submitted 33 self-selected survey responses that were skewed in a similar manner, and tried to make assertions about a large population, I'd be equally hard on them, as I've shown in the past through my balanced criticisms. My focus is on the pure math, regardless of where it came from or what the results said. If/when Lori/Nielsen ever comes back to us with the margins of error that were requested in the other thread, those will help demonstrate things further (although Kurt already took an initial stab at things with his calculations; they'd be different for each slide, though). Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
George (et.al.): Despite an ever-mounting sense of regret for my previous intervention, I wish to make a couple points in response to George’ email below and the discussion generally. My sense of frustration is that parties advocating a position will author or refer to a flawed study or paper - and because that paper cites figures or “facts,” people will tend to believe it. Here are three examples: 1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM. In my opinion, that paper contains platitudes, no independent data, and contains half-truths and, also in my opinion, seems more like a guide for those who wish to conduct illicit behavior with the least possible interference from governments or well-intentioned watchdog organizations. Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise. The EFF does good work and I support many of its positions but not in this case. 2) The ICANN study into the Clearinghouse operations said that Clams notices resulted in a 93% abandonment rate. Somewhere else, in relatively tiny font, the authors admitted that the figure was unreliable. Subsequently, the number published has been debunked. Yet that number is still bandied about as true, at great detriment to objective discussion 3) The recent INTA study, it seems to me, did not work out as hoped for the authors. As Kristine said, the anecdotal comments have value but maybe the report should have just focused on that. The risk is that some of the unsubstantiated claims listed in the reports findings will become tomorrow’s headlines and incorrectly shed a bad light on the multi-stakeholder model and the new gTLD program. It matters little that disclaimers were published on another page of the report, the prose will be gainsaid by any advocate for any side of a discussion. My comment the other day was out of frustrations resulting form the fact that authors of studies and papers go ahead and publish data when the better course would be to withhold it. I know that is a tough course to take when a study has been funded but does not turn out as hoped. As a result, we see hard numbers with soft disclaimers, which can lead in incorrect conclusions. Not to make a big deal of it but my first business school class in statistics started with a full session about ethics. We were going to be trained in the use of numbers and it was our duty not to use them inappropriately to make a point. As I said, it wasn’t a big deal but I remember it to this day. I understand papers and studies cannot be perfect and that good can come from an imperfect study — and I am coming to understand that all of us are capable of filtering out unsupported arguments made by those papers without me being pedantic about it. I am sorry for this mess of an email chain to which my initial comment contributed and the rabbit hole down which we went. Kurt
On Aug 31, 2017, at 5:23 AM, George Kirikos <icann@leap.com> wrote:
Hi Greg,
On Wed, Aug 30, 2017 at 10:54 PM, Greg Shatan <gregshatanipc@gmail.com> wrote:
Random samples are preferable as a starting point, but even there there are various types of biases. One is "non-response bias" -- that members of the selected sample don't respond to the survey. [George seems to ignore the distinction between the initial sample and the responses, but that wouldn't serve his purposes....]
The report made not attempt to adjust for any of those biases, like self-selection bias, or that it was unrepresentative of the larger (1000+ members) group of INTA members.
I didn't "ignore" anything. To anticipate every argument in the first email I sent, and make more in depth would have distracted from the key points. They'd have been lost in a 50 page email.
This is an important point -- the initial sample is really the 1000+ members of INTA, which is in turn a subset of all trademark owners. The group of respondents is a subset of that sample.
How is that an "important point"? We have 33 observations. Is that in dispute? We know those 33 are unrepresentative of the 1000+ members of INTA. And we know from the metrics of INTA that they're unrepresentative of all TM holders (i.e. INTA's membership is skewed large, and the 33 were skewed even larger).
What attempts were made to "correct" (if it's even possible) for that? None. It's a lost cause, because there just wasn't enough data to begin with to make those assertions about the larger groups in a statistically robust manner.
If ICA or EFF had submitted 33 self-selected survey responses that were skewed in a similar manner, and tried to make assertions about a large population, I'd be equally hard on them, as I've shown in the past through my balanced criticisms. My focus is on the pure math, regardless of where it came from or what the results said.
If/when Lori/Nielsen ever comes back to us with the margins of error that were requested in the other thread, those will help demonstrate things further (although Kurt already took an initial stab at things with his calculations; they'd be different for each slide, though).
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/ _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
Hi Kurt, On Thu, Aug 31, 2017 at 5:47 AM, Kurt Pritz <kurt@kjpritz.com> wrote: "1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM.....Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise." I don't recall ever referencing that particular paper in this working group. I advocated elimination of the Sunrise RPM long before that paper was even published. That paper only came out in late July: http://mm.icann.org/pipermail/gnso-rpm-wg/2017-July/002256.html and as per the archives, no one ever replied to that thread. I've searched past emails, and can't find myself referencing it all here. My reasoning for elimination of sunrise isn't based at all on that study, especially since the reasoning predated that study. Not only has it not been "cited often", it's unclear anyone but Jeremy has even mentioned it. If you have a link to me (or anyone else for that matter) referencing that paper on this PDP mailing list, I'd appreciate it. Otherwise, I think you should correct your statement. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/
George: I remember Jeremy describing the paper. I was referring to the email chain you started entitled on the heels of that paper, " 99%+ reduction in sunrise utilization rate per TLD supports EFF call for elimination of sunrise,” that, to me, established a link between your arguments and the paper. Kurt
On Aug 31, 2017, at 11:46 AM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
On Thu, Aug 31, 2017 at 5:47 AM, Kurt Pritz <kurt@kjpritz.com> wrote: "1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM.....Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise."
I don't recall ever referencing that particular paper in this working group. I advocated elimination of the Sunrise RPM long before that paper was even published. That paper only came out in late July:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-July/002256.html
and as per the archives, no one ever replied to that thread. I've searched past emails, and can't find myself referencing it all here.
My reasoning for elimination of sunrise isn't based at all on that study, especially since the reasoning predated that study.
Not only has it not been "cited often", it's unclear anyone but Jeremy has even mentioned it.
If you have a link to me (or anyone else for that matter) referencing that paper on this PDP mailing list, I'd appreciate it. Otherwise, I think you should correct your statement.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
Kurt: That doesn't make any sense. Again, I did not reference that late July paper from EFF until you incorrectly linked my postings to it (i.e. until today). I brought up the topic of elimination of the sunrise period on April 13, see: http://mm.icann.org/pipermail/gnso-rpm-wg/2017-April/001509.html "A Brave New World Without Sunrises or the TMCH" EFF made their formal proposal to eliminate the sunrise on April 19th, after I had already made my views known: http://mm.icann.org/pipermail/gnso-rpm-wg/2017-April/001619.html The August 10th thread you mention: http://mm.icann.org/pipermail/gnso-rpm-wg/2017-August/002323.html was **2 weeks** after the EFF paper, and had absolutely no relation to it. To suggest it came "on the heels of it" is incorrect. I was referencing only the formal call for elimination of sunrise that they made back in April (i.e. an ongoing proposal that our PDP is tasked to consider). That July paper: https://www.eff.org/wp/which-internet-registries-offer-best-protection-domai... is completely unrelated. They don't discuss the 99% reduction in sunrise utilization. I was the one that first discussed that 99% figure, in order to support their April proposal. Sincerely, George Kirikos 416-588-0269 http://www.leap.com/ On Thu, Aug 31, 2017 at 7:38 AM, Kurt Pritz <kurt@kjpritz.com> wrote:
George:
I remember Jeremy describing the paper. I was referring to the email chain you started entitled on the heels of that paper, " 99%+ reduction in sunrise utilization rate per TLD supports EFF call for elimination of sunrise,” that, to me, established a link between your arguments and the paper.
Kurt
On Aug 31, 2017, at 11:46 AM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
On Thu, Aug 31, 2017 at 5:47 AM, Kurt Pritz <kurt@kjpritz.com> wrote: "1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM.....Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise."
I don't recall ever referencing that particular paper in this working group. I advocated elimination of the Sunrise RPM long before that paper was even published. That paper only came out in late July:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-July/002256.html
and as per the archives, no one ever replied to that thread. I've searched past emails, and can't find myself referencing it all here.
My reasoning for elimination of sunrise isn't based at all on that study, especially since the reasoning predated that study.
Not only has it not been "cited often", it's unclear anyone but Jeremy has even mentioned it.
If you have a link to me (or anyone else for that matter) referencing that paper on this PDP mailing list, I'd appreciate it. Otherwise, I think you should correct your statement.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
I wanted to say that RiskIQ is an INTA member, and did not participate in this survey. Before joining RiskIQ, I had a solo practice out of New York, and was an INTA member representing both medium size and fairly large companies in trademark matters. In my personal experience, the combination of sunrise and TMCH plays a vital role to mitigate abuse of the DNS with new gTLDs from a brand security perspective. I have found this personally to be true for both relatively small as well as larger companies with brands to secure from abuse. Jonathan Matkowsky, VP - IP & Brand Security RiskIQ, Inc. On Thu, Aug 31, 2017 at 5:08 AM, George Kirikos <icann@leap.com> wrote:
Kurt:
That doesn't make any sense. Again, I did not reference that late July paper from EFF until you incorrectly linked my postings to it (i.e. until today). I brought up the topic of elimination of the sunrise period on April 13, see:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-April/001509.html
"A Brave New World Without Sunrises or the TMCH"
EFF made their formal proposal to eliminate the sunrise on April 19th, after I had already made my views known:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-April/001619.html
The August 10th thread you mention:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-August/002323.html
was **2 weeks** after the EFF paper, and had absolutely no relation to it. To suggest it came "on the heels of it" is incorrect.
I was referencing only the formal call for elimination of sunrise that they made back in April (i.e. an ongoing proposal that our PDP is tasked to consider).
That July paper:
https://www.eff.org/wp/which-internet-registries-offer- best-protection-domain-owners
is completely unrelated. They don't discuss the 99% reduction in sunrise utilization. I was the one that first discussed that 99% figure, in order to support their April proposal.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Thu, Aug 31, 2017 at 7:38 AM, Kurt Pritz <kurt@kjpritz.com> wrote:
George:
I remember Jeremy describing the paper. I was referring to the email chain you started entitled on the heels of that paper, " 99%+ reduction in sunrise utilization rate per TLD supports EFF call for elimination of sunrise,” that, to me, established a link between your arguments and the paper.
Kurt
On Aug 31, 2017, at 11:46 AM, George Kirikos <icann@leap.com> wrote:
Hi Kurt,
On Thu, Aug 31, 2017 at 5:47 AM, Kurt Pritz <kurt@kjpritz.com> wrote: "1) George refers to the EFF and in past emails has particularly referred to the EFF white paper, "Which Internet registries offer the best protection for domain owners?” as justification for eliminated the Sunrise RPM.....Yet, because it is “published," this is the source cited often on this list as authority for eliminating Sunrise."
I don't recall ever referencing that particular paper in this working group. I advocated elimination of the Sunrise RPM long before that paper was even published. That paper only came out in late July:
http://mm.icann.org/pipermail/gnso-rpm-wg/2017-July/002256.html
and as per the archives, no one ever replied to that thread. I've searched past emails, and can't find myself referencing it all here.
My reasoning for elimination of sunrise isn't based at all on that study, especially since the reasoning predated that study.
Not only has it not been "cited often", it's unclear anyone but Jeremy has even mentioned it.
If you have a link to me (or anyone else for that matter) referencing that paper on this PDP mailing list, I'd appreciate it. Otherwise, I think you should correct your statement.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
-- ******************************************************************* This message was sent from RiskIQ, and is intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not a designated recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. *******************************************************************
Never trust a study that you have not tweaked/interpreted towards the desired results yourself... Am 30.08.2017 um 19:47 schrieb Kurt Pritz:
Hi Everyone:
With the exception of the hyperbole at the outset and some slight garbling on confidence level definitions, I largely agree with George.
The extremely low response rate and (more importantly) lack of randomness in the sample of responders essentially prohibits one from drawing any conclusions about the population as a whole. IF the selection was random, then the survey margin of error would be about +/- 18% with a 95% confidence level. This would mean we are 95% sure the margin of error is 18% or less and there is a 5% chance the margin of error is greater.
However, “random" means that we picked the 33 members that responded essentially out of a hat filled with the names of the 6600 members. But that is not what happened. The survey probably received responses from their most DNS-savvy members - those that found the purpose of the survey interesting or where the questions seemed more straight-forward. This significantly skews the results. Georges email demonstrates this in more detail.
I don’t think the survey deals with the skewed data set fairly or honestly. The survey characterizes the findings as traits of the entire membership rather than as traits of the population that responded. This can’t be defended.
For instance, I don’t think it is correct to say: "Vast majority (97%) of */members/* registered domain names in past 24 months, with 9 in 10 registering new TLDs,“ and "9 in 10 */members/* have registered new TLD domains in the past two years in the Sunrise Period." I think new TLD owners would be very surprised (and happy) to hear this. It would be accurate to say that "97% of the */respondents/* registered….” (See slides 9 and 11)
More harmful to the credibility of the study are statements such as:
* 3 in 4 members (76%) have incurred costs for internet monitoring of trademarks in the past 2 years, with more than half (57%) of the members spending $10k or more. (see slide 12) * On average, INTA members spend $150,000 per year on defensive actions (see slides 10 & 27)
These are the types of quotes that find themselves into print and become believed. (“INTA members spend $150K each in defensive efforts, a ~$1 billion cost to industry!”) As George noted, as a rule, larger companies responded and so it can not established by this survey that each of the remaining (smaller) INTA members average $150,000 per year in defensive spend.
There is another interesting facet to the asserted $150K / year spend rate. One company spent $5.2MM. Assuming this $5.2MM spend was over a two-year period, that means that the other 32 respondents averaged (33 x $292K — $5.2MM) / (32 x 2) = $69,000 / year. So except for one outlier, the per year spend by the brand owners that chose to answer the study is half of what the study states. Why didn’t the study make this clear? (see slide 10).
I am not sure of the purpose of the study but there are uses that can be made of it:
There was one conclusion I could draw. It states that UDRP and Sunrise were the favored rights protection mechanisms, used to a major or moderate extent by 67% and 64% of the respondents respectively. The next most utilized RPMs were Trademark Claims and URS (by 36% and 27% respectively). To me this says that, to those who are in-the-know, Sunrise is a highly-valued RPM and, therefore, should be continued. (Sorry, George) (see slides 15 and 51)
Also, the study makes one fact clear that we have already supposed: that business are not aware of new gTLDs and domain utility in general. There are several data sets that point to this. Rather than education efforts that identify costs and target abuse prevention and mitigation only, Brand education could also describe the benefits of domains as strategic tools, that provide greater access to products and indicia of reliability to brands’ customers.
I know this was way pedantic. Sorry. I can’t be on the call as it is at 4AM my time but I’d be pleased to respond to comments or questions.
Best regards,
Kurt
On Aug 29, 2017, at 10:05 PM, George Kirikos <icann@leap.com <mailto:icann@leap.com>> wrote:
Hi folks,
I'm not sure how many have had a chance to read the INTA materials for tomorrow's call yet, or have any background in statistics, but the survey has truly deep and fatal flaws, making any conclusions drawn from it entirely unreliable and non-robust.
I could write 50 pages on this (I've read the report three times now, in horror), but I'll keep it relatively brief (and make these statements in advance of the call, so that Lori or INTA/Nielsen have a chance to rebut).
The entire basis of statistical inference is that one can make statements about an entire population with a certain level of confidence using only data from a subset of that population (i.e. the sample in question). Prerequisites are that (a) the sample be random, and (b) the sample be of sufficient size. INTA's study fails on both counts (self-selected and unrepresentative sample, and a mere 33 responses).
INTA claims to represent 7,000 organizations as members:
https://www.inta.org/About/Pages/Overview.aspx
While they acknowledge on page 5 of the slides the small sample size and suggest "some caution", alarm bells should be ringing regarding that small sample size. Page 6 then demonstrates how unrepresentative and non-random that sample is, with 52% of the 33 respondents having total revenue exceeding $5 billion/year, and a whopping 77% (27%+52%) having revenues exceeding $1 billion. This is hardly representative of typical TM owners. Similarly, 39% of this sample had 25,000 or more employees, and 78% (39%+39%) had 5,000 or more employees.
All throughout the report, the slides say "INTA members" (i.e. wrongly attempting to extrapolate and assert a truth about the entire population, rather than limiting the statements to be applicable only to the sample of 33 respondents).
Basic sanity checks were not done with those extrapolations/inferences. On page 25, the report asserts that "more than 4 in 10 members have applied to operate a new TLD"?
45% of 7000 members implies 3,150 INTA members applied for new gTLDs. That's not correct. The total applications by everyone was 1930 -- see https://newgtlds.icann.org/en/program-status/statistics, and the number by brand owners is a subset of that total (664 according to https://icannwiki.org/Brand_TLD and that will be a bit high, due to multiple applications). If one extrapolated that to the entire universe of trademark holders (i.e. including non-INTA members), millions of TM owners, it would be even more obvious how unrepresentative and non-random the data in this sample is relative to a "typical" TM holder. This sample is highly skewed to the largest of the large organizations who happened to self-select a response to this survey.
All throughout the report, important data on confidence intervals is missing, obscuring the fact that the level of confidence is extremely low (and the margin of error is high) due to the small sample size. [confidence intervals are statements like "+/- 5%, 19 times out of 20]
There are actually calculators that let one know how big a sample should be, in order to have a certain level of confidence and/or a margin of error.
e.g. see: https://www.surveymonkey.com/mp/sample-size-calculator/
For a population size of 7000 members (INTA's total membership) and a 95% confidence level, with a huge 10% margin of error, you'd still need 95 survey responses. Yet, there were only 33 responses. This is particularly important to be kept in mind for charts with percentages (pp. 17 and beyond), where the margin of error, even if sampled properly, would be enormous. Furthermore, those would have had to have been RANDOMLY sampled responses to be proper, which we know isn't the case. If you wanted smaller margins of error, say +/- 5%, you need an even larger sample size (in this case, 365). Another useful calculator is at:
https://www.surveymonkey.com/mp/margin-of-error-calculator/
ICANN has done surveys, by Nielsen even, that didn't suffer from these deficiencies, e.g. see:
https://newgtlds.icann.org/en/reviews/cct/registrant-survey-faqs-25sep15-en
A key takeaway from that work was "Due to a low response rate to emailed invitations to complete the survey, ICANN then worked with Domain Tools to procure a larger sample of WHOIS records." They took greater care in that study to have *randomized* samples, too, along with the larger sample size.
While it is somewhat interesting to have a glimpse into brand protection of some of the largest companies, ultimately this study is not robust.
In summary, any conclusions from this INTA study really need to be taken with a grain of salt, due to the small sample size, combined with the non-random and unrepresentative sample itself. Indeed, many of the conclusions need to be read as the *opposite* of what the study suggests (i.e. if defensive costs are $150K/year for companies with $5 billion+ in revenues, that's a drop in the bucket, and would be much, much smaller for a "typical" TM owner). To correct these deficiencies, future surveys need to be random (easily done, e.g. random sample the USPTO database or other national registries) and have a much larger sample size. Understandably, that costs money, but that's what it takes to do things properly.
Sincerely,
George Kirikos 416-588-0269 http://www.leap.com/
On Mon, Aug 28, 2017 at 4:02 PM, Mary Wong <mary.wong@icann.org> wrote:
Dear all,
The proposed agenda for our next Working Group call, scheduled for 0300 UTC on Thursday 31 August, is as follows:
Roll call (via Adobe Connect and phone bridge only); updates to Statements of Interest Review and discuss results of INTA Cost Impact Survey Next steps/next meeting
For Agenda Item #2, please review the survey results here: https://community.icann.org/download/attachments/61606864/INTA%20Cost%20Impa...
Lori Schulman of INTA, and a member of this Working Group, also did a presentation of the results to the Competition, Consumer Protection & Consumer Trust (CCT) Review Team recently that may be helpful to review: https://community.icann.org/download/attachments/61606864/ICANN%20New%20gTLD.... We are hopeful that Lori will be able to join us for this call, to facilitate our review and discussion.
Thanks and cheers
Mary
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
_______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org https://mm.icann.org/mailman/listinfo/gnso-rpm-wg
-- Bei weiteren Fragen stehen wir Ihnen gerne zur Verfügung. Mit freundlichen Grüßen, Volker A. Greimann - Rechtsabteilung - Key-Systems GmbH Im Oberen Werk 1 66386 St. Ingbert Tel.: +49 (0) 6894 - 9396 901 Fax.: +49 (0) 6894 - 9396 851 Email: vgreimann@key-systems.net Web: www.key-systems.net / www.RRPproxy.net www.domaindiscount24.com / www.BrandShelter.com Folgen Sie uns bei Twitter oder werden Sie unser Fan bei Facebook: www.facebook.com/KeySystems www.twitter.com/key_systems Geschäftsführer: Alexander Siffrin Handelsregister Nr.: HR B 18835 - Saarbruecken Umsatzsteuer ID.: DE211006534 Member of the KEYDRIVE GROUP www.keydrive.lu Der Inhalt dieser Nachricht ist vertraulich und nur für den angegebenen Empfänger bestimmt. Jede Form der Kenntnisgabe, Veröffentlichung oder Weitergabe an Dritte durch den Empfänger ist unzulässig. Sollte diese Nachricht nicht für Sie bestimmt sein, so bitten wir Sie, sich mit uns per E-Mail oder telefonisch in Verbindung zu setzen. -------------------------------------------- Should you have any further questions, please do not hesitate to contact us. Best regards, Volker A. Greimann - legal department - Key-Systems GmbH Im Oberen Werk 1 66386 St. Ingbert Tel.: +49 (0) 6894 - 9396 901 Fax.: +49 (0) 6894 - 9396 851 Email: vgreimann@key-systems.net Web: www.key-systems.net / www.RRPproxy.net www.domaindiscount24.com / www.BrandShelter.com Follow us on Twitter or join our fan community on Facebook and stay updated: www.facebook.com/KeySystems www.twitter.com/key_systems CEO: Alexander Siffrin Registration No.: HR B 18835 - Saarbruecken V.A.T. ID.: DE211006534 Member of the KEYDRIVE GROUP www.keydrive.lu This e-mail and its attachments is intended only for the person to whom it is addressed. Furthermore it is not permitted to publish any content of this email. You must not use, disclose, copy, print or rely on this e-mail. If an addressing or transmission error has misdirected this e-mail, kindly notify the author by replying to this e-mail or contacting us by telephone.
participants (12)
-
claudio di gangi -
Dorrain, Kristine -
George Kirikos -
Greg Shatan -
icannlists -
jonathan matkowsky -
Kurt Pritz -
Lori Schulman -
Mary Wong -
Paul Tattersfield -
theo geurts -
Volker Greimann