Thanks, Justine. First, we haven't made any claims about statistical significance--that's not usually the kind of claim one makes when studying a universe and not testing a hypothesis (e.g., what might have happened had we provided half the universe/sample with free access to counsel). The percentages that can be seen from the data are the percentages from the universe. One might well argue that we have enough cases now to draw some conclusions about the functioning of the system, but it's not a matter of sampling. For other decisions made during coding, you can see them under the tab for "Fields and Methodology," As you note, AB is a matter of determining what's in the decision and lacks a discretionary component. I don't think that V, W, and X require much if anything in the way of judgment: exact string matching (V) is pretty self-explanatory, and W and X sort among the remaining domain names. We haven't done any percentage analysis relying on V, W, and X, but if you're concerned about categorization, collapsing W and X into "non-identical string" could remove any discretion. Final note: In the page on methodology, you can see that spaces and hyphens were disregarded in determining placement into these categories. Y, Z, and AA: I'd be the last person to claim that there's much objective about sorting terms between descriptive and suggestive! That's got to be a matter of judgment. Rebecca Tushnet Frank Stanton Professor of First Amendment Law, Harvard Law School 703 593 6759 ________________________________ From: Justine Chew <justine.chew@gmail.com> Sent: Wednesday, May 30, 2018 9:27:05 PM To: Tushnet, Rebecca Cc: gnso-rpm-wg@icann.org Subject: Re: [gnso-rpm-wg] Actions & Notes: RPM PDP WG 30 May 2018 Prof. Tushnet, I had a couple of questions regarding the research you/your team undertook:- 1. Looking at the MASTER -- Enter Data worksheet, would I be correct in concluding that only the TM Analysis columns (i.e. columns V to AB, well maybe excluding column AB) called for a subjective judgment on the part of your data entry person / RA? In other words, all data in all columns were based on actual "matches" or "presence of matching or relevant text" except for the following columns: V: Domain name (string) identical to TM W: Domain name contains TM + generic or related term X: Domain name similar to TM Y: TM is fanciful Z: TM is arbitrary/suggestive for category AA: TM is descriptive for category AB: TM owner in TMCH (mentioned in decision) 2. Apologies if I had missed coming across it in earlier emails/calls, is there a compendium report to the research/data that informs on assumptions taken and any resulting patterns of statistical significance? Many thanks , Justine Chew ----- On 31 May 2018 at 05:39, Tushnet, Rebecca <rtushnet@law.harvard.edu<mailto:rtushnet@law.harvard.edu>> wrote: Thanks all. My deepest apologies for missing the meeting--I had conflated it with the later meeting today on my calendar. If there are specific questions I can answer, I'd be happy to do so. For example, there's some mention in the notes of the "assumptions" of the coding--I am unclear what exactly is at issue and I'd like to know so we can discuss in the group. Rebecca Tushnet Frank Stanton Professor of First Amendment Law, Harvard Law School 703 593 6759 ________________________________ From: gnso-rpm-wg <gnso-rpm-wg-bounces@icann.org<mailto:gnso-rpm-wg-bounces@icann.org>> on behalf of Julie Hedlund <julie.hedlund@icann.org<mailto:julie.hedlund@icann.org>> Sent: Wednesday, May 30, 2018 5:12:12 PM To: gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> Subject: [gnso-rpm-wg] Actions & Notes: RPM PDP WG 30 May 2018 Dear All, Please see below the action items and brief notes captured by staff from the Working Group call held on 30 May 2018 (1200 UTC). Staff have posted to the wiki space the action items and notes. Please note that these will be high-level notes and are not meant as a substitute for the transcript or recording. The recording, transcript, WebEx chat, and attendance records are posted on the wiki at: https://community.icann.org/display/RARPMRIAGPWG/2018-05-30+Sub+Team+for+Data<https://urldefense.proofpoint.com/v2/url?u=https-3A__community.icann.org_display_RARPMRIAGPWG_2018-2D05-2D30-2BSub-2BTeam-2Bfor-2BData&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=E-M4OQvQBo8UWqE1LwEiDR3PcWlfM0I-0jiI1c4ous0&m=Fz5mz32nned9r4I18xXJQUzQMbagMGoa3kI6hmEPJMI&s=JKd6pXw7OFfZqDQjtB7OEgFZ16KYACFfJVhoTs1Sd0s&e=>. Best Regards, Julie Julie Hedlund, Policy Director Action Items: 1. Co-Chairs will consider how to proceed with the data from Prof. Tushnet and send a proposal to the WG. 2. Co-Chairs and staff will develop programs for the ICANN62 sessions: * Session 1: Wednesday, 27 June, 10:30-12:00 * Session 2: Thursday, 28 June, 09:00-10:15, and * Session 3: Thursday, 28 June, 10:30 to 12:00 Notes: 1. Discuss what quantitative data from Rebecca’s research can be used to address Documents Sub Team questions about the 250 “response received” and 58 “respondent prevailed” cases, Introduction -- Berry Cobb: -- The intent for today’s agenda was to understand what the WG’s or Sub Teams next steps are for Rebecca Tushnet’s research. Within the Documents Sub Team that began with the foundation document with the charter questions created an additional column on what types of data sets would help to answer those questions. -- When the Document Sub Team was wrapping up it work it was about the time Rebecca sent the result of her coding exercise. -- WG/Sub teams can review the data to see how it might answer the charter questions, how this data could be used – such as put back in the sub Teams – or considered by the full WG. -- Rebecca’s coding data set there were 4-5 potential areas where the data might help the Providers Sub Team answer some questions. -- As it relates to the agenda item, staff has already done some analysis (staff is still reviewing the results of the data) the responses we were able to look at those a little bit more in depth based on the coding exercise. Was the response in 14 days and if not in 14 days was there a response in 6 months and was there a request for an extension. -- In the 58 cases where the respondent prevailed just about half of them there was no response, which creates a smaller subset of cases we could review. That would be around the area as it relates to the standard of proof defenses in 5.7 and 5.8. -- As it relates to Rebecca’s research there is one particular part of that analysis that talks about the domain’s disposition. What happened to it after the determination had been made. Did the domain get suspended and what happened after it was suspended. The WHOIS data I had provided was not a current query. -- I have done a fresh query against WHOIS and got 1400 from automated and manually fetched other 400, but this new data set will tell us whether the name is registered or not. Then I have looked at whether the name is under suspension. Of those that aren’t suspended to categorize them as to whether they are under brand protection, DPML, or whether the original registrant is using the name. Cleaning up that data. -- Having done that exercise there are two things to note: As one example (preliminary results) I only found one instance where domain that the complainant prevailed, went through suspension, became available for registration, and the original registrant re-registered the name. In doing this I noted that there were duplicate domains that had two URS cases filed against them. Some of which where the case that was first filed was either withdrawn or the claim was denied and the trademark owner resubmitted the claim. Another primary use case where the trademark owner prevailed in the original case, the domain was deleted, then it was registered by a different registrant and a different URS case was filed against the same name. It looks like it might be another useful data set. -- Trying to confirm and validate some of the pivot table. -- The question is how to proceed with the use of this data set at the full WG or Sub Team level. Discussion: -- There were 827 URS cases filed to date. Out of those there were responses in about 250. In 58 half of those came in cases where there was no response. The 58 is a subset of all the cases. What conclusions from that high level data from that research could help to draw conclusions about the operation of the URS – the response time, objectiveness of the determinations, etc. So far as whether conclusions to be drawn by the full WG or by the Sub Team, we don’t have enough people on the call to draw that conclusion. -- In the questions from the Practitioners or Providers Sub Team could some of those be addressed by the data? Answer: Don’t think there were potential data sets where the data might be useful for the Practitioners, but there were 4 or 5 for the providers since there were some pretty detailed questions and they would need to review the cases to respond. In the 4-5 instances where this could be helpful the providers could try to use the data. -- Note that it will take time for some analysis to find the appropriate coding columns and create a pivot table. There are several summary tabs that may be able to answer those questions, but likely need some manual analysis such as via pivot tables, to aid the WG or the Sub Team. -- Noticed that there are other questions that we can answer with the data using the cases by respondent tab. See some non-English speaking countries. If we look at the default rate for non-English speaking countries that could help answer some of the language related questions, even the success rate when the do respond. -- While this data is interesting, how are we trying to use it? Make sure that we don’t jump to conclusions based on raw data. We don’t have things we are trying to prove or disprove. Worried about taking months to look at data without context. -- We do have a lot of comprehensive data. There are a certain number of withdrawals as well. We have a lot of questions and a lot of data. We need to see how we can use it. -- It seems that individual members have theories that they would like to prove using this data. We need to be careful with the inferences that we draw from the data and extrapolating. For example, whether the default rate for non-English speakers can tell us anything. We don’t know. -- We need to be very careful about only drawing conclusions that flow naturally from the data and not pull things out that aren’t clearly supported. -- Sub Teams could perhaps review the data and decide which of their questions might be addressed. -- Question: Noted that there are a series of language cases, but not sure what that meant. Also, in the summary tables there are references to default, default/final, and then final. Wonder how those distinctions were made. Are there only 15 cases where there was a final decision in the instance of a default. Or there 566 cases where no final decision was rendered. -- What staff observed: The one tab that related to the country is only from the registrants location and don’t think it has more detailed information. The only two providers where cases were in a language other than English were in MFSD and ADNDRC. Haven’t come across one case from FORUM that wasn’t in English. The same columns of data are similar from the original data set that staff scraped from the providers and that is how they are displayed on the providers site. They will provide two kinds of status at it relates to the cases. What is the ultimate outcome – either claim was denied, domain was suspended, case was withdrawn. Then you had the determination and anywhere were it lists default there was no response, no request for an extension, and it was a default outcome that resulted in the domain being suspended, and/or the claim denied. In cases where there is a final it meant there was a response to that case that could have happened in the 14 days or the 6-month period. There are a few that are default/final, and an additional column that highlights appeals. When you are looking at Prof. Tushnet’s sheet the determination is what staff had scraped from the providers’ web sites. -- Regarding the use of an appeal, is there a place where that type of case has been highlighted? Answer: The Documents Sub Team did an analysis that was part of the summary report that the Sub Team produced. There were 14 appeals that we analyze, compare the case to the appeal, understand the responses, but didn’t dive into the aspects of the elements to prove within those cases other than highlighting what it was and the ultimate determination. -- The appeals are highlighted in the research in column AP – either coded as not applicable, successful, or unsuccessful and column AQ will have a link the appeal. Phil reading from the chat: Agree with Jeff. I think just looking at data by itself does not help without a context. In particular, the data has some assumptions and coding built into it that may not be accurate . So if we are going to look at the data we need to consider Rebecca's underlying assumptions as well so we have a better sense of what the data might or might not show. 05/30/2018 07:32:30 AM from Mary Wong to All Participants: As Berry noted, it is for the WG to decide which parts of the data, and how, is relevant to the scope of its work. Rebecca's RA has done a lot of work, but it is entirely her research that she has kindly provided to us in full. That doesn't mean it is all directly relevant to the WG's work, so that's why some of it may be very useful from the data side but perhaps not all. 05/30/2018 07:34:02 AM from Mary Wong to All Participants: For example, some of it seems very relevant to the Documents Sub Team (which highlighted the Respondent Prevailed and Response Filed cases as the two other categories it considered may be helpful to look at). 05/30/2018 07:35:59 AM from Mary Wong to All Participants: The other two Sub Teams may wish to see if any of the data can supplement, as quantitative measures, responses received from the Providers and/or Practitioners - since, collectively, these three Sub Teams scoped out the URS more or less. -- Suggestion: We have very extensive research by Prof. Tushnet that is not a WG document that is submitted to the WG to consider whether it could help to answer our revised charter questions relating to the URS. Some of it will be relevant to the Sub Teams. Suggest that the Co-Chairs will discuss this about the best way to proceed, which may be to invite members from the three URS Sub Teams to engage with the Professor and report back to the WG by ICANN62 as to the relevance of the data with respect to the URS charter questions. We hope to have the answers back from the practitioners and providers surveys by then. Co-Chairs will consider this and come back to the WG with a proposal for how to proceed. 2. ICANN62 Planning: Three sessions: Session 1: Wednesday, 27 June, 10:30-12:00 Session 2: Thursday, 28 June, 09:00-10:15, and Session 3: Thursday, 28 June, 10:30 to 12:00 -- Staff will work with the Co-Chairs to develop programs for each of the sessions. -- One will be talking about procedural issues leading up the publication of our initial report. _______________________________________________ gnso-rpm-wg mailing list gnso-rpm-wg@icann.org<mailto:gnso-rpm-wg@icann.org> https://mm.icann.org/mailman/listinfo/gnso-rpm-wg<https://urldefense.proofpoint.com/v2/url?u=https-3A__mm.icann.org_mailman_listinfo_gnso-2Drpm-2Dwg&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=E-M4OQvQBo8UWqE1LwEiDR3PcWlfM0I-0jiI1c4ous0&m=Fz5mz32nned9r4I18xXJQUzQMbagMGoa3kI6hmEPJMI&s=qDWDazORF6UAy5MvOTuwpAlRY6LySPSbvNanC-v80JY&e=>