Hi Rebecca, Thanks so much for starting this work! We’d be grateful if you could make your comments in the Google Docs at: Registrants: https://docs.google.com/document/d/18K-zhP541IGCR0bXJpBtdmWdpfQ6BbKhFZaaZsfZ... Potential Registrants: https://docs.google.com/document/d/1iY4KsWpioC8Sacs-pWurfnKxhOsiQA7YaSo1wuxb... This is because the Google docs have also been shared with Analysis Group so they will be able to see your comments and you will be able to see any questions they may have with respect to them. Please let us know if staff can assist you in any way. Thanks again for your help! Best, Mary, Ariel, Berry, and Julie From: Gnso-rpm-data <gnso-rpm-data-bounces@icann.org> on behalf of "Tushnet, Rebecca" <rtushnet@law.harvard.edu> Date: Thursday, June 7, 2018 at 5:43 PM To: Ariel Liang <ariel.liang@icann.org>, "gnso-rpm-data@icann.org" <gnso-rpm-data@icann.org> Subject: Re: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 07 June 2018 I have started working on the registrant/potential registrant questions, as Kristine wisely suggested. Would it make sense to circulate my written comments to the Analysis Group as well as this group? If so, what contact information should I use for them? W/r/t the registrants & potential registrants, it seems that a bit more background is needed, in particular the first date anyone actually could have registered a domain name in a new gTLD--does anyone remember that date offhand? Rebecca Tushnet Frank Stanton Professor of First Amendment Law, Harvard Law School 703 593 6759 From: Gnso-rpm-data <gnso-rpm-data-bounces@icann.org> on behalf of Ariel Liang <ariel.liang@icann.org> Sent: Thursday, June 7, 2018 5:19:34 PM To: gnso-rpm-data@icann.org Subject: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 07 June 2018 Dear All, Please see below the action items and notes captured by staff from the Data Sub Team call held on 07 June 2018 (1700 UTC). Staff have posted to the wiki space the action items and notes. Please note that these will be high-level notes and are not meant as a substitute for the transcript or recording. The recording, AC chat, and attendance records are posted on the wiki at: https://community.icann.org/x/GisFBQ Best Regards, Ariel Ariel Xinyue Liang Policy Analyst | Washington, DC Internet Corporation for Assigned Names and Numbers (ICANN) ---------------- ACTION ITEMS AG to start incorporating edits/comments already discussed in the previous meetings and reflect them on the Google docs, so the Sub Team could review. Staff to Doodle additional call times for the next two weeks, including Wednesdays and Fridays Sub Team to use the Friday meeting to work on the TM/Brand Owner survey Sub Team to continue providing redline comments/suggested edits on the Google Doc, led by the original author of each survey. NOTES General Comments AG comments that the length of the survey needs to be balanced, considering the impact to response rate. Concerns with survey fatigue. Sub Team should also consider what are other ways to gather data for the questions that may be cut from the surveys? The questions related to sunrise pricing may be hard to illicit responses from ROs, based on AG's experience with TMCH review. Sub Team members have strong objection regarding the approach the surveys are handled; rushing through the surveys without letting members thoroughly review the documents is very concerning. There are anecdotal questions that one cannot predict how the respondents would respond. Would it be better to revert back to the anecdotal questions instead of trying to convert them into quantifiable questions to illicit data? Selected number of open ended questions can be very useful. AG could do that, but we need to be cautious about the number of open ended questions. The ideal survey is to have no more than 25% of the questions that are open-ended, otherwise the drop off rate would be high. Sub Team commented that while timeline needs to be respected, it should not be driving the decision of Sub Team. Sufficient time needs to be devoted to get the questions right. Registries who want to talk to the Sub Team have had and continue to have plenty of opportunity to do so. Some charter questions have no basis. The Sub Team don't understand why certain people said what they said in the original charter questions. It is hard to convert such questions into data driven questions. That's why this whole exercise has been such a challenge. Sub Team anticipate weeks to work on the surveys with AG, but this length of time could not be provided. There are budgetary limitations in this. Is there an alternative approach to get valuable information to the AG in real time? There are many gaps, as well as the lack of historic context in the RFP Appendix. Sub Team would need to take AG through all the sections after having some time to review the Google Docs before the next calls. The "new eyes" are certainly helpful because it highlights the gaps. AG would benefit from going through the background before they take another pass. Sub Team are encouraged to provide redline comments/edits in the Google Docs, but be mindful that they have other competing priorities. Instead of having everyone looking at every survey, Sub Team members who were the original authors of the draft survey questions are encouraged to focus on their respective survey and provide redline comments/edits. The draft survey questions do not have 100% consensus in the Sub Team. AG should not treat individual’s comments/edits as something agreed by the whole Sub Team. AG hasn't been invited to provide redline edits on the Google docs yet, but will be able to do so going forward. ACTION ITEM: AG to start incorporating edits/comments already discussed in the previous meetings and reflect them on the Google docs, so the Sub Team could review. Subject to the co-chairs' preferences and the Sub Team's availability, is there any objection to using the next two weeks to finish the surveys? F2F meeting in Panama may be useful but cannot expect everything be done there. We have to adjust expectations that not everyone can be on all calls in 2 hour blocks. Kristine volunteered to work with AG to answer questions and provide clarifications on the RO related questions. Not to wordsmith but to provide context/background. Michael volunteered for the TM/Brand Owner survey questions. Of course, the whole Sub Team would have the final say about the final survey questions. Q16/Q17 Original questions are on page 3 of the RFP Appendix Consider combining these two questions. The aspects that Q16/17 don't cover are the potential benefits if the sunrise period length is changed. The original questions not covered here are -- What were the benefits (to the registry or to brand owners)? What were the drawbacks? Were there any complaints or was anyone confused? (Include complaints from potential nonbrand owner registrants). We cannot ask TM owners what benefit they are getting in this question. We are only asking ROs. Registrars may not be able to answer these questions with any degree of certainty. It is difficult to come up with different options to make these questions more quantifiable. What Sub Team want to know is what they are missing. Why do some people want to extend the Sunrise period? It is hard to get the answer through a grid/tabulate format. Sub Team specifically try to illicit responses on how Sunrise period is impacted. Sub Team would like to know where the complaints went to and what they were. Sub Team are more concerned with the 25% of survey questions that are open ended. AG would like to request the Sub Team to provide suggestions and include additional questions/options to Q16/Q17 to the Google Doc. AG could maintain the grid format of the current questions, but to also include a separate question to illicit anecdotal responses. It may be helpful to also include questions illiciting "yes, no, maybe" type of responses. ROs don't have much of the technical costs of Sunrise. It's the registrars who have most of the technical costs. There are also marketing issues related to the sunrise. It may be helpful to have a couple of Sub Team members preview the survey and see how the questions work. Q18 The current question is not asking the critical aspect of the original draft question (e.g., duration, perpetual) Would it be better to ask respondents to explain why they rank the requirements the way they do? AG will include this sub question. Not particularly in favor of this question, and the Sub Team are not sure whether it is getting what the Sub Team were trying to ask. Original charter question: Should there be a different rule for some registries, such as specialized gTLDs (e.g. community or geo TLDs), based on their published registration/eligibility policies? Sub Team needs better wording for this question. Context of the original charter question: when you have specialized TLDs, should brand owners get into the door first? There are other ways to allocate TLDs that are not contingent to brands. ROs experience a lot of pushbacks related to certain TLDs. Include a "would you still offer sunrise" after Q18 AG said that the comment in the survey document is meant to show the charter questions that this section is intended to address It may be helpful to include examples to clarify the intention of the original charter question, but the examples should not be limiting type of examples. Q21/Q21a These questions are vague. Some TLDs don't have registration eligibility restrictions. Some ROs may not understand what these questions are asking. We may need some qualification for these questions. Rephrase the question to “Have you had requests for Sunrise from parties that do not have the eligibility?” or something else to that effect. As worded, the answer should be "yes" for each RO with a QLP In the instances where the wording is vague/confusing, AG would welcome direct feedback/redline suggestion in the Google Doc. AG could help rephrase questions to convey the meaning/context of the questions. Sub Team's input/clarification for content would be helpful. How can we develop the surveys and word the questions in a way to illicit meaningful data, instead of anecdotal responses. For Q21 in particular, it would be helpful to know what ideal type of response would look like. Q22 It may be placed before Q20 and Q21 Original question on page 4 of Appendix: How could the ICANN brand protection policies like Sunrise or Claims be altered to better accommodate restricted TLDs (like Community or GeoTLDs)? "How could" is the key, not just "should they". Respondents should suggest practical solutions and identify what they think would work better. AG could include an open-ended sub question to illicit such response. Q24 Respondents only need to answer Q24 if they answer yes to Q23 The word "startup" can be removed. Q26 While the original draft question is reflected in this proposed question, there is a follow up question asking for suggestions. May add a follow up question asking respondents what suggestion they may have to resolve the issues, but not to change the tone of the proposed question. Respondents only answer the additional question if they have encountered problems. AG can add a question that is more forward looking with open text field, but remain concerned with the length of the survey. Q27 Add a gating question. Spell out the acronyms. Q29b ROs do not send Claims notices. Registrars may provide more effective responses. Q30 Similar to Q16/Q17, all the tables descend to a whole range of issues that the Sub Team may not have the time to develop a comprehensive list for. We should not toss all the grid questions all together. Possibly there is a way to come up with a metric while still gathering anecdotal data. This question is just asking opinions, so respondents would rank based on things that matter to them. Not sure what we would get after sending this question, rather than seeing respondents advancing their own positions. This holds true for all target groups that would express their own opinions. It may be helpful to give respondents an opportunity to explain the basis of their opinions and “why”. Far more important are the actual stats and metrics that the WG could come up with. The analysis of the survey results would be crucial. The format/style of the same type of grid questions need to be consistent among all surveys. Q32 This question doesn’t belong in the Q23-26 section and is perhaps covered by the expansion of other questions now. Questions related to eligibility-restricted TLDs scattered in the survey. Grouping these questions would be more meaningful and can possibly shorten the survey. Ask respondents to indicate which program they are referring to in their answers (e.g., LRP, QLP, ALP). Most ROs won’t offer just QLP or ALP individually. Many ROs offer more than one program. Sub Team specifically want to know what changes the respondents would like to see and are less concerned about the problems that they encountered, as we know there are problems. Procedural Matters ACTION ITEM: Sub Team to use the Friday meeting to work on the TM/Brand Owner survey ACTION ITEM: Staff to Doodle additional call times for the next two weeks, including Wednesdays and Fridays