Dear all,

Thank you to Rebecca, Kristine and Michael for continuing the discussion. At the end of the last sub team call, the following appeared to be the decision points under consideration for the actual and potential registrant surveys:

• The “actual registrant” and “potential registrant” surveys are two separate surveys; however, for the “actual registrant” target group, it may be helpful to also have a section that mirrors part of the “potential registrant” survey, so as to capture the views of those who succeeded in registering one/some domain(s) but decided not to proceed with other attempted registrations.
• In both the “actual registrant” and “potential registrant” surveys, it will be important (possibly even necessary) to make clear that the respondent knows what a Claims Notice looks like – this may entail actually showing the respondent (during the survey?) a copy of the Claims Notice.
• While “use cases” and illustrative examples using a hypothetical domain name can be considered but may not ultimately be necessary, what remains critical is eliciting responses where the respondent understands what a Claims Notice is and knows what it looks like.
• Some reordering/reformatting of the questions may be necessary in order to present respondents with a logical process flow and to facilitate responses.

If it will be of assistance, do you wish staff to request that Greg and/or Stacey provide a more specific description, on one of your next calls with them,  as to what methods they and their subcontractor will be using to identify, target and reach the category of respondents that we are describing as “actual registrants” (i.e. those that succeeded in registering domain names especially in the new gTLDs) and “potential registrants” (i.e. those who either attempted but did not complete a new gTLD registration, or would consider a new gTLD registration in the future)?

Thanks and cheers

Mary, Julie, Ariel & Berry

From: Gnso-rpm-data <gnso-rpm-data-bounces@icann.org> on behalf of "Michael Graham (ELCA)" <migraham@expedia.com>
Date: Saturday, June 16, 2018 at 06:38
To: "Dorrain, Kristine" <dorraink@amazon.com>, "Tushnet, Rebecca" <rtushnet@law.harvard.edu>, Ariel Liang <ariel.liang@icann.org>, "gnso-rpm-data@icann.org" <gnso-rpm-data@icann.org>
Subject: Re: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 15 June 2018

Thanks for taking the time to go through all this, Kristine – especially since my attendance today was somewhat sporadic.  I do understand the scope of what we are trying to get, but I agree that we should ask specific questions about actual responses to the Claims Notice, and not ask hypotheticals.

So, I can go along with asking registrants, applicants who did not obtain a registration, and potential domain name applicants what they understand the Claims Notice to say, what their reaction to it would be, etc.  However, I think we need to ask the same question of registrants in two classes: Those who have received Trademark Claims Notices, and those who have not.  More importantly, in asking, parsing, analyzing and attempting to draw any conclusions from the results, we must be sure that AG draws clear distinctions between the different groups being asked these questions, their answers, and the significance of their answers in light of their different levels and types of experience and concern.

I am therefore agreeable, but extremely cautious in light of past surveys and analyses.

Michael R.

From: Dorrain, Kristine [mailto:dorraink@amazon.com]
Sent: Friday, June 15, 2018 2:49 PM
To: Michael Graham (ELCA) <migraham@expedia.com>; Tushnet, Rebecca <rtushnet@law.harvard.edu>; Ariel Liang <ariel.liang@icann.org>; gnso-rpm-data@icann.org
Subject: RE: Actions & Notes: RPM Data Sub Team 15 June 2018

Hi Michael,

I do understand your concern but I think we ARE providing the Analysis Group with clear instructions about who to survey and how to ask the questions – and they’ve told us how they will get the participants.  That’s the point of the calls and that’s indeed what we’re discussing here, now.

Correct me if I’m misunderstanding, but I take your suggestion to mean you oppose surveying potential registrants at all.  I don’t think we are at liberty to do that and I’ll explain: 

1. We have charter questions that ask how understandable the Claims Notice is and we have some (at least anecdotal, abandonment) information that people may have been scared away by the Claims Notice.
2. As a result, we are expressly asked by the full WG to figure out if the Claims Notice is scary or needs to be re-worded.
3. We are also asked to figure out how people from different places of the world perceive the Claims Notice, as presented and worded (ideally, in a variety of languages- I have not seen how AG plans to address this)

^^These, as I understand it, are not negotiable.  These are the pieces of information the WG asked to get survey data on.

Therefore, we MUST present the Claims Notice to some people (ideally, people who would actually register a domain name) to learn how people react to it.  Now, we can for sure get some of this information from people who succeeded in registering domain names – they completed the process.  But we are also looking to get this information from people who opted not to proceed for whatever reason, when presented with a claims notice.  These are “potential registrants.”  And yes, for people who actually didn’t try yet or didn’t proceed, these questions are somewhat hypothetical, but what we’re attempting to get is a REACTION and an idea of how it might affect future behavior, not past data.  Think of this part as market research, not historical data.

The “hypothetical” disagreement is just in the level detail, as I understand it. I’ve been an outspoken critic of hypothetical questions. 

My solution is to give some general use cases (you want a domain name for a blog or for your business or as an investment) and then ask if the Claims Notice would be scary or a deterrent and why.  Rebecca likes the idea of being more specific (providing an example name at least, maybe more of a detailed scenario). 

If I’ve misunderstood, I hope you’ll set me straight.

Best,

Kristine

From: Gnso-rpm-data [mailto:gnso-rpm-data-bounces@icann.org] On Behalf Of Michael Graham (ELCA)
Sent: Friday, June 15, 2018 2:42 PM
To: Tushnet, Rebecca <rtushnet@law.harvard.edu>; Ariel Liang <ariel.liang@icann.org>; gnso-rpm-data@icann.org
Subject: Re: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 15 June 2018

Understood.  However, this sort of research requires a clear identification of the universe to be studied, the wording that will be used, and the environment in which it will be asked.  I think we need some clear parameters and questions, and I don’t think we have those yet.  Hopefully after our call today, AG can provide some guidance.

Michael R.

From: Tushnet, Rebecca [mailto:rtushnet@law.harvard.edu]
Sent: Friday, June 15, 2018 2:27 PM
To: Michael Graham (ELCA) <migraham@expedia.com>; Ariel Liang <ariel.liang@icann.org>; gnso-rpm-data@icann.org
Subject: Re: Actions & Notes: RPM Data Sub Team 15 June 2018

As I said on the call, this is standard comprehension research, relied on by courts and marketing departments.  When you want to know what members of a group take away from a message, you show it to them and ask them about it, even if they haven't seen it before.  It is precisely the kind of data driven research this working group is supposed to conduct. 

Rebecca Tushnet

Frank Stanton Professor of First Amendment Law, Harvard Law School
703 593 6759

---

From: Michael Graham (ELCA) <migraham@expedia.com>
Sent: Friday, June 15, 2018 5:12:17 PM
To: Tushnet, Rebecca; Ariel Liang; gnso-rpm-data@icann.org
Subject: RE: Actions & Notes: RPM Data Sub Team 15 June 2018

Thanks, Rebecca.  From your explanation, then, we do not want to ask hypothetical questions, but want to ask what subjects’ understanding of the Claims Notice is?  That makes sense if asked of actual registrants or applicants who received the notices.  I don’t think it makes sense, however, to ask this question of those who have not gone through the process.  Then it becomes a hypothetical:  If you received this notice, what does it tell you? 

Without the context of involvement in the process I don’t think “potential registrants” would provide relevant or useful information. 

I’ll go back to the questions after the weekend to see how and who they address the issue of effectiveness of the actual Claims Notice to.  That will better answer my concerns I hope.

Michael R.

From: Tushnet, Rebecca [mailto:rtushnet@law.harvard.edu]
Sent: Friday, June 15, 2018 2:01 PM
To: Michael Graham (ELCA) <migraham@expedia.com>; Ariel Liang <ariel.liang@icann.org>; gnso-rpm-data@icann.org
Subject: Re: Actions & Notes: RPM Data Sub Team 15 June 2018

Surveying potential registrants (defined in the survey as those who either attempted but did not complete a new gTLD registration, or would consider a new gTLD registration in the future) for their understanding of the meaning of the claims notice is not hypothetical--it is a standard reception survey of the target audience. The question is not generally what issues kept some potential registrants away--that is a good deal broader than our task, I believe. The question is what people considering new gTLD registrations take away from the Claims Notice when they read it.  That is one of the key questions this working group has about the functioning of the Claims Notice.  

Past and future new gTLD registrants are our universe.  Depending on their past experiences, including receipt of claims notices (if we get any such respondents) we may have different questions for different subgroups. But they are the target group as a whole, and should get the same questions, except for those that don't apply to them.

Rebecca Tushnet

Frank Stanton Professor of First Amendment Law, Harvard Law School
703 593 6759

---

From: Gnso-rpm-data <gnso-rpm-data-bounces@icann.org> on behalf of Michael Graham (ELCA) <migraham@expedia.com>
Sent: Friday, June 15, 2018 4:50:32 PM
To: Ariel Liang; gnso-rpm-data@icann.org
Subject: Re: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 15 June 2018

Thanks, Ariel.  I will review the Tape (as they say), but believe I disagree with the last point unless the meaning is that we would send a single survey to both registrants and “potential registrants” (although I would like to understand better how this last universe will be identified/selected) but according to which group they fall into, they will follow separate branches – registrants being asked whether they received Claims Notices and what they did/how they reacted; and potential registrants being asked why they did not apply for registration – i.e. what hypothetical issues could arise that kept them away.

I do not think we should mix them or that we should ask purely hypothetical questions of anyone since these will only obtain opinions, beliefs, and perceptions – not what the effects of the RPMs actually have been.

Michael R.

From: Gnso-rpm-data [mailto:gnso-rpm-data-bounces@icann.org] On Behalf Of Ariel Liang
Sent: Friday, June 15, 2018 11:56 AM
To: gnso-rpm-data@icann.org
Subject: [Gnso-rpm-data] Actions & Notes: RPM Data Sub Team 15 June 2018

Dear All,

Please see below the action items and notes captured by staff from the Data Sub Team call held on 15 June 2018 (1600 UTC).  Staff have posted to the wiki space the action items and notes.  Please note that these will be high-level notes and are not meant as a substitute for the transcript or recording. The recording, AC chat, and attendance records are posted on the wiki at: https://community.icann.org/x/ZYMpBQ

Have a great weekend!

Best Regards,

Ariel

Ariel Xinyue Liang

Policy Analyst | Washington, DC 

Internet Corporation for Assigned Names and Numbers (ICANN) 

----------------

ACTION ITEM

• Staff to ask Sub Team members’ availability to join the Monday, 18 June meeting at 17:00 UTC; if low availability, staff to propose an alternative time. (Done)

NOTES

Q2

• 2nd option: the first new gTLD was launched in July 2013, so change 3 years to 5 years 
• If the following questions do not apply to the registrants, redirect them to the potential registrant survey section.

Q2b

• Change 3 years to 5 years. 

Q2c

• Change 3 years to 5 years

Q2e

• "If Yes, did you register the domain name for which you received a Claims Notice?" is a core question. We want to know why they went ahead despite receiving a Claims Notice. We want to know qualitatively why they did so. It goes with the same set of questions below. Same rationale may apply to the second time they receive the Claims Notice, or more. 
• We spent a lot of time asking why people did NOT proceed, and there could be a lot of reasons. We do have a set of questions asking people why they DO proceed. If they did so, they may have made some serious decisions here, and we need to know why. Perhaps provide some multiple-choice options listing possible reasons why they proceed in order to dive deeper. 
• Suggestions for options to be included in a multiple-choice question

- I would not be sued if I continued

- I would not be subject to an action to take the domain if I continued

- No one else had a legal right to the name

- I had legitimate or legal right to the domain

- It does not seem like too much trouble to continue

- I consulted with an attorney and it was fine

- I was committed to the domain name for person/business reasons

- It matched investments names I already own in a portfolio  (going with "theme" registrants)

- I didn't understand the notice

- Something else [explain]

• There are portfolio registrants and they may have received many notices. May ask a quantitative question, "How many times this has happened to you?" and let them choose via multiple-choice options. Follow up with a question "For any of the domain name that you decide to proceed with registration, check all the reasons that apply". Bearing in mind that some people could have big portfolios.
• Is there anyway to encourage the survey taker to write a narrative and explain themselves? If we do get the response, it would be very helpful. 
• Registrants with big portfolio might actually prefer a narrative option.

Q2g

• There is an agreement to funnel this type of questions into potential registrant survey. 

Q2g(i) 

• The structure is a bit confusing 
• Need to be clear that this question should apply to anyone who is presented a Claims Notice, regardless whether he/she completed the registration or not 
• Which of the following best describes your understanding of the purpose of the Claims Notice? Don’t know/Not sure -- we need a "what a heck" option there too

Q2g(ii) – Q2h(i)

• Re virtue.food - oppose to specify as people may give a false hypothetical
• May want to specify "I plan to start a business or already have a business under that name"
• If we specify the business aspect, how would that help us? We don't know the background of the registrant why they proceed or not proceed. 
• We don't know what the registrants are thinking, so we are asking them, at least we will get some insight and useful answers if we give them an example to think about. If we just give them a notice, they would not know what to do with that. 
• We did give examples on other surveys to target non-expert. 
• How do we know there is no brand using virtue.food. How do we know survey takers’ level of knowledge whether it would be an issue or not? 
• We can ask clarifying questions at the end, but ask examples first to provide some context. You may need to place the “demographic” type of questions first to get into the mindset of the survey takers, and we can get a sense of the trend:

- Would you be nervous to register the domain name? -- provide options

- If yes, what actions would you take? -- provide options 

- If no, what is the reason? -- provide options 

- Does trademark law matter? 

• Concerned if asking those questions first may have a bad effect on the survey takers, e.g., asking them about trademark law knowledge. It may be super hard to ask each survey taker to consider each scenario. Worth asking trademark only after asking some hypothetical questions, otherwise it would trigger some negative reaction. 
• Don't need to limit this question for the people who have not completed the registration.
• Proposed new order:

- Show trademark claims notice.

- Q2g(iv) – confidence?

- Would you consult with an attorney? [maybe also “would you talk with domain name registrar, etc?]

- Q2h(i) – Why you abandoned?

- Q2g(iii) – trademark law knowledge

• We are making assumption that the user is going to assume the claims has something to do with the trademark. Most people don't know the difference between trademark and copyright. That's a mistake in the premise of each Q that the analysis group needs to resolve. Maybe this is the FIRST question and there is a matrix. so different paths of questioning need to be clarified... 
• One possibility is to ask registrants what they imagine their use would be. If we will have enough respondents, we could sub group them. 
• Those general Qs should appear after the more specific responses to virtue.food

Q2h(i)

• We are asking why they did not proceed, and it is not with respect to virtue.food. Where are we asking them what they are doing with virtue.food? 
• This question is for people who have abandoned. Everyone should be asked for this question. It is doubtful that we will get a lot of registrant that receives claims notice. 

Discussion about the open-ended questions  

• Narrative responses can become cumbersome. It is fine to have a few in the survey, but the more complicated question is, the more loaded the responses would be. To have a lot of these questions would make it difficult for the respondent to respond, and can lower response rate and increase dropout rate. 
• Some open text questions may be better for interview survey. Form survey would not have the opportunity to redirect the answer. 
• Wide range of responses in open text field can make the analysis of the responses difficult and it would be hard to summarize the results in a simple way. 
• There is difference between "Other___" and the larger open text field asking very open-ended question. 

Discussion about hypothetical questions

• Reference page 28-29 of the RFP Appendix regarding the comments for/against hypothetical questions
• Getting into use cases is different from hypothetical. There are general categories that people fall into. Give people the general categories. 
• If we ask hypothetical questions of Registrants, I would like to return to TM/Brand owner survey to include some hypos.  What would be the difference?
• I feel like that is a different kind of request for information. That would be something to ask in the design phase of a project not necessarily an evaluation phase.
• I'm ok with the page 29 scenario since we get an understanding of what the respondent is thinking when they answer. Consider the alternative to hypothetical listed on page 29
• The questions about actual events and actions will provide metrics; hypos will prove opinions.  We must somehow separate the two if we include them.  
• Why the hypothetical question falls into the registrant survey? Q2g, Q2g(i), Q2g(ii) are inspired by the potential registrant survey questions. 
• We want to know the set of people who register domain name, and when they see the trademark claims notices, what they are doing with them. To provide use cases is because it is hard for registrant to understand in the abstract. There is no way to ask such question without something to think about.
• The only way to get an answer to the question is by showing claims notice and see what they think. If you got a claims notice.... [Qs] AND here is a claims notice... [ Q]
• AG suggestion:  

- Keep the hypo question for the potential registrant survey

- In the registrant survey, ask respondents did they receive claims survey, what they did to it, what was the purpose of their registration, etc. elicit the insight without providing a hypothetical scenario?

• Is it possible that most people taking the survey are legacy domain name registrants? 
• We account for those potential respondents and try to filter them in the survey.
• It does not make a lot of sense to differentiate the registrants and potential registrants in two different surveys. There are a lot of overlapping paths. It should be one survey with branching paths. The people who said “I have received a claims notice, but didn't have the use case” should be given the opportunity to provide useful responses.