Draft WG Self-Assessment Design, Objectives, and Questionnaire
SCI Team Members: Since our last teleconference, I have been working on the development of a WG Self-Assessment instrument. Rather than inundate the SCI email list with multiple drafts and iterations during the incubation period, I thought it would be more helpful to post my work products to the ICANN Community Wiki. I have now reached a point where I think the material is ready for your review. In particular, you will find the following pages under the GNSO tab (category) and Working Group Resources space. Below is the hierarchy including links: WG Self-Assessments: https://community.icann.org/x/2Cp-Ag . Design Considerations: https://community.icann.org/x/USt-Ag . Learning Objectives: https://community.icann.org/x/hit-Ag . Questionnaire (Draft v1): https://community.icann.org/x/EC1-Ag If you want to follow my developmental logic path, I suggest starting at the top and proceeding in the sequence shown. If you want to skip to the actual draft questionnaire, it is contained in the bottom linked page. I included a few notes which you should bear in mind before perusing the actual questions. At this stage, I would like to receive your feedback before proceeding further to be sure that I am on a productive track. You will certainly notice that this instrument is no longer focused on the WG Guidelines documentation (the original survey scope); instead, on the effectiveness of the WG's operations, norms, logistics, decision-making, and outputs. I included my rationale for that shift at the top of the Design Considerations <https://community.icann.org/x/USt-Ag> page. Feel free to comment on the email list or add comments to the individual Wiki pages. I will attend to both sources over the next week or so as you have time to review and evaluate my recommended approach to the WG Self-Assessment. Regards, Ken Bour
hi Ken, i really like the work you've done. i've sprinkled reactions inline, but overall this is a great start. thanks, mikey On Jun 10, 2013, at 3:43 PM, "Ken Bour" <ken.bour@verizon.net> wrote:
SCI Team Members:
Since our last teleconference, I have been working on the development of a WG Self-Assessment instrument. Rather than inundate the SCI email list with multiple drafts and iterations during the incubation period, I thought it would be more helpful to post my work products to the ICANN Community Wiki. I have now reached a point where I think the material is ready for your review.
In particular, you will find the following pages under the GNSO tab (category) and Working Group Resources space. Below is the hierarchy including links:
WG Self-Assessments: https://community.icann.org/x/2Cp-Ag
really good introduction. on second reading i realized that one thing that might need expanding in the "lower-level" chunks of work is the LOs and questions that try to tease out the effectiveness of the formally-documented processes (the newly-changed PDP, developed by all the WGs that came before us). not a lot more, but just a bit. i think one of the reactions i had to the first questionnaire was that it was too focused on the formal-process stuff. so there's some balancing to be found. i could certainly live with the balance you've struck in this first pass -- but if there was a graceful way to add just a little more review of the PDP framework, i'd like that too. not too much though. we're going for "just right." :-)
· Design Considerations: https://community.icann.org/x/USt-Ag
individual vs collective -- i like where you're going in the highlighted box -- some kind of hybrid between the two, maybe up to the WG. if they get really stuck arriving at consensus, just let the discussion serve as documentation rather than trying for consensus. i agree, it would be a shame to run a working group through another tough consensus process at the very end. especially tough if they weren't at consensus on their findings and were reopening old wounds. demographic information -- all good -- wisecracks about the NSA notwithstanding. although another approach to consider might be to have the information be public. i don't know -- tradeoffs. something to puzzle about. size, complexity, length -- yep, this is where i came into the discussion -- the previous questionnaire was really long/hard to fill out. but it would be interesting to see if there's a way to build in "optional" sections if people had a lot to say. my post-mortem on my experience chairing the Fast Flux WG took several days to write. if somebody's got a will to do something like that, we should provide a way. methodology -- no strong opinions either way here.
· Learning Objectives: https://community.icann.org/x/hit-Ag
first pass -- WOW! i like these a lot. there may be more, but this is a really good start. good for: starting off the WG to set norms and expectations, periodically referring back to "mid flight" to offer course-corrections to the process as its's going, and reflection on opportunities for improvement at the end. since we're a PDP-process focused group, i'm wondering if there's a way to work a couple of LOs in about the framework itself. the goal being to see if there WG members noticed flaws in the process that got in their way. a good source of ideas might be the questionnaire i was so cranky about. i was mostly cranky because it was hard to answer, but maybe there's a way to rework some of those into LO's that fit in here?
· Questionnaire (Draft v1): https://community.icann.org/x/EC1-Ag
this looks like it flows pretty directly out of the LO's -- so maybe we focus on those first? i really like how the number of LOs is longer than the number of questions -- this looks like a questionnaire that could be filled out pretty quickly. and maybe it's just distributed as a fill-in-blanks PDF or spreadsheet (to make scraping the answers easier)? same reactions on the "policy-process" part being a little light, perhaps. not a bunch more questions, but maybe one or two for those reflective types who want a place to note something in the PDP that got in the WG's way?
If you want to follow my developmental logic path, I suggest starting at the top and proceeding in the sequence shown. If you want to skip to the actual draft questionnaire, it is contained in the bottom linked page. I included a few notes which you should bear in mind before perusing the actual questions.
At this stage, I would like to receive your feedback before proceeding further to be sure that I am on a productive track. You will certainly notice that this instrument is no longer focused on the WG Guidelines documentation (the original survey scope); instead, on the effectiveness of the WG’s operations, norms, logistics, decision-making, and outputs. I included my rationale for that shift at the top of the Design Considerations page.
Feel free to comment on the email list or add comments to the individual Wiki pages. I will attend to both sources over the next week or so as you have time to review and evaluate my recommended approach to the WG Self-Assessment.
Regards,
Ken Bour
PHONE: 651-647-6109, FAX: 866-280-2356, WEB: www.haven2.com, HANDLE: OConnorStP (ID for Twitter, Facebook, LinkedIn, etc.)
Mikey just noticed that I replied only to him on this email and not to the entire group. Remedying that inadvertent error now. From: Ken Bour [mailto:ken.bour@verizon.net] Sent: Monday, June 10, 2013 7:30 PM To: 'Mike O'Connor' Subject: RE: [gnso-improvem-impl-sc] Draft WG Self-Assessment Design, Objectives, and Questionnaire Mikey: Thanks for another blazingly fast response. RE: PDP Framework You mentioned this item several times in your initial comments, but I'll address it once here. This is an interesting question as to whether the PDP framework itself is so intricately linked to the WG process that it could, on its own merits, influence the overall effectiveness of the WG. If I were to rephrase your inquiry, would this be close: "If the PDP process is cumbersome and difficult to follow, a WG's Mission could be challenged (if not inhibited) from the outset. If we are intending to evaluate a WG's effectiveness, one of the issues may be the very subject/content (and attendant processes) it is required to follow, e.g., the PDP framework." If I'm near the mark, I had not thought about that linkage until you raised it. Most of the WGs I have supported were not PDP-related, so it escaped my attention. I will be interested to see what others have to add on this matter. Another option would be to develop a separate survey-like instrument to evaluate the effectiveness of the PDP Framework itself although, as you point out, it is the WG construct that is asked to navigate those waters. Hmmm. RE: Individual vs. Collective For consistency (analytical, historical, design), I think we should have one defined process vs. having each WG roll its own. That design constraint would not preclude identifying a couple of overarching questions that we want each WG to address every time (via separate format). Furthermore, it is not inconceivable to have a couple of questions that are addressed both collectively and individually. Then, we could compare both sets of answers to see whether the WG answered similarly or differently from the individuals that made up the team (very intriguing and reminiscent of the Abilene Paradox puzzle). RE: Demographics I agree that private vs. public is complex and there are pros/cons to be weighed. I tend to favor the collection of personal identifying info for the reasons provided, but safeguarding that info in order to promote honesty and candor (still requires trust). Even in a largely volunteer organization, there are some people who will not be comfortable offering what may be perceived as negative commentary unless they can do so under the cover of anonymity. I think it might be cybernetics that advises: in dynamic systems, we learn primarily from error and, if failures go unreported and undiscovered, then those improvement opportunities are forever lost. RE: Size, Length, Complexity Hopefully, providing several open-ended text boxes will be generally sufficient for those who have a "lot to say." I'm not sure that any online survey instrument could be developed that would accommodate the exhaustive and insightful analysis you authored on your experience with the Fast Flux WG. From my recollection, that was and probably will always be remembered as a ONE-OF-A-KIND! J RE: Learning Objectives You are exactly correct. I started by extracting LOs from the WG Guidelines and Charter material; then, I built the questionnaire. Once I got into it, I began seeing that I could fold a couple of topics together (similar content) and ended up with fewer questions. Thanks for the input! I am energized by the responses thus far and am looking forward to continuing this process. Ken From: owner-gnso-improvem-impl-sc@icann.org [mailto:owner-gnso-improvem-impl-sc@icann.org] On Behalf Of Mike O'Connor Sent: Monday, June 10, 2013 5:27 PM To: Ken Bour Cc: gnso-improvem-impl-sc@icann.org; Hoggarth, Robert (ICANN) Subject: Re: [gnso-improvem-impl-sc] Draft WG Self-Assessment Design, Objectives, and Questionnaire hi Ken, i really like the work you've done. i've sprinkled reactions inline, but overall this is a great start. thanks, mikey On Jun 10, 2013, at 3:43 PM, "Ken Bour" <ken.bour@verizon.net> wrote: SCI Team Members: Since our last teleconference, I have been working on the development of a WG Self-Assessment instrument. Rather than inundate the SCI email list with multiple drafts and iterations during the incubation period, I thought it would be more helpful to post my work products to the ICANN Community Wiki. I have now reached a point where I think the material is ready for your review. In particular, you will find the following pages under the GNSO tab (category) and Working Group Resources space. Below is the hierarchy including links: WG Self-Assessments: <https://community.icann.org/x/2Cp-Ag> https://community.icann.org/x/2Cp-Ag really good introduction. on second reading i realized that one thing that might need expanding in the "lower-level" chunks of work is the LOs and questions that try to tease out the effectiveness of the formally-documented processes (the newly-changed PDP, developed by all the WGs that came before us). not a lot more, but just a bit. i think one of the reactions i had to the first questionnaire was that it was too focused on the formal-process stuff. so there's some balancing to be found. i could certainly live with the balance you've struck in this first pass -- but if there was a graceful way to add just a little more review of the PDP framework, i'd like that too. not too much though. we're going for "just right." :-) . Design Considerations: <https://community.icann.org/x/USt-Ag> https://community.icann.org/x/USt-Ag individual vs collective -- i like where you're going in the highlighted box -- some kind of hybrid between the two, maybe up to the WG. if they get really stuck arriving at consensus, just let the discussion serve as documentation rather than trying for consensus. i agree, it would be a shame to run a working group through another tough consensus process at the very end. especially tough if they weren't at consensus on their findings and were reopening old wounds. demographic information -- all good -- wisecracks about the NSA notwithstanding. although another approach to consider might be to have the information be public. i don't know -- tradeoffs. something to puzzle about. size, complexity, length -- yep, this is where i came into the discussion -- the previous questionnaire was really long/hard to fill out. but it would be interesting to see if there's a way to build in "optional" sections if people had a lot to say. my post-mortem on my experience chairing the Fast Flux WG took several days to write. if somebody's got a will to do something like that, we should provide a way. methodology -- no strong opinions either way here. . Learning Objectives: <https://community.icann.org/x/hit-Ag> https://community.icann.org/x/hit-Ag first pass -- WOW! i like these a lot. there may be more, but this is a really good start. good for: starting off the WG to set norms and expectations, periodically referring back to "mid flight" to offer course-corrections to the process as its's going, and reflection on opportunities for improvement at the end. since we're a PDP-process focused group, i'm wondering if there's a way to work a couple of LOs in about the framework itself. the goal being to see if there WG members noticed flaws in the process that got in their way. a good source of ideas might be the questionnaire i was so cranky about. i was mostly cranky because it was hard to answer, but maybe there's a way to rework some of those into LO's that fit in here? . Questionnaire (Draft v1): <https://community.icann.org/x/EC1-Ag> https://community.icann.org/x/EC1-Ag this looks like it flows pretty directly out of the LO's -- so maybe we focus on those first? i really like how the number of LOs is longer than the number of questions -- this looks like a questionnaire that could be filled out pretty quickly. and maybe it's just distributed as a fill-in-blanks PDF or spreadsheet (to make scraping the answers easier)? same reactions on the "policy-process" part being a little light, perhaps. not a bunch more questions, but maybe one or two for those reflective types who want a place to note something in the PDP that got in the WG's way? If you want to follow my developmental logic path, I suggest starting at the top and proceeding in the sequence shown. If you want to skip to the actual draft questionnaire, it is contained in the bottom linked page. I included a few notes which you should bear in mind before perusing the actual questions. At this stage, I would like to receive your feedback before proceeding further to be sure that I am on a productive track. You will certainly notice that this instrument is no longer focused on the WG Guidelines documentation (the original survey scope); instead, on the effectiveness of the WG's operations, norms, logistics, decision-making, and outputs. I included my rationale for that shift at the top of the <https://community.icann.org/x/USt-Ag> Design Considerations page. Feel free to comment on the email list or add comments to the individual Wiki pages. I will attend to both sources over the next week or so as you have time to review and evaluate my recommended approach to the WG Self-Assessment. Regards, Ken Bour PHONE: 651-647-6109, FAX: 866-280-2356, WEB: www.haven2.com, HANDLE: OConnorStP (ID for Twitter, Facebook, LinkedIn, etc.)
participants (2)
-
Ken Bour -
Mike O'Connor