Rubens,
"Except for RSP Evaluation, all the other evaluations existed in 2012, and all of them were done by outside contractors, not ICANN itself."
I'm not entirely sure this is correct. I seem to recall that Background Screening (or at least part of that) was undertaken by ICANN staff, but I am happy to be corrected if I am mistaken.
https://www.icann.org/en/system/files/files/program-review-29jan16-en.pdf
Page 58 states that Pricewaterhouse (PwC) was the independent third-party that did background screening.
In any case, an additional thought has come to mind on Anne's point - I seem to recall that ICANN org may be contemplating moving some of the evaluations in-house. I can't recall which ones exactly but insofar as that is a possibility, would any of the arbiters being considered in Annex, then change? Or are we to keep our recommendations / implementation guidance simply on the basis of what happened for the 2012 round?
I think we should look at future trends, starting with what happened. Considering all my dealings with ICANN Org I was genuinely surprised when they mentioned the possibility of moving some evaluations in-house, but they also mentioned that this would be more in response to a continuing application process, not a round based one.
So, we would need to look whether the post-transition RfR is now as effective as a redress mechanism as being thought in the limited appeals process. The central issue is whether substantive redress is possible, in order to not have "we followed procedure" responses.
Rubens