That will only work if a crawler reads robots.txt and respects the published directive(s). Not all do.
All of the search engines used by consumers do. I'm still having trouble understanding what the problem is here. The specific set of records that Scott noticed are in fact just some examples linked from a public web page, and I see no reason to think that it'd be hard to keep RDAP info out of the usual search engines if that's what you want to do. For a very long time, Domaintools and others have scraped WHOIS info and provide a little of it for free and more for pay. RDAP doesn't change that. If you want to redact information beyond what's in WHOIS, that's a reasonable discussion to have, but it's exactly the same for WHOIS or RDAP. R's, John