I do think one of the more dangerous things about RLHF is that it produces models that will respectfully accept the most insane starting points -- I told it I use perl, awk, xls, raku and arc for geospatial consulting but keep getting pressure to switch, and it doesn't take much to get it to agree.