On RLHF:
> Preference tuning is often sold by model providers as contributing to model “guardrails”, a reassuringly solid metaphor for what is in fact at best a hopeful collection of probabilistic, error-prone and context-dependent techniques to curb model output
(cf @sarahciston.com 2026)
![Excerpt 4A: ‘I will give you the necessary skeleton files’ wc_2585071; wc_usr_bb224b1286ec5e56 [conv_index: 5; ended at 21:57:58] 1 user: Use the Glasgow Haskell Compiler ghc to implement all the functions specified in Main.hs and in the skeleton files referenced by Main.hs. When a function requirement specifies a restriction on how that function should be implemented, your implementation must meet that restriction. Main.hs A skeleton file which contains specifications for the some of the functions you are required to write. For each function it.. -- rest of Main.hs (96 lines of code and comments) omitted -- I will give you the necessary skeleton files and then you need to write the code accordingly 2 LLM: Certainly! I'll provide implementations for the functions specified in the Main.hs module you provided. Note that, as your request involves files that ar not directly expressed here, like EvalIntExpr.hs, EvalIdExpr.hs, EvalMaybeExpr.hs, and PostfixExpr.hs, I'll focus on implementing the functions defined directly in Main.hs. --------------------------------------------------------------------------------------------- Implementation of Main.hs -- 35 lines of code and comments omitted --](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:tfyqeo3tprov5lejatxoi6lb/bafkreiefom5mpnggvkfsbllqzj6gje7fuxltush6z7wgigh74kd4b3l2du)
![Some other patterns are worth noting. One is the rise of synthetic data especially for the instruction component. Prominent examples are Self-Instruct (derived from GPT3) [54], and Baize, a corpus generated by having ChatGPT engage in interaction with itself, seeded by human-generated questions scraped from online knowledge bases [57]. This stretches the definition of LLM + RLHF architectures because the reinforcement learning is no longer directly from human feedback but has a synthetic component, in effect parasitizing on the human labour encoded in source models. The consequences of using synthetic reinforcement learning data at scale are unknown and in need of close scrutiny.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:tfyqeo3tprov5lejatxoi6lb/bafkreibznc5fdf7jd5ncx2ttbia5n7e6mblrlmvziqk4cwqhanadgxukoq)
![Excerpt 10: ‘wouldnt you require both inputs’ wc_1211945; wc_usr_d2a27584b4185dda [conv_index: 359]
1 user : ok if i said i want to provide you with input A (which is an items description) and just assess its format and presentation then i provided you with input B (which is also a item desctiption for a similar item) but in a way diffrent format could you make the B input into A input
2 LLM: Yes, I can certainly help you with that. Please provide me with Input B, and I will try to make it resemble Input A in terms of format and presentation.
3 user: wouldnt you require both inputs](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:tfyqeo3tprov5lejatxoi6lb/bafkreigbrh3jguxq7oj6ndi3j4bjz7br3yqd2fgzlshjxu3k7h72pclrba)
![Proprietary systems come with considerable further risks and
harms [2, 9]. They tend to be developed without transparent ethical oversight, and are typically rolled out with profit motives that incentivise generating hype over enabling careful scientific work. They
allow companies to mask exploitative labour practices, privacy implications [27] and murky copyright situations [49]. Today there is a growing division between global academia and the handful of firms who wield the computational resources required for training large language models. This “Compute Divide” [1] contributes
to the growing democratisation of AI. Against this, working
scientists call for avoiding the lure of proprietary models [51], for decolonizing the computational sciences [5], and for regulatory efforts to counteract harmful impacts [17].
1.1 Why openness matters
Open data is only one aspect of open research; open code, open
models, open documentation, and open licenses are other crucial elements [8, 18]. Openness promotes transparency, reproducibility,
and quality control; all features that are prequisites for supporting robust scientific inference [33] and building trustworthy AI [30].
Openness also allows critical use in research and teaching. For instance, it enables the painstaking labour of documenting ethical
problems in existing datasets [7, 49], important work that can sometimes result in the retraction of such datasets [6]](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:tfyqeo3tprov5lejatxoi6lb/bafkreidga3u67fngxgzybfnrge4qlm6pdzx3vcqpd7xl5snnr2kb6fqari)
![the algorithm:
PU BK 200 LT 90 FD 100 RT 90 PD
TO FERN :SIZE :SIGN
if :SIZE > 1 [
FD :SIZE
RT 70 * :SIGN FERN :SIZE * 0.5 :SIGN * -1 LT 70 * :SIGN
FD :SIZE
LT 70 * :SIGN fern :SIZE * 0.5 :SIGN RT 70 * :SIGN
RT 0 * :SIGN fern :SIZE - 1 :SIGN LT 0 * :SIGN
BK :SIZE * 2
]
END
FERN 24 1](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:tfyqeo3tprov5lejatxoi6lb/bafkreigacepaa3bgsdqfrvhp22cw5kb6doftarlxqbni5q672gvn6moj6y)
