iiinova — where your digital self keeps evolving

It’s more like % better for X the cost (or time) to train/tune tradeoff . For example gpt4 is significantly slower and 15x more expensive than 3.5-turbo

100

Reposted by Jared Palmer

men used to go to war, now they build auth startups

1123

men used to go to war, now they build auth startups

1123

Different models for different things. Cost and speed matters.

There are other solutions: use smaller or faster models and move inferencing closer to the end user. But my hot takes are that 1) raw LLMs are a pretty bad API for building generative UI 2) the JS DX of these AI providers is a hot mess 3) there is are a lot opportunities in this space

000

That works with fast models, but with gpt4 or other big bois your user is gonna be staring at a spinner for 20 seconds

But even this is still not massively useful because the response would still need to be completed in order to parse. Instead, I want the provider/model to let me specify named variables to be completed and then give me back an endpoint for each variable so that I can stream/fetch independently

210

For chat/assistant models, I want a System Prompt AND an Envelope Prompt that’s respected. The model should know about JSON when doing token affinity calculations

The problem is that parse/validate/reflection doesn’t work with streaming. So you can’t have nice things. 🚮🚮🚮

100

And yes, I’m aware that some providers offer System Prompts, these are still insufficient when variable output is necessary. The current SOTSA approach is to provide few shot examples, parse/validate the response and use reflection (another prompt to the LLM with the error) to get corrected output

Streaming text is not conducive to structured output (JSON, YAML, etc.) as you cannot parse and validate on the fly. Furthermore, if you need both structure and variable responses, you must provide more examples in your prompts (and pay more)

130

Having now used almost every major AI language model hosting provider/SaaS over the past few weeks, my conclusion is that the current programming model of manipulating a firehose of streaming text is extremely restrictive and incredibly annoying

360

Thank you for your interest in shitposting on Bluesky. While we appreciate your enthusiasm, we must respectfully decline your request at this time. We hope you understand our position and appreciate your cooperation.

Clubhouse was hot too for a minute. Hopeful about this one though

Yeah but it’s fun to start from scratch! Feels like a new email account