Prompting Principle: If you're fighting the model, STOP!

A variant of "if you can't fix it, feature it."

Sorry for the delay. Lots of stuff came up around clients and Augmentoolkit.

“Wait, clients?”

Yes! I’m a prompt engineering consultant now, helping companies move from the OpenAI API to cheaper and more customizable open-source model APIs while improving their output quality. Consulting’s hard, and there’s a lot to figure out, so… bear with me. There might be some inconsistency in the posting schedule. Then again, Prompting Weekly kinda also serves as a lead magnet/proof of authority, so maybe it’ll get a lot more love in the coming months, when I expand my leadgen beyond warm outreach.

Anyway, this isn’t Consulting Weekly, it’s Prompting Weekly, so let’s get to prompting and AI!

Here’s one of the many many prompts in Augmentoolkit. You might notice that the question passed as input is difficult to answer, if you don’t know what text it’s referring to.

One of the great problems of Augmentoolkit’s question generation is that, for many models, it’s very insistent on adding “according to the text” or “in the text” or some variant thereof. Not one few-shot example contains “the text” in a question, or an answer. But still a huge number of the generated questions and answers reference the damnable text. Maybe the chain of thought is at fault, I don’t know. This could also be one of the cases where it’s just a quirk of the model that makes it generate questions with this kind of contextual reference.

When there are a lot of prompts, the “specific cause” of your consistent behavior can be hard to find. Finding the causes of model behavior is an instinctual skill you build with practice — it relies more on your mental bird’s eye view of every example in the prompt you’re working on, and the ability to say “oh I bet that THIS part is the cause”, than clear logic. Using AI a lot is a great way to build this skill, but sometimes it fails nonetheless, or there’s simply too much text to look through. And so we need to adapt in other ways.

“If you can’t fix it, feature it” is one of the great lessons from Gerald M. Weinberg’s “Secrets of Consulting” book. And it suits prompting very well.

Put another way, for this field specifically:

If you can’t get the model to do what you want, change “what you want” to something you can get the model to do.

It may seem cheap to call moving the goalposts a principle, but when your job involves trying to cajole blackboxes with room temperature IQ, flexibility is important. The trick is to move the goalposts to something that still achieves the overall goal, and to be ruthless with your original vision — it’s expendable.

There’s a quote by William Faulkner, “in writing, you must kill all your darlings.” Prompting takes the darling-murder up to 11. The first idea you have for an LLM powered system will almost always be too artistic, take too much time to implement, be too fragile, clash with the fundamental nature of the LLM, be too convoluted… the list goes on. This might work if you’re working in an ivory tower and on a hobby open-source project that hasn’t shipped yet (and might never do so), but all practical use cases have real-world constraints. So you need to boil your goal down to its most fundamental form.

With the prompt I showed earlier, the fundamental goal wasn’t “remove mention of the text so that questions become answeranble” it was “give questions enough context to be answered.” So the “remove mention of the text” prompt (surprisingly finicky, I was fighting the model a lot on that one) became “add the actual name of the text in, where appropriate.” Goalposts moved? Yes. Goal achieved? Also yes.

Context achieved. Mission accomplished. Onto the next prompt.

This principle flows naturally from “Consistent behavior has a specific cause.” Sometimes the behavior is in the dataset or RLHF of the model. In that case, even if you do manage to prompt in such a way that the model sometimes acts the way you want, it’ll come at such a cost in reliability (because your prompt is fighting the model, and sometimes the model will win) that the “solution” is unsuitable for production. The goal is not to get the model capable of outputting a correct generation, you could use zero-shot for that. The goal is to get it doing so 90, or even 80, times out of 100. In my experience, if you fight the model, even if you “win”, your prompt will end up being fragile. Models are like humans in that consistency is the hard part (just look at my posting schedule).

So in short: if the model is being stubborn, and you don’t think it’s your prompts, change the way your LLM-powered system reaches its goal.

There’s actually a paper on this, it’s one of my favorites. In-Context Learning Learns Lable Relationships but Is Not Conventional Learning. They set out to validate or invalidate three hypotheses, including “Is ICL powerful enough to overcome prediction preferences originating from pretraining?” In more simple terms, this is: “Is prompting enough to overcome bias introduced by a model’s training data?” The answer they come up with in their very-comprehensive paper is “It can reduce it, but can’t overcome it.” Yet more evidence that if you fight the model, you lose in the end. Frustratingly, it can reduce the pretrained biases of the model, meaning that it gives you the illusion you’re winning, too. And maybe for your use case, you can accept a bit of instability, and that’s fine. But if you’re ever trying to get a model to do something, and it just. won’t. listen., maybe consider getting it to do something easier. The goal is what you’re working to see realized, not the process you thought of first.

Some misc news:

• Augmentoolkit’s getting some major overhauls, including a Python script, a configuration file (YAML), and the option to use an OpenAI-compatible API (so stuff like together.ai or mistral.ai that offer Open Source model endpoints will be usable). This update is inspired by the illustrious MrDragonFox (you might know him as a moderator on the Mistral and TheBloke’s Discord servers) who has become a contributor to the project. So if you have thousands of dollars of API credits lying around, or if you simply hate Jupyter Notebooks, then Augmentoolkit’s going to get a lot better for you sometime around the end of this month. Oh and you also won’t need to title all your input files descriptively.

• I mentioned this earlier, but, I’m consulting now! If you can think of any companies that use the OpenAI API, and would rather not be using the OpenAI API because it’s costly and inflexible and easily detectable as AI-written and annoying, let me know!

• Having written this article I realize how relatively easy it is to write substack articles and am now hating myself for not posting the previous two weeks. If this feeling of frustration manages to endure ‘till next Sunday, expect the resumption of weekly posting. Topic for the next week… maybe decomposition of prompts into smaller tasks?

Anyway, that’s all for today! You know, I’ve realized that I could stand to be promoting more of the stuff I program or write, I feel like nerds fundamentally undervalue the ideas of “image” and “reputation”. Being active online opens a lot of doors. And you know what one very easy way to show you’re active is? Showing other people that you’re reading interesting things by featuring them on your socials:

Have a good one and I’ll see you next week!