The safety net most AI workflows rely on has a serious flaw
Why oversight is less reliable than most people assume
Getting better at prompting improves your outputs. It doesn’t fix the gap between what the model knows and what you can verify.
There’s a phrase that gets used a lot in AI circles to reassure people: “human in the loop.” It means a person is involved in the process somewhere, reviewing outputs, catching mistakes, steering things back on track. It sounds like oversight. It sounds like control.
I’ve been thinking about how often it’s neither.
Most of what we call prompt engineering is a negotiation with a system we can’t fully see, using tools we can’t fully evaluate, to produce outputs we often can’t independently verify. We type instructions. Something happens inside the model. Text comes out. We read the text and decide if it’s good. And somewhere in that last step, a very quiet assumption sneaks in: that we’d notice if something was wrong.
Often we wouldn’t.