The model does the part that used to take a PhD and a weekend. It does not do the part that took a senior engineer and a Tuesday: input validation, retry behavior, rate limits, the eval set nobody wrote, and the prompt that works until someone pastes in a PDF.
None of that is new work. It is the same work - logging, tests, backpressure, graceful degradation - with a nondeterministic function call in the middle instead of a deterministic one. Teams that treat that call as magic skip the parts that make every other dependency safe to use.
The tell is always the same: someone ships the demo, the demo works, and then nobody writes the harness that would tell them when it stops working. That is not an AI problem. That is the same problem you would have shipping any external API without a contract, just with better marketing.