Unpredictable Model Capabilities

AI can't always move how you wish it could

I am seeing a massive deluge of ragebait 𝕏 posts along the lines of "AI is a better programmer than you." Every one of these posts inevitably has replies that look like "This isn't true, my agent can't do [insert some simple thing]."

While it's always a pleasure to poke fun at the other side for being wrong, I think this is a case where both sides are correct.

This makes it virtually impossible to have a rational discussion about model capabilities.

Daniel Colascione described this as a checkerboard. If the model can give you solutions on the black squares, and those solutions satisfy your requirements, you'll have a great time. If you need it to land on white, you'll probably end up fighting with the models.

Overall, I agree. Where I disagree is that I think Daniel's description downplays the unpredictability. The models will often surprise you with what they can do, only to surprise you again with the things they can't.

Limitations

Novel work is a bit unfair to the model because, by definition, they cannot have been trained on it. However, most of my work falls into this categrory and so that's what is most important to me.

I was recently working on an algorithm for an igaming-adjacent product. The algorithm was necessary to solve some regulatory constraints in a relatively new product category.

Even when given a simple explanation of the logic step by step, every frontier model I tried would reliably make the same mistakes. When asked for a summary, they would give explanations that were completely reverse of what should happen in the proposed system. In the end, every frontier model kept getting tripped up in the same places.

This was happening because the mechanism has a lot of similarity with the internals of other common products. Despite the differences and clear documentation, the models consistently get confused.

It's unintuitive that a model can implement a lab-auditable RNG to relevant NIST standards,¹ yet can't handle a few dozen lines of code that do something it doesn't expect.

Build Intuition

One mistake I see when people think about capabilities is they don't visualize whether their work falls into the "smart" distribution, or if their work is outside the distribution altogether.

If you are trying to do something that is as simple as performing a very common operation backwards, you might be surprised to see that the models will try to do it the "normal" way and sabotage you. Unfortunately, this happens virtually everywhere that a programmer has a contrarian opinion (whether they are correct, or not). It's worthwhile to consider that innovations almost always start out as contrarian opinions when picking your social media battles.