AI Advances Fastest When We Find Unnatural Ways of Doing Things

LLMs3 have been making more and more tasks look “like chess”, amenable to efficient automation. There are still many tasks they can’t handle, but the boundary keeps moving.

One could interpret this as the unfolding of a single cluster of innovations around LLM architectures and training. In this scenario, the transformer architecture provided a single leap forward in “squishy skills”, and over the last couple of years we’ve just been applying it at increasing scale and in cleverer ways. In that case, we might eventually hit a wall, with tasks like childrearing and corporate strategy waiting for another breakthrough that could be many years away.

Alternatively, perhaps the road from GPT-3.5 to Claude 3.5 and Gemini 2 and o3 has included further progress on the squishy side of things. I’m not sure how to rigorously define this, let alone measure it, but I believe there is a real question here. If we are seeing progress on that side, then even tasks that require deep judgement may soon look “like chess”, where AIs can outdo us, and human capability won’t seem so impressive. If not, then some skills may continue to elude AI for a while, and we’ll say that humans are uniquely suited to those things.

I’m not sure which track we’re on. I wish we had more information about how OpenAI’s latest models have racked up such impressive scores on FrontierMath and Humanity’s Last Exam. That might shed a bit of light. But I’m sure we’ll learn more soon. Whichever skills continue to seem like the best fit for the still-mysterious workings of the human brain, will be the last to fall to AI.

Subscribed