LLM's $3 Trillion Dollar Problem

I'm starting to feel like LLMs are now commodity. If I asked you what the best model is right now, the quote-unquote best model, most of you would probably give me very different answers. There's a lot of people in the audience who think Codex models are the best, and there's also a lot of people who think Claude models are the best. Heck, there's also a lot of people who use Gemini for everything and think that's the best. The differences between these models are becoming harder and harder to see, right? And not just across companies, but also in the same company or the same LLM provider. There's no meaningful, clear improvement between Opus 4.6, 4.7, 4.8, right? Like, if you just ignore all the hype of every single new model release on Twitter and everything, and you actually use these things in real-world tasks, you will notice none of them are actually improving at a very fast rate, right? There's no clear noticeable improvement between any of these new models at all. And you might think you can just look at the benchmarks for every single model, right? Because benchmarks are just raw numbers and don't have any subjective bias in them, right? You might think that, but the benchmarks don't show real-world usage. Like, you can look at all of these numbers and maybe decide which model is right for you, but these benchmarks aren't going to show real-world usage, right? One, because a lot of companies are just trying to benchmark max, right? They're trying to do some benchmark maxing where they're trying to purposely create models that get higher and higher scores on benchmarks instead of real-world task performance, right? And the second reason why I don't think these benchmarks really matter all that much is because the harness around the models is so much more important than the model itself, right? And we'll talk more about this in just a second. So, for 99% of tasks that you might want to automate or whatever, as long as you're using any reasonably frontier model, right, you're going to see no difference between the outputs of all of these different models, right? As long as you're building a good harness around it. Hey, just give it another six months, and nobody's even going to care what model you're using. Nobody's going to care about it because it's going to become a commodity, right? And a harness around the model, that's going to become more important than the model itself. And when I say harness, I mean things like the user experience around the harness, the context provided to the model, and the tools available to the model. And these things are becoming more important than the actual model itself. Because if you nail down these three things, you can get 100 times better output from even a quote-unquote worst model as compared to the quote-unquote best model with none of these things available. Right. Since models are becoming harder and harder to distinguish, right, and it can just be swapped for another model very easily now, right, the user experience and the harness around the model just became much, much, much more important. Like, just look at how Cursor has so many more users than, say, Anti-Gravity, right? Even though in both of these IDEs, in both of these editors, you can have Claude models, you can have Codex models, all of these things, right? But Cursor as a harness, as an AI agentic code editor, is so much better, right? That's what I mean when I say the harness is more important than the actual model itself. Like, I don't know if you guys saw the new Anti-Gravity 2.0 app, but it's basically just a complete ripoff of Codex, right? And Codex still gets many more users. So really, even though the models don't really seem to matter that much anymore, the harness around the model is just as much important, right? The harness around the model is so much more important, right? Even open-source models, like, I don't know, Gemma 4. I've been using this a lot. And DeepSeek, again, another popular model. These models are so good that you can literally use a tiny 12 billion parameter model for so many different complex tasks, right? As long as you provide it with the right tools and the right context that the model needs to get the job done, right? Exactly what we discussed here with the harness. And a lot of people are getting great results with DeepSeek, right? You can do the same tasks as the latest Codex and Claude models at a fraction of the cost, right? And for most business tasks, you don't even need the latest frontier models. You don't need these Opus 5.6 or whatever, right? It's a waste to use such expensive models for the common business tasks that most companies are trying to automate. You can achieve the same result at a fraction of the cost by using these open-source models. So really, I see a future in the next six months where nobody really gives a fuck what model you're using. The specific model that you're using is going to become less and less relevant. And what's going to become more important is the harness around the model, the tools available to the model, the context provided to the model. So, quick pro tip: if you're someone who builds AI agents, right, let me give you some quick tips over here. You need to start focusing more on the harness around the models, as opposed to the model itself. Start paying more attention to harness engineering. OpenAI has a really awesome article on their blog about this exact topic. I'll try to link it in the description. Go read that as well. And just to give you an example, let's say you're building an AI-powered app over here, right? You need to give it the tools it needs to take actions on users' behalf, right? So the users have to do less work themselves. Also, make it so it can automatically pull the context and the information it needs from the app to give personalized, relevant outputs without the user having to manually provide that context himself. The harness or the app should be able to pull that context automatically. And just a quick sales pitch before I end the video. If you're a business owner and you don't understand what I'm talking about here and would like to integrate AI into your company, look no further than this channel. This is the work I do. I'm a freelance software developer, and I build AI workflows and agents as well as websites and mobile apps for businesses just like yours. And if that's you, check out the top link in the description. You can book a call with me right now, and let's get you integrated with AI. That's all for this one. Thanks for watching, and I'll see you in the next one.