LLMs in 2024
2024 was a great year in terms of LLM progress.
Here are some of the notable progress. It’s primarily based on the summary written by Simon Willison
🔥 Biggest Changes:
- LLM prices crashed dramatically (up to 27x cheaper)
- Powerful models can now run on laptops (Llama 3.3 70B, Qwen2.5)
- Voice/camera interactions is becoming mainstream. It’s not science friction anymore
- Open Source models are on par with proprietary AI models. Unlike last year, OpenAI doesn’t seem to competitive advantage with current progress
💪 Key Improvements:
- Multimodal became standard (vision, audio, video support) and almost all the new models supports them
- Context lengths increased massively (up to 2M tokens)
- Training efficiency improved significantly (DeepSeek v3 trained for ~$6M)
- Inference-scaling models emerged (like OpenAI’s o1/o3 series)
- It uses Chain of Thoughts under the hood
⚠️ Ongoing Challenges:
- “Agents” still haven’t materialized meaningfully. And similar to fine-tuning in 2023, everyone started using “Agents” as next big thing (buzz word)
- Models remain difficult to use effectively
- Environmental concerns shifted from per-prompt cost to infrastructure
🔮 Notable Trends:
- Synthetic training data proved highly effective
- Prompt-driven app generation became commonplace
- Apps like Bolt.new, Vercel’s v0, etc becoming more and more popular