Vercel Removed 80% of Agent Tools and Accuracy Went Up
Vercel’s engineering team has built their own internal text-to-SQL agent (d0) with many custom tools, prompt engineering, context management and all the magic that we need to build an AI app.
Though the app worked it was kind of fragile and needed constant maintenance.
So what they did is they stripped away all the tools and just gave arbitrary bash execution tool powered by their Sandbox feature (obviously!)
Based on their benchmarks here are the results:
- Success rate jumped to 100% (from 80%)
- Fewer tokens - fast response & cost
- AI model interacted with raw data instead of something over engineered.
Addition by subtraction is real. The best agents might be the ones with the fewest tools. Every tool is a choice you’re making for the model. Sometimes the model makes better choices.
The key takeaway is that more tools does not mean better result.
Sometime, it could be reason why we don’t get correct result. So less tools are better 😅
Reference
