Need to start writing more...
A benchmark for long-horizon software building: seventeen full-stack apps, 241 feature requests, and 544 test plans, showing that even top models only ship fully working products 49–65% of the time.
Why letting frontier models choose their own effort level, handoffs, and subagents beats prescriptive routing — the bitter lesson of harness design, applied to Replit Agent.
How evaluation at Replit went from launch check to improvement loop — learning from production traces to ship a better Replit Agent, week over week.
I spent a couple of years building email and voice agents for freight carrier sales. Three takeaways stuck with me.
Do chat templates matter at inference time? We test the theory with an instruction-following benchmark and the best open-source LLMs of early 2024.