Review of AI Coding Bots — May 2024

I Can Write Excellent Detailed Tech Specifications — Is There an AI out There yet That Can Generate Good Code to Match It ?

Muse
9 min readMay 7, 2024
This was Claude 2’s first attempt at creating the form based on my spec

I’ve worked for 25 years as a software developer and project manager.

I’ve been loving the boost to productivity that GitHub Copilot and ChatGPT have given to my work over the last few years.

But are we at the point yet where an AI can produce the code for a full (small) system from a detailed specification?

Summary of AI Tools Assessed

Here are the tools I tried (and the summary results):

ChatGPT4 — not up to the task, but could make an excellent start

Tabnine Pro— what does this even do?!

Amazon CodeWhisperer / Amazon Q Developer / Amazon CodeCatalyst — unable to get it/them working

GitHub Copilot Workspace — not available yet

Program AI — cannot read specifications, nor read long instructions

Smol Developer — unable to get it working

❕Claude 2 — an improvement on GPT4, but still requiring some spoon feeding and hand holding

--

--

Muse

Muse spent his youth searching the world for meaning, and good parties. He has now settled down to the quiet life of a writer, business owner, and father of 5.