- Purdue University researchers found ChatGPT incorrectly answered over half of 517 software programming prompts from Stack Overflow.
- Participants struggled to identify incorrect ChatGPT answers 39.34% of the time, suggesting extensive work needed to address its shortcomings.
- The researchers suggest focusing on enhancing ChatGPT’s reasoning capabilities to improve its reliability in programming context.