LLMs vs Human Experts in Requirements Engineering
The short version: AI held its own and even outperformed humans in some areas of requirements engineering.
5 February 2025
I’ve been building custom software in a services capacity for the better part of two decades. With varying roles from founder, engineer, solutions architect and consultant—you name it, I’ve seen my share of projects.
As many know, any project starts with trying to understand what the customer wants and translating it into requirements you can build from. As many people may not know, this requirement-gathering portion is ALWAYS the first thing that gets compressed, prioritized, defunded or otherwise bastardized. Regardless of the project size, you always try to start with two sprints dedicated to requirements. Oftentimes, projects of medium complexity bleed into three sprints, with larger projects taking four+ sprints.
This is the i-d-e-a-l case, meaning you have plenty of time to iterate with the customer and really nail down what they want.
However, if you’re an SE/PM/PjM/BA reading this, you know that never happens. It ALWAYS gets compressed down the shortest possible timeframe because customers “know what they want,” with account reps trying to lower the price to win the deal. Bollocks.
LLMs, with their vast array of Jedi force-like knowledge, offer a great way to help speed up requirements gathering and definition. They’ve scored well on language tests, performed great on coding challenges and can pass bar exams. So, how well do they actually do at creating software requirements as compared to a human expert? How well do these infant technologies stack up against grizzled experts? That’s the question we wanted to ask and was the tee-up for this study.
The complete process and results are available in our research whitepaper, “Analysis of LLMs vs Human Experts in Requirements Engineering,” but the short version is this: AI held its own and even outperformed humans in some areas of requirements engineering.
Here’s what you need to know:
The Challenge: Early Requirements
Here’s how the experiment worked: Participants began by submitting ideas for software projects they wanted to build. An LLM (in this case, ChatGPT-4.0) was given a single-shot prompt to generate requirements based on each idea. Meanwhile, a human expert was given an hour to create requirements for the same project.
Both sets of requirements were formatted as epics and user stories (epics are large pieces of work broken down into smaller pieces, or “user stories”). The participants then reviewed each document without knowing which was AI-generated and answered a series of survey questions.
Key Findings
Here’s what we found:
What This Tells Us About the Future of Software Development
Our study clearly shows the role that AI is capable of playing in software development, particularly when it comes to early-stage requirements gathering. Benefits could include:
While AI’s potential in software engineering is clear, we should note that the study had some limitations. For example, we compared only single instances of AI and human-generated requirements, and the time constraints (one-shot prompt versus one hour) may not always reflect real-world conditions. Additional research could explore how AI performs over extended periods across multiple iterations of the requirements engineering process.
The Bottom Line: AI Augments Human Development
While the study showcased AI’s ability to generate initial software requirements, it’s unlikely to entirely replace humans (for now). What’s more likely near term is AI and humans collaborating, where AI provides a sturdy foundation or starting point that humans can refine and expand upon. The classic human-on-the-loop or possibly human-as-validator architecture.
As AI rapidly evolves, it will continue to make its mark on software development. Professionals in this field, from developers to project managers to business analysts, should explore how AI tools can augment their work in requirements engineering and other aspects of the development lifecycle or risk being left behind.