Crowdbotics Logo

About Us arrow

We are on a mission to radically transform the software development lifecycle.

Home Blog ...

Announcements App Development CodeOps Product Development

LLMs vs Human Experts in Requirements Engineering

The short version: AI held its own and even outperformed humans in some areas of requirements engineering.

5 February 2025

by Cory Hymel

I’ve been building custom software in a services capacity for the better part of two decades. With varying roles from founder, engineer, solutions architect and consultant—you name it, I’ve seen my share of projects. 

As many know, any project starts with trying to understand what the customer wants and translating it into requirements you can build from. As many people may not know, this requirement-gathering portion is ALWAYS the first thing that gets compressed, prioritized, defunded or otherwise bastardized. Regardless of the project size, you always try to start with two sprints dedicated to requirements. Oftentimes, projects of medium complexity bleed into three sprints, with larger projects taking four+ sprints. 

This is the i-d-e-a-l case, meaning you have plenty of time to iterate with the customer and really nail down what they want. 

However, if you’re an SE/PM/PjM/BA reading this, you know that never happens. It ALWAYS gets compressed down the shortest possible timeframe because customers “know what they want,” with account reps trying to lower the price to win the deal. Bollocks. 

LLMs, with their vast array of Jedi force-like knowledge, offer a great way to help speed up requirements gathering and definition. They’ve scored well on language tests, performed great on coding challenges and can pass bar exams. So, how well do they actually do at creating software requirements as compared to a human expert? How well do these infant technologies stack up against grizzled experts? That’s the question we wanted to ask and was the tee-up for this study.

The complete process and results are available in our research whitepaper, “Analysis of LLMs vs Human Experts in Requirements Engineering,” but the short version is this: AI held its own and even outperformed humans in some areas of requirements engineering.

Here’s what you need to know:

The Challenge: Early Requirements

Here’s how the experiment worked: Participants began by submitting ideas for software projects they wanted to build. An LLM (in this case, ChatGPT-4.0) was given a single-shot prompt to generate requirements based on each idea. Meanwhile, a human expert was given an hour to create requirements for the same project.

Both sets of requirements were formatted as epics and user stories (epics are large pieces of work broken down into smaller pieces, or “user stories”). The participants then reviewed each document without knowing which was AI-generated and answered a series of survey questions.

Key Findings

Here’s what we found:

  1. AI requirements were generally more aligned: Participants, on average, rated AI-generated requirements as more aligned (+1.12 points) with their original ideas compared to human-generated ones (based on a scale of 1-10, with 10 being most aligned and one being least aligned).
  2. AI is faster and cheaper: Staggeringly, the LLM produced requirements 720 times faster than humans and at literally a fraction—0.06%–of the cost.
  3. AI-generated requirements were often more complete: The AI-generated documents were 10.2% more complete on average than human-generated ones.
  4. Humans couldn’t always tell the difference: Participants often believed the requirements they thought were more aligned were also created by human experts, even when AI-generated.

What This Tells Us About the Future of Software Development

Our study clearly shows the role that AI is capable of playing in software development, particularly when it comes to early-stage requirements gathering. Benefits could include:

While AI’s potential in software engineering is clear, we should note that the study had some limitations. For example, we compared only single instances of AI and human-generated requirements, and the time constraints (one-shot prompt versus one hour) may not always reflect real-world conditions. Additional research could explore how AI performs over extended periods across multiple iterations of the requirements engineering process.

The Bottom Line: AI Augments Human Development

While the study showcased AI’s ability to generate initial software requirements, it’s unlikely to entirely replace humans (for now). What’s more likely near term is  AI and humans collaborating, where AI provides a sturdy foundation or starting point that humans can refine and expand upon. The classic human-on-the-loop or possibly human-as-validator architecture. 

As AI rapidly evolves, it will continue to make its mark on software development. Professionals in this field, from developers to project managers to business analysts, should explore how AI tools can augment their work in requirements engineering and other aspects of the development lifecycle or risk being left behind.