At qodo, we began to consider an AI-powered solution for Pull Requests (PRs) following dozens of interviews with developers and team leaders, across a range of industries. We wanted to get a deeper understanding of the requirements, difficulties, and problems that are frequently encountered during a PR’s standard review procedure. The primary issues mentioned were the following:
1) Reviewing PRs is not fun: it’s a time-consuming and repetitive task, that is frequently seen as a “chore”, rather than a task with interesting technical content. As a result, reviewing PRs is frequently prioritized lower than other tasks, resulting in low-quality reviews and longer review times.
2) The longer the PR – the shorter the feedback: short PRs are more manageable. The reviewer can get a comprehensive look and provide detailed feedback. In contrast, long PRs (more than ~100 lines of code changes) can overwhelm reviewers, both in terms of technical difficulty, and the actual time it will take to properly review all the code. In addition, providing feedback covering all the changes in a long PR may result in tens of comments, which would appear to be excessive and unreasonable.
The following graph, adapted from a study by Graphite, demonstrates this point – the more code lines there are in a PR, the fewer comments it gets (per every 100 lines reviewed).
This implies that many code lines in a long PR practically remain “unreviewed”.
3) No one likes to write PR descriptions: a 10x engineer can open several PRs per week, for which consistently writing detailed PR descriptions is a difficult and monotonous task. This frequently results in PRs with minimal descriptions, that provide few details, if any. For example, a research study found that in a collection of over 330k PRs, more than 34% had no descriptions at all (!).
Even when attempting to write a longer description, developers frequently fail to cover all changes made. They often create PRs containing multiple changes, but only address in the PR description the most important change. This increases the likelihood that a buggy PR will be approved, since the reviewer is not informed of all the changes made from the PR description, and must instead “figure it out” himself by reading the diff code, which is a difficult and time-consuming technical task.
4) Conflicts on a team level: when writing technical code, engineers often work alone, or occasionally in pairs. A PR is the first time they share their new code on a team level. This new scenario, which is distinct from the more common day-to-day work, can lead to conflicts. For example, merge problems and inconsistencies, long review processes, failure of existing tests, lack of tests for the new code, unclear code (for other users), style differences, and more. These conflicts might create bottlenecks when integrating new code, prevent the engineers from taking on additional technical tasks, or lead to general dissatisfaction.
By combining these pain points into a high-level view of a typical PR, we can depict three common personas involved in the PR review cycle. Every persona has unique objectives, difficulties, and preferences, and contradictions frequently occur.
A. The Developer
The individual who writes the code and initiates the PR. He usually wants the PR approved as fast as possible, so that his code can be merged and have an impact. He is also frequently blocked from taking on additional tasks while his PR is waiting to be reviewed.
The developer will still appreciate clear and concise technical feedback, that focuses on identifying and resolving important issues and bugs. However, he would like to avoid a lengthy and unjustified review process, as well as receiving review comments that he considers irrelevant or trivial.
B. The Reviewer
The reviewer is tasked with giving feedback and approving the PR (sometimes multiple people need to approve). Often the reviewer will feel overwhelmed and underequipped. Both the developer and the reviewer share the responsibility of not introducing problems and bugs to production. Yet the reviewer frequently lacks the time, technical knowledge, or focus to fully comprehend the PR, and provide adequate feedback.
The dynamics between the developer, more eager for quick approval, and the reviewer, prioritizing code integrity, can lead to communication barriers, and limit the reviewer’s ability to have an open discussion with the developer. This situation is exacerbated if the reviewer is less technically acquainted with the PR content than the developer, potentially leading to superficial approvals without in-depth review and feedback.
Reviewers, under pressure to prevent issues in production, may feel underequipped and undervalued, navigating a complex balance between efficiency and thoroughness.
C. The Mid-level Technical Manager
Typically, a mid-level technical manager no longer writes code, and is not even directly responsible for reviewing PR code. In reality, managers would find it difficult to read and comprehend the majority of PR code diffs because of knowledge and daily-practice gaps.
Despite this, there’s a strong preference among managers for a system that includes high-level summaries of each PR, alongside periodic compilations of technical achievements over a given time period (weekly or monthly). This approach ensures that managers remain connected with the team’s day-to-day activities and ongoing coding projects, without needing to delve into the details of code themselves.
Qodo Merge to the rescue:
When building Qodo Merge (qodo’s (formerly Codium) git plugin!), we tried to address the needs and the pain points of each persona described above.
For the developer we provide concise, high-quality code and PR feedback, aimed at identifying issues early and enhancing the PR, ensuring their time is used effectively and efficiently.
“Maximized Efficiency: Streamlining PR Preparation, and High-Quality Code Suggestions”
For the reviewer, we offer a comprehensive feedback package including automatic PR description, file walkthrough, identifying possible issues, possibly for a chat on the entire PR or on specific code lines, and many more features, aiming to help him close the knowledge gap, and safeguard the main branch from bugs and problems.
“Closing Knowledge Gaps and Safeguarding Main Branch Against Bugs and Issues”
For the mid-level technical manager, we offer in each PR a high-level bullet point summary, PR labels, automatic changelog, and other concise feedback mechanisms that will enable him to “stay in touch” with the code changes and understand what’s going on, without needing to read technical code.
“Enabling Managers to Quickly Grasp Code Changes Without Deep Technical Dives”
Elana Krasner
Marin Levrel
