Code Integrity Supercharges Code Generation

TL;DR:

Gen AI is a new standard, at least for software development.

How to stay ahead and exploit AI coding assistant tools smartly?

Code Generation tools: AI assistants or agents for generating code enable you to code faster. However, they can also create new problems for development teams, like introducing hidden bugs and reducing familiarity, understanding, and responsibility of the code.

Code Integrity tools: AI assistants or agents for verifying that the code fits the intent or spec, improving code coverage, improving code quality, and helping developers get familiar with the code.

Code generation and integrity tools, working together, suggest a powerful combination and a glimpse into the future of intelligent software development.

Examining two leading AI coding solutions

In the preliminary blog “GPT-4 vs. AlphaCode, comparing two leading code generation tools”, we discussed and analyzed how AlphaCode [2] achieved ~x4 better accuracy over GPT-4 [0] on Codeforeces Ratings [1].

GPT-4 is mostly a code generation tool, while the AlphaCode solution is composed of both code generation and code integrity tools.

Let’s further discuss Code Generation and Integrity tools and how they can be combined to create powerful software development agents.

Code Generation Tools

Examples: GitHub Copilot, Google Codey [3], JetBrains AI, Sourcegraph Cody, Tabnine

Code generation tools and agents offer a powerful way to produce initial drafts of relevant code snippets or even entire sections of code according to a developer’s prompt or high-level specifications.

Main value proposition:

Speed up the creation of repetitive and boilerplate code. Instead of searching in Google and StackOverflow, for example, you get code snippets inside your IDE.
Speed up the drafting of entire code sections. Without spending excessive time on manual coding and project creation, you get an initial draft.

Side value proposition:

Learn. With a couple of “strokes”, you get various code snippets or solutions.
Rapid prototyping. Quickly create prototypes or proof-of-concept implementations where high-quality code is less important.
Someone to talk to. Chat with a coding assistant to discuss almost anything, from software design to specific implementation details.
Code documentation, translation, refactor, or enhancement.

Some Drawbacks:

One-line code completion is great. The accuracy seems to be great, and the mental load is low. However, a full function or module code generation does not pass tests, as shown in AlphaCode work, for example.
Over-reliance and shrinking familiarity with the code. Developers might not be as familiar with the generated code as they are with manually written code. They might also change how they develop to favor code that is easier to produce with code generators.
Misalignments with specific standards, guidelines, patterns, and best practices. Code generation tools follow predefined patterns and logic according to the data they were trained on. Each dev team and/or company may have additional or specific standards, guidelines, patterns, and best practices.

Anti-DRY (“don’t-repeat-yourself”). With the easiness of generating code, developers might repeat similar code. This is a challenge we see and feel ourselves at Qodo (formerly Codium) and also hear from our clients that use code generation tools.

Mitigations might be introduced in the future for most of the drawbacks. For example, code-generation might search and find similar code and suggest using or merging it. In another example, code-gen tools can combine forces with code-integrity tools – this will be covered later in this post.

In conclusion, code generation tools can be highly valuable for enhancing productivity and automating repetitive coding tasks. However, carefully considering their limitations and appropriate use cases is necessary to ensure their high quality and maintainable code bases.

We’ve heard from developers that code-gen tools alleviate the mental strain by offering relevant code suggestions within their IDE. However, they also mention that prolonged use of these tools has led to increased fatigue due to the shift towards more code reading and reviewing rather than active writing.

Code Integrity Tools

Examples: Qodo (formerly Codium), some features of GitHub Copilot-X

What are code integrity tools?

These tools help ensure that the code is complete, correct, and consistent, ultimately contributing to the overall functionality and reliability, as well as the efficiency and security of the software system. Code integrity tools assist developers and development teams in identifying and rectifying errors, vulnerabilities, and issues in the codebase.

To further read about Code Integrity, please see posts at Qodo (formerly Codium) and wiki [4].

Code integrity tools’ capabilities may include static code analysis, test generation and analysis, code review, enhanced documentation, and more.

Main value proposition:

Analyze and increase code coverage and behavior coverage.
Verify code behavior vs intent. Reduce bugs.
Improve familiarity with the code. This is done mainly through code explanation that is grounded in code behaviors and analysis as appropriate to code integrity tools.
Increase confidence when merging code and deploying software. See this post, “Revolutionizing Code Integrity” [5]

Side value proposition:

Learn. Code integrity and testing are swiped behind the carpet in computer science studies, with a focus on coding and architecture. Code integrity tools provide developers with an opportunity for just-in-time learning on how to compose comprehensive tests.
Someone to talk to about code integrity, such as testing and code review.
Code documentation, translation, refactoring, or enhancement that is grounded in testing and code analysis methods
Consistency: code integrity tools may adhere to predefined templates and standards, ensuring consistency in coding practices across the project. This reduces the likelihood of errors caused by manual coding inconsistencies.

Can code generation tools act as code integrity tools?

Code Integrity tools may differ in various aspects from Code Generation tools: UX/UI, integrations, solution architecture, underlying ML models, and more.

Code-generation tools may partially provide code-integrity-related features. However, due to various differences, it is very likely that the results will be inferior. For example, ChatGPT can be used to create unit tests; however, they will very likely be naive compared to those created by a code integrity tool. Check out this comparison between Qodo (formerly Codium) vs ChatGPT [6].

What to expect from Code Integrity tools that are absent in Code Generation tools?

As an example, Code Integrity tools should be able to consider existing test suites when generating additional tests.

In another example, Code Integrity tools should manage existing and new mocks in particular and doubles in general.

Yet in another example, Code Integrity tools should measure the quality of generated tests, e.g., with advanced techniques such as mutation testing.

Combining Generation & Integrity Tools

Code generation and integrity assistants or agents can be used simultaneously to exploit the advantage of each other and even to compensate for and cover each other disadvantages.

We’ve covered such an example, the AlphaCode solution mentioned above.

Let’s explore how you can start exploiting these tools today and in the future.

Software developments with coding assistants:

Developers’ common software development flows with AI assistants:

in the IDE – code generation tools may live side by side. For example, code generation can be used to suggest code, while code integrity will generate and analyze test suites.
in pull requests – code integrity tools can assist in code reviewing, while code generation can be used to update documentation and changelogs.
when writing software specs – writing assistants can help create coding specs. You can try more generic ones, like Notion or ChatGPT, or more designated ones (disclaimer and spoiler: future roadmap of Qodo (formerly Codium))

Software developments with coding agents:

A glimpse into the (near?) future – software development with agents:

Code integrity and generation agents will interact among themselves and with developers to generate, test, and align code according to the developers’ specifications. This is somewhat similar to the primary (and somewhat primitive) architecture used in AlphaCode.

Think about a scenario where a developer writes a code spec with the help of a coding assistant. The code generation and integrity agents work in a duet to generate a full project while analyzing, testing, and fixing it. During the process, the agents may go back to the developer to ask for clarification or request to make decisions on various aspects.

Agents’ capabilities are progressing, and we expect to see breakthroughs in the near future. For example, expect to see improved tools, chain-of-thoughts, designated memory, etc… for further reading on this point, see [7].

About Qodo (formerly Codium):

Qodo’s (formerly Codium) vision is to:

Create an intelligent
software development world
where any developer can
develop fast with zero bugs

At first, our mission is to simplify code integrity.

With Qodo (formerly Codium), developers can:

True to Aug 23’, Qodo (formerly Codium) offers IDE extensions and a pull-request agent (left to right respectfully; on the left a screenshot of the IDE extension, and on the right a diagram of Qodo Merge (formerly PR-Agent) software architecture).

References:

[0] GPT-4: GPT-4 Technical Report (paper, blog)
[1] Codeforces Rating (post)
[2] AlphaCode: Competition-Level Code Generation with AlphaCode (paper, blog)
[3] Codey (API doc)
[4] Code Integrity (post, wiki)
[5] Revolutionizing Code Integrity (post)
[6] Qodo (formerly Codium) ChatGPT-4 (blog)
[7] 6 important AI near-future breakthroughs (blog)
[8] GPT-4 vs. AlphaCode, comparing two leading code generation tools (blog)