ChatGPT for Automated Testing: Examples and Best Practices

ChatGPT or Generative Pre-trained Transformer is an artificial intelligence system that is able to do a lot of amazing things including writing essays, poems and stories, answering various questions in natural language and engaging in a dialogue. As a language model, ChatGPT may not be a universal tool for test automation in all cases. It is designed to generate human-like responses to text inputs, and while it is trained on a variety of tasks and domains, it is primarily used for natural language processing tasks such as text generation, language translation, question-answering, and summarization. However, there are multiple scenarios where ChatGPT’s generative AI capabilities can be utilized for code testing and may make life of testers and QA specialists easier as well as support automated testing. In this article we will discuss some of the ways to use ChatGPT for automating and speeding up the test lifecycle.

Advantages of using ChatGPT in software testing

Here are some of the reasons why you may want to leverage ChatGPT capabilities in your testing process.

• Automate test data and test cases generation for a faster and more efficient testing process. ChatGPT can analyze given requirements and generate relevant test data and test cases based on the context and the desired outcome. This can greatly speed up the testing process and free up testers’ time for other tasks. It can also be used to simulate user interactions with a system, which can be helpful in testing user interfaces and chatbots.
• Improve testing quality and test coverage. ChatGPT can be used to generate more realistic and complex test cases that can help identify a wider range of software bugs and problems. It is also able to generate more tests in a given time than a human, which leads to better test coverage.
• Overcome lack of knowledge in a specific technology. ChatGPT is a valuable resource that can help to overcome the constraints related to having limited knowledge of a specific technology. If you do not know where to start it could generate initial code for you. Of course, you cannot rely fully on these generated results, but it could be a good starting point if you are stuck. In some cases, it can even support debugging and give you the exact reason why the code is failing.

It is important to note that ChatGPT should not be used as a replacement for traditional automated testing tools which are specifically designed for testing software systems and have a wide range of features and capabilities that are not available in ChatGPT, e.g. test environment management, test integration, test reporting and some others.

Disadvantages of using ChatGPT in software testing

While ChatGPT is a powerful tool, there are several limitations for its usage in the testing process.

• Training limitations
  ChatGPT is only trained on the already available data, which can limit its ability to create more creative and complex test cases. If the training data was not representative of the software application being tested, ChatGPT will not manage to generate accurate test cases.Also since it is based on statistical patterns, often it does not fully understand the context, hidden meaning, or have in-depth knowledge of a system to be tested.
• Need for additional validation
  ChatGPT may generate test cases that are not always correct and accurate, so additional results validation is required. Sometimes the results are also not complete. Therefore, you still need to understand the system’s response and be able to refine and finalize it.
• High complexity of integration in the existing process
  ChatGPT is a complex technology and is not specifically designed for utilization in the test process. Integration of this tool in the existing process may require additional implementation efforts.

Examples of using ChatGPT for testing automation

Let’s check some examples of how ChatGPT can be used for higher testing process automation.

1. Automated test cases generation
   As previously mentioned, ChatGPT can generate test cases based on natural language descriptions.For example, let’s imagine we are testing an authentication form which includes username and password fields. There are some requirements which a password and a username should fulfill: the username must be at least 4 characters but not more than 15 and can include letters and numbers; the password must be at least 8 characters and contain at least one letter and one number or special character. The password can not be the same as the username. We can then provide this information to ChatGPT and ask it to generate some tests to check functionality. Here is what it has suggested.
   
   If this is not enough, you can ask ChatGPT to generate you further tests. Once generated test cases cover all possible scenarios, the system could start to suggest additional requirements to increase security of authentication data.As already stated, it is important that you check generated results, because they may contain errors. In the following example output, ChatGPT decided that the username contains less than 4 symbols, although this is not the case.
   
2. Automatic unit test generation in a chosen programming language
   ChatGPT can write simple code in different programming languages including Python, javascript and others.  This may support the testing team in unit test automation and save enormous time for implementation of code tests. Below we will provide two examples and check their correctness.First, following the previous example we may not only ask ChatGPT to generate use cases but also write automated unit tests in a chosen language e.g. Python.

   import unittest
   from authentication_form_generator import generate_authentication_form
   
   class TestAuthenticationFormGenerator(unittest.TestCase):
   
       def test_valid_input(self):
           """Test the generator with valid input."""
           form = generate_authentication_form('user123', 'password123')
           self.assertIn('Username: <input type="text" name="username" minlength="4" maxlength="15" value="user123">', form)
           self.assertIn('Password: <input type="password" name="password" minlength="8" required>', form)
   
       def test_username_too_short(self):
           """Test the generator with a username that is too short."""
           with self.assertRaises(ValueError):
               generate_authentication_form('abc', 'password123')
   
       def test_username_invalid_characters(self):
           """Test the generator with a username that contains invalid characters."""
           with self.assertRaises(ValueError):
               generate_authentication_form('user*123', 'password123')
   
   if __name__ == '__main__':
       unittest.main()

   ChatGPT does not only produce the code, but also provides explanations about the solution and how you could use it.
   
   In the second example we ask ChatGPT to write another simple test in Python to check if API GET-request to qodo (formerly Codium) is successful. The system produces again the correct result:

   import requests
   
   def test_api_status_code():
       url = "https://qodo.ai"
       response = requests.get(url)
       assert response.status_code == 200

   You can also provide ChatGPT with the code of the program or function you need to test and ask it to write some automated tests to check its correctness.

3. Write documentation
   Although ChatGPT is capable of writing code, its biggest strength is generating texts and explanations. This can be successfully utilized in writing testing documentation. For example, if you need to write team instructions about unit testing you can ask ChatGPT to do this for you. Ideally you should provide the tool with a list of topics which should be covered in your documentation.For our unit test example we asked ChatGPT to write what it is, how it can be performed and what are the results. Let’s check what it suggests:
   
4. Help to make a decision based on best practices
   ChatGPT may help you answer a wide range of questions around test automation, e.g. which tests should be automated first or other questions on organization of the testing process. Here is its answer about automated tests in a new project:

In conclusion, ChatGPT works well for standard daily tasks which do not require much understanding of underlying processes or deep dive into software being developed.Still there is a big potential in using it for speeding up and improving the testing life cycle in conjunction with other automated testing tools to ensure the quality and accuracy of the testing process. With AI developing, learning, and becoming more and more sophisticated, reliability of ChatGPT results will also increase.
And here is what ChatGPT itself says about its future in software testing.
“While AI technology is advancing rapidly and some tasks can be automated, testing is a complex activity that requires a human perspective, creativity, and critical thinking.

Therefore, I see myself as a tool to assist testers and other professionals in their work, rather than a replacement for them.”