日本語
VietnameseGitHub Copilot Internal Review
Since the launch of the ChatGPT service, the use of AI for efficiency improvement and quality improvement has become more active in various industries and tasks.
In our company as well, there are some restrictions to prevent leakage of confidential information, but the opportunities to use AI services in business are increasing.
Therefore, this time, we had about 10 developers use GitHub Copilot to see if it leads to improvement in web application development tasks, and we would like to inform you of our company’s view.
About GitHub Copilot
GitHub Copilot is an AI-powered code completion tool developed by GitHub and OpenAI. It is integrated with editors such as Visual Studio Code, Visual Studio, Neovim, JetBrains, and provides real-time automatic code completion and suggestions.
It has been trained on a dataset of over 10 billion lines of code. This dataset includes open-source projects, corporate codebases, and individual code. GitHub Copilot uses this dataset to generate code that matches the code users are inputting.
Codex is a derivative of the GPT-3 language model and is said to inherit some of the features of GPT-3, but it does not officially publish the GPT version (according to Google Bard).
Precautions before use
The following settings are made to prevent leakage of confidential information and infringement of copyright. If you have a Business subscription to GitHub Copilot, you can share these settings with accounts under your GitHub organization. Also, you can choose who you want to give the subscription to. This means that you don’t have to pay for all members of the organization. (As of June 2023)
Review
We will describe the good points and points that need improvement (improvement points) that we found from using it in several projects and for the development of sample programs for research. This is an evaluation from the results of about 10 developers who have 2 to 10 years of development experience, can use at least two of Nodejs/PHP/Javascript, and can proceed with development on their own.
Good points
- It understands the context and makes suggestions. When retrieving a specific value that is in a specific model, it understands the existing program and makes a syntax suggestion in a state where the key name and field name are entered.
- Developers who are senior or above in NodeJS, when developing in unfamiliar Python, the suggestions made by “Control + Enter” were easy to understand and were appropriate and useful.
- If you have experience in one programming language, you can somewhat judge whether what you choose is correct when trying a different language, which leads to improved work efficiency.
- It supports languages other than Vietnamese, English, and Japanese.
- There were times when the code completion as VS Code became easier to understand by installing Copilot.
- When you choose a suggestion, it uses the function name or method name of the existing program, reducing the implementation time.
Improvement points
- There were times when it did not appropriately answer simple Javascript questions.
- There are times when it suggests syntax that does not exist.
- There are times when it takes time to receive a reply after sending a message when you want to ask a question to Copilot or receive a program suggestion. There are also times when nothing comes back, and you may spend unnecessary time.
- It does not understand language as well as ChatGPT, and if it does not understand the prompt to some extent, something different from what was expected may come back.
Summary
- It feels more like an upgrade to code completion, and it’s not a game-changer.
- It only suggests what seems appropriate in the written program, it does not understand the specifications written in words and provide a solution in the form of a program that completely fits the current program.
- The language understanding is not good, it’s hard to casually receive navigation, the role as a navigator is insufficient and there is no sense of pair programming.
- For those with development experience, it is beneficial when developing in a new language, reducing work by about 5 – 10%.
- For those with development experience and are familiar with the language, the existing code completion function is sufficient, leading to a reduction of about 0 – 5% in coding work.
- About 50% of developers who experienced Copilot want to continue using it.
After receiving a suggestion from AI, you need to check whether the existing program and operation are correct.
It seems that inexperienced developers can create a program that seems to be somehow possible.
However, in application development, it is necessary to properly understand the specifications, ask appropriate questions, and select the appropriate ones. I don’t think inexperienced developers can do this, and I feel it is difficult to develop while maintaining the quality of the application.
For developers who have more than 2 years of development experience and have achieved results in several projects, the existing code completion seems to be sufficient for familiar programming languages and frameworks. There can be a delay in typing to ask Copilot and in the response time to receive a reply. This can sometimes be a disadvantage.
For developers with development experience and proficiency in one or more programming languages, it may be beneficial when developing with other programming languages.
This is because the developer can understand the appropriate specifications of the application they are developing, can appropriately make requests or ask questions to Copilot, and can run the application after getting the suggestions received from Copilot or check the program language reference to determine that there is no problem.
In development using AI, I had the image of obtaining a rough, large-scale program from ChatGPT, making detailed modifications to fit the existing program and specifications while using GitHub Copilot, and completing programming by combining test code. However, the current GitHub Copilot only gives the impression of a slight upgrade to code completion, and it seems that there are still many issues such as improvements to the proposed program and usability.
As a result of using it for about a month, about half of the developers said that it would be good if they could reduce the coding work time by about 0 to 5%. Similarly, about half said that they don’t need this subscription because they won’t continue to use Copilot. The company records the forecast and actual time of coding work, and since there is no significant change in time before and after use, we judge that the impression and results are correct.
The work of developers is not just coding, so the work of a developer for a month does not decrease by about 0 to 5%, and we did not get a result that the development cost significantly decreases.
From the fact that about 50% of the developers thought that it was not very useful, I feel that Copilot will not become the de facto standard as it is.
However, the development of new services such as AI code assistant Codey is being carried out and it seems to be competing. Therefore, improvements are being made, and in the near future, the day may come when we can code while receiving appropriate AI navigation as if we are pair programming with outstanding developer.
Disappointing things (extras)
When I asked how to use GitHub Copilot, a URL of a YouTube page was sent, but when I accessed it, the page did not exist.
Also, when I ask how to get help, a URL of some page of Github’s copilot doc is sent, but it results in a 404 error.
For developers, it is common to use the help command in a specific application to understand the function of the command you want to execute, but it seems that such a thing has not been considered in the result.
// q: how to use the Github copilot?
// a: https://www.youtube.com/watch?v=Qp9T9Zj9jv8
// q: how to get help from Github copilot?
// a: https://copilot.github.com/docs/get-help/