Jetbrains (Jetbrains AI Assistant) and Microsoft (GitHub Copilot, GitHub Copilot Chat) recently announced new AI tools that work directly in your IDE. I decided to see how well they worked when doing XP development. Note that to use the AI tools, you will need a subscription.
In this article, let’s see how the tools do at generating commit messages in Intellij IDEA and Visual Studio. You can also read about how the tools do at generating unit tests and refactoring code.
To create a Git change set, I started with the GildedRose kata and simply made the initial failing test pass. Then I asked the AI tools to generate the commit message.
Let’s start by generating a commit message in Intellij IDEA using the Jetbrains AI Assistant.
Jetbrains AI Assistant in Intellij IDEA
To generate a commit message in Intellij IDEA with the Jetbrains AI Assistant plugin, you must open the commit window. Then you select the AI Assistant icon to the right of the Amend commit option, right above where you enter the commit message. If you hover over the icon you will see a message “Generate Commit Message with AI Assistant.”
The resulting commit message is shown below. It summarizes the change on one line and then adds a more detailed summary after that. I would use that commit message.
Here is how that commit message looks in the git log. I wish the tool would limit the line length to 72 characters, but otherwise it looks pretty good. I can just read the summary and if I need more information, I can read the detailed explanation as well.
Now let’s try generating the commit message in Intellij IDEA using the GitHub Copilot Chat tool.
GitHub Copilot Chat in Intellij IDEA
To generate a commit message in Intellij IDEA using the GitHub Copilot Chat plugin, you need to open the chat window and ask the tool to do so.
As you can see, the result was a bit underwhelming. I asked the tool to “generate a commit message for the changes” and it gave me back “Added a new test case in GildedRoseTest.”
That’s not very good. At a minimum, I thought it would explain what scenario I was trying to test. I wouldn’t use this commit message.
Now let’s give the same tools a try in Visual Studio with the same code kata and see how the tools do.
Jetbrains AI Assistant in Visual Studio
To generate the commit message in Visual Studio using the Jetbrains AI Assistant, I entered a prompt of “generate a commit message.”
The result was not what I expected. The Jetbrains AI Assistant basically asked me to give it “…a brief description of the changes…” which is what I asked it to do. That is not helpful at all. I would just enter the commit message myself in this case.
Let’s try again, with a better prompt. This time I used “generate a commit message for the code changes.”
This time I got a commit message. However, it’s still not very good. At a minimum, I thought it would explain what scenario I was trying to test. I wouldn’t use this commit message.
Finally, let’s try generating the commit message in Visual Studio using the GitHub Copilot Chat tool.
GitHub Copilot Chat in Visual Studio
To generate a commit message in Visual Studio using the GitHub Copilot Chat extension, you must open the git changes window. Then you select the sparkle pen icon to the right of the Amend commit option, right below where you enter the commit message. If you hover over the icon you will see a message “Add AI generated commit message.”
The resulting commit message is shown below. Not great, but not horrible either. The summary is very long but the detailed description is a little more helpful.
Here is how that commit message looks in the git log. The message is really long. It would be better if there was a short summary at the top. Also, I wish the tool would limit the line length to 72 characters.
Ok, that’s it. Now let’s review the results of these experiments.
Conclusion
Interestingly, the Jetbrains AI Assistant generated a much better commit message in Intellij IDEA and the GitHub Copilot Chat tool generated a much better commit message in Visual Studio.
That said, I think in this case I would still just write my own commit message for this change set. Something like “fix: test was using wrong value for assertion.”
The reason is that I want my commit message to explain why I made the change, not what the change was. Anyone can see what the change was by looking at the git diff.
How does your experience compare to mine? Which commit message do you like better?
