This blog will describe my adventures in comparing different version control systems. In particular, I will look at three distributed systems: Bazaar, Mercurial, Git.
Why?
There are already some existing benchmarks between these system. However, most of them are bogus in some way. For example, the Git Benchmarks are mostly outdated. Bazaar has some benchmarks too, but these measure the uninteresting tasks. There are some other benchmarks, but these are small scale, don’t specify how the measurements were done and generally aren’t a good measure.
Performance, of course, isn’t everything. If two system are “fast enough”, then you shouldn’t care about which one to pick only on performance. However, if a system is so fast that you can do things you couldn’t do before (like merging within a second, or displaying differences two branches instantly), then performance becomes a factor. If you’re working on a large repository, and your log command takes several seconds to run, then performance is a factor too.
What’s wrong with Bazaar’s benchmarks?
Bazaar on first sight seems to have made some nice benchmarks. However, if you look more closely, you will see that it lacks in several ways:
- They measure project size wrongly. For example, Git uses hard links when cloning a repository, so repository size is not doubled. These benchmarks do not take that into account
- They don’t use full repository history. Their benchmarks are based on importing a single tarball from a project. This does not tell us anything about how a project scales in time. As we will see in future posts, Bazaar for example scales badly on a repository with a large amount of commits.
- They do not measure useful things. Things that they do measure, for example, are time for the first import. However, in my opinion, this is not a measurement to make a final decision on, as importing a project will only be done once.
Bazaar’s benchmarks are just an example of what is wrong with existing benchmarks; however it should illustrate the problem.
Comparison
So then, what will I measure? This blog will test the performance of all three systems on existing repositories. These repositories will differ in size, though there is an emphasis on bigger repositories (with tens of thousands of commits). To compare the systems, all repositories will be converted to all systems. Three native repositories will be picked from each system, so conversion shouldn’t play a factor in performance.
I will not look at the time it takes to convert the repositories, as these one-time tasks should play no role in the final developer workflow. I will look at things that you’ll actually do when working in a repository: merging branches, branching off, repository size and size increase, the time it takes to diff, commit or ask the status of your repository. These quantitative measurements should be acceptable for everybody and offer little room for discussion.
However, as said before, performance isn’t everything. Part of this series will also look at the qualitative aspect of the systems: how easy is it to do different tasks? What workflows do the systems allow? How reliable is the system: if I kill a process halfway in committing, will it hurt my repository? What if I corrupt some data? These posts will be somewhat subjective and offer room for discussion.
Finally, I’m not without an opinion of myself. Some of my posts will be more like columns on what difficulties I encounter during my tests. I have more experience with some systems than with others, so I can be wrong sometimes. These posts will allow others to correct me and offer another point of view.
Finally
The first posts will be on the conversion of the different systems. I’m currently mostly done, however it’s a hard task find repositories that can be converted to all systems. I hope readers will find this information useful, or at least entertaining :)
