Plugins in Version Control

What is up with these version control systems that use plugins to do everything? I understand that having an extensible system is very useful if users want to create something custom. Mercurial and Bazaar both support plugins written in Python, while Git has an extensive set of commands that allow you to easily write custom scripts in any language.

However, being able to extend a program is not an excuse to ship a product missing features. One of the things that makes Git powerful is that it ships with a lot of capabilities built in.

Take Bazaar for example. If you want to commit only part of the changes in a file, you need the BzrTools extension. To rebase revisions you need the Rebase extension. You need extensions to show diffs in colour, to push all branches at once, and even to remove untracked files in your working tree! Why aren’t these included by default? Are they afraid a user might be afraid to see a coloured diff? Or is it too much work to support these tools directly?

This exactly the problem with plugins: it separates the functionality from the main program. Who is going to install some weird plugin on some website to gain functionality? Who created the plugin anyway, can it be trusted as much as the program itself? How is it updated? What if there’s an incompatibility between the plugin version and the program version? What if I’m using some feature, but others that I’m working with can’t do the same, because they don’t have the plugin installed yet?

Having an extensive set of tools shipped by the program isn’t a bad thing, it’s good. It allows you to use the same workflow and suggest the same solutions to everybody else, without requiring a certain plugin. It also makes features easier to use, as you don’t have to manually install something first.

Git is easily capable of supporting its full feature set release after release, and keeps adding useful features. When upgrading, you can be sure that all your previous commands are still usable. As they are integrated into the system, you can be sure that they get the same amount of attention as anything else.

This last point is more important than you might think. After all, you’re talking about your source code here. You want the program that you use to be reliable, to work consistently across releases. You don’t want to depend on something that you don’t fully trust.

Relying on plugins is just a way for lazy programmers to maintain less code. I really prefer Git’s builtin tools more than some lousy plugin that doesn’t even have it’s own website. If you stick to only core functionality, at least be the fastest ;).

Git, Mercurial, Bazaar Repository Size Benchmark

I have finished all conversions of the repositories that I’m going to test. The first interesting metric is of course the repository size. This is interesting because Bazaar claims on its homepage that “Bazaar’s default storage format is highly efficient, better than its main competitors in most cases”. They include benchmarks which should show that they are more efficient than specifically Git and Mercurial.

As I mentioned before, this benchmark is bogus because it does not include project history. While Bazaar acknowledges this, they still mention space efficiency as one of their prime benefits. They also mention that their benchmarks are done on real use cases, not arbitrary processes.

At the same time, Mercurial states it has an “Extremely high-performance delta-compressed storage scheme”. Git mentions “It also uses an extremely efficient packed format for long-term revision storage that currently tops any other open source version control system.”

Test method

For all repositories, only a single branch was converted. For all repositories except Samba, this meant the development branch. As Samba has multiple development branches, I chose the v3-3-test branch.

I made use of the fast-export/import interface where possible. This means that all conversion were done using fast-export, except Git to Mercurial, for which I used “hg convert”, and Bazaar to Mercurial. This last one was a bit tricky, as Mercurial has no Bazaar importer. I therefore converted the Git repository of the same repo to Mercurial.

After conversion, I ran a pack command for the repositories that support this. For Git, this meant a “git repack -adf —window=250”, for Bazaar it meant a “bzr pack”, and removing the obsolete packs.

Tests were done using Git v1.5.5.3, Bzr v1.5, Hg v1.0

Results

So then, using actual, real world data, which system has the best storage efficiency? Below is the table of all projects.

Repository	Git	Mercurial	Bazaar
Git
Cairo	15MB	24MB	30MB
Coreutils	29MB	44MB	76MB
Samba	82MB	146MB	310MB
Mercurial
Octave	22MB	49MB	57MB
Mozilla	78MB	205MB	255MB
Dovecot	9MB	14MB	23MB
Bazaar
Emacs	120MB	163MB	300MB
Mailman	42MB	75MB	73MB
Pkgconfig	1.1MB	1.3MB	1.8MB
Total	398MB	721MB	1125MB
Relative	1	1.8	2.8

Finally

As can be seen from the table, Git really is the most efficient in storing the data. Next up is Mercurial, which also does a nice job. Bazaar is the least efficient by far, taking on average 2.8 times the space of an equivalent Git repository.

Bazaar, Git, Mercurial comparison: Introduction

This blog will describe my adventures in comparing different version control systems. In particular, I will look at three distributed systems: Bazaar, Mercurial, Git.

Why?

There are already some existing benchmarks between these system. However, most of them are bogus in some way. For example, the Git Benchmarks are mostly outdated. Bazaar has some benchmarks too, but these measure the uninteresting tasks. There are some other benchmarks, but these are small scale, don’t specify how the measurements were done and generally aren’t a good measure.

Performance, of course, isn’t everything. If two system are “fast enough”, then you shouldn’t care about which one to pick only on performance. However, if a system is so fast that you can do things you couldn’t do before (like merging within a second, or displaying differences two branches instantly), then performance becomes a factor. If you’re working on a large repository, and your log command takes several seconds to run, then performance is a factor too.

What’s wrong with Bazaar’s benchmarks?

Bazaar on first sight seems to have made some nice benchmarks. However, if you look more closely, you will see that it lacks in several ways:

• They measure project size wrongly. For example, Git uses hard links when cloning a repository, so repository size is not doubled. These benchmarks do not take that into account
• They don’t use full repository history. Their benchmarks are based on importing a single tarball from a project. This does not tell us anything about how a project scales in time. As we will see in future posts, Bazaar for example scales badly on a repository with a large amount of commits.
• They do not measure useful things. Things that they do measure, for example, are time for the first import. However, in my opinion, this is not a measurement to make a final decision on, as importing a project will only be done once.

Bazaar’s benchmarks are just an example of what is wrong with existing benchmarks; however it should illustrate the problem.

Comparison

So then, what will I measure? This blog will test the performance of all three systems on existing repositories. These repositories will differ in size, though there is an emphasis on bigger repositories (with tens of thousands of commits). To compare the systems, all repositories will be converted to all systems. Three native repositories will be picked from each system, so conversion shouldn’t play a factor in performance.

I will not look at the time it takes to convert the repositories, as these one-time tasks should play no role in the final developer workflow. I will look at things that you’ll actually do when working in a repository: merging branches, branching off, repository size and size increase, the time it takes to diff, commit or ask the status of your repository. These quantitative measurements should be acceptable for everybody and offer little room for discussion.

However, as said before, performance isn’t everything. Part of this series will also look at the qualitative aspect of the systems: how easy is it to do different tasks? What workflows do the systems allow? How reliable is the system: if I kill a process halfway in committing, will it hurt my repository? What if I corrupt some data? These posts will be somewhat subjective and offer room for discussion.

Finally, I’m not without an opinion of myself. Some of my posts will be more like columns on what difficulties I encounter during my tests. I have more experience with some systems than with others, so I can be wrong sometimes. These posts will allow others to correct me and offer another point of view.

Finally

The first posts will be on the conversion of the different systems. I’m currently mostly done, however it’s a hard task find repositories that can be converted to all systems. I hope readers will find this information useful, or at least entertaining :)