The concept of diversity, a follow up to Jonathan Carter’s reply.

Jonathan Carter posted a blogged reply to my post Why we need more, not less GNU/Linux distributions. FIrst off, just a note Jonathan, I didn’t actually disable comments on the post, but to prevent spam the blog automatically disables them after ten days.

Now on to Jonathan’s argument, which is the major argument I have heard against my post: more distributions mean duplication of effort. He then goes on to claim my diversity argument is invalid, though he doesn’t back that up with any indication as to why it is invalid. Since I gave real arguments in favor of it, even finding a very valid analogue with a system that has been successful for billions of years and produced hundreds of millions of lifeforms, all of whom are perfectly adapted to their environment and have a perfect balance with the rest of the organisms around them – I would have at least expected a solid piece of evidence that my analogy is flawed.

Since Jonathan didn’t actually give me any reason to believe that diversity doesn’t improve overall quality and allow for the maximum rate of innovation, I will simply ask a few questions: by that logic – why do we have python and PHP ? Neither does anything that Perl didn’t already do – and Perl itself doesn’t do much you couldn’t do in the C-shell if you were sufficiently masochistic. But Python is simply a much better application language than Perl ever was, PHP is a much better web development language, and RoR is even better for some uses. So why develop Ruby-on-Rails in the first place ? For a lot of web-app work it’s far less suitable than PHP, for a lot of others it is a much better language… it’s good that we have both – it means for any given project we can choose the one that is most suitable.

Now let’s look at a brief history of some of the major distributions today. Suse was based on slackware in the beginning, it has very little resemblence to it’s parent today but if slackware hadn’t split off and basically redone Yggdrasil from scratch, Suse would never have existed. RedHat basically started from scratch, so did Debian, and both spawned hundreds of other distributions.

Apparently nobody sees a contradiction in their own words though. It was apparently good for Ubuntu to fork from debian – but it’s not good for anybody to fork from anywhere else in the tree ? If you base your distro on Slackware (like I did) – you cannot end up with something as good as OpenSuse is today (as good at something else I mean)… so how on earth did OpenSuse do it then ?

If you start from RedHat – you cannot get through Mandriva to PCLinuxOS ? Oh, and for the life of me I cannot figure out why LFS and Gentoo even exist ! I mean why would anybody want to build his own GNU/Linux system from scratch ? Or build packages from source ? But it turns out that for a lot of people that is exactly what they want. These are not the same people who use Ubuntu – and no Ubuntu derivative will ever please the kind of user who prefers to build from source. So what if they are not the majority ? The point about GNU/Linux is that you don’t need to be the majority to get what you want !

So where does Kongoni fit in ? Well Kongoni is being built to be a very close cousin of the BSD systems, but on a GNU/Linux base. Hence it made sense to use slackware as a parent, since slackware is the most BSD-Linux distro out there (erm… when did people get the idea I’m building from nothing… I never suggested that, ever distro is ultimately a compilcation of other work, it’s how you compile it that makes it unique… like mix-tapes I guess). But I wanted to add the single nicest thing about the BSD systems that isn’t in slackware – ports.

That there are people who like ports is clear from the massive userbase of gentoo/funtoo. That there are people who like the BSD systems, and think they can make decent desktops is clear from the recent proliferation of BSD-based desktop distributions (PC-BSD and it’s ilk).
Everyone of these systems are trying to marry the best of one tree of thinking, with the best of an entirely different one – to produce something unique, that will appeal to a certain subset of the users from both.

I completely disagree that there are very few use-cases which cannot be met by an Ubuntu/Debian derivative – I say that it is virtually impossible for such a system to even reach the kind of use-cases I am talking about, and like it or not it is from those systems with those users that the great innovations in the GNU/Linux world has consistently come. We have Udev today because Richard Gooch created devfs (why reinvent the /dev filesystem.. why reinvent it yet again a year later ?) – and Richard uses an LFS based system – to name just one example.

This is not some idle statement, it’s a basic logical consequence: these systems attract the most competent programmers as users, people who like to tweak and modify their systems down to the very lowest levels, and if you aren’t a person like that, you are never going to innovate anything at that level.

Sorry, Mac-killer desktops isn’t enough – it’s good, and those desktops are likely to come from places like OpenSuse, PCLOS and Ubuntu because they are what is needed by their userbase, but they would be useless without equally impressive innovation happening at every level of the system form the kernel upwards.

If that means a bit of duplicated effort, so be it. That’s the price you pay for allowing every niche to to be filled, ever need to be met, ever idea to be explored – and I think it’s a far smaller price than people like Jonathan think. Now don’t get me wrong, I fully understand his sentiment, as a programmer myself I feel horrified whenever I think of the idea of anything inefficient. I get annoyed by badly timed traffic lights, and I get thoroughly pissed at bureaucracy. I started out feeling much like Jonathan does now, but over many years of observation I came to the conclusion that this is not a matter of inefficiency at all, it’s a matter of diversification and specialization.

When you look at what has been achieved through this process in it’s roughly 25 years, well it’s absolutely amazing isn’t it ? That’s not inefficiency, at all – that’s the worlds smartest ever allocation of resources, it looks inefficient over the short term – but over the medium to long term it achieves what would otherwise be impossible.

This is like suggesting that if there was some specialist gearbox company that supplied all car-makers we’d have one perfect gearbox, and how much better would it be if there was just one car-maker with various models right ? After all, Audi has nothing that makes it more suitable for certain people than Mercedes has (for the same market segment) does it… so why are both companies rich then ? Do you really think a world like that would have had tiptronic gearboxes and permanent-four-wheel drive ? Sure most of us don’t need it, but it’s something we all would love to have, and because of diversity it will ultimately be in every car. Car companies of course are not FOSS, and ideas cannot spread until their patents expire, so the process is slow – FOSS takes it to a level unmatched by any industry because we can take our tiptronic gearboxes and put them everywhere they make sense the day after they are invented.

Oddly, us GNU/Linux people are strange sometimes – on one side we promote our systems for giving user’s choice, and letting the market expand beyond single providers, with the other hand we complain about those other providers stealing some of the thunder from our favorites. We tell people choice is good, then we go and tell one another that people will never use our stuff if we don’t take away the choices ! Apparently Jonathan (who loves to catch me saying apparently contradictory things by taking single sentences out of context) doesn’t realize that he himself as a proponent of software choice, is contradicting himself when he tries to stifle projects that want to expand the choices.

I want to end off by blowing the favorite quote of those who disagree with us more-choice proponents out of the water. They always tell us “do not reinvent the wheel”… I’m sorry, would you like your car to run on two sawn-down trees ? Because that was the “wheel” – somebody reinvented it by adding spokes, somebody reinvented it by adding tires and air, since then it’s be reinvented a thousand times over to produce wheels designed for snow, or sand, or roads, wheels for formula1 tracks and wheels for bycicles… what duplication of effort that they aren’t all the same wheel, and don’t get me started on tanks… now they really reinvented the entire idea of the wheel. Who would ever want caterpillar tracks on a car ? Oh but hold on, car’s cannot go where tanks and cranes go… that reinvented wheel is useful for a small minority, but thank goodness we live in a world where that doesn’t mean they don’t get made.

These things don’t look to us like reinventions since we see it all in the right-now (not to mention the phrase was coined when very few of these reinventions existed) – to us they are all just “the wheel” – a round thing to make moving heavy objects easier. Well new GNU/Linux distribution don’t (often) replace GNU or Linux… they just put them together a little differently, they don’t (often) “reinvent the wheel” – they customize it for different roads. Ubuntu is a formula-one tire, a very good one (perhaps the best one) but if you are trying to build a tractor-tyre, a formula-one tyre would be a very stupid starting point – so no, the only use-cases for which Ubuntu is a good starting point is specialized subsets of it’s own market. Other non-technical desktop users – it will never be a good starting point for people who like BSD features (which are really good – at other things).

And just occasionally, we have to make a wheel that isn’t even a wheel anymore, like the caterpillar tracks, those Linux systems on cellphones and routers that don’t even contain GNU but use only custom-built versions of busybox ? They are somewhat like this, they really aren’t suitable for well almost anything – but where they do work, nothing else could have done the job. Heck busybox itself is about as big a duplication of effort as you can get, it’s a less-powerful single-app-with-symlinks replacement for most of the low-level GNU-toolset… but for tiny systems where space is crucial – it’s actually a godsend.

We need to realize that the single best thing GNU/Linux can offer on the desktop is a system as diverse as the personalities and needs of it’s users. There isn’t, and never can-be, a one-size-fits-all operating system, Windows have tried and failed to do one for years, GNU/Linux however could change this – by creating a custom-tailoring business around operating systems. It’s an adjustment for users, it may slow down adoption a little – but over the medium to long term, it will be the best thing that has ever happened to the computing industry, a revolution that will make the Internet pale by comparison.

I think that trying to stop this process because we are afraid of a bit of extra work, because we fear it will be duplication of effort would be incredibly stupid. I hear the arguments people raise, and the answer to them, the compromise if you will, is to keep channels of communication between developers open – and to improve them. So that within the various structures, we don’t recreate things that already exist (I mean at the small scale) – we just put them together in a billion different ways. It would be duplication of effort if we each used our own building blocks – but that’s not what we’re seeing, what we are seeing is lego-pieces being used to build everything from the Eiffel Tower to the planet Mars…