The Blog From Hell

Hi again and welcome to another chapter of the Foss Archeologist. My apologies for the delay in posting this but I was away for a death in the family and took some time off from blogging afterward. Now it's time to get back into it and finish this series.

Midnight Commander (or 'mc' as the program launcher was called) was one of the first truly modern file-managers in the free software world – all the more spectacular because it was a console app, not a graphical one. The two-pane interface with hotkey control was copied from the old Dos program Norton Commander (hence the name) but midnight commander would come to be far more powerful than it's ancestor in many ways.

MC had it's own built in shell which was compatible with bash though it didn't support tab-completion (because tab was the hotkey for switching between panes), it could mass-select, copy, paste, rename and delete – all the standard functions you want in a file manager – but it went further. It implemented the first filemanager virtual filesystem on a Unix-like platform allowing you to work directly with the contents of tarballs or browse and manage files on samba and ftp servers. In fact when working on the console MC provided one of the best ways to transfer files to and from network servers, including automatic support for resumes, full and proper handling of Unix permissions and more.

The brains behind MC was Miguel De Icaza – nowadays better known as the head of the Gnome project – this was before he became "the great pragmatist" and was one of the most hardcore believers in the ethics of free-software. De Icaza of course was not alone, and many other programmers worked on, and expanded, MC over the years – it was after all, a tool they loved as much as their sys-admin brethren.

One of those programmers is worthy of a special mention. Paul Sheer developed midnight commander's builtin text-editor mcedit. It could also run standalone and was in many was the most userfriendly text editor available on the Linux console – and still is. Unlike the other programmer's editors it eschewed dual-mode editing and power-features for a simple arrow-controlled editing environment reminiscent of the editors popular in the dos world at the time.

It is quite a surprise to me that mcedit is not the editor by default suggested in Ubuntu documentation for system text editing, preferring nano instead as nano is a much harder editor to use with much more difficult key-stroke controls and an interface that is, unintuitive to say the least. I can only surmise that mcedit fell out of favor because it depends on mc itself which is a bit of a bulky program and these days with our graphical file managers and graphical editors no longer generally included in a distribution's default packages (though it's almost always available in the repositories since so many older users still love it).

Part of why I wanted to mention mcedit is because Paul Sheer is a South African, one of the founding members of Obsidian systems (and he was the first trainer in their then newly-opened training department) and mcedit counts as one of the very first (if not the first) major contributions to a (then) incredibly important free software project to come out of Africa at all. 

Midnight Commander's look and feel has been replicated by numerous graphical file managers but for the most part none of them are very popular and non have ever become a user default. The two-pane view which is so incredibly useful on the console is cumbersome to work with in a world controlled by a mouse. Nevertheless it's legacy lives on, the gtk-view of mc (which was never it's default) was the first official file manager of the gnome project and some of it's code directly inspired early code in nautilus.

When you browse an sftp server, or deep into a tarball in nautilus today (or for that matter in a KDE file manager) this is done using virtual-fileystem support, a concept that was largely pioneered by mc and which lives on as a lasting legacy on the free software world even when we don't use it anymore. 

Having said that, unlike most of the other projects in this series mc is very much alive and well and I highly recommend every Linux user installs it. Even if you never work on the console, the one time you have to – it could make your life a million times simpler and (in a major break with tradition) it could do so while increasing rather than decreasing your productivity.

E-Mail wasn't always something that was done on the internet. Most networks since the late 70s had e-mail services and early ISP's often had mail services that only fed their own subscribers. These e-mail systems which allowed you to send mail only to people on your own company network were themselves predated by systems which only worked within the same machine.

That wasn't so useless as it sounds – since back then computers were generally large backroom mainframe servers and various users were on terminals all over the building/campus or whatever – so it had all the advantage of modern corporate mail services between colleagues (if not with outside customers and such).

Modern e-mail systems and their protocols can be directly traced back to those early versions – and they were mostly built on Unix – with a set of systems that Linux inherited and later greatly expanded. The early GNU/Linux killer app was apache and it made Linux the operating system of choice for ISPs – which made it the OS on which internet e-mail was created. 

At the heart of those early in-machine mail systems was sendmail.  Sendmail basically was a command to send mail from one user to another, mail was sent to qeues in /var, to be picked up by the user's mail client. Later it became network aware and could send to from itself to the sendmail on another machine using SMTP – the other machine sendmail would again deliver it into the user's qeue. This process actually still gets used by almost all ISP mail systems – though sendmail is hardly ever the server anymore.

As internet mail became popular, it became needed for people to have a way to download one user's mailqeue only to another machine – for ISP mail to function. POP was developed as a protocol to do so – and fetchpop was written as a tool to collect your mail from another machine and deliver it to your own qeue on your own machine.

Fetchpop stagnated in development by the mid-90's however and Eric Raymond took it over, further developing it. In doing so he added support for newer protocols like IMAP and renamed the program to fetchmail to reflect this change. 

That was internet mail by the mid-90's sendmail to get mail out, using the network when it moves between machines and fetchmail to collect it to individual machines in this post-mainframe age. Fetchmail could collect entire qeues so some companies even had a single qeue at an ISP which would be collected regularly and delivered to their local server from which individual users would then collect. 

Such setups sound cumbersome but to a skilled linux admin at the time they were easy and commonplace setups, many even used more obscure protocols like UUCP to transfer those mailqeues. 

These days ISP pop mail probably still use a qeue system exactly like that, but fetchmail has fallen into disuse as most machines are now genuinely single-user so people don't want their pop/imap mail going through a local multi-user-capable qeue, they just download it directly into a mail-client like evolution or kmail. 

On the sending side, sendmail got hit in the late 80's with several massive security exploits. The famous sendmail worm of 1989 actually led to the infant internet of the time being shut down for 48 hours (it was small enough to still do so) to get the worm to choke itself to death without connections. And that worm was an accident – the author had never intended for it to be released into the wild, he was doing research on computer viruses.

It got worse, sendmail was unlike almost all later mail systems in how it was configured. Not using the typical unix text-config files as such, it basically used a configuration file written in the M4 macro language which had to be compiled before it could be used. Massively powerful as this was, it made the configuration very complex and hard to get right-  and very easy to make very insecure.

Sendmail insecurities flourished and it became the plague of the internet. By the late 90's however other mail clients like postfix were starting to appear and getting better. They had simpler but equally powerful configurations and were built with designs that considered the security issues which had plagued sendmail.

Sendmail's security issues shouldn't be held against it too much – it was written in an age of small networks and the problems were a matter of scale as the internet exploded, things happened which could not have been reasonably foreseen when it was designed so it wasn't that the designers didn't care about security – they just didn't understand it because there was no reference for the future to come.

Ultimately today most Linux users will never run sendmail or fetchmail, even most ISP servers no longer depend on them – but once they were a core part of how even home Linux users worked with e-mail. It was just the way it worked – and even such standard incompliant absolute trash as exchange are built on the foundation that was laid by sendmail and fetchmail – the e-mail programs that once drove the internet.

Both programs are still actively developed though the userbase has shrunk, and can be found in the repositories of the vast majority of GNU/Linux distributions – especially those that have a server focus – but these days other programs have become the default tools of choice.