Swedish ISP Loopia offers free email forwarding for any domain for which they handle DNS.  Unfortunately, this includes a spam filter that the customer cannot turn off, unless they pay, which means it’s not free anymore.

So what’s wrong with some free spam filtering?  Well, this is what happened:

1. A prospective client sent an email with a proposition.
2. Loopia’s spam filter found the proposition suspect and bounced it.
3. The client saw a bounce and thought that they had the wrong email address and gave up.

This probably lead to lost business.  I can’t blame Loopia, but I can stop using them.

All hackers want to live on the edge. What we rarely reflect on is the fact that most things have more than one edge. Adaptation curves have two: one where the early adopters build up their reputation for cool and one where I found myself this afternoon.

Apparently I should have let go of Fedora 6 some time ago. I installed this particular server in December 2006 when FC6 was the cool new girl on the block, and it has been running smoothly since then. But today when I tried to (yum) install a package, there were no servers to be found:

# yum search libxslt-devel
Loading "installonlyn" plugin
Setting up repositories

http://archive.kernel.org/fedora-archive/fedora/linux/core/6/i386/os/repodata/repomd.xml:

  [Errno 14] HTTP Error 404: Server: nginx/0.5.0
...
Trying other mirror.
core                      100% |=========================| 1.1 kB    00:00
Error: Cannot find a valid baseurl for repo: extras

Oops. I looked in /etc/yum.repos.d/fedora-extras.repo which is provides the base URL:

[extras]
name=Fedora Extras $releasever - $basearch
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/extras/$releasever/$basearch/
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-$releasever&arch=$basearch
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

In earlier experience, the problem has been that the DNS service for the repo that mirrorlist recommends is down, and this is what some googling suggested too. So I went to mirrors.fedoraproject.org to see what it said. The answer surprised me:

# repo = extras-6 arch = i386 error: invalid repo or arch
# following repositories are available:
# repo=core-2, arch=i386
# repo=core-2, arch=x86_64
# ...
# repo=core-6, arch=x86_64
# ...

So core-6 is in the list but extras-6 is not. I tried some different combinations thinking I had got it wrong, but to no avail. Instead, on a whim, I checked out the site’s root where I could navigate to a human-readable list of mirrors and I started checking them out one by one. Uni-Bayreuth seemed to have what I wanted, so I edited fedora-extras.repo like this:

[extras]
name=Fedora Extras $releasever - $basearch
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/extras/$releasever/$basearch/
#mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=extras-$releasever&arch=$basearch
baseurl=http://ftp.uni-bayreuth.de/linux/fedora/linux/extras/$releasever/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

and ran yum search libxslt-devel again. This time it complained about updates instead (Error: Cannot find a valid baseurl for repo: updates). Unfortunately Bayreuth didn’t have the updates catalog which sent me searching in a few other places until I finally decided to try to make do without it by setting enabled=0 in fedora-updates.repo:

[updates]
name=Fedora Core $releasever - $basearch - Updates
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/core/updates/$releasever/$basearch/
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-fc$releasever&arch=$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

It worked, and I was able to install the packages I wanted, but I really need to get myself a newer OS. I’ll combine that with a move to another host; FS-data have let me down once too many, and I’ve never fancied upgrading the OS on a production server anyway.

I acquired this server in December 2006 to host a few Rails projects and one of the first things I did was to disable Apache and install Nginx.  Once I decided to use WordPress for this blog, I needed PHP, and while I could have tried to set up PHP for Nginx, I decided to go the easy route and use the familiar LAMP combo.  I have after all used Apache in various guises ever since it was the Cern httpd back in 1993, so it should be no match for me.

Not so.  But to make a long story short and save myself some time next time around, here are the mistakes I made:

Private DNS

My wardrobe server (on my home network) hosts a number of public-facing web servers; little things that I set up for friends before I started doing web work professionally.  So, for instance www.norbusam.org points to my fixed home IP, and all traffic on port 80 is forwarded to my server.  The WGR614 that I use for NAT and routing is not smart enough to let me use the public interface from the private part of the network, i.e. if I sit at my desk at home and enter www.norbusam.org into my web browser, I won’t get anywhere.  That would have annoyed me if I hadn’t liked the solution so much: I set up a DNS daemon on the home server and let it trick all home computers to use the internal interface (192.168.x.y) for norbusam.org and others.  Easy as pie.

This came back to bite me for hours on end last night.  notetoself.vrensk.com was pointing at my home server, so while I was hacking the Apache settings on this server and reloading my browser furiously, all I got was my home server.  And since both run Apache, I never got suspicious—it just looked like my changes didn’t take.  Ouch.

SELinux

This server is running Security Enhanced Linux out of the box.  I asked for a vanilla install of FC6, and apparently SELinux is switched on by default.  I decided to keep it on when I got the server since I saw it as a learning opportunity.  It caused me some problems when I set up Postfix when the server was new, but I sorted it out and haven’t thought about it since then.

This, too, came back to bite me for a couple of hours.  It turns out that the default setting of SE for Apache are somewhere between frugal and paranoid.  There are various security bits to be set or cleared, and the default setting assumes that I only want to publish pages from a root-blessed directory and that I don’t want to run any scripts, use a database or have httpd talk to any other servers out there.  Rather than go through all these settings one by one and reverse them (and risk disabling something that was actually permissive to start with) I decided to turn off SELinux for Apache:

# setsebool -P httpd_disable_trans 1

Followed by a restart (not reload) of Apache.  I’m indebted to an article at Begin Linux for this solution.  I can’t recommend reading the actual article though, as it is just a long recapitulation of man pages and other documentation without a trace of explanation or even a try to put things in context.

Oh well.  Next time I will make sure to double check my IPs and disable SELinux at least while setting up a system.