Archive for the ‘linux and unix’ Category

system monitoring with munin

Sunday, November 15th, 2009

A while back a friend and colleague of mine introduced me to the server monitoring tool called munin which he had installed on one of the servers he maintains. It looked interesting enough for me to stick it on my “to do” list for my VPSs. Having a bunch of relevant stats presented in graphical form all in one place would be useful. So this weekend I decided to install it on both my mail and web VPS and my tor node.

Munin can be installed in a master/slave configuration where one server acts as the main monitoring station and periodically polls the others for updated stats. This is the setup I chose, and now this server (my web and mail host) acts as the master and my tor node is a slave. Each server in the cluster must be set to run the munin-node monitor (which listens by default on port 4949) to allow munin itself to connect and gather stats for display. The configuration file allows you to restrict connections to specific IP addresses. On the main node I limit this to local loopback whilst on the tor node I allow the master to connect in addition to local loopback. And just to be on the safe side, I reinforced this policy in my iptables rules.

The graphs are drawn using RRDtool, which can be a little heavy on CPU usage, certainly too heavy for the slugs which ruled out my installing the master locally rather than on one of the VPSs. But the impact on my bytemark host looks perfectly acceptable so far.

One of the neatest things about munin is its open architecture. Statistics are all collected via a series of plugins. These plugins can be written in practically any scripting language you care to name. In the plugins which came by default with the standard debian install of munin I found plugins mostly written as shell scripts with the occasional perl script. However, a couple of the additional scripts I installed were written in php and python. The standard set of plugins covers most of what you would expect to monitor on a linux server (cpu, memory i/o, process stats, mail traffic etc). but there were two omissions which were quite important to me. One was for lighttpd, the other for tor. I found suitable candidates on-line pretty quickly though. The tor monitor plugin can be found on the munin exchange site (a repository of third party plugins). I couldn’t find a lighttpd plugin there but eventually picked one up from here (thomas is clearly not a perl fan).

Most plugins (at least those supplied by default in the the debian package) “just work”, but some do need a little extra customisation. For example the “ip_ ” plugin (which monitors network traffic on specified IP addresses) gets its stats from iptables and assumes that you have an entry of the form:

-A INPUT -d 192.168.1.1
-A OUTPUT -s 192.168.1.1

at the top of your iptables config file. You also need to ensure that the “ip_” plugin is correctly named with the suffix formed of the IP address to be monitored (e.g. “ip_” becomes “ip_192.168.1.1″). The simplest way to do this (and certainly the best way if you wish to monitor multiple addrresses) is to ensure that the symlink from “/etc/munin/plugins/ip_” to “/usr/share/munin/plugins/ip_” is named correctly. Thus (in directory /etc/munin/plugins):

ln -s /usr/share/munin/plugins/ip_ ip_192.168.1.1

The lighttpd plugin I found also needs a little bit of work before you can see any useful stats. The plugin connects to lighty’s “server status” URL to gather its information. So you need to ensure that you have loaded the mod_status module in your lighty config file and you have specified the URL correctly (any name will do, it just has to be connsistent in both the lighty config and the plugin). It is also worth restricting access to the URL to local loopback if you are not going to access the stats directly from a browser from elsewhere. This sort of entry in your config file should do:

server.modules += ( “mod_status” )

$HTTP["remoteip"] == “127.0.0.1″ {
status.status-url = “/server-status”
}

The tor plugin connects to the tor control port (9051 by default) but this port is normally not configured because it poses a security risk if configured incorrectly. Unless you also specify one of “HashedControlPassword” or “CookieAuthentication”, in the tor config file, then setting this option will cause tor to allow any process on the local host to control it. This is a “bad thing” (TM). If you choose to use the tor plugin, then you should ensure that access to the control port is locked down. The tor plugin assumes that you will use “CookieAuthentication”, but the path to the cookie is set incorrectly for the standard debian install (which sets the tor data directory to /var/lib/tor rather than the standard /etc/tor).

So far it all looks good, but I may add further plugins (or remove less useful ones) as I experiment with munin over the next few weeks.

OSS shouldn’t frighten the horses

Wednesday, November 11th, 2009

Since I first read that Nokia were adding much needed telephony capability to their N8x0 range of internet tablets I have been watching the development of the new Nokia N900 with much interest. It looks to be potentially the sort of device I would buy. Despite all the hype around the iPhone, I really dislike Apple’s proprietary approach to locking in its customers and I hate even more its use of DRM. So the emergence of a device which uses Linux based software such as Maemo and which is obviously targetted at the iPhone’s market looks to me to be very interesting. But some of the advertising is starting to look scary….

(I still want one though.)

logrotate weirdness in debian etch

Sunday, October 18th, 2009

I have two VPSs, both running debian. One runs lenny, the other runs etch. The older etch install runs fine, and is much as the supplier delivered it. Until now I have not had cause to consider the need to upgrade the etch install to lenny because it “just worked”. But today I noticed for the first time a very odd difference between the two machines. A difference which had me scratching my head, and reading too many man entries, for some long time before I found the answer.

For reasons I don’t need to go into, I log all policy drops in my iptables config to a file called “/var/log/firewall”. This file is (supposedly) rotated weekly. The logrotate and cron entries on both machines are identical.. The entry in “/etc/logrotate.d/firewall” looks like this:

/var/log/firewall {
rotate 6
weekly
mail logger@baldric.net
compress
missingok
notifempty
}

The (standard) file “/etc/logrotate.conf” simply calls the firewall logrotate file out of the included directory “/etc/logrotate.d”. The “/etc/cron.daily/logrotate” file (which calls the logrotate script) is also standard and simply says:

#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

and the (again standard) crontab file says:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don’t have to run the `crontab’
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user command
54 * * * * root cd / && run-parts –report /etc/cron.hourly
55 4 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.daily )
36 5 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.weekly )
3 3 5 * * root test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.monthly )
#

So far so simple, and you would expect the file “/var/log.firewall” to be rotated once a week at 04.55 on a sunday morning. Wouldn’t you.

Well, on lenny, you’d be right. But on the etch machine the file was rotated daily at a time completely unrelated to the crontab entry. It turns out that there is a bug in the way etch handles logrotation because syslog doesn’t use logrotate and overrides the logrotatiion entries run out of cron. I found this after much searching (and swearing).

See the bug report at bugs.debian.org and this entry which pointed me there.

I love standards.

debian on a DNS-313

Saturday, October 3rd, 2009

I bought another new toy last week – a D-Link DNS 313 NAS.

D-Link DNS-313

D-Link DNS-313

Actually, this was a mistake because what I really wanted was the DNS-323. I just wasn’t careful enough at the time. Quite apart from having space for two 3.5″ SATA hard drives instead of just one, the 323 is a very different beast to its smaller (and much cheaper) sibling.

Martin Michlmayr has a nice guide to installing debian on a 323. Given that the 323 has a faster processor and more RAM than a slug and it can take two internal SATA disks rather than just the external USBs it looks like an attractive option for a new debian based server. Pity then that I bought the wrong one. My excuse is that I thought the only difference was in the disk capacity and I was prepared to settle for just 1TB of store. The (normally reliable) owner of the shop where I bought the beast was also adamant that the disk capacity was the only difference between the two devices. I should have known better than to succumb to what was essentially an impulse buy when I wasn’t really intending to buy a new NAS at the time (I was in the shop for something else and picked up the 313 because I recalled reading Martin’s pages recently).

Once I got the 313 home of course and checked the specs it looked as if I would be stuck with the D-Link supplied OS. However, a bit of searching turned up the DSM-G600, DNS-323 and TS-I300 Hack Forum which has a series of articles on installing debian on a 313 (despite the forum title). A forum contributor called “CharminBaer” has put together a nice tarball of debian lenny which allows the user to replace the D-Link OS without actually reflashing the device. This means that the original bootloader is retained but the device boots into the replacement system on disk. The nicest part of this installation method is that there is almost no risk of bricking the device because the installation simply entails copying the tarball onto the disk over a USB connection, extracting the files and then booting into a shiny new Lenny install. Result.

The tarball can be found here, and the installation instructions are here.

Many thanks to “CharminBaer”.

wordpress on lighttpd

Saturday, September 12th, 2009

I have commented in the past how I prefer lighttpd to apache, particularly on low powered machines such as the slug. I used to be a big apache fan, in fact I think I first used it at version 1.3.0 or maybe 1.3.1, having migrated from NCSA 1.5.1 (and before that Cern 3.0) back in the day when I ran web servers for a living. However, those days are long gone and my web server requirements are now limited to my home network and VPSs so I don’t need, nor do I want, the power of an industrial strength apache installation. In fact, my primary home web server platform (the slugs) struggles with a standard apache install. Lighttpd works very well on machines which are low on memory.

Having got used to lighttpd, it seemed a natural platform to use on my VPSs. And it performs very well on those machines for the kind of traffic I see. Moving trivia to my bytemark VPS meant that I had to take care of some minor configuration issues myself – most notably the form of permalinks I use. Most of the documentation about running your own wordpress blog assumes that you will be using apache (since that is the most popular web server software provided by shell account providers). For those of you who, like me, want to use lighttp instead, the configuration details from my vhosts config file are below. Lighttpd is remarkably easy to configure for both virtual hosting in general, and for wordpress in particular. Note that I also choose to restrict access to wp-admin to my home IP address, this helps to keep the bad guys out.

Extract of “conf-enabled/10-simple-vhost.conf” file:

# redirect www. to domain (assumes that “mod_redirect” is set in server.modules in lighttpd.conf)

$HTTP["host"] =~ “www.baldric.net” {
url.redirect = ( “.*” => “http://baldric.net”)
}

#
# config for the blog
#
$HTTP["host"] == “baldric.net” {
# turn off dir listing (you can do this globally of course, but I choose not to.)
server.dir-listing = “disable”
#
# do the rewrite for permalinks (it really is that simple)
#
server.error-handler-404 = “/index.php”
#
# reserve accesss to wp-admin directory and wp-login to our ip address
#
$HTTP["remoteip"] !~ “123.123.123.123″ {
$HTTP["url"] =~ “^/wp-admin/” {
url.access-deny =(“”)
}
$HTTP["url"] =~ “^/wp-login.php” {
url.access-deny =(“”)
}
}

}

# end

Enjoy.

we’ve moved

Wednesday, September 9th, 2009

As I mentioned in the last post. I decided to move trivia from its old home on a shared hosting platform to my own VPS at bytemark. I also mentioned that this was proving trickier than it should – for no real good reason. However, the move is now complete and the blog is now completely under my control on my own VPS. So if anything goes wrong, I have only myself to blame.

So why did it take so long? Apart from the fact that I went on holiday immediately after the last post, the main reasons are twofold; firstly the difference in versions between that on my old host (2.2.1) and the current release (2.8.4) were sufficiently great to make the upgrade process trickier than it need have been; but secondly, and more importantly, my old provider’s DNS management process was less than helpful.

Before committing to the move, I naturally tested the installation and migration first on my new platform. This raised the problem of how I could install as “baldric.net” without clashing with the existing blog (I didn’t want to install under a different domain name for fairly obvious reasons). Changing my local DNS settings to point the domain name at my new IP address solved this problem (changing /etc/hosts would also have worked) but that meant that I could not have both old and new blogs on screen for comparison at the same time. Irritating, but not ultimately an insuperable problem. In moving to 2.8.4 I discovered that none of my (blogroll) links migrated properly and I had to recreate them all by hand. This took rather longer than I had anticipated, but it proved a useful exercise because I found some broken links in the process. They are currently still broken but at least I know that and I’ll fix them shortly. Because I use lighttpd and not the more usual apache I also had to address the problem of getting permalinks to work properly, but that didn’t prove too difficult – I’ll cover that in a separate post about wordpress on lighttpd.

Having got the new installation up and running to my satisfaction, I now wanted to point my domain name at the new blog. This is where I ran into some oddities in the way 1and1 set up their blog hosting and domain management. Ordinarily it is pretty easy to switch the A record for a 1and1 hosted domain (I have several) from the default to a new address. Not so if you have a blog hosted on that domain – the domain becomes “unmodifiable”. Technical support were initially not particularly helpful since they didn’t seem to understand my problem (and there were worrying echoes of my experiences with BT “support”). But this simply reaffirmed my belief that I was better off controlling my own destiny in future.

Eventually I was told that the only way I could unlock the domain to allow me to point to a new A record was to a) move the blog to a new domain (tricky if you don’t have one, and a pretty dumb idea anyway) or b) delete the blog (an even dumber idea if. like me, you are cautious enough to want to test the transition before committing). Eventually I decided to move the blog to a spare domain. I’ll delete it in the next week or so. Meanwhile, if you find an apparent duplicate of trivia on a completely different domain, you know why.

wordpress woes

Wednesday, August 26th, 2009

As is common with many blogs, my public ramblings on this site are made possible through the ease of use and flexibility of the mysql/php based software known as wordpress.. And again, as is common to much php/mysql based software, that package has vulnerabilities – sometimes serious, remotely exploitable vulnerabilities. When vulnerabilities are made public, and a patch to correct the problem becomes available, the correct response is to apply the patch, and quickly. In the world of mission critical software, or even in the world where your business or reputation depends upon correctly functional, dependable, and “secure” (for some arguable definition of secure) software it is absolutely essential that you patch to correct known faults. If you don’t, and as a result you get bitten, then your business or reputation, or both will suffer accordingly.

Yet again, as is common, I have to date used the services offered by a third party to host my blog rather than go to the trouble of managing my own installation. Many bloggers simply sign up to one of the services such as is offered by wordpress itself on the related wordpress.com site. Such sites tend to give you a blog presence of the form “yourblogname.wordpress.com” or “myblogname.blogger.com” etc. Other, usually paid for, service providers such as the one I use offer a blog with your own domain name. Whatever service you choose though, you are inevitably reliant on the service provider to ensure that the software used to host your site is patched and up to date. My own provider uses a template based approach to its blog service, This limits me (and others) to the functionality they choose to provide. In return, I expect them to ensure that the version of software they provide and support is as secure as is reasonably possible to expect for the sum I pay each month.

A couple of recent events have caused me to question this arrangement though and I am now in the process of moving this blog to one of my own servers. Firstly, wordpress itself has recently suffered from a particularly embarrassing remote exploit which allows an attacker to reset the admin password, and secondly, as I discussed at z05 below, the servers belonging to some supposedly security conscious individuals were compromised largely because poor patch management practices (amongst other things) left them exposed.

Time to rethink my posture.

I currently have three separate VPSs with different providers and I figure it is time I took responsibility for my own configuration management rather than relying on my current provider (which, incidentally, hasn’t updated its wordpress version for some long time despite both this current and many earlier security updates being released). However, for a variety of “interesting” and ultimately annoying reasons, this is proving to be trickier than it should be.

I’ll post an update when I have made the transition. Meanwhile, I hope not to see any break in service – unlike the self-inflicted cock-up in transfer of one of my domains.

Watch this space.

z05

Sunday, August 2nd, 2009

I really missed the old phrack magazine. Some of the “loopback” entries in particular are superb examples of technical nous, complete irreverance and deadpan humour. One of my favourites (from phrack 55) appears in my blogroll under “network (in)security”. I am particularly fond of the observation that details of how to exploit old vulnerabilities are “[ As useless as 1950's porn. ] “. As I said, sorely missed (but now, with issue 66 back in action after over a year since the last release).

It would seem, however, that I have been missing a new kid on the block who follows in phrack’s footsteps. A group called z0 appears to publish a ‘zine in the mold of the phrack of old. And their latest release, z05.txt has been causing something of a stir because it relates details of the compromise of systems owned and/or managed by some high profile and well known personages such as Dan Kaminsky and Kevin Mitnick.

The ‘zine bears reading. The style is unmistakably “underground” and “down with the kids” and it is (unnecessarily in my view) filled with unix-geek listings of bash history files and such like, but its authors still manage to make the sort of pertinent comments that I so loved in phrack.

“It’s the simple stuff that works now, and will continue to work years into the future. Not only is it way easier to dev for simple mistakes, but they are easier to find and are more plentiful.”

How well patched are you?

aspire one bios updates

Sunday, July 5th, 2009

This post is partly for my own benefit. It records some of the most useful references to bios updates for the AAO. My own AAO is actually running a fairly early bios (3114) and deliberately so. I upgraded to 3309 in (yet another futile) attempt to get sony memory sticks to work but found that screen brightness suffered for no perceptible gain in any other functionality. So I reverted. That aside, the references here may be useful to others.

Firstly, there is considerable discussion of the relative merits of various bios versions on the Aspire One User Forum. That site also has a lot of other useful (and sometimes not so useful) discussion points. Broadly, though, I have found that forum a good starting point for searching for AAO related information.

Next up is the very useful blog site run by Macles. Macles gave a very good set of instructions on flashing the bios in this post, which has been added to. and improved with the addition of lots of step by step pictures, by “flung” at netbooktech in this posting. Note however that the link to the acerpanam site supposedly hosting the bios images is broken. The images, along with many other driver downloads can currently be found by following links on the gd.panam.acer.com pages. Alternatively, I have found the images on the Acer Europe ftp site a good source (though it has yet to show the 3310 release).

No – my sony memory sticks still don’t work.

tor on a vps

Sunday, July 5th, 2009

I value my privacy – and I dislike the increasing tendency of every commercial website under the sun to attempt to track and/or profile me. Yes, I know all the arguments in favour of advertising, and well targeted advertising at that, but I get tired of the Amazon style approach which assumes that just because I once bought a book about subject X, I would also like another book about almost the same subject. I don’t much like commercial organisations profiling me (and, incidentally, I find it highly ironic that we in the UK seem to make a much bigger fuss about potential “big brother” Government than we do about commercial data aggregation, but hey).

Sure, I routinely bin cookies, block adware and irritating pop up scripts, and use all the, now almost essential, firefox privacy plugins, but even there we still have a problem. I don’t know who wrote those plugins, I just have to trust them. That worries me. Some of the best known search engines are even more scary if you think carefully about the aggregate information they have about you.

Sometimes I care about the footprint I leave, sometimes I don’t, but the point is that I should be in control of that footprint. Increasingly that is becoming difficult. Besides being tracked by sites I visit, last year’s controversy about BT’s use of phorm is also worrying. If my ISP can track everything I do, then I face another level of difficulty in protecting my fast vanishing privacy.

Besides using a locked down browser, DNS filtering which blocks adware, cutting cookies and all the other tedious precautions I now feel are necessary to make me feel comfortable, I often use anonymous proxies when I don’t want the end site to know where I came from. But even that now looks problematic. If you use a single anonymising proxy, all you are doing is shifting the knowledge about your browsing from the end site to an interrmediary. That intermediary may (indeed should) have a very strict security policy. Ideally, it should log absolutely nothing about transit traffic. But if that intermediary does log traffic data and then sells that data to a third party, you may be in an even worse position than if you had not attempted to become anonymous. Back in january of this year, Hal Roberts of Harvard University, posted a blog item about GIFC selling user data. If sites such as Dynaweb are prepared to sell user data, then the future for true anonymity looks problematic. As Doc Searle said in this blog posting,

We live in a time when personalized advertising is legitimized on the supply side. (It has no demand side, other than the media who get paid to place it.) Worse, there’s a kind of gold rush going on. Even in a crapped economy, a torrent of money is flowing into online advertising of all kinds, including the “personalized” sort. No surprise that companies in the business of fighting great evils rationalize the committing of lesser ones. I’m sure they do it it the usual way: It’s just advertsing! And it’s personalized, so it’s good for you!

No, as Searle well knows, it is not good for you.

What to do? Enter tor and privoxy.

I first used tor some years ago in its earlier incarnation as “the onion router” (hence its name) and until recently had used it only sporadically since. The main drawback of the early tor network was its speed, or lack of it. Tor gets is strength (anonymity) from the way it routes traffic.

how tor works

Tor traffic passes through a series of nodes before exiting at a node which cannot be linked back to the original source. So tor performance depends on a large number of both fast intermediate relays and a large number of exit nodes. Since not all tor users are prepared to run relays, let alone an exit node (it can be bandwidth expensive and in the case of an exit node can lead to your system being mistaken for a hostile, or compromised, site) tor can be slow, at times painfully slow. But recently tor has been getting faster as more relay and exit nodes are added. It is now at a state which is probably usable most of the time, so long as you are prepared to wait a little longer than is customary for some web pages to load (and you don’t use youtube…..).

When using tor recently I have tended to follow the well trodden path of local installation alongside privoxy. Because I believe in giving something back to the community if I am gaining benefit, I also set my local configuration to run as a relay. But that caused some difficulty. If we assume that my tor usage was fairly representative of the majority of tor users out there, then the fact that my relay was only operational when my client system was up and running meant that the relay would be seen by the tor network as unstable and probably slow, Indeed, the fact that I had to throttle tor usage to the minimum to stop the network from impacting unduly on my ADSL bandwidth, meant that I was not entirely happy with the setup. So I stopped relaying. But that leaves me feeling that I am taking advantage of a free good when I could be contributing to that good.

Some while back I bought myself a VPS from Bytemark (an excellent, technically savvy, UK based hosting company) to run a couple of webs and an MTA. I use it now largely as a mail server (running postfix and dovecot) and the traffic is relatively low volume. That VPS is pretty small (though actually way better specced than some real servers I have run in the past) but I reasoned that I could easily run a tor relay on that machine and then connect to it remotely from my client system. I did, and it worked fine. But I soon found that the tor network seems to have a voracious appettite for bandwidth, Even with a fairly strict exit policy (no torrents allowed!) and some tight bandwidth shaping, I still found that I was using about 2 Gig of traffic per day (vnstat is useful here). Any more than that would start to encroach on my bandwidth allowance for my VPS and possibly impact on the main business use of that server. Monthly rates for VPSs are now less than I pay for my mobile phone contract (and arguably more useful than a phone contract too) so I decided to specialise and buy another VPS just for tor. I now run an exit node on a VPS with 384 MB of RAM and 150 Gig monthly traffic allowance. That server is currently throttled to about 2 Gig of traffic per day, but I will double that very shortly.

Now one of the nicest things about running a tor relay is the fact that your own tor usage is masked and you may get better anonymity. I therefore run privoxy on my tor relay and proxy through from my client to that proxy which in turn chains to tor internally on my relay. However, if you simply configure your local client to proxy through to your relay in clear you are allowing your ISP (and anyone else who cares to look) to see your tor requests – not smart. So I tunnel my requests to the tor relay through ssh. My local client has an ssh listener which tunnels tor requests through to the relay and connects to privoxy on port 8118 bound to localhost on the relay. I also have a separate browser on my desktop which has as its proxy the ssh listener on my client system. For a good description of how to do this see tyranix’s howto on the tor wiki site. Now whenever I want to use tor myself I simply switch browser (and that browser is particularly dumb and stripped, and has no plugins or toolbars which could leak information). Of course, should I get really paranoid, I could always run the local browser in a VM on my desktop and reload the VM after each session.

But I’m not that paranoid.