trivia

During the new year period when I was having a little local difficulty with thrustVPS, I started looking around for alternative providers. My first port of call was lowendbox. That site lists many VPS providers and is often used by suppliers to advertise “special deals” for short periods. Indeed, I think I intially found thrust (or damnvps as it then was) on that site. My criteria were: UK based, cheap, high bandwidth, with a choice of OS. I prefer debian but many providers offer only Centos or a limited choice of sometimes outdated distros (ubuntu 8.04 anyone?). Disk capacity is not an issue here – tor won’t use it and my tails mirror will only need enough storage for some recent .iso images.

I much prefer a virtualisation product which will allow me to fully update the OS (including kernel) so that means I tend also to look for Xen in preference to openVZ. Other than that the spec should include a minimum of 256 Meg of RAM (not shared or “burstable”) and a reasonable share of a decent processor. Even given that requirement, there is still a range of offerings around for anywhere from £5.00 to £10.00 pcm. Since I effectively give both services away, I’m prepared to pay a reasonable sum, but I like cheap :-)

Crucially however, the supplier must allow tor on his network. This tends to reduce the choice. Many ISPs don’t like tor, particularly if you operate an exit node. In my experience, it’s not the bandwidth consumption they object to, more the time they have to devote to answering dumb, often automated, “abuse” complaints from others. One of the main reasons I have tended to favour a UK operation is that they seem more tolerant (and understanding) of how tor operates than some competitors overseas. And I really don’t need the aggravation of dealing with a US based DMCA takedown complaint routed through a US provider.

But for the last six months or so, I’ve only operated a relay node. With no exit permitted, I can’t be implicated in any so-called abuse, so both my provider and I get a quieter life but I still provide resources to tor. This means that I could probably afford to look more widely and perhaps source a VPS in another country. The US is particularly well served with cheap offerings, but I still confess a prejudice towards european data centres. Since I already have three VPSs in the UK, perhaps one, or maybe two in mainland europe might be useful.

Whilst I was checking a variety of potential new suppliers, I also checked with the tor-relays mail list for recommendations. One of the suppliers I was pointed to was a relatively new startup called DigitalOcean. Their offering (256 Mb RAM, 1 core, 20 Gig disk, and totally free bandwidth on a 1 Gig network) seemed astonishingly good. One guy on the list reported that he had transitted 15 TiB in one month on a server on their network. Whilst they are US based, DigitalOcean also offer servers in Amsterdam, so of course I just had to try them out. Boy are they fast.

Although they offer you a free trial for a few hours, I figured I could afford $5.00 (or less than £3.20 at current exchange rates) for a month’s trial. At the time of writing (13 January) I have been up just over 12 days and have transitted 3.57 TiB through one of their cheapest servers. My vnstat stats today show:

for a total of around 1200 established tor circuits. My top statistics report:

So I am ticking over quite healthily and shipping a very useful traffic load at just under 25 Mbit/s. Astonishing.

For comparison, a few days ago (on the 4th of January during early testing) I took snapshots of both my existing tor server with thrust and the new one at digitalocean. The image below compares the iptraf stats for each server (digitalocean at the top).

This second image compares the vnstat stats at the same time.

Note that both vnstat and the iptraf snapshots show that the new server is handling around 20 times the traffic going through my thrust server.

However, there is a downside to “unlimited” bandwidth (and besides, it won’t really be unlimited forever). Tor is voracious and will happily use all available resource on a server which does not throttle its connection rate. In early testing I did not set any bandwidth limits on tor. After just a few days I was running at around 35 Mbit/s for over 1600 tor circuits and tor was complaining “Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy.” So I started to explore options which would give the maximum freedom to tor without overloading the VPS.

As an aside, I should say that the “MaxAdvertisedBandwidth config option” doesn’t actually do what the tor manual suggests it should, so setting it in the configuration file is practically useless. However, I found a useful post on the tor-relays archive which suggested that I should advertise a CPU limit (“NumCPU 1″) and set “MaxOnionsPending 250″ (up from the default 100) since this would reduce the load on my server of a backlog of “onionskin queue overflows” (apparently often caused by a tor node being temporarily chosen as the best route to a hidden service). For a while, this worked – well at least the log messages went away, which may not actually mean the same thing. A couple of days later however, tor died with no entry in the logfile to explain why.

On further exploration, I found entries in the kernel log of the form “tor invoked oom-killer” followed by details of processes which had been killed as a result, so clearly we were running out of memory. Eventually tor itself died. Time to start setting some bandwidth limits.

I started fairly high (BandwidthRate 4 MB with a burst capability to 4.2 MB) and gradually wound it down over a period of two or three days as I watched top stats and log files. Whilst the VPS eventually seemed able to handle a traffic load of around 2 MB (that is each direction) I finally settled on 1.5 MB with a burst capability of 1.8 MB in the interests of both long term stability and fairness on the VPS providers – after all, $5.00 a month is peanuts for a total traffic load of nearly 9 TiB for that same month.

When first signing up to the trial at digitalocean I had specifically asked if they were happy with a tor node. The answer I got suggested that they would be happy with a transit relay, but less happy with an exit node. Not an unusual response. Following my early testing I decided it might also be smart to ask them if they really didn’t mind my using all that bandwidth for only $5.00. After all, it would be better to find out their actual stance in advance if I were to contemplate a long term service arrangement. They were candid in their reply, which I can summarise as: “it’s OK at the moment, but we are just building our service and the free bandwidth model won’t last. We will be charging in future. We just don’t know details yet”.

So, I’ll stick with them during this phase and see what they come up with. Whatever it is, on a network that fast, and a VPS that cheap, it looks worthwhile my keeping a presence in Amsterdam.