Mar 26 2013

using an ssh reverse tunnel to bypass NAT firewalls

There is usually more than one way to solve a problem.

Back in October last year I wrote about using OpenVPN to bypass NAT firewalls when access to the firewall configuration was not available. I have also written about using ssh to tunnel out to a tor proxy. What I haven’t previously commented on is the ability to use ssh to set up reverse tunnels. By this I mean the capability to set up an outbound connection from one machine to a remote machine which can then be used as a tunnel back from the remote machine to the original.

You might ask why on earth anyone would want to do that. In fact this is a very useful trick. If we think back to the problem outlined in my october post, we were faced with the requirement to connect from a client machine on one network to a remote host on another network which was behind two NAT firewalls. The scenario was as below:

openvpn-scenario

As I said in my previous post, ordinarily all we would have to do is run an SSH daemon (or indeed openVPN) on Host A and set up port forwarding rules on routers A and B to forward the connection to that host. But in the case we were considering, the owner of Host A was not able to set up the requisite port forwarding rules on the outermost router (because he didn’t own it). Indeed, this scenario is quite common in cases such as mobile data connections where the client device (say your smart phone or tablet) is given an RFC 1918 reserved address by the telco and is firewalled (often transparently) from the real internet by their NAT proxy.

We previously tackled this problem by setting up an intermediary openVPN server on a cheap VPS and using that as a relay between the two devices. However, ssh allows us to set up a reverse tunnel directly to the machine marked as “client” from the machine marked as “Host A” in the diagram above. We could, of course, also set up the tunnel to the VPS as an intermediate if we so desired. This might be necessary in cases where we don’t have access to router C for example and cannot configure inbound port forwarding there either.

Again, here’s how to accomplish what we want, this time using ssh.

On “client” (or the intermediary VPS) ensure that we have an unprivileged user such as “nobody” who has no login shell but does have a valid password. I set the shell on all unused accounts to /bin/false anyway, but you may choose otherwise. The sshd daemon itself uses /usr/sbin/nologin. The nobody account does not need a shell because we are only using it to set up the tunnel, not to actually login to the remote machine. Of course the “client” machine must have an ssh daemon listening on a port reachable by “Host A” or the initial tunnel connection will fail.

Host A must also have an ssh daemon listening in order to accept the back connection. Note that this latter daemon must be listening on “localhost” as well as the machine’s external IP address. This can most easily be accomplished by setting “ListenAddress 0.0.0.0” (rather than say, “ListenAddress 192.168.3.12”) in Host A’s /etc/ssh/sshd_config file. Localhost is necessary because that is the address on which Host A will see the incoming connection over the tunnel when it is established

Now on “Host A” open the tunnel to “client” thus:

ssh -N -f -R 2020:localhost:222 nobody@client

Here -N means no command at the remote end, i.e we are simply tunnelling; -f means background the process; -R means remote (or reverse); then listen on port 2020 on localhost on the client and connect back to port 222 on Host A). Note that nowhere in this command do we specify the ssh port used by the daemon on “client”. That is taken care of by default by ssh itself (ssh will assume port 22 unless we have changed that behaviour by modifying the port for “client” in /etc/ssh/ssh_config or a local .ssh configuration). Alternatively we could add the -p switch, followed by the requisite port number, to the above command line. Port 222 is the port on which the ssh daemon on Host A listens. Port 2020 can actually be any port which is currently unused on the “client” machine. Pick a port above 1024 to avoid the need for root privileges.

On running the command above we should be prompted for “nobody’s” password on the machine “client”. After supplying the password we will be returned to the command line on Host A. Now we can check that we have actually opened a connection by running netstat where we should see an established connection to ssh on the remote machine.

On the remote machine called “client”, we should similarly see the established ssh connection from Host A, but we should also be able to see a new ssh process listening on port 2020 on localhost. It is this process listening on port 2020 that allows us to connect back to Host A.

Now that we have an open tunnel, we can connect from “client” to “Host A” thus:

ssh -p 2020 localhost

This will default to the user we are currently logged in as on “client” and we will be prompted for the password for that userid “@localhost” (actually, the password for that user on Host A). Of course, if we wish to specify a different user we could add the “-l userid” switch to the ssh command line. If the password is accepted we will now be logged in to “Host A” from “client” and we have completely bypassed the NAT firewalls.

I strongly recommend that you do not attempt to use this mechanism (or openVPN) to circumvent a corporate security policy unless you wish to jeopardise your career.

Permanent link to this article: https://baldric.net/2013/03/26/using-an-ssh-reverse-tunnel-to-bypass-nat-firewalls/

Mar 13 2013

impolite spam

Most blogs get hit by spammers aiming to get their URLs posted in the comments section. Like most wordpress based blogs, I use the default Akismet antispam plugin. I don’t like it, I don’t like the fact that it is shipped by default, I don’t like the fact that it is increasingly becoming non-free (as in beer), I don’t like the way my blog has to call home to Akismet servers. I particularly don’t like the way that all comments (including the genuine ones) are thus passed to Akismet servers, giving them the IP and email addresses of my commenters (which is another good reason for you to lie to me and use an anonymising service to mask your IP address. But Akismet has the one huge redeeming factor that it works. It stops a vast amount of crud from reaching my blog.

However, some still gets through and hits my moderation queue (I will never allow unfettered commentary on trivia). Most of the stuff that does get through has fooled the Akismet filters by using vaguely plausible (if usually ungrammatical) phrasing. Typical comments are bland, generic staments along the lines of “Nice blog, interesting discussion I have bookmarked this for later and will tell all my friends about you”. That comment will then have a URL somewhere which points to a site flogging knock-off chinese imitations of western luxury goods (or porn, though this is becoming less frequent).

I was therefore delighted to find the following in my queue a couple of days ago:

“The next time I read a weblog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my option to read, but I in fact thought youd have some thing fascinating to say. All I hear is really a bunch of whining about some thing that you could fix when you werent too busy seeking for attention.”

Way to go. Having managed to get through a spam filter and hit my moderation queue, the poster then shoots him or herself in the foot by insulting me.

No way will I post that……

Permanent link to this article: https://baldric.net/2013/03/13/impolite-spam/

Feb 28 2013

touching update

I have recently upgraded the internal disk on my main desktop from 1TB to 2TB. I find it vaguely astonishing that I should have needed to do that, but I do have a rather large store of MP4 videos, jpeg photos and audio files held locally. And disk prices are again coming down so the upgrade didn’t cost too much. One noticable improvement following the upgrade is the reduction in noise. The disk I chose is one of Western Digital’s “Green Desktop” range which is remarkably quiet. Thoroughly recommended. But the point of this post is the consequence of the upgrade.

In order to minimise disruption after installation of the new disk (and of course a fresh install of Mint) I simply slotted the new disk into a spare slot in the PC chassis and hooked it up to the SATA port used by the old disk. I then hooked the old disk up to a separate spare SATA port. (I could, of course, have changed the boot order in BIOS to achieve the same effect.) Having installed Mint, I then rebooted the machine from the old disk for one final check of my old configuration before copying my data from old-disk/home/mick to new-disk/home/mick. Despite the fact that my data occupied over 900GB, the copy went reasonably quickly and painlessly – one of the advantages of a disk to disk copy over SATA, even if it is only SATA 2.0 (3Gb/s) – (Note to self. Next build should include SATA 3.0).

However, what happened next certainly wasn’t quick. In my haste to copy my old data and get back to using my PC, I stupidly forgot to preserve the file attributes (-p or -a switch) in my recursive cp. This meant of course that all my files on the new disk now had a current date attached to them. Worse, I didn’t immediately notice until I came to backup my desktop to my NAS. I do this routinely on a daily basis using rsync in a script like so:

/usr/bin/rsync -rLptgoDvz –stats –exclude-from=rsync-excludes /home/mick nas:/home/mick-backup

Guess what? Since all my desktop files now had a current modification time, rsync seemed to want to recopy them all to the NAS. This was going to take a /lot/ longer than a local cp. So I killed it so that I could figure out what had gone wrong (that didn’t take long when I spotted the file timestamps) and could find a simple fix (that took longer).

Now I had thought that rsync was smart enough to realise that the source and destination files were actually the same, regardless of the file timestamp change. Realising that I didn’t actually know what I /thought/ I knew about rsync I explained to colleagues on the ALUG mailing list what I had done and sought advice. They didn’t laugh (well not publicly anyway) and a couple of them offered very helpful suggestions to sort my problem. Wayne Stallwood first pointed out that “the default behaviour for rsync is to only compute file differences via the checksums if the modification dates between source and destination indicate the file has changed (source is newer than the destination) So it’s not actually recopying everything (though if things like permissions have changed on the source files they will now overwrite those on the target and naturally the timestamps will be updated). Previously when you ran your backup it would have just skipped anything that had the same or older timestamp than the target.”

So, what I saw as a possibly very long re-copy exercise was actually rsync comparing files and computing checksums. Needless to say, that computation on a fairly low powered NAS was going to take a long time anyway. And besides, I didn’t /want/ the timestamps of my backups all changed to match the (now incorrect) timestamps on my desktop. I wanted the original timestamps restored.

Then Mike Dorrington suggested that I could simply reset the timestamps with a “find -exec touch” approach. As Mike pointed out, touch could be made to use the timestamp of one file (in this case my old original file on the 1TB disk) to update the timestamp of the target file without any need for copying whatsover. But I confess I couldn’t at first see how I could get touch to recurse down one filetree whilst find recursed down another. Mark Rogers put me out of my misery by suggesting the following:

cd /old-disk/home/mick

find . -exec touch /new-disk/home/mick/\{\} –reference=\{\} –no-dereference –no-create \;

I think this is rather elegant and it certainly would not have occurred to me without prompting.

Permanent link to this article: https://baldric.net/2013/02/28/touching-update/

Jan 25 2013

now that is what an isp should be like

In my post about the astonishing speed of the DigitalOcean network compared to the appalling service I was getting at ThrustVPS, I mentioned that the free bandwidth model didn’t look sustainable in the long run. Indeed, DigitalOcean told me themselves that they would move to a more normal commercial model when they had a better idea of their customer profile. So I was fully prepared to be told eventually that the good times were over and I should now expect to pay a lot more for my traffic. Regardless of that though, I intended to stay in Amsterdam because the network was so refreshingly fast.

Today I was delighted to receive an email from DigitalOcean telling me that as an early customer they had “grandfathered” me in to their new commercial arrangements on the same terms as I orginally signed up to – i.e. free bandwidth forever. Not quite believing my luck I asked for clarification – particularly whether that meant my account as a whole, or just the current VM. My reason for asking was that I am about to create two new VMs with them following their upgrade of the basic $5.00 pcm offering to 512MB RAM and 20 Gig of SSD disk space. I will use the first new VM to move my new tor node and the second to replace my tails mirror (which eats about 900 GiB pcm on the slow, unreliable Thrust network).

They replied:

It’s true, since you have an older account you have unlimited bandwidth for your full account, meaning every droplet you create is set to unlimited bandwidth too!

Now /that/ is what I call service. And a truly fantastic committment from DigitalOcean.

So, if you want a fast VM on a /really/ fast network. Take a look at DigitalOcean. Even without the bandwidth allowance I have, they are still astonishing value for money.

Permanent link to this article: https://baldric.net/2013/01/25/now-that-is-what-an-isp-should-be-like/

Jan 14 2013

a long time ago in a galaxy far far away

The ascii art in sl was quite cute, this is far more elaborate.

star-wars

To see it in action, telnet to towel.blinkenlights.nl.

Why waste your time browsing rubbish websites – fire up a terminal session and waste even more time with telnet. More fun places to try can be found at telnet.org.

Permanent link to this article: https://baldric.net/2013/01/14/a-long-time-ago-in-a-galaxy-far-far-away/

Jan 13 2013

what a difference a gig makes

During the new year period when I was having a little local difficulty with thrustVPS, I started looking around for alternative providers. My first port of call was lowendbox. That site lists many VPS providers and is often used by suppliers to advertise “special deals” for short periods. Indeed, I think I intially found thrust (or damnvps as it then was) on that site. My criteria were: UK based, cheap, high bandwidth, with a choice of OS. I prefer debian but many providers offer only Centos or a limited choice of sometimes outdated distros (ubuntu 8.04 anyone?). Disk capacity is not an issue here – tor won’t use it and my tails mirror will only need enough storage for some recent .iso images.

I much prefer a virtualisation product which will allow me to fully update the OS (including kernel) so that means I tend also to look for Xen in preference to openVZ. Other than that the spec should include a minimum of 256 Meg of RAM (not shared or “burstable”) and a reasonable share of a decent processor. Even given that requirement, there is still a range of offerings around for anywhere from £5.00 to £10.00 pcm. Since I effectively give both services away, I’m prepared to pay a reasonable sum, but I like cheap :-)

Crucially however, the supplier must allow tor on his network. This tends to reduce the choice. Many ISPs don’t like tor, particularly if you operate an exit node. In my experience, it’s not the bandwidth consumption they object to, more the time they have to devote to answering dumb, often automated, “abuse” complaints from others. One of the main reasons I have tended to favour a UK operation is that they seem more tolerant (and understanding) of how tor operates than some competitors overseas. And I really don’t need the aggravation of dealing with a US based DMCA takedown complaint routed through a US provider.

But for the last six months or so, I’ve only operated a relay node. With no exit permitted, I can’t be implicated in any so-called abuse, so both my provider and I get a quieter life but I still provide resources to tor. This means that I could probably afford to look more widely and perhaps source a VPS in another country. The US is particularly well served with cheap offerings, but I still confess a prejudice towards european data centres. Since I already have three VPSs in the UK, perhaps one, or maybe two in mainland europe might be useful.

Whilst I was checking a variety of potential new suppliers, I also checked with the tor-relays mail list for recommendations. One of the suppliers I was pointed to was a relatively new startup called DigitalOcean. Their offering (256 Mb RAM, 1 core, 20 Gig disk, and totally free bandwidth on a 1 Gig network) seemed astonishingly good. One guy on the list reported that he had transitted 15 TiB in one month on a server on their network. Whilst they are US based, DigitalOcean also offer servers in Amsterdam, so of course I just had to try them out. Boy are they fast.

Although they offer you a free trial for a few hours, I figured I could afford $5.00 (or less than £3.20 at current exchange rates) for a month’s trial. At the time of writing (13 January) I have been up just over 12 days and have transitted 3.57 TiB through one of their cheapest servers. My vnstat stats today show:

vnstat-13-01-13

for a total of around 1200 established tor circuits. My top statistics report:

top-stats-13-01-13

So I am ticking over quite healthily and shipping a very useful traffic load at just under 25 Mbit/s. Astonishing.

For comparison, a few days ago (on the 4th of January during early testing) I took snapshots of both my existing tor server with thrust and the new one at digitalocean. The image below compares the iptraf stats for each server (digitalocean at the top).

vm-comparison-1

This second image compares the vnstat stats at the same time.

vm-comparison-2

Note that both vnstat and the iptraf snapshots show that the new server is handling around 20 times the traffic going through my thrust server.

However, there is a downside to “unlimited” bandwidth (and besides, it won’t really be unlimited forever). Tor is voracious and will happily use all available resource on a server which does not throttle its connection rate. In early testing I did not set any bandwidth limits on tor. After just a few days I was running at around 35 Mbit/s for over 1600 tor circuits and tor was complaining “Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy.” So I started to explore options which would give the maximum freedom to tor without overloading the VPS.

As an aside, I should say that the “MaxAdvertisedBandwidth config option” doesn’t actually do what the tor manual suggests it should, so setting it in the configuration file is practically useless. However, I found a useful post on the tor-relays archive which suggested that I should advertise a CPU limit (“NumCPU 1”) and set “MaxOnionsPending 250” (up from the default 100) since this would reduce the load on my server of a backlog of “onionskin queue overflows” (apparently often caused by a tor node being temporarily chosen as the best route to a hidden service). For a while, this worked – well at least the log messages went away, which may not actually mean the same thing. A couple of days later however, tor died with no entry in the logfile to explain why.

On further exploration, I found entries in the kernel log of the form “tor invoked oom-killer” followed by details of processes which had been killed as a result, so clearly we were running out of memory. Eventually tor itself died. Time to start setting some bandwidth limits.

I started fairly high (BandwidthRate 4 MB with a burst capability to 4.2 MB) and gradually wound it down over a period of two or three days as I watched top stats and log files. Whilst the VPS eventually seemed able to handle a traffic load of around 2 MB (that is each direction) I finally settled on 1.5 MB with a burst capability of 1.8 MB in the interests of both long term stability and fairness on the VPS providers – after all, $5.00 a month is peanuts for a total traffic load of nearly 9 TiB for that same month.

When first signing up to the trial at digitalocean I had specifically asked if they were happy with a tor node. The answer I got suggested that they would be happy with a transit relay, but less happy with an exit node. Not an unusual response. Following my early testing I decided it might also be smart to ask them if they really didn’t mind my using all that bandwidth for only $5.00. After all, it would be better to find out their actual stance in advance if I were to contemplate a long term service arrangement. They were candid in their reply, which I can summarise as: “it’s OK at the moment, but we are just building our service and the free bandwidth model won’t last. We will be charging in future. We just don’t know details yet”.

So, I’ll stick with them during this phase and see what they come up with. Whatever it is, on a network that fast, and a VPS that cheap, it looks worthwhile my keeping a presence in Amsterdam.

Permanent link to this article: https://baldric.net/2013/01/13/what-a-difference-a-gig-makes/

Jan 11 2013

dovecot failure

Today I ran a routine apt-get update/apt-get upgrade on my mailserver and dovecot failed afterwards. This is a “bad thing” (TM). No routine software upgrade should cause a failure of the kind I experienced.

Two things happened which should not have done. Firstly the SSL certificates appeared to have changed (which meant that mail clients immediately complained and flagged the error). Secondly, the back end authentication in my mysql database also seemed to fail. Actually, neither of these symptoms was really true, but nevertheless it took me some time to figure out what had happened and then to fix it.

The SSL certificate error occured because the upgrade replaced one of the configuration files (/etc/dovecot/conf.d/10-ssl.conf) without telling me. This meant that my locally configured SSL certificates, which are stored in a non-standard directory, were ignored and instead the standard /etc/ssl/certs/dovecot.pem and /etc/ssl/private/dovecot.pem were substituted. This is particularly annoying because apt-get normally flags any changed configuration files on an upgrade and asks if you wish to retain your locally changed file or replace it with the new one distributed. It didn’t this time and I got caught out.

Having fixed that problem, I then hit the apparent authentication failure. My mail client said that authentication with the server had failed whilst the server logs said: “Initialization failed: namespace configuration error: inbox=yes namespace missing“. This didn’t initally mean much to me, but a search for the error message brought me examples of others who had hit this problem. In particular, Bill Ryder’s email on the dovecot email list archive gave me the answer I needed to fix the problem. Once I had added the configuration option he suggests to /etc/dovecot/conf.d/10-mail.conf and reloaded dovecot, my problems disappeared.

But this shouldn’t happen. OK, maybe I should have read the changelog before upgrading, but no minor upgrade should so change a previously working configuration that it fails. I nearly dumped dovecot back in 2010 when version 2.xx release superceded
the earlier 1.2.xx version. The changes between those branches was significant and it took me some time to figure out what I needed to do to get a working migration between the two. But at least I expected changes between a version 1.xx and a version 2,xx.

I may now reconsider my decision to stick with dovecot and look again at courier, or even look for another IMAP/POP server entirely.

Permanent link to this article: https://baldric.net/2013/01/11/dovecot-failure/

Jan 10 2013

my top ten

I have been collecting statistics on trivia’s visitors using counterize for some time. The lastest version, which I have been using for about five months now, allows the user to graph, and publish, site statistics in a nice form. I have today created a new page (listed under “longer trivia” on the right) called trivial statistics. I am gratified that my “how-to” type of content is so popular, but my readership profile is interesting.

Permanent link to this article: https://baldric.net/2013/01/10/my-top-ten/

Jan 07 2013

thrust respond

This afternoon I received an email from Andrew Sinclair at thrustVPS in response to my post below. In the interests of openness I have added his email as a comment to the post. My thanks to Andrew for that email. It is just a shame that I did not get such a helpful response in earlier correspondence.

Permanent link to this article: https://baldric.net/2013/01/07/thrust-respond/

Jan 06 2013

lies, damned lies and VPS traffic

I had an interesting time over the new year period.

For some time now I have run a tor node on a VPS at ThrustVPS. I also run my tails mirror on a VPS at the same provider. Their network has always struck me as pretty poor (response times to ssh login is particularly bad, and the servers sometimes go off-line for no apparent reason) but over the last month I had worse than average performance, and frankly appalling service response.

I initially chose thrust (then known as damnvps for some odd reason) because they offered “1TB of transfer” per month on all their VPSs. Tor is a particularly voracious bandwidth hog and needs to be throttled to keep it in check. Fortunately the configuration file allows this fairly easily and I had mine set to limit my traffic to no more than 30 GiB per day – i.e. less than the supposed monthly limit. My tails mirror is similarly throttled, though I do that in lighttpd and it is a bit more tricky and less accurate

Until recently I have monitored all my servers through munin. But a while back I moved the VPS which acted as the munin master from one place to another and did not get around to reconfiguring munin on that node. Big mistake. It meant that I had no records of the activity on my tails mirror and tor node over a critical period when thrust seemed to be even more unreliable than usual. Munin monitor notwthstanding, I still check my servers pretty regularly and I found that the thrust VPSs seemed to be going off-line too frequently. One server, the tor node, went down on the 19th, 22nd, and 29th of december and the tails mirror also went down on the 22nd. They came back up when rebooted through the web administration interface, but performance was pretty dire. So I complained to support.

My first email to support on the 29th of december elicited an interesting response. They replied:

“We have stopped your server 109.169.77.131 since it is causing DDOS attack to the hostnode 15xnuk. It is related to the performance issue of the host node we have no other choice without stopping you.Please do take proper measures, install the proper anti virus on the server make through scan on your server and update us.”

DDOS eh? So I asked for more detail and got the following in response:

“I have been started your server when you opened the ticket. And we are sorry for you to stop the VPS without any information. We are monitoring this node for the past few hours and if your IP DDOS again we may forced to stop it again. Please do take measures and monitor your VPS.”

In my email to support I had explained that the VPS in question ran a tor node (though they should have known that because I specifically told them that when I paid for it) and that heavy traffic from multiple IP addresses inbound to my tor port was perfectly normal. Moreover, I could not believe that my tor mode would be the only one targeted by a DDOS if someone didn’t like tor (and lots of people don’t) so I checked with colleagues on the tor-relays mailing list. No-one else seemed to be having problems. Nevertheless, I shut tor down while I investigated and fired off another email to support asking them again to answer my earlier questions. In particular I said:

“Specifically,

– is the “DDOS” aimed at my IP address, and my IP address alone (and only to port 443)?

– is all the traffic coming to my IP address from tor nodes, and only tor nodes (this is easy to check, the addresses are public)?

If the traffic has now dropped, and the answer to either question is yes, then this is not a DDOS, it is expected behaviour of
a tor node.

I am about to turn tor back on. Watch the traffic build again.”

I received another unhelpful reply:

“I could see that most traffic coming from https. So please monitor the https ports. The host node is stable now so I think the issue get compromised.

Please back to us if you need any further assistance.

Thanks for the understanding.”

Note that at this stage there has been no answer to my questions, nor has there been any explanation of why my tails mirror went down, or the tor node went down on the days before the 29th. But by now I suspected simply that the thrust network was so heavily oversold that whenever my servers came close to using anywhere near their alloted bandwidth, they were arbitrarily shut down. This was confirmed for me by another email from support which said:

“Please note that there exists a script in the node to reboot the servers having high load (thereby causing load spike in the node) inorder to get the load in the node to be stable. And we are considered the high resource using is also as a DDOS attack.”

I interpreted this to mean that any high traffic is seen by thurst as a DDOS and they simply shut down the target IP. Way to go thrust. If there /is/ a DDOS attack on a node then you have just done the attacker’s job for them. Worse, they don’t bother to tell the customer.

By now (the 31st of December) my server was back on-line, but it went down again later that day. So I complained again and got another stupid reply:

“Your server was suspended due to high ddos attack on the host node and it affected our host node performance heavily.

Also it is related to the performance issue of the host node we can’t unsuspend your server for the next 24 hours , please update us after the 24 hours complete then only I can unsuspend your server.”

Which I took to mean that they would not turn my server back on again for another 24 hours.

Along with the stupid reply, I was given a snippet of log traffic to/from my server. Guess what, it showed traffic from high numbered ports on a range of IP addresses to/from my port 443 (I run tor on 443 – thrust support seemed to think that this meant it must be https). To anyone who knows anything about IP networking this is indicative of client systems communicating with a server. In my case, it is tor clients trying to connect through my tor node on their way somewhere else. Thrust don’t get this, or don’t want to. All they see is lots of traffic so they shut it down.

Now note carefully here that at no time have they proved DDOS. My vnstat traffic stats on the VPS show that I am well within my bandwidth allowance for the month and the daily traffic is not unusually high either. The only possible problem is that I am attempting to use the bandwidth they advertise as available, and that I have paid for. I confess I was a little abrupt in my reply to this nonsense and I asked that the issue be escalated to management.

Things then went quiet. I think I pissed off the support team.

As an aside it is worth taking a look at the thrustVPS sales page:

thrustVPS-1

I particularly like the “Friendly, knowledgeable support available 24×7” and “above all we care about you and your customers“.

and the page advertising the service I actually paid for:

thrustVPS-2

Note carefully the claimed “512MB RAM, 15GB Disk Space, 1TB Transfer, 1 IP, UK or US” because this is where it gets interesting. On the 2nd of January I received an email from one Gareth Akehurst saying

“Hello Mick,

Escalations was unavailable during the holiday period, how can I help?
Regards,
Gareth Akehurst”

(so clearly “escalations” don’t work 24/7)

In answer to my response querying why my server had been shut down, Gareth replied:

“This is standard procedure for our clients, all nodes are on a 100mBits network, the node you are currently on shares that connection with 59 other virtual servers, if you are requiring a high traffic service it may be worth considering a low contention vps which the nodes are limited to no more than 25 vpses.”

Note that “all nodes are on a 100mBits network”. So at best, my VPS is on a machine which has 60 VPS instances trying to share a 100Mbit/s network, and at worst, the whole damned network of hardware is on a 100 Mbit backbone. I’ve got one of those at home for pitys sake. Now, since my vnstat daily stats on that VPS show I was running at 3.1 Mbit/s for a daily throughput of around 30 GiB, there is no way on earth that all 60 customers could /ever/ get their full monthly allowance. In fact, if all of us tried, we would be running at nearly twice the total available bandwidth of the backbone. No wonder the network response was crap. And no wonder thrust arbitrarily shut down customers with “high” loads. The network is so hugely oversold it is criminal.

The ISP industry has a pretty shoddy reputation for advertising (see 8 Mbit/s broadband advertising for example) but this is frankly taking the piss. If I buy a product that says I should get 1TB of transfer, I expect to get that. If they can’t provide that (and on their maths, they can’t) then I have been missold. Worse, I have been treated like an idiot and lied to about DDOS attacks. If I hadn’t paid in advance for the two servers I have with thrust, I’d close the account now. But oddly, they do not refund unhappy customers. So I’ll sit and use as much as possible under my current contract and then leave.

Meanwhile, if anyone else is considering thrustvps – just don’t. Look elsewhere. There are much better providers out there. Bytemark is by far the best. Unfortunately they don’t offer a high bandwidth allowance at a price point I am prepared to pay for a service which I give away.

(In the interests of transparency and honesty, I have posted here the entire email exchange between myself and thrust “support”. Read it and weep.)

Happy New Year.

Permanent link to this article: https://baldric.net/2013/01/06/lies-damned-lies-and-vps-traffic/

Dec 24 2012

another anniversary

This time last year I noted that christmas eve was trivia’s birthday. Guess what, it’s that time of year again, and what am I again doing?

Merry Christmas to all^h^h^h both my readers.

Mick

Permanent link to this article: https://baldric.net/2012/12/24/another-anniversary/

Dec 19 2012

no sites are broken

Or so the wordpress post at wordpress.org would have us believe. However, I think there is flaw in both their logic, and their decision making here.

I spotted the problem following an upgrade to wordpress 3.5 on a site I use. One of the plugins on that site objected to the upgrade with the following error message:

“Missing argument 2 for wpdb::prepare(), called in /path/to/some/function.php”

That error message was splattered all over the page which called that function. The page is public.

A quick search brought me to the wordpress announcement above. But crucially, and worryingly, it also threw up nearly 10,000 other references. Almost all the referenced pages I checked showed websites with that error message (with variations on the path/to/function) displayed fairly prominently.

The wordpress article explains that the error occurs because some function written by a developer (probably a developer of a theme or plugin) called the “$wpdb->prepare()” wordpress function with only one argument when it is supposed to be called with two. Worse, the article explains that calling the “$wpdb->prepare()” function in this way is insecure because the SQL query it is passing is not properly prepared (despite what the developer may think).

The article says:

“But here’s where the problem lies:

 

$wpdb->prepare( “SELECT * FROM table WHERE id = $id” );

 

See the problem? That query isn’t secure! You may think you are “preparing” this query, but you’re not — you’re passing $id

directly into the query, unprepared. And this, right here, is why $wpdb->prepare() now issues a warning if it isn’t called with

more than one argument. Because you can’t prepare a query without more than one argument. Here’s a correct example:

 

$wpdb->prepare( “SELECT * FROM table WHERE id = %d”, $id );

 

This wasn’t a decision done lightly. We don’t like shoving PHP warnings into the faces of users and developers. But given

the potential security risks, we wanted everyone to immediately look at how they are running queries. And, of course, always

prepare them properly.

I love that last paragraph (I could argue with the grammar too, but hey). If they “don’t like shoving PHP warnings into the faces of users and developers” why do so? In doing this they have drawn attention to the following:

  • the website is running wordpress 3.5 (a rather new version which may have other problems than this).
  • the website manager has not turned off php error reporting (and thus may be inexperienced or otherwise lax in his/her security stance).
  • the site is running one (or more) plugins which may be vulnerable to SQL injection attacks.
  • internal path structures on those websites are now exposed for all to see.

They have also made a lot of websites look bloody awful. Take this as an example:

wordpress-error

Thanks guys. Way to go. And despite what you say, lots of sites now look broken.

Permanent link to this article: https://baldric.net/2012/12/19/no-sites-are-broken/

Dec 14 2012

password theft

I have mentioned odd postings to bugtraq before. Today, one “gsuberland” added to the canon with a gem about the Netgear WGR614 wireless router. He says in his post that he has been “reverse engineering” this router. Now for most bugtraq posters (and readers) this would mean that he has been disassembling the firmware. But no, Mr “gsuberland” has actually been pulling apart the hardware.

He says:

After disassembling the device and identifying the components, I noticed that the firmware and all settings are stored on a

single EEPROM (flash) chip, specifically a Macronix MX25L1605 SPI 16Mbit EEPROM. Other variants of the device may

use slightly different chips, but they all seem to use SPI EEPROMs with identical I/O commands. I de-soldered the IC and

hooked it up to a BusPirate, and used it to extract the entire contents of the chip.

He went on to say

I quickly discovered two interesting things:

First, I found a hard-coded credential used for direct serial programming. Using it requires direct physical access and you

have to solder wires onto the board. Despite this not being particularly interesting, this issue has been assigned as

CVE-2012-6340 anyway. It’s always good to have the information out there.

Oh yes. Indeed it is.

Second, I noticed that there were multiple copies of my config file, and all passwords (for both control panel and wifi) within

them are plain-text. It turns out that, in order to prevent config file corruption, the router re-generates the entire config file

and writes a new copy directly after the previous one. It then activates the new config, and soft-deletes the old file by

removing its entry from a list. Once you’ve changed the config several times (about 11 on this device), it hits the end of the

flash chip’s storage and cycles back to the original address. However, it does not actually wipe the old config files.

This issue, assigned CVE-2012-6341, results in the ability to recover all previously used passwords on the device, for both

the control panel and WEP/WPA/WPA2, in plaintext. A factory reset does not fix this; it simply restores a default config file

onto the lower address.

Now the best bit.

As such, an attacker who steals the device may recover the last-used passwords and config, as well as many previous

passwords and configuration data. There also seems to be some storage of DHCP client information, but the data I have is

inconclusive due to it being partially overwritten.

Now I can see that it is not good that Netgear should store passwords in clear. But if the only way anyone can get these passwords is to physically steal my router and pull it apart, I might notice that it had gone. Moreover, the stolen passwords could not be used in a remote attack against me because I no longer have the damned thing.

Permanent link to this article: https://baldric.net/2012/12/14/password-theft/

Dec 12 2012

horse-terity

Steve Bell’s If… cartoon in yesterday’s Guardian was wonderful, and wonderfully opportune in its timing given my letter to Jeff Bezos. I wanted to include a copy here so I emailed Steve this morning asking for permission. Astonishingly he replied, saying yes, only minutes after I had sent my request (clearly he wasn’t very busy…..)

I am immensely grateful to Steve for his permission to republish here.

Permanent link to this article: https://baldric.net/2012/12/12/horse-terity/

Dec 11 2012

moonlighting in parliament

Yesterday I followed a link from Duncan Campbell’s Reg article on the joint parliamentary committee’s scrutiny of the Communications Data Bill referred to in my post below. That link took me to the UK Parliamentary website which I confess I haven’t visited in a while. I was initially irritated that the video format used on that site used Microsoft Windows Media for both video and audio. This meant that I would need to use either Windows Media Player, or install Microsoft silverlight, which is sort of tricky on non-Microsoft platforms. However, I was pleasantly surprised to find that the Parliamentary authorities recognised that some viewers would be using such platforms. The page including the embedded video referred specifically to the fact that the video required the use of Microsoft Silverlight or Microsoft Media Player and pointed non-Microsoft users to their help page. That page in turn points Linux users to the Novell Moonlight site where they can obtain the open source implementation of Microsoft’s silverlight.

Now much as I might deplore the use of proprietary software (and related codecs) on a site such as the UK Parliament’s, they are to be congratulated in actually recognising that there are viewers who will not be using Microsoft products and in providing pointers to alternatives (even if the debian purists will still be excluded).

Interesting video nonetheless. Worth watching.

Permanent link to this article: https://baldric.net/2012/12/11/moonlighting-in-parliament/

Dec 10 2012

tor and the UK data communications bill

As a Tor node operator, I have an interest in how the draft UK Data Communications Bill would affect me should it be passed into law. In particular, I would be worried if Tor ended up being treated as a “telecommunications operator” within the terms of the Act (should it become an Act).

Fortunately, Steven Murdoch, Tor’s Chief Research Officer believes that on the evidence to date, that is not likely because the nodes which carry data are not run by the Tor Project itself, rather they are run by volunteers (i.e. people like me). He records in a blog entry the evidence he submitted on the Tor project’s behalf to the Joint Parliamentary Select Committee (JPSC) on the Draft Data Communications Bill. In that blog post, Steven also points to the transcript of the session (PDF file) at which he gave oral evidence to that committee. It makes for interesting reading (and not just for Steven’s comments. I was intrigued by some of the evidence offered by representatives from Facebook, Twitter and Skype/Microsoft).

When asked specifically by committee member Dr Julian Huppert what the consequences of the Bill would be for Tor, Steven said:

In some sense that is difficult to say, because the Bill does not go into very much detail. The Tor Project or architecture is very different from these other systems. We do not process the communications of the users. People who use the Tor network download the software from us, but their communications go over servers that are operated by volunteers to which we do not have access. Because we have designed the system to have privacy from the start, we would not technically be able to hand over any communications data, regardless of whether we were ordered to do so. Whether we fall under the legislation at all is an open question, and we have not had any guidance as to what they believe. If we did, we would have to substantially change the architecture of Tor, and basically rewrite it from scratch, (my emphasis) before we could do anything useful to provide communications data. Long before that happened, our funders would pull out and the users would pull out and the project would effectively cease.

Moreover, when questioned later by another member, Lord Strasburger, for clarification about Tor’s capability to pass meaningful information to UK LEAs following a request, he went on to say:

under the current design we do not have access to those data. We would have to build something different from the beginning before we could be in a position to collect any of them.

Steven clearly believes that the JPSC understands some of the difficulties inherent in the Bill as currently drafted. He concludes in his blog article:

Based on the discussions which have taken place, it appears that the committee has serious reservations about the bill but we will not know for sure until the committee publishes their report.

According to the Parliamentary website, the Committee has agreed its report and it is currently being prepared for publication.

Permanent link to this article: https://baldric.net/2012/12/10/tor-and-the-uk-data-communications-bill/

Dec 07 2012

an open letter to Jeff Bezos

Dear Jeff

You don’t know me(*), but I used to be a customer of yours. I say “used to be” because I am no more, nor shall I be again. I have today closed my Amazon account (and I must say that you do not make that process very easy).

You may be aware that the UK, in common with many other economies, is currently going through a rather tough recession. Indeed, our Chancellor of the Exchequer, one George Gideon Oliver Osborne MP (salary c. £145,000, estimated personal fortune of around £4 million from a family trust), is finding the economic situation in the UK so dire that he has been forced to introduce measures to cap benefits to some of the most disadvantaged people in our country. In bringing this action, he has said that it was about “being fair to the person who leaves home every morning to go out to work and sees their neighbour still asleep, living a life on benefits… we have to have a welfare system that is fair to the working people who pay for it”. Personally, I think that increasing taxation on those companies and individuals who can afford it would be a better idea, but hey, that’s just me.

The benefits cap will apparently save some £1.43bn a year by 2014-15. One of our major charities has calculated that this saving to the exchequer will cost jobless families with two children, who are already living on the breadline, some £430 per annum.

“So what has this to do with me?” And “Why are you telling me this?” you may ask.

Simple. You have arranged Amazon’s corporate structure to minimise the tax paid on economic activity in the UK.

Since moving ownership of the UK operation to Luxembourg in 2006, the UK operation has been classed as an “order fulfilment’ business”, a delivery organisation. That has enabled you to avoid paying any UK corporation tax on turnover in the UK estimated at £3.3bn last year alone.

I recognise that this is perfectly legal, and that as a public company you are obliged to minimise liabilities so that you can maximise income for your shareholders, but as Margaret Hodge, the chair of the UK Parliamentary Public Accounts Committee said recently:

“Global corporations with huge operations in the UK generating significant amounts of income are getting away with paying

little or no corporation tax here. This is an insult to British business and individuals who pay their fair share.

“The inescapable conclusion is that multinationals are using structures and exploiting current tax legislation to move offshore

profits that are clearly generated from economic activity in the UK,”

Ms Hodge also seemed less than impressed with your director of public policy, Andrew Cecil. Perhaps you should have a word with him.

Now since I have contributed for some time to the income on which you avoid paying UK taxation, I decided to check just how much I had contributed. Astonishingly, I seem to have spent over £1700 with you over the past few years, Indeed in 2011 alone I spent nearly £500. That is quite a lot of CDs and books. And £500 is more than the sum which is about to be taken from the families on benefits described above. Of course, my contribution alone, even were it taxed correctly, would not do much to help our hard pressed Chancellor of the Exchequer. But if lots of people like me, who can afford to spend £500 per annum on what a person on benefits might see as unaffordable luxuries, were to stop spending that money with you and instead spend it with companies which do contribute their fair share to UK taxation, it might help a little, don’t you think?

So I have decided to do exactly that.

Goodbye and Merry Christmas.

Mick

(Oh and BTW, I have deregistered the Kindle. Not that I ever bought anything on it from you anyway.)

(*) Actually, you do know quite a bit about me. And more than I am comfortable about. But I was somewhat reassured some time ago when you apparently got me very wrong.

Permanent link to this article: https://baldric.net/2012/12/07/an-open-letter-to-jeff-bezos/

Nov 29 2012

why I read el reg

The Register today carries a story about the discovery of a truly massive black hole named “SDSS J1106+1939” at the center of a dying galaxy. That black hole is reported to be emitting energy “equivalent to two trillion times the power output of the sun.”

The story itself is classic Reg material and is covered in only rather scant detail, but the commentary by Reg readers make the story worthwhile.

“frank ly” says:

Big and complicated science made accessible.

” … while our Milky Way galaxy has a relatively piddly black hole at its heart, the one found in quasar SDSS J1106+1939 is

whopping.”

That’s why I read The Register.

“gizmo23” follows with some explanatory arithmetic:

Head crunching numbers

So, off the top of my head the sun’s power is about 3 x 10^26 W and there are about 10^11 stars in our galaxy…

and to top it all, it’s 100 times more power than that which is 3 x 10^39 W, divide by c^2 (lets say 10^17) equals 3 x 10^22

kg of matter converted to energy per second… And as the Earth’s mass is approx 6 x 10^24 kg that’s 1/200 the mass of the

earth converted from matter to energy every second.. ooh my brain hurts

and Neil Barnes notes:

Energy output like that…

You could make toast really really quickly.

Just sayin’.

That quality of discussion is just not available in run of the mill publications such as Nature, or Scientific American.

Permanent link to this article: https://baldric.net/2012/11/29/why-i-read-el-reg/

Nov 27 2012

what gives with dban?

Recently I have been faced with the need to wipe a bunch of hard disks removed from some old (indeed, in one or two cases, very old) PCs before disposal. Normally I would have used DBAN to do this because it gives me a nice warm feeling that I have taken all reasonable steps and no-one (short of a very specialist forensic recovery laboratory) is likely to be able to read the disks thereafter. The latest version of DBAN (post version 2.0) even (supposedly) supports USB disks. I say supposedly because I couldn’t get it to work on my setup.

I have a USB to SATA/IDE disk converter which is very useful for quickly mounting disks which would otherwise need to be installed internally. The fact that DBAN can now handle such disks made it a no brainer choice when I was faced with the task of wiping half a dozen assorted disks of varying vintage. Howver, every disk I tried resulted in a DBAN failure followed by an advertising page for the commercial disk erasure product called blancco. Worse, it appears that Darik’s own web page now advertises blancco as a preferred product. Wierd.

Given the DBAN failure I had to fall back on dd. At least that works. And a wipe once with dd (dd if=/dev/zero of=/dev/sdb), followed by a repartition and reformat with gparted then another dd, this time with random data (dd if=/dev/urandom of=/dev/sdb) is probably enough to stop the the 1992 Parish Council minutes appearing anywhere they shouldn’t.

But oh boy, dd is slow when using /dev/urandom – so I cheated and stuck to /dev/zero.

Permanent link to this article: https://baldric.net/2012/11/27/what-gives-with-dban/

Nov 22 2012

why mysql crashed

Update to previous post

According to bytemark, a bug in the bigV system caused some disk corruptions for a small number of customers. It seems that I was just one of the unlucky ones. I am assured that all is now well.

Permanent link to this article: https://baldric.net/2012/11/22/why-mysql-crashed/

Nov 16 2012

forcing innodb recovery in mysql

Today I had a nasty looking problem with my mysql installation. At first I thought I might have to drop one or more databases and re-install. Fortunately, I didn’t actually have to do that in the end.

I first noticed a problem at around 15.45 today when I couldn’t collect my mail. My mail system on this VM uses postfix as the smtp component and dovecot for IMAPS/POP3S delivery. A quick check showed that the server was up and postfix and dovecot were both running. Nothing immediately obviously wrong. However, a check on the mail log showed multiple entries of the form:

dovecot: auth-worker(3078): Error: mysql(localhost): Connect failed to database (mail): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’

An attempt to connect to trivia also resulted in a “database error” message. So on checking again I noticed that of course mysql wasn’t running. Unfortunately, all my attempts to restart it failed. Further investigation of my logs (in particular, an ominously large /var/log/error) showed that I had a corrupt InnoDB page. As soon as mysql reached this it barfed and fell over. Not good. I don’t like database errors because I’m no database expert. But at least that explained the mail error. Both components of my mail system rely on a running mysql server because I store users, passwords, aliases, domain details etc. in mysql databases.

The largest database on this VM is, of course, the blog. I keep regular backups of that so in extremis I could dump the old database and reload from a backup and loose only my daily stats, But I was reluctant to do that without first checking to see if a repair was possible. I then spent some long time reading man pages, my O’Reily MySQL book, and the on-line MySQL reference pages. The best clue I received was the message in the error log:

Nov 16 15:57:05 pipe mysqld: InnoDB: If the corrupt page is an index page
Nov 16 15:57:05 pipe mysqld: InnoDB: you can also try to fix the corruption
Nov 16 15:57:05 pipe mysqld: InnoDB: by dumping, dropping, and reimporting
Nov 16 15:57:05 pipe mysqld: InnoDB: the corrupt table. You can use CHECK
Nov 16 15:57:05 pipe mysqld: InnoDB: TABLE to scan your table for corruption.
Nov 16 15:57:05 pipe mysqld: InnoDB: See also https://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Nov 16 15:57:05 pipe mysqld: InnoDB: about forcing recovery.
Nov 16 15:57:05 pipe mysqld: InnoDB: Ending processing because of a corrupt database page.

That message appeared just before a load of stacktrace information which looked horribly meaningless to me but spoke of deep, unfathomable database wrongness. A read of the referenced mysql manual page didn’t initially reassure me over much either. It starts:

If there is database page corruption, you may want to dump your tables from the database with SELECT … INTO OUTFILE. Usually, most of the data obtained in this way is intact.

“Usually” eh? Hmmm.

As suggested in the documentation I added the line “innodb_force_recovery = 1” to the [mysqld] section of my /etc/mysql/mysql,cnf config file and restarted the server. I started with the lowest non-zero option in the hope that this would cause the least corruption if I had to dump the tables. This time, mysql came up when I started it, but the error log contained the following:

Nov 16 18:09:26 pipe mysqld: InnoDB: A new raw disk partition was initialized or
Nov 16 18:09:26 pipe mysqld: InnoDB: innodb_force_recovery is on: we do not allow
Nov 16 18:09:26 pipe mysqld: InnoDB: database modifications by the user. Shut down
Nov 16 18:09:26 pipe mysqld: InnoDB: mysqld and edit my.cnf so that newraw is replaced
Nov 16 18:09:26 pipe mysqld: InnoDB: with raw, and innodb_force_… is removed.

Now there is nothing in my my.cnf about “raw” or “newraw”, so I simply removed the “innodb_force_recovery” line and shutdown and restarted mysql. Mysql started up without error, and without my having to dump any database. And no, I have no idea why. I can only assume that the force_recovery option forced some database repair as well as the documented forcing of the InnoDB storage engine to start.

And I don’t yet know what caused the problem in the first place, but since my logs show that the VM went down and restarted at around 14.36 I conclude that a failure somewhere in the VM system occurred during a database write and that screwed things up. I’m waiting for a response to my support call.

So. If you too ever face a database corruption similar to mine, do not panic. And do not attempt unnecessarily drastic database drops and reloads until you have at least tried the “innodb_force_recovery” option in your my.cnf configuration.

It may of course just be magic. Most database incantations are as far as I am concerned.

Permanent link to this article: https://baldric.net/2012/11/16/forcing-innodb-recovery-in-mysql/

Oct 27 2012

using openvpn to bypass NAT firewalls

OpenVPN is a free, open source, general purpose VPN tool which allows users to build secure tunnels through insecure networks such as the internet. It is the ideal solution to a wide range of secure tunnelling requirements, but it is not always immediately obvious how it should be deployed in some circumstances.

Recently, a correspondent on the Anglia Linux User Group (ALUG) mailing list posed a question which at first sight seemed easy to answer. He wanted to connect from one internet connected system to another which was behind a NAT firewall (actually, it turned out to be behind two NAT firewalls, one of which he didn’t control, and therein lay the difficulty).

The scenario was something like this:

He wanted to connect from the system called “Client” in the network on the left to the secured system called “Host A” on the network on the right of the diagram. We can assume that both networks use RFC 1918 reserved addresses and that both are behind NAT firewalls (so routers A and C at the least are doing NAT).

Ordinarily, this would be pretty straightforward. All we have to do is run an SSH daemon (or indeed openVPN) on Host A and set up port forwarding rules on routers A and B to forward the connection to the host. So long as we have appropriate firewall rules on both the host and the routers, and the SSH/OpenVPN daemons are well configured we can be reasonably confident that the inbound connection is secure (for some definition of “secure”). This is exactly the setup I use on my home network. I have an openVPN server running on one of my debian systems. When out and about I can connect to my home network securely over that VPN from my netbook.

However, as I noted above, the problem in this scenario is that the owner of Host A did not control the outermost NAT device (Router A) so could not set up the necessary rule. Here is where openVPN’s flexibility comes in. Given that both networks are internet connected, and can make outbound connections with little difficulty, all we need to do is set up an intermediary openVPN host somewhere on the net. A cheap VPS is the obvious solution here (and is the one I used). Here’s how to accomplish what we want:

  • install openVPN server on the VPS;
  • install openVPN client on Host A;
  • set up the openVPN tunnel from Host A to the VPS;
  • connect over SSH from Client to VPS;
  • connect (using SSH again) over the openVPN tunnel from the VPS to Host A.

Using SSH over an already encrypted tunnel may seem like overkill, but it has the advantage that we can leverage the existing security mechanisms on Host A (and we really don’t want a telnet daemon listening there).

Installing openVPN

There are already a huge range of openVPN “HowTos” out there so I won’t add to that list here. The very comprehensive official HowTo on the openVPN website lists all that you need to know to install and configure openVPN to meet a wide variety of needs, but it can be a bit daunting to newcomers. OpenVPN has a multitude of configuration options so it is probably best to follow one of the smaller, distro specific, howtos instead. Two I have found most useful are for debian and arch. And of course, Martin Brooks, one of my fellow LUGgers has written quite a nice guide too. Note, however, that Martin’s configuration does not use client side certificates as I do here.

By way of example, the server and client configuration files I built when I was testing this setup are given below. Note that I used the PKI architectural model, not the simpler static keys approach. As the main howto points out, the static key approach doesn’t scale well, is not as secure as we’d like (keys must be stored in plain text on the server), but most importantly, it doesn’t give us perfect forward secrecy so any key compromise would result in complete disclosure of all previous encrypted sessions. Note also that I chose to change the keysize ($KEY_SIZE) in the file “vars” to 2048 from the default 1024. If you do this, the build of the CA certificate and server and client keys warns you that the build “may take a long time”. In fact, on a system with even quite limited resources, this only takes a minute or two. Of course, it should go without saying that the build process should be done on a system which is a secure as you can make it and which gives you a secure channel for passing keys around afterwards. There is little point in using a VPN for which the keys have been compromised. It is also worth ensuring that the root CA key (“ca.key” used for signing certificates) is stored securely away from the server. So if you build the CA and server/client certificates on the server itself, make sure that you copy the ca key securely to another location and delete it from the server. It doesn’t need to be there. I chose /not/ to add passwords to the client certificates because the client I was testing from (emulating Host A) is already well secured (!). In reality, however, it is likely that you would wish to strengthen the security of the client by insisting on passwords.

server.conf on the VPS

# openvpn server conf file for VPS intermediary

# Which local IP address should OpenVPN listen on (this should be the public IP address of the VPS)

local XXX.XXX.XXX.XXX

# use the default port. Note that we firewall this on the VPS so that only the public IP address of the

# the network hosting “Host A” (i.e. the public address of “Router A”) is allowed to connect. Yes this exposes

# the VPN server to the rest of the network behend Router A, but it is a smaller set than the whole internet.

port 1194

# and use UDP as the transport because it is slightly more efficient than tcp, particularly when routing.

# I don’t see how tcp over tcp can be efficient.

proto udp

# and a tun device because we are routing not bridging (we could use a tap, but it is not necessary when routing)

dev tun

# key details. Note that here we built 2048 bit diffie hellman keys, not the default 1024 (edit $KEY_SIZE in the

# file “vars” before build)

ca /etc/openvpn/keys/ca.crt

dh /etc/openvpn/keys/dh2048.pem

cert /etc/openvpn/keys/vps-server-name.crt

key /etc/openvpn/keys/vps-server-name.key

# now use tls-auth HMAC signature to give us an additional level of security.

# Note that the parameter “0” here must be matched byy “1” at the client end

tls-auth /etc/openvpn/keys/ta.key 0

# Configure server mode and supply a VPN subnet for OpenVPN to draw client addresses from.

# Our server will be on 172.16.10.1. Be careful here to choose an RFC1918 network which will not clash with

# that in use at the client end (or Host A in our scenario)

server 172.16.10.0 255.255.255.0

# Maintain a record of client virtual IP address associations in this file. If OpenVPN goes down or is restarted,

# reconnecting clients can be assigned the same virtual IP address from the pool that was previously assigned.

ifconfig-pool-persist ipp.txt

# We do /not/ push routes to the client because we are on a public network, not a reserved internal net.

# The default configuration file allows for this in the (example) stanzas below (commented out here)

# push “route 192.168.1.0 255.255.255.0”

# push “route 192.168.2.0 255.255.255.0”

# Nor do we tell the client (Host A) to redirect all its traffic through the VPN (as could be done). The purpose of this

# server is to allow us to reach a firewalled “client” on a protected network. These directives /would/ be useful if we

# wanted to use the VPS as a proxy to the outside world.

# push “redirect-gateway def1”

# push “dhcp-option DNS XXX.XXX.XXX.XXX”

# Nor do we want different clients to be able to see each other. So this remains commented out.

# client-to-client

# Check that both ends are up by “pinging” every 10 seconds. Assume that remote peer is down if no ping

# received during a 120 second time period.

keepalive 10 120

# The cryptographic cipher we are using. Blowfish is the default. We must, of course, use the same cipher at each end.

cipher BF-CBC

# Use compression over the link. Again, the client (Host A) must do the same.

comp-lzo

# We can usefully limit the number of allowed connections to 1 here.

max-clients 1

# Drop root privileges immediately and run as unpriviliged user/group

user nobody

group nogroup

# Try to preserve some state across restarts.

persist-key

persist-tun

# keep a log of the status of connections. This can be particularly helpful during the testing stage. It can also

# be used to check the IP address of the far end client (Host A in our case). Look for lines like this:

#

# Virtual Address,Common Name,Real Address,Last Ref

# 172.16.10.6,client-name,XXX.XXX.XXX.XXX:nnnnn,Thu Oct 25 18:24:18 2012

#

# where “client name” is the name of the client configuration, XXX.XXX.XXX.XXX:nnnnn is the public IP address of the

# client (or Router A) and port number of the connection. This means that the actual client address we have a connection

# to is 172.16.10.6. We need this address when connecting from the VPS to Host A.

status /var/log/openvpn-status.log

# log our activity to this file in append mode

log-append /var/log/openvpn.log

# Set the appropriate level of log file verbosity

verb 3

# Silence repeating messages. At most 20 sequential messages of the same message category will be output to the log.

mute 20

# end of server configuration

Now the client end.

client.conf on Host A

# client side openvpn conf file

# we are a client

client

# and we are using a tun interface to match the server

dev tun

# and similarly tunneling over udp

proto udp

# the ip address (and port used) of the VPS server

remote XXX.XXX.XXX.XXX 1194

# if we specify the remote server by name, rather than by IP address as we have done here, then this

# directive can be useful since it tells the client to keep on trying to resolve the address.

# Not really necessary in our case, but harmless to leave it in.

resolv-retry infinite

# pick a local port number at random rather than bind to a specific port.

nobind

# drop all priviliges

user nobody

group nogroup

# preserve state

persist-key

persist-tun

# stop warnings about duplicate packets

mute-replay-warnings

# now the SSL/TLS parameters. First the server then our client details.

ca /home/client/.openvpn/keys/server.ca.crt

cert /home/cient/.openvpn/keys/client.crt

key /home/client/.openvpn/keys/client.key

# now add tls auth HMAC signature to give us additional security.

# Note that the parameter to ta.key is “1” to match the “0” at the server end.

tls-auth /home/client/.openvpn/keys/ta.key 1

# Verify server certificate by checking.

remote-cert-tls server

# use the same crypto cypher as the server

cipher BF-CBC

# and also use compression over the link to match the server.

comp-lzo

# and keep local logs

status /var/log/openvpn-status.log

log-append /var/log/openvpn.log

verb 3

mute 20

# end

Tunneling from Host A to the VPS.

Now that we have completed configuration files for both server and client, we can try setting up the tunnel from “Host A”. In order to do so, we must, of course, have openVPN installed on the Host A and we must have copied the required keys and certificates to the directory specified in the client configuration file.

At the server end, we must ensure that we can forward packets over the tun interface to/from the eth0 interface. Check that “/proc/sys/net/ipv4/ip_forward” is set to “1”, if it is not, then (as root) do “echo 1 > /proc/sys/net/ipv4/ip_forward” and then ensure this is made permanent by uncommenting the line “net.ipv4.ip_forward=1″ in /etc/sysctl.conf”. We also need to ensure that neither the server, nor the client block traffic over the VPN. According to the openVPN website, over 90% of all connectivity problems with openVPN are caused not by configuration problems in the tool itself, but by firewall rules.

If you have an iptables script on the server such as I use, then ensure that:

  • the VPN port is open to connections from the public address of Host A (actually Router A in the diagram);
  • you have modified the anti-spoof rules which would otherwise block traffic from RFC 1918 networks;
  • you allow forwarding over the tun interface.

The last point can be covered if you add rules like:

$IPTABLES -A INPUT -i tun0 -j ACCEPT
$IPTABLES -A OUTPUT -o tun0 -j ACCEPT
$IPTABLES -A FORWARD -o tun0 -j ACCEPT

This would allow all traffic over the tunnel. Once we know it works, we can modify the rules to restrict traffic to only those addresses we trust. If you have a tight ruleset which only permits ICMP to/from the eth0 IP address on the VPS, then you may wish to modify that to allow it to/from the tun0 address as well or testing may be difficult.

Many “howtos” out there also suggest that you should add a NAT masquerade rule of the form:

“$IPTABLES -t nat -A POSTROUTING -s 172.16.10.0/24 -o eth0 -j MASQUERADE”

to allow traffic from the tunnel out to the wider network (or the internet if the server is so connected). We do not need to do that here because we are simply setting up a mechanism to allow connection through the VPS to Host A and we can use the VPN assigned addresses to do that.

Having modified the server end, we must make similarly appropriate modifications to any firewall rules at the client end before testing. Once that is complete, we can start openVPN at the server. I like to do this using the init.d run control script whilst I run a tail -f on the log file to watch progress. Watch for lines like “TUN/TAP device tun0 opened” and “Initialization Sequence Completed” for signs of success. We can then check that the tun interface is up with ifconfig, or ip addr. Given the configuration used in the server file above, we should see that the tun0 interface is up and has been assigned address 172.16.10.1.

Now to set up the tunnel from the client end, we can run (as root) “openvpn client.conf” in one terminal window and we can then check in another window that the tun0 interface is up and has been assigned an appropriate IP address. In our case that turns out to be 172.16.10.6. It should now be possible to ping 172.16.10.1 from the client and 172.16.10.6 from the server.

Connecting over SSH from Client to VPS.

We should already be doing this! I am assuming that all configuration on the VPS has been done over an SSH connection from “Client” in the diagram. But the SSH daemon on the VPS may be configured in such a way that it will not permit connections over the VPN from “Host A”. Check (with netstat) that the daemon is listening on all network addresses rather than just the public IP address assigned to eth0. If it is not, then you will need to modify the /etc/ssh/sshd_config file to ensure that “ListenAddress” is set to “0.0.0.0”. And if you limit connections to the SSH daemon with something like tcpwrappers, then check that the /etc/hosts.allow and /etc/hosts.deny files will actually permit connections to the daemon listening on 172.16.10.1. Once we are convinced that the server SSH configuration is correct, we can try a connection from the Host A to 172.16.10.1. If it all works, we can move on to configuring Host A.

Connect (using SSH again) over the openVPN tunnel from the VPS to Host A.

This is the crucial connection and is why we have gone to all this trouble in the first place. We must ensure that Host A is running an SSH daemon, configured to allow connections in over the tunnel from the server with address 172.16.10.1. So we need to make the same checks as we have undertaken on the VPS. Once we have this correctly configured we can connect over SSH from the server at 172.16.10.1 to Host A listening on 172.16.10.6.

Job done.

Addendum: 26 March 2013. For an alternative mechanism to achieve the same ends, see my later post describing ssh tunnelling.

Permanent link to this article: https://baldric.net/2012/10/27/using-openvpn-to-bypass-nat-firewalls/

Oct 23 2012

book theft

Back in August 2011. I wrote about my preference for real books over the emerging electronic version. In that post I noted that Amazon had famously deleted copies of Orwell’s 1984 and Animal Farm purchased by some customers. It now appears that Amazon has gone even further and deleted the entire contents of a Norwegian customer’s kindle. In Cory Doctorow’s post he points out that:

Reading without surveillance, publishing without after-the-fact censorship, owning books without having to account for your ongoing use of them: these are rights that are older than copyright. They predate publishing. They are fundamentals that every bookseller, every publisher, every distributor, every reader, should desire. They are foundational to a free press and to a free society.

Now I’d be pretty angry if someone sold me a book, but later stole that book back on the grounds that I had somehow infringed some sales condition buried in a contract I had implicitly (and forcedly) entered into by the act of purchase. But I would be absolutely livid if, in the act of stealing back “their” book, they also removed the rest of my library. Amazon, however, seems to find this acceptable.

Doctorow went on to say that encrypting storage on mobile devices was much preferable to the option of remote deletion in case of loss. I agree. Unfortunately I also agree with his view that users will have difficulty with password protected encrypted filesystems, and I am completely with him when he says:

If it’s a choice between paving the way for tyranny and risking the loss of your digital life at the press of a button by some deceived customer service rep, and having to remember a password, I think the password is the way to go. The former works better, but the latter fails better.

My own kindle only has the DRM free content I originally uploaded (over a USB connection) after my wife bought it for me. And the wifi is resolutely turned off. But I don’t know why I bothered, because I still haven’t used it, despite taking it on holiday. And now, like Adrian Short, I never will.

Permanent link to this article: https://baldric.net/2012/10/23/book-theft/

Oct 20 2012

grep -R doesn’t search amazon

Towards the end of last month, following the release of the unity lens in ubuntu which searches amazon, “akeane” posted a bug report on launchpad complaining that “grep -R doesn’t automatically search amazon”. In his first posting he said:

Dear “root owning” overlords,

When using grep recursively I only get local results:

grep -R fish_t /home/noob/fish_game/*

/home/noob/fish_game/fish.h: struct fish_t { /home/noob/fish_game/fish.c: struct fish_t eric_the_ fish;

or worse:

grep -R shark_t /home/noob/fish_game/*/home/noob/fish_game/fish.h: struct shark_t { /home/noob/fish_game/fish.c: struct

shark_t_t mark_sw;

I declare this a bug for two reasons:

1. The output is boring.

2. The terminal has more than 2 lines!!! It’s an unefficient use of my screenspace.

I believe the reason for this is that the grep command only searches locally for things I am actually looking for, I kind of

expect the results I get from my codebase and as such it removes any sense of mystery or something new and exciting to

spice up my dull geek existence. That’s boring, grep -R should also search amazon, so I get more exciting results such as:

Shark Season 1 Starring Steven Eckholdt, Nora Dunn, Patrick Fabian, et al.

Amazon Instant Video to buy episodes: $1.99 to buy season: $34.99 ($1.59 per episode)

This bug report has been added to over the past few weeks, particularly after a reference was posted to reddit. As at today’s date, the bug is reported to “affect 169 people”.

The main thrust of akeane’s complaint is that the command line is lacking functionality available in the GUI. He finds that annoying and others agree. On 21 October, “rupa” wrote

I’ve been following this bug with some interest. A lot of good ideas here, but there are some technical issues that haven’t

been addressed.

Until all the common unix utilities can be upgraded to be affiliate-aware, common pipe operations might suffer from related

amazon results. if you pipe grep into awk for example, without being able to be sure of the results, things can get messy.

I feel this issue can be solved by introducing a new ‘default’ file descriptor. In addition to ‘stdout’ and ‘stderr’, I propose a

‘stdaffiliate’ file descriptor, and amazon results can go there. This would allow us to see relevant results in the terminal, but

keep them out of pipes unless purposely redirected.

Here I must disagree. In my view, all amazon (or related adware) results should be piped directly to /dev/null. Of course, stdaffiliate could be linked directly to /dev/null and all would be well.

Permanent link to this article: https://baldric.net/2012/10/20/grep-r-doesnt-search-amazon/