Jan 14 2013
The ascii art in sl was quite cute, this is far more elaborate.
To see it in action, telnet to towel.blinkenlights.nl.
Why waste your time browsing rubbish websites – fire up a terminal session and waste even more time with telnet. More fun places to try can be found at telnet.org.
Permanent link to this article: https://baldric.net/2013/01/14/a-long-time-ago-in-a-galaxy-far-far-away/
Jan 13 2013
During the new year period when I was having a little local difficulty with thrustVPS, I started looking around for alternative providers. My first port of call was lowendbox. That site lists many VPS providers and is often used by suppliers to advertise “special deals” for short periods. Indeed, I think I intially found thrust (or damnvps as it then was) on that site. My criteria were: UK based, cheap, high bandwidth, with a choice of OS. I prefer debian but many providers offer only Centos or a limited choice of sometimes outdated distros (ubuntu 8.04 anyone?). Disk capacity is not an issue here – tor won’t use it and my tails mirror will only need enough storage for some recent .iso images.
I much prefer a virtualisation product which will allow me to fully update the OS (including kernel) so that means I tend also to look for Xen in preference to openVZ. Other than that the spec should include a minimum of 256 Meg of RAM (not shared or “burstable”) and a reasonable share of a decent processor. Even given that requirement, there is still a range of offerings around for anywhere from £5.00 to £10.00 pcm. Since I effectively give both services away, I’m prepared to pay a reasonable sum, but I like cheap :-)
Crucially however, the supplier must allow tor on his network. This tends to reduce the choice. Many ISPs don’t like tor, particularly if you operate an exit node. In my experience, it’s not the bandwidth consumption they object to, more the time they have to devote to answering dumb, often automated, “abuse” complaints from others. One of the main reasons I have tended to favour a UK operation is that they seem more tolerant (and understanding) of how tor operates than some competitors overseas. And I really don’t need the aggravation of dealing with a US based DMCA takedown complaint routed through a US provider.
But for the last six months or so, I’ve only operated a relay node. With no exit permitted, I can’t be implicated in any so-called abuse, so both my provider and I get a quieter life but I still provide resources to tor. This means that I could probably afford to look more widely and perhaps source a VPS in another country. The US is particularly well served with cheap offerings, but I still confess a prejudice towards european data centres. Since I already have three VPSs in the UK, perhaps one, or maybe two in mainland europe might be useful.
Whilst I was checking a variety of potential new suppliers, I also checked with the tor-relays mail list for recommendations. One of the suppliers I was pointed to was a relatively new startup called DigitalOcean. Their offering (256 Mb RAM, 1 core, 20 Gig disk, and totally free bandwidth on a 1 Gig network) seemed astonishingly good. One guy on the list reported that he had transitted 15 TiB in one month on a server on their network. Whilst they are US based, DigitalOcean also offer servers in Amsterdam, so of course I just had to try them out. Boy are they fast.
Although they offer you a free trial for a few hours, I figured I could afford $5.00 (or less than £3.20 at current exchange rates) for a month’s trial. At the time of writing (13 January) I have been up just over 12 days and have transitted 3.57 TiB through one of their cheapest servers. My vnstat stats today show:
for a total of around 1200 established tor circuits. My top statistics report:
So I am ticking over quite healthily and shipping a very useful traffic load at just under 25 Mbit/s. Astonishing.
For comparison, a few days ago (on the 4th of January during early testing) I took snapshots of both my existing tor server with thrust and the new one at digitalocean. The image below compares the iptraf stats for each server (digitalocean at the top).
This second image compares the vnstat stats at the same time.
Note that both vnstat and the iptraf snapshots show that the new server is handling around 20 times the traffic going through my thrust server.
However, there is a downside to “unlimited” bandwidth (and besides, it won’t really be unlimited forever). Tor is voracious and will happily use all available resource on a server which does not throttle its connection rate. In early testing I did not set any bandwidth limits on tor. After just a few days I was running at around 35 Mbit/s for over 1600 tor circuits and tor was complaining “Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy.” So I started to explore options which would give the maximum freedom to tor without overloading the VPS.
As an aside, I should say that the “MaxAdvertisedBandwidth config option” doesn’t actually do what the tor manual suggests it should, so setting it in the configuration file is practically useless. However, I found a useful post on the tor-relays archive which suggested that I should advertise a CPU limit (“NumCPU 1”) and set “MaxOnionsPending 250” (up from the default 100) since this would reduce the load on my server of a backlog of “onionskin queue overflows” (apparently often caused by a tor node being temporarily chosen as the best route to a hidden service). For a while, this worked – well at least the log messages went away, which may not actually mean the same thing. A couple of days later however, tor died with no entry in the logfile to explain why.
On further exploration, I found entries in the kernel log of the form “tor invoked oom-killer” followed by details of processes which had been killed as a result, so clearly we were running out of memory. Eventually tor itself died. Time to start setting some bandwidth limits.
I started fairly high (BandwidthRate 4 MB with a burst capability to 4.2 MB) and gradually wound it down over a period of two or three days as I watched top stats and log files. Whilst the VPS eventually seemed able to handle a traffic load of around 2 MB (that is each direction) I finally settled on 1.5 MB with a burst capability of 1.8 MB in the interests of both long term stability and fairness on the VPS providers – after all, $5.00 a month is peanuts for a total traffic load of nearly 9 TiB for that same month.
When first signing up to the trial at digitalocean I had specifically asked if they were happy with a tor node. The answer I got suggested that they would be happy with a transit relay, but less happy with an exit node. Not an unusual response. Following my early testing I decided it might also be smart to ask them if they really didn’t mind my using all that bandwidth for only $5.00. After all, it would be better to find out their actual stance in advance if I were to contemplate a long term service arrangement. They were candid in their reply, which I can summarise as: “it’s OK at the moment, but we are just building our service and the free bandwidth model won’t last. We will be charging in future. We just don’t know details yet”.
So, I’ll stick with them during this phase and see what they come up with. Whatever it is, on a network that fast, and a VPS that cheap, it looks worthwhile my keeping a presence in Amsterdam.
Permanent link to this article: https://baldric.net/2013/01/13/what-a-difference-a-gig-makes/
Jan 11 2013
Today I ran a routine apt-get update/apt-get upgrade on my mailserver and dovecot failed afterwards. This is a “bad thing” (TM). No routine software upgrade should cause a failure of the kind I experienced.
Two things happened which should not have done. Firstly the SSL certificates appeared to have changed (which meant that mail clients immediately complained and flagged the error). Secondly, the back end authentication in my mysql database also seemed to fail. Actually, neither of these symptoms was really true, but nevertheless it took me some time to figure out what had happened and then to fix it.
The SSL certificate error occured because the upgrade replaced one of the configuration files (/etc/dovecot/conf.d/10-ssl.conf) without telling me. This meant that my locally configured SSL certificates, which are stored in a non-standard directory, were ignored and instead the standard /etc/ssl/certs/dovecot.pem and /etc/ssl/private/dovecot.pem were substituted. This is particularly annoying because apt-get normally flags any changed configuration files on an upgrade and asks if you wish to retain your locally changed file or replace it with the new one distributed. It didn’t this time and I got caught out.
Having fixed that problem, I then hit the apparent authentication failure. My mail client said that authentication with the server had failed whilst the server logs said: “Initialization failed: namespace configuration error: inbox=yes namespace missing“. This didn’t initally mean much to me, but a search for the error message brought me examples of others who had hit this problem. In particular, Bill Ryder’s email on the dovecot email list archive gave me the answer I needed to fix the problem. Once I had added the configuration option he suggests to /etc/dovecot/conf.d/10-mail.conf and reloaded dovecot, my problems disappeared.
But this shouldn’t happen. OK, maybe I should have read the changelog before upgrading, but no minor upgrade should so change a previously working configuration that it fails. I nearly dumped dovecot back in 2010 when version 2.xx release superceded
the earlier 1.2.xx version. The changes between those branches was significant and it took me some time to figure out what I needed to do to get a working migration between the two. But at least I expected changes between a version 1.xx and a version 2,xx.
I may now reconsider my decision to stick with dovecot and look again at courier, or even look for another IMAP/POP server entirely.
Permanent link to this article: https://baldric.net/2013/01/11/dovecot-failure/
Jan 10 2013
I have been collecting statistics on trivia’s visitors using counterize for some time. The lastest version, which I have been using for about five months now, allows the user to graph, and publish, site statistics in a nice form. I have today created a new page (listed under “longer trivia” on the right) called trivial statistics. I am gratified that my “how-to” type of content is so popular, but my readership profile is interesting.
Permanent link to this article: https://baldric.net/2013/01/10/my-top-ten/
Jan 07 2013
This afternoon I received an email from Andrew Sinclair at thrustVPS in response to my post below. In the interests of openness I have added his email as a comment to the post. My thanks to Andrew for that email. It is just a shame that I did not get such a helpful response in earlier correspondence.
Permanent link to this article: https://baldric.net/2013/01/07/thrust-respond/
Jan 06 2013
I had an interesting time over the new year period.
For some time now I have run a tor node on a VPS at ThrustVPS. I also run my tails mirror on a VPS at the same provider. Their network has always struck me as pretty poor (response times to ssh login is particularly bad, and the servers sometimes go off-line for no apparent reason) but over the last month I had worse than average performance, and frankly appalling service response.
I initially chose thrust (then known as damnvps for some odd reason) because they offered “1TB of transfer” per month on all their VPSs. Tor is a particularly voracious bandwidth hog and needs to be throttled to keep it in check. Fortunately the configuration file allows this fairly easily and I had mine set to limit my traffic to no more than 30 GiB per day – i.e. less than the supposed monthly limit. My tails mirror is similarly throttled, though I do that in lighttpd and it is a bit more tricky and less accurate
Until recently I have monitored all my servers through munin. But a while back I moved the VPS which acted as the munin master from one place to another and did not get around to reconfiguring munin on that node. Big mistake. It meant that I had no records of the activity on my tails mirror and tor node over a critical period when thrust seemed to be even more unreliable than usual. Munin monitor notwthstanding, I still check my servers pretty regularly and I found that the thrust VPSs seemed to be going off-line too frequently. One server, the tor node, went down on the 19th, 22nd, and 29th of december and the tails mirror also went down on the 22nd. They came back up when rebooted through the web administration interface, but performance was pretty dire. So I complained to support.
My first email to support on the 29th of december elicited an interesting response. They replied:
“We have stopped your server 109.169.77.131 since it is causing DDOS attack to the hostnode 15xnuk. It is related to the performance issue of the host node we have no other choice without stopping you.Please do take proper measures, install the proper anti virus on the server make through scan on your server and update us.”
DDOS eh? So I asked for more detail and got the following in response:
“I have been started your server when you opened the ticket. And we are sorry for you to stop the VPS without any information. We are monitoring this node for the past few hours and if your IP DDOS again we may forced to stop it again. Please do take measures and monitor your VPS.”
In my email to support I had explained that the VPS in question ran a tor node (though they should have known that because I specifically told them that when I paid for it) and that heavy traffic from multiple IP addresses inbound to my tor port was perfectly normal. Moreover, I could not believe that my tor mode would be the only one targeted by a DDOS if someone didn’t like tor (and lots of people don’t) so I checked with colleagues on the tor-relays mailing list. No-one else seemed to be having problems. Nevertheless, I shut tor down while I investigated and fired off another email to support asking them again to answer my earlier questions. In particular I said:
“Specifically,
– is the “DDOS” aimed at my IP address, and my IP address alone (and only to port 443)?
– is all the traffic coming to my IP address from tor nodes, and only tor nodes (this is easy to check, the addresses are public)?
If the traffic has now dropped, and the answer to either question is yes, then this is not a DDOS, it is expected behaviour of
a tor node.I am about to turn tor back on. Watch the traffic build again.”
I received another unhelpful reply:
“I could see that most traffic coming from https. So please monitor the https ports. The host node is stable now so I think the issue get compromised.
Please back to us if you need any further assistance.
Thanks for the understanding.”
Note that at this stage there has been no answer to my questions, nor has there been any explanation of why my tails mirror went down, or the tor node went down on the days before the 29th. But by now I suspected simply that the thrust network was so heavily oversold that whenever my servers came close to using anywhere near their alloted bandwidth, they were arbitrarily shut down. This was confirmed for me by another email from support which said:
“Please note that there exists a script in the node to reboot the servers having high load (thereby causing load spike in the node) inorder to get the load in the node to be stable. And we are considered the high resource using is also as a DDOS attack.”
I interpreted this to mean that any high traffic is seen by thurst as a DDOS and they simply shut down the target IP. Way to go thrust. If there /is/ a DDOS attack on a node then you have just done the attacker’s job for them. Worse, they don’t bother to tell the customer.
By now (the 31st of December) my server was back on-line, but it went down again later that day. So I complained again and got another stupid reply:
“Your server was suspended due to high ddos attack on the host node and it affected our host node performance heavily.
Also it is related to the performance issue of the host node we can’t unsuspend your server for the next 24 hours , please update us after the 24 hours complete then only I can unsuspend your server.”
Which I took to mean that they would not turn my server back on again for another 24 hours.
Along with the stupid reply, I was given a snippet of log traffic to/from my server. Guess what, it showed traffic from high numbered ports on a range of IP addresses to/from my port 443 (I run tor on 443 – thrust support seemed to think that this meant it must be https). To anyone who knows anything about IP networking this is indicative of client systems communicating with a server. In my case, it is tor clients trying to connect through my tor node on their way somewhere else. Thrust don’t get this, or don’t want to. All they see is lots of traffic so they shut it down.
Now note carefully here that at no time have they proved DDOS. My vnstat traffic stats on the VPS show that I am well within my bandwidth allowance for the month and the daily traffic is not unusually high either. The only possible problem is that I am attempting to use the bandwidth they advertise as available, and that I have paid for. I confess I was a little abrupt in my reply to this nonsense and I asked that the issue be escalated to management.
Things then went quiet. I think I pissed off the support team.
As an aside it is worth taking a look at the thrustVPS sales page:
I particularly like the “Friendly, knowledgeable support available 24×7” and “above all we care about you and your customers“.
and the page advertising the service I actually paid for:
Note carefully the claimed “512MB RAM, 15GB Disk Space, 1TB Transfer, 1 IP, UK or US” because this is where it gets interesting. On the 2nd of January I received an email from one Gareth Akehurst saying
“Hello Mick,
Escalations was unavailable during the holiday period, how can I help?
Regards,
Gareth Akehurst”
(so clearly “escalations” don’t work 24/7)
In answer to my response querying why my server had been shut down, Gareth replied:
“This is standard procedure for our clients, all nodes are on a 100mBits network, the node you are currently on shares that connection with 59 other virtual servers, if you are requiring a high traffic service it may be worth considering a low contention vps which the nodes are limited to no more than 25 vpses.”
Note that “all nodes are on a 100mBits network”. So at best, my VPS is on a machine which has 60 VPS instances trying to share a 100Mbit/s network, and at worst, the whole damned network of hardware is on a 100 Mbit backbone. I’ve got one of those at home for pitys sake. Now, since my vnstat daily stats on that VPS show I was running at 3.1 Mbit/s for a daily throughput of around 30 GiB, there is no way on earth that all 60 customers could /ever/ get their full monthly allowance. In fact, if all of us tried, we would be running at nearly twice the total available bandwidth of the backbone. No wonder the network response was crap. And no wonder thrust arbitrarily shut down customers with “high” loads. The network is so hugely oversold it is criminal.
The ISP industry has a pretty shoddy reputation for advertising (see 8 Mbit/s broadband advertising for example) but this is frankly taking the piss. If I buy a product that says I should get 1TB of transfer, I expect to get that. If they can’t provide that (and on their maths, they can’t) then I have been missold. Worse, I have been treated like an idiot and lied to about DDOS attacks. If I hadn’t paid in advance for the two servers I have with thrust, I’d close the account now. But oddly, they do not refund unhappy customers. So I’ll sit and use as much as possible under my current contract and then leave.
Meanwhile, if anyone else is considering thrustvps – just don’t. Look elsewhere. There are much better providers out there. Bytemark is by far the best. Unfortunately they don’t offer a high bandwidth allowance at a price point I am prepared to pay for a service which I give away.
(In the interests of transparency and honesty, I have posted here the entire email exchange between myself and thrust “support”. Read it and weep.)
Happy New Year.
Permanent link to this article: https://baldric.net/2013/01/06/lies-damned-lies-and-vps-traffic/
Dec 24 2012
This time last year I noted that christmas eve was trivia’s birthday. Guess what, it’s that time of year again, and what am I again doing?
Merry Christmas to all^h^h^h both my readers.
Mick
Permanent link to this article: https://baldric.net/2012/12/24/another-anniversary/
Dec 19 2012
Or so the wordpress post at wordpress.org would have us believe. However, I think there is flaw in both their logic, and their decision making here.
I spotted the problem following an upgrade to wordpress 3.5 on a site I use. One of the plugins on that site objected to the upgrade with the following error message:
“Missing argument 2 for wpdb::prepare(), called in /path/to/some/function.php”
That error message was splattered all over the page which called that function. The page is public.
A quick search brought me to the wordpress announcement above. But crucially, and worryingly, it also threw up nearly 10,000 other references. Almost all the referenced pages I checked showed websites with that error message (with variations on the path/to/function) displayed fairly prominently.
The wordpress article explains that the error occurs because some function written by a developer (probably a developer of a theme or plugin) called the “$wpdb->prepare()” wordpress function with only one argument when it is supposed to be called with two. Worse, the article explains that calling the “$wpdb->prepare()” function in this way is insecure because the SQL query it is passing is not properly prepared (despite what the developer may think).
The article says:
“But here’s where the problem lies:
$wpdb->prepare( “SELECT * FROM table WHERE id = $id” );
See the problem? That query isn’t secure! You may think you are “preparing” this query, but you’re not — you’re passing $id
directly into the query, unprepared. And this, right here, is why $wpdb->prepare() now issues a warning if it isn’t called with
more than one argument. Because you can’t prepare a query without more than one argument. Here’s a correct example:
$wpdb->prepare( “SELECT * FROM table WHERE id = %d”, $id );
This wasn’t a decision done lightly. We don’t like shoving PHP warnings into the faces of users and developers. But given
the potential security risks, we wanted everyone to immediately look at how they are running queries. And, of course, always
prepare them properly.
I love that last paragraph (I could argue with the grammar too, but hey). If they “don’t like shoving PHP warnings into the faces of users and developers” why do so? In doing this they have drawn attention to the following:
They have also made a lot of websites look bloody awful. Take this as an example:
Thanks guys. Way to go. And despite what you say, lots of sites now look broken.
Permanent link to this article: https://baldric.net/2012/12/19/no-sites-are-broken/
Dec 14 2012
I have mentioned odd postings to bugtraq before. Today, one “gsuberland” added to the canon with a gem about the Netgear WGR614 wireless router. He says in his post that he has been “reverse engineering” this router. Now for most bugtraq posters (and readers) this would mean that he has been disassembling the firmware. But no, Mr “gsuberland” has actually been pulling apart the hardware.
He says:
After disassembling the device and identifying the components, I noticed that the firmware and all settings are stored on a
single EEPROM (flash) chip, specifically a Macronix MX25L1605 SPI 16Mbit EEPROM. Other variants of the device may
use slightly different chips, but they all seem to use SPI EEPROMs with identical I/O commands. I de-soldered the IC and
hooked it up to a BusPirate, and used it to extract the entire contents of the chip.
He went on to say
I quickly discovered two interesting things:
First, I found a hard-coded credential used for direct serial programming. Using it requires direct physical access and you
have to solder wires onto the board. Despite this not being particularly interesting, this issue has been assigned as
CVE-2012-6340 anyway. It’s always good to have the information out there.
Oh yes. Indeed it is.
Second, I noticed that there were multiple copies of my config file, and all passwords (for both control panel and wifi) within
them are plain-text. It turns out that, in order to prevent config file corruption, the router re-generates the entire config file
and writes a new copy directly after the previous one. It then activates the new config, and soft-deletes the old file by
removing its entry from a list. Once you’ve changed the config several times (about 11 on this device), it hits the end of the
flash chip’s storage and cycles back to the original address. However, it does not actually wipe the old config files.
This issue, assigned CVE-2012-6341, results in the ability to recover all previously used passwords on the device, for both
the control panel and WEP/WPA/WPA2, in plaintext. A factory reset does not fix this; it simply restores a default config file
onto the lower address.
Now the best bit.
As such, an attacker who steals the device may recover the last-used passwords and config, as well as many previous
passwords and configuration data. There also seems to be some storage of DHCP client information, but the data I have is
inconclusive due to it being partially overwritten.
Now I can see that it is not good that Netgear should store passwords in clear. But if the only way anyone can get these passwords is to physically steal my router and pull it apart, I might notice that it had gone. Moreover, the stolen passwords could not be used in a remote attack against me because I no longer have the damned thing.
Permanent link to this article: https://baldric.net/2012/12/14/password-theft/
Dec 12 2012
Steve Bell’s If… cartoon in yesterday’s Guardian was wonderful, and wonderfully opportune in its timing given my letter to Jeff Bezos. I wanted to include a copy here so I emailed Steve this morning asking for permission. Astonishingly he replied, saying yes, only minutes after I had sent my request (clearly he wasn’t very busy…..)
I am immensely grateful to Steve for his permission to republish here.
Permanent link to this article: https://baldric.net/2012/12/12/horse-terity/
Dec 11 2012
Yesterday I followed a link from Duncan Campbell’s Reg article on the joint parliamentary committee’s scrutiny of the Communications Data Bill referred to in my post below. That link took me to the UK Parliamentary website which I confess I haven’t visited in a while. I was initially irritated that the video format used on that site used Microsoft Windows Media for both video and audio. This meant that I would need to use either Windows Media Player, or install Microsoft silverlight, which is sort of tricky on non-Microsoft platforms. However, I was pleasantly surprised to find that the Parliamentary authorities recognised that some viewers would be using such platforms. The page including the embedded video referred specifically to the fact that the video required the use of Microsoft Silverlight or Microsoft Media Player and pointed non-Microsoft users to their help page. That page in turn points Linux users to the Novell Moonlight site where they can obtain the open source implementation of Microsoft’s silverlight.
Now much as I might deplore the use of proprietary software (and related codecs) on a site such as the UK Parliament’s, they are to be congratulated in actually recognising that there are viewers who will not be using Microsoft products and in providing pointers to alternatives (even if the debian purists will still be excluded).
Interesting video nonetheless. Worth watching.
Permanent link to this article: https://baldric.net/2012/12/11/moonlighting-in-parliament/
Dec 10 2012
As a Tor node operator, I have an interest in how the draft UK Data Communications Bill would affect me should it be passed into law. In particular, I would be worried if Tor ended up being treated as a “telecommunications operator” within the terms of the Act (should it become an Act).
Fortunately, Steven Murdoch, Tor’s Chief Research Officer believes that on the evidence to date, that is not likely because the nodes which carry data are not run by the Tor Project itself, rather they are run by volunteers (i.e. people like me). He records in a blog entry the evidence he submitted on the Tor project’s behalf to the Joint Parliamentary Select Committee (JPSC) on the Draft Data Communications Bill. In that blog post, Steven also points to the transcript of the session (PDF file) at which he gave oral evidence to that committee. It makes for interesting reading (and not just for Steven’s comments. I was intrigued by some of the evidence offered by representatives from Facebook, Twitter and Skype/Microsoft).
When asked specifically by committee member Dr Julian Huppert what the consequences of the Bill would be for Tor, Steven said:
In some sense that is difficult to say, because the Bill does not go into very much detail. The Tor Project or architecture is very different from these other systems. We do not process the communications of the users. People who use the Tor network download the software from us, but their communications go over servers that are operated by volunteers to which we do not have access. Because we have designed the system to have privacy from the start, we would not technically be able to hand over any communications data, regardless of whether we were ordered to do so. Whether we fall under the legislation at all is an open question, and we have not had any guidance as to what they believe. If we did, we would have to substantially change the architecture of Tor, and basically rewrite it from scratch, (my emphasis) before we could do anything useful to provide communications data. Long before that happened, our funders would pull out and the users would pull out and the project would effectively cease.
Moreover, when questioned later by another member, Lord Strasburger, for clarification about Tor’s capability to pass meaningful information to UK LEAs following a request, he went on to say:
under the current design we do not have access to those data. We would have to build something different from the beginning before we could be in a position to collect any of them.
Steven clearly believes that the JPSC understands some of the difficulties inherent in the Bill as currently drafted. He concludes in his blog article:
Based on the discussions which have taken place, it appears that the committee has serious reservations about the bill but we will not know for sure until the committee publishes their report.
According to the Parliamentary website, the Committee has agreed its report and it is currently being prepared for publication.
Permanent link to this article: https://baldric.net/2012/12/10/tor-and-the-uk-data-communications-bill/
Dec 07 2012
Dear Jeff
You don’t know me(*), but I used to be a customer of yours. I say “used to be” because I am no more, nor shall I be again. I have today closed my Amazon account (and I must say that you do not make that process very easy).
You may be aware that the UK, in common with many other economies, is currently going through a rather tough recession. Indeed, our Chancellor of the Exchequer, one George Gideon Oliver Osborne MP (salary c. £145,000, estimated personal fortune of around £4 million from a family trust), is finding the economic situation in the UK so dire that he has been forced to introduce measures to cap benefits to some of the most disadvantaged people in our country. In bringing this action, he has said that it was about “being fair to the person who leaves home every morning to go out to work and sees their neighbour still asleep, living a life on benefits… we have to have a welfare system that is fair to the working people who pay for it”. Personally, I think that increasing taxation on those companies and individuals who can afford it would be a better idea, but hey, that’s just me.
The benefits cap will apparently save some £1.43bn a year by 2014-15. One of our major charities has calculated that this saving to the exchequer will cost jobless families with two children, who are already living on the breadline, some £430 per annum.
“So what has this to do with me?” And “Why are you telling me this?” you may ask.
Simple. You have arranged Amazon’s corporate structure to minimise the tax paid on economic activity in the UK.
Since moving ownership of the UK operation to Luxembourg in 2006, the UK operation has been classed as an “order fulfilment’ business”, a delivery organisation. That has enabled you to avoid paying any UK corporation tax on turnover in the UK estimated at £3.3bn last year alone.
I recognise that this is perfectly legal, and that as a public company you are obliged to minimise liabilities so that you can maximise income for your shareholders, but as Margaret Hodge, the chair of the UK Parliamentary Public Accounts Committee said recently:
“Global corporations with huge operations in the UK generating significant amounts of income are getting away with paying
little or no corporation tax here. This is an insult to British business and individuals who pay their fair share.
“The inescapable conclusion is that multinationals are using structures and exploiting current tax legislation to move offshore
profits that are clearly generated from economic activity in the UK,”
Ms Hodge also seemed less than impressed with your director of public policy, Andrew Cecil. Perhaps you should have a word with him.
Now since I have contributed for some time to the income on which you avoid paying UK taxation, I decided to check just how much I had contributed. Astonishingly, I seem to have spent over £1700 with you over the past few years, Indeed in 2011 alone I spent nearly £500. That is quite a lot of CDs and books. And £500 is more than the sum which is about to be taken from the families on benefits described above. Of course, my contribution alone, even were it taxed correctly, would not do much to help our hard pressed Chancellor of the Exchequer. But if lots of people like me, who can afford to spend £500 per annum on what a person on benefits might see as unaffordable luxuries, were to stop spending that money with you and instead spend it with companies which do contribute their fair share to UK taxation, it might help a little, don’t you think?
So I have decided to do exactly that.
Goodbye and Merry Christmas.
Mick
(Oh and BTW, I have deregistered the Kindle. Not that I ever bought anything on it from you anyway.)
(*) Actually, you do know quite a bit about me. And more than I am comfortable about. But I was somewhat reassured some time ago when you apparently got me very wrong.
Permanent link to this article: https://baldric.net/2012/12/07/an-open-letter-to-jeff-bezos/
Nov 29 2012
The Register today carries a story about the discovery of a truly massive black hole named “SDSS J1106+1939” at the center of a dying galaxy. That black hole is reported to be emitting energy “equivalent to two trillion times the power output of the sun.”
The story itself is classic Reg material and is covered in only rather scant detail, but the commentary by Reg readers make the story worthwhile.
“frank ly” says:
Big and complicated science made accessible.
” … while our Milky Way galaxy has a relatively piddly black hole at its heart, the one found in quasar SDSS J1106+1939 is
whopping.”
That’s why I read The Register.
“gizmo23” follows with some explanatory arithmetic:
Head crunching numbers
So, off the top of my head the sun’s power is about 3 x 10^26 W and there are about 10^11 stars in our galaxy…
and to top it all, it’s 100 times more power than that which is 3 x 10^39 W, divide by c^2 (lets say 10^17) equals 3 x 10^22
kg of matter converted to energy per second… And as the Earth’s mass is approx 6 x 10^24 kg that’s 1/200 the mass of the
earth converted from matter to energy every second.. ooh my brain hurts
and Neil Barnes notes:
Energy output like that…
You could make toast really really quickly.
Just sayin’.
That quality of discussion is just not available in run of the mill publications such as Nature, or Scientific American.
Permanent link to this article: https://baldric.net/2012/11/29/why-i-read-el-reg/
Nov 27 2012
Recently I have been faced with the need to wipe a bunch of hard disks removed from some old (indeed, in one or two cases, very old) PCs before disposal. Normally I would have used DBAN to do this because it gives me a nice warm feeling that I have taken all reasonable steps and no-one (short of a very specialist forensic recovery laboratory) is likely to be able to read the disks thereafter. The latest version of DBAN (post version 2.0) even (supposedly) supports USB disks. I say supposedly because I couldn’t get it to work on my setup.
I have a USB to SATA/IDE disk converter which is very useful for quickly mounting disks which would otherwise need to be installed internally. The fact that DBAN can now handle such disks made it a no brainer choice when I was faced with the task of wiping half a dozen assorted disks of varying vintage. Howver, every disk I tried resulted in a DBAN failure followed by an advertising page for the commercial disk erasure product called blancco. Worse, it appears that Darik’s own web page now advertises blancco as a preferred product. Wierd.
Given the DBAN failure I had to fall back on dd. At least that works. And a wipe once with dd (dd if=/dev/zero of=/dev/sdb), followed by a repartition and reformat with gparted then another dd, this time with random data (dd if=/dev/urandom of=/dev/sdb) is probably enough to stop the the 1992 Parish Council minutes appearing anywhere they shouldn’t.
But oh boy, dd is slow when using /dev/urandom – so I cheated and stuck to /dev/zero.
Permanent link to this article: https://baldric.net/2012/11/27/what-gives-with-dban/
Nov 22 2012
Update to previous post
According to bytemark, a bug in the bigV system caused some disk corruptions for a small number of customers. It seems that I was just one of the unlucky ones. I am assured that all is now well.
Permanent link to this article: https://baldric.net/2012/11/22/why-mysql-crashed/
Nov 16 2012
Today I had a nasty looking problem with my mysql installation. At first I thought I might have to drop one or more databases and re-install. Fortunately, I didn’t actually have to do that in the end.
I first noticed a problem at around 15.45 today when I couldn’t collect my mail. My mail system on this VM uses postfix as the smtp component and dovecot for IMAPS/POP3S delivery. A quick check showed that the server was up and postfix and dovecot were both running. Nothing immediately obviously wrong. However, a check on the mail log showed multiple entries of the form:
dovecot: auth-worker(3078): Error: mysql(localhost): Connect failed to database (mail): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’
An attempt to connect to trivia also resulted in a “database error” message. So on checking again I noticed that of course mysql wasn’t running. Unfortunately, all my attempts to restart it failed. Further investigation of my logs (in particular, an ominously large /var/log/error) showed that I had a corrupt InnoDB page. As soon as mysql reached this it barfed and fell over. Not good. I don’t like database errors because I’m no database expert. But at least that explained the mail error. Both components of my mail system rely on a running mysql server because I store users, passwords, aliases, domain details etc. in mysql databases.
The largest database on this VM is, of course, the blog. I keep regular backups of that so in extremis I could dump the old database and reload from a backup and loose only my daily stats, But I was reluctant to do that without first checking to see if a repair was possible. I then spent some long time reading man pages, my O’Reily MySQL book, and the on-line MySQL reference pages. The best clue I received was the message in the error log:
Nov 16 15:57:05 pipe mysqld: InnoDB: If the corrupt page is an index page
Nov 16 15:57:05 pipe mysqld: InnoDB: you can also try to fix the corruption
Nov 16 15:57:05 pipe mysqld: InnoDB: by dumping, dropping, and reimporting
Nov 16 15:57:05 pipe mysqld: InnoDB: the corrupt table. You can use CHECK
Nov 16 15:57:05 pipe mysqld: InnoDB: TABLE to scan your table for corruption.
Nov 16 15:57:05 pipe mysqld: InnoDB: See also https://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Nov 16 15:57:05 pipe mysqld: InnoDB: about forcing recovery.
Nov 16 15:57:05 pipe mysqld: InnoDB: Ending processing because of a corrupt database page.
That message appeared just before a load of stacktrace information which looked horribly meaningless to me but spoke of deep, unfathomable database wrongness. A read of the referenced mysql manual page didn’t initially reassure me over much either. It starts:
If there is database page corruption, you may want to dump your tables from the database with SELECT … INTO OUTFILE. Usually, most of the data obtained in this way is intact.
“Usually” eh? Hmmm.
As suggested in the documentation I added the line “innodb_force_recovery = 1” to the [mysqld] section of my /etc/mysql/mysql,cnf config file and restarted the server. I started with the lowest non-zero option in the hope that this would cause the least corruption if I had to dump the tables. This time, mysql came up when I started it, but the error log contained the following:
Nov 16 18:09:26 pipe mysqld: InnoDB: A new raw disk partition was initialized or
Nov 16 18:09:26 pipe mysqld: InnoDB: innodb_force_recovery is on: we do not allow
Nov 16 18:09:26 pipe mysqld: InnoDB: database modifications by the user. Shut down
Nov 16 18:09:26 pipe mysqld: InnoDB: mysqld and edit my.cnf so that newraw is replaced
Nov 16 18:09:26 pipe mysqld: InnoDB: with raw, and innodb_force_… is removed.
Now there is nothing in my my.cnf about “raw” or “newraw”, so I simply removed the “innodb_force_recovery” line and shutdown and restarted mysql. Mysql started up without error, and without my having to dump any database. And no, I have no idea why. I can only assume that the force_recovery option forced some database repair as well as the documented forcing of the InnoDB storage engine to start.
And I don’t yet know what caused the problem in the first place, but since my logs show that the VM went down and restarted at around 14.36 I conclude that a failure somewhere in the VM system occurred during a database write and that screwed things up. I’m waiting for a response to my support call.
So. If you too ever face a database corruption similar to mine, do not panic. And do not attempt unnecessarily drastic database drops and reloads until you have at least tried the “innodb_force_recovery” option in your my.cnf configuration.
It may of course just be magic. Most database incantations are as far as I am concerned.
Permanent link to this article: https://baldric.net/2012/11/16/forcing-innodb-recovery-in-mysql/
Oct 27 2012
OpenVPN is a free, open source, general purpose VPN tool which allows users to build secure tunnels through insecure networks such as the internet. It is the ideal solution to a wide range of secure tunnelling requirements, but it is not always immediately obvious how it should be deployed in some circumstances.
Recently, a correspondent on the Anglia Linux User Group (ALUG) mailing list posed a question which at first sight seemed easy to answer. He wanted to connect from one internet connected system to another which was behind a NAT firewall (actually, it turned out to be behind two NAT firewalls, one of which he didn’t control, and therein lay the difficulty).
The scenario was something like this:
He wanted to connect from the system called “Client” in the network on the left to the secured system called “Host A” on the network on the right of the diagram. We can assume that both networks use RFC 1918 reserved addresses and that both are behind NAT firewalls (so routers A and C at the least are doing NAT).
Ordinarily, this would be pretty straightforward. All we have to do is run an SSH daemon (or indeed openVPN) on Host A and set up port forwarding rules on routers A and B to forward the connection to the host. So long as we have appropriate firewall rules on both the host and the routers, and the SSH/OpenVPN daemons are well configured we can be reasonably confident that the inbound connection is secure (for some definition of “secure”). This is exactly the setup I use on my home network. I have an openVPN server running on one of my debian systems. When out and about I can connect to my home network securely over that VPN from my netbook.
However, as I noted above, the problem in this scenario is that the owner of Host A did not control the outermost NAT device (Router A) so could not set up the necessary rule. Here is where openVPN’s flexibility comes in. Given that both networks are internet connected, and can make outbound connections with little difficulty, all we need to do is set up an intermediary openVPN host somewhere on the net. A cheap VPS is the obvious solution here (and is the one I used). Here’s how to accomplish what we want:
Using SSH over an already encrypted tunnel may seem like overkill, but it has the advantage that we can leverage the existing security mechanisms on Host A (and we really don’t want a telnet daemon listening there).
Installing openVPN
There are already a huge range of openVPN “HowTos” out there so I won’t add to that list here. The very comprehensive official HowTo on the openVPN website lists all that you need to know to install and configure openVPN to meet a wide variety of needs, but it can be a bit daunting to newcomers. OpenVPN has a multitude of configuration options so it is probably best to follow one of the smaller, distro specific, howtos instead. Two I have found most useful are for debian and arch. And of course, Martin Brooks, one of my fellow LUGgers has written quite a nice guide too. Note, however, that Martin’s configuration does not use client side certificates as I do here.
By way of example, the server and client configuration files I built when I was testing this setup are given below. Note that I used the PKI architectural model, not the simpler static keys approach. As the main howto points out, the static key approach doesn’t scale well, is not as secure as we’d like (keys must be stored in plain text on the server), but most importantly, it doesn’t give us perfect forward secrecy so any key compromise would result in complete disclosure of all previous encrypted sessions. Note also that I chose to change the keysize ($KEY_SIZE) in the file “vars” to 2048 from the default 1024. If you do this, the build of the CA certificate and server and client keys warns you that the build “may take a long time”. In fact, on a system with even quite limited resources, this only takes a minute or two. Of course, it should go without saying that the build process should be done on a system which is a secure as you can make it and which gives you a secure channel for passing keys around afterwards. There is little point in using a VPN for which the keys have been compromised. It is also worth ensuring that the root CA key (“ca.key” used for signing certificates) is stored securely away from the server. So if you build the CA and server/client certificates on the server itself, make sure that you copy the ca key securely to another location and delete it from the server. It doesn’t need to be there. I chose /not/ to add passwords to the client certificates because the client I was testing from (emulating Host A) is already well secured (!). In reality, however, it is likely that you would wish to strengthen the security of the client by insisting on passwords.
server.conf on the VPS
# openvpn server conf file for VPS intermediary
# Which local IP address should OpenVPN listen on (this should be the public IP address of the VPS)
local XXX.XXX.XXX.XXX
# use the default port. Note that we firewall this on the VPS so that only the public IP address of the
# the network hosting “Host A” (i.e. the public address of “Router A”) is allowed to connect. Yes this exposes
# the VPN server to the rest of the network behend Router A, but it is a smaller set than the whole internet.
port 1194
# and use UDP as the transport because it is slightly more efficient than tcp, particularly when routing.
# I don’t see how tcp over tcp can be efficient.
proto udp
# and a tun device because we are routing not bridging (we could use a tap, but it is not necessary when routing)
dev tun
# key details. Note that here we built 2048 bit diffie hellman keys, not the default 1024 (edit $KEY_SIZE in the
# file “vars” before build)
ca /etc/openvpn/keys/ca.crt
dh /etc/openvpn/keys/dh2048.pem
cert /etc/openvpn/keys/vps-server-name.crt
key /etc/openvpn/keys/vps-server-name.key
# now use tls-auth HMAC signature to give us an additional level of security.
# Note that the parameter “0” here must be matched byy “1” at the client end
tls-auth /etc/openvpn/keys/ta.key 0
# Configure server mode and supply a VPN subnet for OpenVPN to draw client addresses from.
# Our server will be on 172.16.10.1. Be careful here to choose an RFC1918 network which will not clash with
# that in use at the client end (or Host A in our scenario)
server 172.16.10.0 255.255.255.0
# Maintain a record of client virtual IP address associations in this file. If OpenVPN goes down or is restarted,
# reconnecting clients can be assigned the same virtual IP address from the pool that was previously assigned.
ifconfig-pool-persist ipp.txt
# We do /not/ push routes to the client because we are on a public network, not a reserved internal net.
# The default configuration file allows for this in the (example) stanzas below (commented out here)
# push “route 192.168.1.0 255.255.255.0”
# push “route 192.168.2.0 255.255.255.0”
# Nor do we tell the client (Host A) to redirect all its traffic through the VPN (as could be done). The purpose of this
# server is to allow us to reach a firewalled “client” on a protected network. These directives /would/ be useful if we
# wanted to use the VPS as a proxy to the outside world.
# push “redirect-gateway def1”
# push “dhcp-option DNS XXX.XXX.XXX.XXX”
# Nor do we want different clients to be able to see each other. So this remains commented out.
# client-to-client
# Check that both ends are up by “pinging” every 10 seconds. Assume that remote peer is down if no ping
# received during a 120 second time period.
keepalive 10 120
# The cryptographic cipher we are using. Blowfish is the default. We must, of course, use the same cipher at each end.
cipher BF-CBC
# Use compression over the link. Again, the client (Host A) must do the same.
comp-lzo
# We can usefully limit the number of allowed connections to 1 here.
max-clients 1
# Drop root privileges immediately and run as unpriviliged user/group
user nobody
group nogroup
# Try to preserve some state across restarts.
persist-key
persist-tun
# keep a log of the status of connections. This can be particularly helpful during the testing stage. It can also
# be used to check the IP address of the far end client (Host A in our case). Look for lines like this:
#
# Virtual Address,Common Name,Real Address,Last Ref
# 172.16.10.6,client-name,XXX.XXX.XXX.XXX:nnnnn,Thu Oct 25 18:24:18 2012
#
# where “client name” is the name of the client configuration, XXX.XXX.XXX.XXX:nnnnn is the public IP address of the
# client (or Router A) and port number of the connection. This means that the actual client address we have a connection
# to is 172.16.10.6. We need this address when connecting from the VPS to Host A.
status /var/log/openvpn-status.log
# log our activity to this file in append mode
log-append /var/log/openvpn.log
# Set the appropriate level of log file verbosity
verb 3
# Silence repeating messages. At most 20 sequential messages of the same message category will be output to the log.
mute 20
# end of server configuration
Now the client end.
client.conf on Host A
# client side openvpn conf file
# we are a client
client
# and we are using a tun interface to match the server
dev tun
# and similarly tunneling over udp
proto udp
# the ip address (and port used) of the VPS server
remote XXX.XXX.XXX.XXX 1194
# if we specify the remote server by name, rather than by IP address as we have done here, then this
# directive can be useful since it tells the client to keep on trying to resolve the address.
# Not really necessary in our case, but harmless to leave it in.
resolv-retry infinite
# pick a local port number at random rather than bind to a specific port.
nobind
# drop all priviliges
user nobody
group nogroup
# preserve state
persist-key
persist-tun
# stop warnings about duplicate packets
mute-replay-warnings
# now the SSL/TLS parameters. First the server then our client details.
ca /home/client/.openvpn/keys/server.ca.crt
cert /home/cient/.openvpn/keys/client.crt
key /home/client/.openvpn/keys/client.key
# now add tls auth HMAC signature to give us additional security.
# Note that the parameter to ta.key is “1” to match the “0” at the server end.
tls-auth /home/client/.openvpn/keys/ta.key 1
# Verify server certificate by checking.
remote-cert-tls server
# use the same crypto cypher as the server
cipher BF-CBC
# and also use compression over the link to match the server.
comp-lzo
# and keep local logs
status /var/log/openvpn-status.log
log-append /var/log/openvpn.log
verb 3
mute 20
# end
Tunneling from Host A to the VPS.
Now that we have completed configuration files for both server and client, we can try setting up the tunnel from “Host A”. In order to do so, we must, of course, have openVPN installed on the Host A and we must have copied the required keys and certificates to the directory specified in the client configuration file.
At the server end, we must ensure that we can forward packets over the tun interface to/from the eth0 interface. Check that “/proc/sys/net/ipv4/ip_forward” is set to “1”, if it is not, then (as root) do “echo 1 > /proc/sys/net/ipv4/ip_forward” and then ensure this is made permanent by uncommenting the line “net.ipv4.ip_forward=1″ in /etc/sysctl.conf”. We also need to ensure that neither the server, nor the client block traffic over the VPN. According to the openVPN website, over 90% of all connectivity problems with openVPN are caused not by configuration problems in the tool itself, but by firewall rules.
If you have an iptables script on the server such as I use, then ensure that:
The last point can be covered if you add rules like:
$IPTABLES -A INPUT -i tun0 -j ACCEPT
$IPTABLES -A OUTPUT -o tun0 -j ACCEPT
$IPTABLES -A FORWARD -o tun0 -j ACCEPT
This would allow all traffic over the tunnel. Once we know it works, we can modify the rules to restrict traffic to only those addresses we trust. If you have a tight ruleset which only permits ICMP to/from the eth0 IP address on the VPS, then you may wish to modify that to allow it to/from the tun0 address as well or testing may be difficult.
Many “howtos” out there also suggest that you should add a NAT masquerade rule of the form:
“$IPTABLES -t nat -A POSTROUTING -s 172.16.10.0/24 -o eth0 -j MASQUERADE”
to allow traffic from the tunnel out to the wider network (or the internet if the server is so connected). We do not need to do that here because we are simply setting up a mechanism to allow connection through the VPS to Host A and we can use the VPN assigned addresses to do that.
Having modified the server end, we must make similarly appropriate modifications to any firewall rules at the client end before testing. Once that is complete, we can start openVPN at the server. I like to do this using the init.d run control script whilst I run a tail -f on the log file to watch progress. Watch for lines like “TUN/TAP device tun0 opened” and “Initialization Sequence Completed” for signs of success. We can then check that the tun interface is up with ifconfig, or ip addr. Given the configuration used in the server file above, we should see that the tun0 interface is up and has been assigned address 172.16.10.1.
Now to set up the tunnel from the client end, we can run (as root) “openvpn client.conf” in one terminal window and we can then check in another window that the tun0 interface is up and has been assigned an appropriate IP address. In our case that turns out to be 172.16.10.6. It should now be possible to ping 172.16.10.1 from the client and 172.16.10.6 from the server.
Connecting over SSH from Client to VPS.
We should already be doing this! I am assuming that all configuration on the VPS has been done over an SSH connection from “Client” in the diagram. But the SSH daemon on the VPS may be configured in such a way that it will not permit connections over the VPN from “Host A”. Check (with netstat) that the daemon is listening on all network addresses rather than just the public IP address assigned to eth0. If it is not, then you will need to modify the /etc/ssh/sshd_config file to ensure that “ListenAddress” is set to “0.0.0.0”. And if you limit connections to the SSH daemon with something like tcpwrappers, then check that the /etc/hosts.allow and /etc/hosts.deny files will actually permit connections to the daemon listening on 172.16.10.1. Once we are convinced that the server SSH configuration is correct, we can try a connection from the Host A to 172.16.10.1. If it all works, we can move on to configuring Host A.
Connect (using SSH again) over the openVPN tunnel from the VPS to Host A.
This is the crucial connection and is why we have gone to all this trouble in the first place. We must ensure that Host A is running an SSH daemon, configured to allow connections in over the tunnel from the server with address 172.16.10.1. So we need to make the same checks as we have undertaken on the VPS. Once we have this correctly configured we can connect over SSH from the server at 172.16.10.1 to Host A listening on 172.16.10.6.
Job done.
Addendum: 26 March 2013. For an alternative mechanism to achieve the same ends, see my later post describing ssh tunnelling.
Permanent link to this article: https://baldric.net/2012/10/27/using-openvpn-to-bypass-nat-firewalls/
Oct 23 2012
Back in August 2011. I wrote about my preference for real books over the emerging electronic version. In that post I noted that Amazon had famously deleted copies of Orwell’s 1984 and Animal Farm purchased by some customers. It now appears that Amazon has gone even further and deleted the entire contents of a Norwegian customer’s kindle. In Cory Doctorow’s post he points out that:
Reading without surveillance, publishing without after-the-fact censorship, owning books without having to account for your ongoing use of them: these are rights that are older than copyright. They predate publishing. They are fundamentals that every bookseller, every publisher, every distributor, every reader, should desire. They are foundational to a free press and to a free society.
Now I’d be pretty angry if someone sold me a book, but later stole that book back on the grounds that I had somehow infringed some sales condition buried in a contract I had implicitly (and forcedly) entered into by the act of purchase. But I would be absolutely livid if, in the act of stealing back “their” book, they also removed the rest of my library. Amazon, however, seems to find this acceptable.
Doctorow went on to say that encrypting storage on mobile devices was much preferable to the option of remote deletion in case of loss. I agree. Unfortunately I also agree with his view that users will have difficulty with password protected encrypted filesystems, and I am completely with him when he says:
If it’s a choice between paving the way for tyranny and risking the loss of your digital life at the press of a button by some deceived customer service rep, and having to remember a password, I think the password is the way to go. The former works better, but the latter fails better.
My own kindle only has the DRM free content I originally uploaded (over a USB connection) after my wife bought it for me. And the wifi is resolutely turned off. But I don’t know why I bothered, because I still haven’t used it, despite taking it on holiday. And now, like Adrian Short, I never will.
Permanent link to this article: https://baldric.net/2012/10/23/book-theft/
Oct 20 2012
Towards the end of last month, following the release of the unity lens in ubuntu which searches amazon, “akeane” posted a bug report on launchpad complaining that “grep -R doesn’t automatically search amazon”. In his first posting he said:
Dear “root owning” overlords,
When using grep recursively I only get local results:
grep -R fish_t /home/noob/fish_game/*
/home/noob/fish_game/fish.h: struct fish_t { /home/noob/fish_game/fish.c: struct fish_t eric_the_ fish;
or worse:
grep -R shark_t /home/noob/fish_game/*/home/noob/fish_game/fish.h: struct shark_t { /home/noob/fish_game/fish.c: struct
shark_t_t mark_sw;
I declare this a bug for two reasons:
1. The output is boring.
2. The terminal has more than 2 lines!!! It’s an unefficient use of my screenspace.
I believe the reason for this is that the grep command only searches locally for things I am actually looking for, I kind of
expect the results I get from my codebase and as such it removes any sense of mystery or something new and exciting to
spice up my dull geek existence. That’s boring, grep -R should also search amazon, so I get more exciting results such as:
Shark Season 1 Starring Steven Eckholdt, Nora Dunn, Patrick Fabian, et al.
Amazon Instant Video to buy episodes: $1.99 to buy season: $34.99 ($1.59 per episode)
This bug report has been added to over the past few weeks, particularly after a reference was posted to reddit. As at today’s date, the bug is reported to “affect 169 people”.
The main thrust of akeane’s complaint is that the command line is lacking functionality available in the GUI. He finds that annoying and others agree. On 21 October, “rupa” wrote
I’ve been following this bug with some interest. A lot of good ideas here, but there are some technical issues that haven’t
been addressed.
Until all the common unix utilities can be upgraded to be affiliate-aware, common pipe operations might suffer from related
amazon results. if you pipe grep into awk for example, without being able to be sure of the results, things can get messy.
I feel this issue can be solved by introducing a new ‘default’ file descriptor. In addition to ‘stdout’ and ‘stderr’, I propose a
‘stdaffiliate’ file descriptor, and amazon results can go there. This would allow us to see relevant results in the terminal, but
keep them out of pipes unless purposely redirected.
Here I must disagree. In my view, all amazon (or related adware) results should be piped directly to /dev/null. Of course, stdaffiliate could be linked directly to /dev/null and all would be well.
Permanent link to this article: https://baldric.net/2012/10/20/grep-r-doesnt-search-amazon/
Oct 14 2012
But we may ask you for a contribution.
Canonical have made another move in what is beginning to look ever more like a monetary commercialisation of ubuntu. On 9 October 2012, they added a new page to the “download” section titled “Tell us what we should do more……and put your money where your mouth is ;)” The page looks like this:
The sliders allow you to “target” your contribution to those areas of ubuntu which you feel deserve most reward (or conversely, you believe need most effort in improvement). The default is $2.00 for each of the eight radio button options (for a total of $16.00).
Now $16.00 is not a huge amount to pay for a linux distro of the maturity of ubuntu, but I’m not sure I like the way this is being done. Most distros offer a “donate” button somewhere on their website, but no other has placed it as prominently in the download process as canonical has chosen to do. I’m also a little bothered by the size and placement of the “Not now, take me to the download” option and I have a sneaking feeling that will become even less prominent over time.
Not surprisingly, some of the commentariat have taken great umbrage at this move (witness the comment over at El Reg of the form “Where is the option for “fix the known damn bugs and quit pissing around with GUI”?” and I expect more hostility as and when users start fetching the new 12.10 release.
But an earlier move to monetise the ubuntu desktop worries me even more. Canonical’s link with Amazon through the ubuntu desktop search was, according to Mark Shuttleworth, perfectly sensible, because “the Home Lens of the Dash should let you find *anything* anywhere. Over time, we’ll make the Dash smarter and smarter, so you can just ask for whatever you want, and it will Just Work.” (So that’s alright then.) But the problem, which Shuttleworth clearly doesn’t understand, is that people don’t generally like having advertising targetted at them based on their search criteria. (cf. Google…..). What was worse, the search criteria were passed to Amazon in clear. Think about that.
I share the views of Paul Venezia over at Infoworld where he says:
“But the biggest problem I have with the Amazon debacle is another comment by Shuttleworth: “Don’t trust us? Erm, we have root. You do trust us with your data already.” That level of hubris from the founder of Ubuntu, in the face of what is clearly a bad idea badly implemented, should leave everyone with a bad taste in their mouth. If this idea can make it to the next Ubuntu release, then what other bad ideas are floating around? What’s next? Why should we maintain that trust?
So fine, Mr. Shuttleworth. You have root. But not on my box. Not anymore.”
Ubuntu is already in decline following the way unity was foisted on the userbase. And Canonical has been likened to Apple in the past. Things can only get worse for ubuntu from hereon. Way past time to move on.
Permanent link to this article: https://baldric.net/2012/10/14/ubuntu-is-free-and-it-always-will-be/
Oct 13 2012
One of my fixed term savings accounts matured at the end of last week. This means that the paltry “bonus” interest rate which made the account ever so slightly more attractive than the pathetic rates generally available 12 months ago now disappears and I am left facing a rate so far below inflation that I have contemplated just stuffing the money under my mattress. Current rates generally on offer at the moment are pretty terrible all round, but I was certainly not going to leave the money where it was so I decided to move it to a (possibly temporary) new home.
After checking around, I found a rate just about more attractive than my mattress and so set about opening the new account on-line. Bearing in mind that this account will be operated solely on-line and may hold a significant sum of money (well, not in my case, but it could) one would expect strong authentication mechanisms. I was therefore not reassured to be greeted by a sign up mechanism that asked for the following:
a password which must:
(oh, and it is not case sensitive. That’s good then.)
Further, I am asked to provide:
I should note here that I initially failed the last hurdle because the place name I chose had fewer than the required 8 characters, and when I tried a replacement I found that I wasn’t allowed to use a place name with spaces in it (so, something like “Reading” or “Ross on Wye” are unacceptable to this idiot system).
I haven’t tried yet (the account is in the process of being set up and I will receive details in the post) but from experience with other similar accounts, I guess that the log-on process will ask for my password, then challenge me to enter three characters drawn from one of my memorable date/name/place. Oh, and the whole process is secured by a 128bit SSL certificate.
My friend David wrote a blog piece a while ago about stupid password rules. The ones here are just unbelievable. Why must the password be limited to 8-10 characters? Why can’t I choose a long passphrase which fits my chosen algorithm (like David, I compute passwords according to a mechanism I have chosen which suits me). Why must it only be alphanumeric? And why for pity’s sake should it be case insensitive? Are they deliberately trying to make it easy to crack?
As for the last three requirements, what proportion of the population do you think are likely to choose their birthdate, mother’s maiden name and place of birth (unless of course, they were born in Reading, or London, or York, or Glasgow, or Burton on Trent or…)
Answers on a postcard please.
Permanent link to this article: https://baldric.net/2012/10/13/password-lunacy/
Oct 05 2012
Whenever my logs show evidence of unwanted behaviour I check what has happened and, if I decide there is obviously hostile activity coming from a particular address I will usually bang off an email to the abuse contact for the netblock in question. Most times I never hear a thing back though I occasionally get an automated response.
Today, after finding over 23,000 automated attempts to access the admin page of trivia I sent off my usual notification to the netblock owner (“Hey, spotted this coming from you, a bit annoying”). Within a couple of hours I got an automated acknowledgement asking me to authenticate myself by response. A couple of hours after that, I got a human response saying “We’ve dealt with it. Your address is now blocked”. I’ve never had that helpful a response before.
The ISP was Russian.
Permanent link to this article: https://baldric.net/2012/10/05/a-positive-response/
Sep 09 2012
I paid for a new VPS to run tor this week. It is cheaper, and offers a higher bandwidth allowance than my existing tor server so I may yet close that one down – particularly as I recently had trouble with the exit policy on my existing server.
In setting up the new server, the first thing I did after the base installation of debian and the first apt-get update/upgrade was to install my default minimum iptables firewall ruleset. This rule simply locks down the server to accept inbound connections only to my SSH port and only from my home trusted network. All other connections are denied. I have a variety of different iptables rules depending upon the system (rules for headless servers are clearly different to those needed on desktops running X for example). In reviewing my policy stance for this new server, I started comparing the rules I was using on other servers, both externally on the ‘net and internally on my LAN. I found I was inconsistent. Worse, I was running multiple rulesets with no clear documentation and no obvious commonality where the rules should have been consistent, or any explanation of the differences. In short I was being lazy, but in doing so I was actually making things more difficult for myself because a) I was reinventing rulesets each time I built a server, and b) the lack of documentation and consistency meant that checking the logic of the rules was unnecessarily time consuming.
To add to my woes, I noted that in one or two cases I was not even filtering outbound traffic properly. This is a bad thing (TM), but not untypical of the approach I have often seen used elsewhere. Indeed, a quick check around the web will show that most sites offering advice about iptables rulesets concentrate only on the input chain of the filter table and ignore forwarding and output. To be fair, many sites discussing iptables seem to assume that ipforwarding is turned off in the kernel (or at least recommend that it should be) but very few that I could find even consider output filtering.
In my view, output fitering is almost as important, if not as important as input filtering. Consider for example how most system compromises occur these days. Gone are the days when systems were compromised by remote attacks on vulnerable services listening on ports open to the outside world. Today, systems are compromised by malicious software running locally which calls out to internet based command and control or staging servers. That malicious software initially reaches the desktop through email or web browsing activity. This “first stage” malware is often small, aimed at exploiting a very specific (and usually completely unpatched) vulnerability and is unnoticed by the unsuspecting desktop user. The first stage malware will then call out to a server (over http or https usually) to both register its presence and obtain the next stage malware. That next stage will give the attacker greater functionality and persistence on the compromised system. It is the almost ubiquitous ability of corporate desktops to connect to any webserver in the world that has led to the scale of compromise we now routinely see.
But does output filtering matter on a server? And does it really matter when that server is running linux and not some other proprietary operating system? Actually, yes, it matters. And it matters regardless of the operating system. There is often a disconcerting smugness from FOSS users that “our software is more secure than that other stuff – we don’t need to worry”. We do need to worry, And as good net citizens we should do whatever we can to ensure that any failures on our part do not impact badly on others.
I’m afraid I was not being a good net citizen. I was being too lax in places.
If your linux server is compromised and your filtering is inadequate, or non-existent, then you make the attacker’s job of obtaining additional tools easy. Additionally, you run the risk of your server being used to attack others because you have failed to prevent outbound mailicious activity, from port scanning to DOS, to email spamming to running IRC or other services he wants, on your server (for which you pay the bills). Of course if the attacker has root on your box, no amount of iptables filtering is going to protect you. He will simply change the rules. But if he (or she) has not yet gained root, and his privilege escalation depends upon access to the outside world, then your filters may delay him enough to give you time to take appropriate recovery action. Not guaranteed of course, but at least you will have tried.
So how can your server be compromised? Well, if you get your input filtering wrong and you run a vulnerable service, you could be taken over by a ‘bot. There are innumerable ‘bots out there routinely scanning for services with known vulnerabilities. If you don’t believe that, try leaving your SSH port open to the world on the default port number and watch your logs. Fortunately for us, most distros these days ship with the minimum of services enabled by default, often not even SSH. But how often have you turned on a service simply to try something new? And how often did you review your iptables rules at the same time? And have you ever used wget to pull down some software from a server outside your distro’s repository? And did you then bother to check the MD5 sum on that software? Are you even sure you know fully what that software does? Do you routinely su to root to run software simply because the permissions require that? Do you have X forwarding turned on? Have you ever run X software on your server (full disclosure – I have)? Ever run a browser on that? In the corporate world I have even seen sysadmins logged in to servers which were running a full desktop suite. That way lies madness.
Believe me, there are innumerable ways your server could become compromised. What you need to do is minimise the chances of that happening in the first place, and mitigating the impact if it does happen. Which brings me back to iptables and my configuration.
The VM running trivia is also my mailserver. So this server has the following services running:
My tails mirror only has port 80 and my nonstandard SSH port open, my tor server has ports 80, 9001 and my non standard SSH port open, and of course some of my internal LAN servers listen on ports such as 53, 80, 443, 2049, (and even occasionally on 139 and 445 when I decide I need to play with samba, horrible though that is). I guess this is not an unusual mix,
My point here though, is that not all of those ports need to be accessible to all network addresses. On my LAN, none of them need to be reachable from anywhere other than my internal selected RFC1918 addresses. My public servers only need to be reachable over SSH from my LAN (if I need to reach one of them when I am out, I can do so from a VPN back into my LAN) and given that my public servers are on different networks, they in turn do not need to reach the same DNS servers or distro repositories (one of my ISPs runs their own distro mirror. I trust that. Should I?). Whilst inevitably the iptables rules for each of these servers needs to be different, the basic rule configuration should really be the same (for example, all should have a default drop policy, none need allow inbound connections to any non existent service, none need allow broadcasts, none need access to anything other than named DNS servers, or NTP servers etc.) so that I am sure it does what I think it should do. My rules didn’t conform to that sort of approach. They do now.
Having spent some time considering my policy stance, I decided that what I needed was a single iptables script that could be modified quite simply, and clearly, in a header which stated the name of the server, the ports it needed open or which it needed access to and the addresses of any servers which it trusted or it needed access to. This turned out to be harder to implement than I at first thought it should be.
Consider again this server. It should be possible to nail it down so that it only allows inbound new or established connections to the ports listed and only allows oubound established connections to those inbound. Further, it should not call out to any servers other than my DNS/NTP and distro repositories. Easy. But not so. Mail is awkward for example because we have to cater for inbound to port 25 from anywhere as well as outbound to port 25 anywhere. That feels a bit lax to me, but it is necessary unless we connect only to our ISP’s mailserver as a relay. Worse, as I discovered when I first applied my new tight policy, I found that my wordpress installation slowed to a crawl in certain circumstances. Here it transpired that I had forgotten that I run the akismet plugin which needs access to four akismet servers (Question. Do I need to continue to run akismet? What are the costs/benefits?) It is conceivable that other plugins will have similar requirements. I also noticed that I had over thirty entries for rpc servers in my wordpress “Update Services” settings (this lists rpc servers you wish to automatically notify about posts/updates on your blog). Of course WP was attempting to reach those servers and failing. So I found myself adding exceptions to an initially simple rulebase. I don’t like that. And what if the IP addresses of those servers change?
So I actually ended up with two possible policy stances, which I called “tight” and “loose”. The first attempts to limit all access to known services and servers (with the obvious exception of allowing inbound to public services). The second takes a more permissive stance in that it recognises that it may not be possible to list all the servers we must allow connection to, but limits those connections to particular services (so for example, whilst it will allow out connection only to DNS on one or two servers, it will allow out new connections to any server on say port 80 (I actually don’t like this, for fairly obvious reasons, but it is at least more restrictive than the usual “allow anything to anywhere”).
Others may find these scripts useful so I have posted them here: iptables.tight.sh and iptables.loose.sh Since the scripts must be run at boot time they should be run out of one your boot run control scripts (such as /etc/init.d/rc.local) or at network initialisation as a script in /etc/network/if-up.d. Before doing so, however, I strongly advise you to test them on a VM locally, or at least on a machine to which you have console access. Locking yourself out of a remote VM can be embarrassing.
By way of explanation of the policy stances taken, I have posted separate descriptions of each at tight and loose.
Comments. feedback, suggestions for improvement or criticism all welcome.
Permanent link to this article: https://baldric.net/2012/09/09/iptables-firewall-for-servers/