Posts Tagged ‘networks’

« Older Entries

t-mobile resets its policy?

Thursday, January 12th, 2012

As I have mentioned in other posts here, I run my own mail server on one of my VMs. I do this for a variety of reasons, but the main one is that I like to control my own network destiny. Back in October last year I noticed an interesting change in my mail experience with my HTC mobile (actually my wife first noticed it and blamed me, assuming that I had “twiddled with something” as she put it). Heaven forfend.

My mail setup is postfix/dovecot with SASL authentication and TLS protecting the mail authentication exchange. My X509 certs are self generated (and so not signed by any CA). I pick up mail over IMAPS (when mobile) and POP3S (at home – for perverse reasons of history I like to actually download mail to my main desktop over POP3 and archive it to two separate NAS backups). I send via the standard SMTP port 25 but require authentication and protect the exchange with TLS.

My mail had been working fine ever since I set it up some years ago, but as I said, back in October my wife complained that she could no longer send email from her HTC mobile (we both use t-mobile as the network provider). She was at work at the time so away from my home network. Both our phones are setup to use use wifi for connectivity where it is available (as it is at home of course). When my wife complained I checked my phone and it could send and receive without problem. But when I switched wifi off, thus forcing the data connection though the mobile network, I got the same problem as my wife reported. On checking my mail server logs I read this:

postfix/smtpd[28089]: connect from unknown[149.254.186.120]
postfix/smtpd[28089]: warning: network_biopair_interop: error reading 11 bytes from the network: Connection reset by peer
postfix/smtpd[28089]: SSL_accept error from unknown[149.254.186.120]:-1
postfix/smtpd[28089]: lost connection after STARTTLS from unknown[149.254.186.120]
postfix/smtpd[28089]: disconnect from unknown[149.254.186.120]

(the ip address is one of t-mobile’s servers on their “TMUK-WBR-N2″ network)

Everything I could find about that sort of message suggested that the client was tearing down the connection because there was something wrong with the TLS handshake and it was not trusted. Checking earlier logs, I found that t-mobile’s address had apparently changed (to the address above) recently. So I assumed that some recent network change following the Orange/T-mobile merger had been badly managed and all would be well again as soon as the problem was spotted. Wrong. It persisted. So I had to investigate further. As part of my investigation of the error, I tried moving mail from port 25 to 587 (submission) because that sometimes gets around the problem of ISPs blocking, or otherwise interfering, with outbound connections from their networks to port 25, No deal. In fact it looked as if t-mobile were blocking all connections to port 587 (I assumed a whitelisting policy block, or again, a cockup).

So, the scenario was: mail works when connecting over wifi and using my domestic ISP’s network, but doesn’t when using t-mobile’s 3G network. Symptoms point to a lack of trust in the TLS handshake. Tentative conclusion? There is an SSL/TLS proxy somewhere in the mobile operator’s chain. That proxy sucessfully negotiates with our phones, but when it gets my self certified X509 cert from the server. it can’t authenticate it and decides that the connection is untrusted so tears it down. My server sees this as the client (my phone) tearing down the connection. [As it turns out, this conclusion was completely wrong, but hey].

I said in an email at the time to a friend whose advice I was seeking, “I suspect cockup rather than outright conspiracy, but if my telco is dumb enough to stick a MITM ssl proxy in my mail chain, they really ought to have thought about handling self signed certs a little better. Otherwise it sort of gives the game away.”

In response, he very sensibly suggested that I should run a sniffer on the server and check what was going on. At that time, I was busy doing something else so I didn’t. And because the problem was intermittent (and my wife stopped complaining) I never got around to properly investigating further. (I should explain that I rarely send mail from my mobile nowadays. I just read mail there and wait until I get home to a decent keyboard and can reply to whatever needs handling from there. My wife just gave up bothering to try).

I should have persisted because of course I wasn’t the only one to experience this problem.

Back in November, a member of the t-mobile discussion forum called “dpg” posted a message complaining that he could not connect to port 587 over t-mobile’s 3G network. In response, a member of the t-mobile forum team suggested that dpg might reconfigure his email so that it was relayed via t-mobile’s own SMTP server. Not unreasonably, dpg didn’t think this was an acceptable response – not least because he would then have to send his email in clear. He then posted again saying that “the TLS handshake fails when the mail client receives a TCP packet with the reset (RST) flag set.” (This is a bad thing (TM). Further, he posted again saying that he had set up his own mail server and repeated earlier tests so that he could see both ends of the connection. At the client side he posted mail from his laptop tethered to his phone which was connected to the t-mobile 3G network. By running sniffers at both ends of the connection he was able to prove to his own satisfaction that something in the t-mobile network was sending a RST and tearing down any connection when a STARTTLS was seen. Again, in a later post in response to one from another poster who apparently manages several mail servers and had been looking at the same issue for a client, dpg says:

“I must say I’m not too pleased to discover that T-Mobile may be snooping all traffic to check for SMTP messages. I have demonstrated that they may be doing this by running a SMTP server on a non-standard port and finding that they still sent TCP reset packets during TLS negotiation – so they must be examining all packets and not just those destined for TCP ports 25 and 587.

I’m also not that keen on T-Mobile spoofing/forging TCP resets. This is the sort of tactic resorted to by the Great Firewall of China (http://www.lightbluetouchpaper.org/2006/06/27/ignoring-the-great-firewall-of-china/) and also by Comcast back in 2007 (https://www.eff.org/wp/packet-forgery-isps-report-comcast-affair) until the US FCC told them to stop (http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-08-183A1.pdf).”

Then 9 days ago, dpg posted this message:

“I finally got to the bottom of this. I was contacted by T-Mobile technical support today and was told that they are now actively looking for and blocking any TLS-secured SMTP sessions. So, it is a deliberate policy after all, despite what the support staff have been saying on here, twitter and on 150. They told me it is something they have been rolling out over the last three months – which explains why it was intermittent and dependent on IP address and APN to begin with.

So, the only options for sending email over T-Mobile’s network are:
- unencrypted but authenticated SMTP (usually on port 25)
- SSL-encrypted SMTP (usually on port 465)
- unauthenticated and unencrypted email to smtp.t-email.co.uk

TLS-encrypted SMTP sessions are always blocked whether or not they are on the default port of 587.”

(As an aside, there is, of course, another alternative. You can ditch t-mobile as your provider and pick one which doesn’t use DPI to screw your connections. You pays your money….)

Following this, a new poster called “mickeyc” said this:

“I’ve been experiencing this exact same problem. I run my own mail server which has SSL on port 465 and also uses TLS on port 587. I used wireshark to confirm that the RST packets are being spoofed. This is the exact same technology used by “The Great Firewall of China”. I have two t-mobile sims. One is about a year old and doesn’t experience this problem (yet), one is a few weeks old and does.”

He went to say that he had also experienced problems with his OpenVPN connections and would be blogging about the problem (damned bloggers get everywhere) and sure enough, Mike Cardwell did so at grepular.com. That blog post is worth reading because it has an interesting set of comments and responses from Mike appended.

Mike’s post seems to have been picked up by a few others (El Reg has one, and as Mike himself has pointed out, boingboing.net has a particularly OTT post which seems to say that he is accusing t-mobile of something he clearly isn’t.

Finally, two days ago, dpg posted this:

“I’m pleased to report that T-Mobile is no longer blocking TLS-secured email on port 587. As a follow-up to an email exchange over the Christmas period I was contacted today to say that, contrary to what I had been told previously, it was never a deliberate policy to block TLS-secured outgoing email. There was a problem with some equipment after all, which was resolved yesterday.”

I tried again myself today. Initially, I got the same old symptoms (“lost connection after STARTTLS”) then I rebooted my ‘phone and lo and behold I could send email.

Like Mike, I tend to the cockup over conspiracy theory, it’s more likely for one thing. IANAL, but it seems to me that it would be in breach of RIPA part I, Unlawful Interception, for the telco to intercept my SMTP traffic in the way it seems to have been doing. That is not likely to be a deliberate act by a major UK mobile network provider.

But I’ll still keep an eye on things.

Tags: , , , , ,
Posted in network (in)security, networks and networking, privacy, security, trivial musing | No Comments »

who are you going to call

Monday, July 18th, 2011

Like most email users I get my fair share of spam and other internet crud. Mostly I ignore it, but I received an intriguing email a couple of days ago which purported to be a mailer daemon “Delivery Status Notification” informing me of a failed delivery to some address I had not even heard of. Mostly this sort of junk results from backscatter from mailers responding to spammers spoofing your email address in the outgoing mail. I get a lot of this to the “info@baldric.net” address as I’ve mentioned below.

However, this particular email interested me because the message-id in the headers said it came from a machine calling itself mx1.rlogin.net. Now I own and control that domain and I know that I have no such machine. So I concluded that the machine at the address in question (which looks to be at the end of a commercial cable line) was either deliberately being used incorrectly by its owner, or (much more likely) had been compromised and was being used illegally to send spam mail designed to look like it came from my domain. Either way, I’m not overly happy about that so I decided to contact the ISP and let them know that they might have a customer with a problem.

The normal way to do this is to send email to the “abuse@domain.name” address which is usually listed in the whois record for the network owner. There is even an RFC which codifies this practice. However, on searching a variety of whois records I couldn’t find any obvious address to use. Worse, speculative email to “abuse@the-likeliest-relevant-domain” was simply returned as undeliverable. What to do?

Well as it happens I found a very helpful utility called abuseEmail which automates searches of whois records for likely addresses. Better yet, some helpful people at cyberabuse.org give a web based front end to the abusEmail php script so that you don’t have to run your own.

Tags: , , , , ,
Posted in mail and mail lists, network (in)security, trivial musing | No Comments »

no police here

Tuesday, February 1st, 2011

The UK Home Office launched a new crime statistics website today at www.police.uk. The site is supposed to show “Local crime and policing information for England and Wales”.

I’m not entirely convinced of the merit of the site in the first place (and can see all sorts of potential objections arising in some of the more rabid tabloid newspapers), but I thought I would try it out before making any form of judgement of my own. Unfortunately I’m not impressed.

The opening page of the new service invites the user to “Enter your postcode, town, village or street into the search box below, and get instant access to street-level crime maps and data, as well as details of your local policing team and beat meetings.”

I have tried various combinations of the suggestions, scaling outwards and upwards from my precise postcode to the whole of that part of the County in which I live. I was not reassured to get the following message:

screenshot of www.police.uk website

Discussion elsewhere on the ‘net suggests that this result is not unusual. It appears to be a badly worded (or badly coded) response to an error condition resulting from system overload following the launch. At least I sincerely hope that is the case and we are not really completely devoid of policing services in the whole of South Norfolk.

Examination of the HTML source for the webpage generated suggests that the service is running on Amazon’s Web Services. Certainly some of pages are retrieved from S3 servers, and the IP address of the site appears to be on Amazon’s AWS (see dig and whois results below *). If the site is, as it appears to be, cloud based, then either the supplier (Rock Kitchen Harris, Leicester) or the Home Office has seriously undersized the requirement. Regardless of who is at fault here, there is an evident need to pull in some more resource pretty quickly. This should be a good test of the much vaunted flexibility of cloud based services such as Amazon’s EC2. I expect the service to be running quickly and cleanly by this time tomorrow.

* dig www.police.uk returns:

; <<>> DiG 9.6-ESV-R3 <<>> www.police.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10557
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.police.uk. IN A

;; ANSWER SECTION:
www.police.uk. 1251 IN CNAME policeuk-167782603.eu-west-1.elb.amazonaws.com.
policeuk-167782603.eu-west-1.elb.amazonaws.com. 60 IN A 46.137.113.146

;; Query time: 268 msec
;; SERVER: 80.68.80.24#53(80.68.80.24)
;; WHEN: Tue Feb 1 14:16:23 2011
;; MSG SIZE rcvd: 107

and a whois lookup of 46.137.113.146 returns:

% Information related to ’46.137.0.0 – 46.137.127.255′

inetnum: 46.137.0.0 – 46.137.127.255
netname: AMAZON-EU-AWS
descr: Amazon Web Services, Elastic Compute Cloud, EC2, EU

Tags: , ,
Posted in coding and admin, networks and networking, trivial musing | 1 Comment »

damn, I think I got hit by a 419er

Sunday, January 23rd, 2011

I am normally pretty careful about my on-line security and privacy. I take a lot of care to ensure that my home network is nailed down tightly and all the clients and servers on it are also nailed down as well as I know how. I don’t use software which is susceptible to the majority of the malware out there; my browser is nailed down as tightly as I can get it whilst still allowing it to be useful (roll on HTML5, I hate flash, but it is so damned useful); I do some, very specific, browsing (such as on-line banking) from within a VM and do not use that browser or machine for anything except that specific activity; I routinely bin cookies and flash LSOs (in fact I find it better to disallow all LSOs in the first place); this blog does not include any email addresses harvestable by ‘bots; my email client is a niche (i.e. minority) product and is configured only to allow text (no HTML or embedded images or webbugs); I use tor when I want to be as anonymous as possible; my local DNS server blocks access to a whole range of addresses I don’t like; and I never respond to unsolicited email.

But I got phished. Damn.

Here’s what happened.

I advertised an unwanted mobile phone on gumtree. I chose gumtree in preference to ebay because a) adverts are free, and b) gumtree allows you to target the advertising to a specific location. I like this idea because it means you can say “I’ve got a doohickey for sale in South London. Come and see it and pay cash if you want it”. My ad gave details of the item for sale and, as is recommended, I chose to have responses emailed to me. Here I made mistake number one – I used my normal email address rather than a disposable one. To be fair, gumtree don’t expose any of your private details, they just forward any responses to the address you give. Here’s where I made mistake number two, I responded to queries about the ad from the address given to gumtree. Damn. Idiot. So stupid.

So why do I think the responses weren’t kosher? Well there were a number of giveaways. Firstly the requests were for information already in the ad (“how much do you want?”); secondly, there were a suspiciously high number of “spilling misteaks” in the emails; thirdly, the correspondent wanted me to mail the ‘phone to a location outside the UK (“Thanks for your quick respond actually i will love to buy the Ad for my Daughter who is currently studying at British international college (BIC) in West Africa so am willing to pay you additional £48.76 for the shipping via Express Air Mail.” (sic)); fourthly, the respondents all seemed desperately keen for me to accept paypal as the preferred payment option. I’m normally quicker on the uptake than this, but sadly it took me four or five emails to realise that there was a pattern here and that the people after the phone seemed to be following a script and were completely ignoring my responses. Here’s a sample:

Someone calling him or her self “Janet Mason”:

“Hello Seller,
Can I know the condition of the item? I think you will accept PayPal. And I will pay the postage and packing cost for the item. If you can send me paypal payment request now and I will make the payment straight away without any delay. Hope to hear from you very soon.”

My response:

“Janet

As the ad say, the phone is in “as new condition”. This means what it says. The phone is completely clean and has no visible markings or scratches.”

“Janet’s” reply:

“Ok send me your paypal Payment request now so that I can make the payment now.”

My response:

“I’d like a bit more detail first please.

Where are you? (Full address and telephone mumber so that I can confirm that I am sending to “Janet Mason”.

Details of your confirmed paypal account (so that I know that Paypal have verifed you).

If you want to know why I am concerned please read the paypal guidance for sellers – particularly the bit about sending only to UK or US based addresses and getting signatures on receipt of goods.

I have received several requests to send the phone to “my daughter/son/nephew” or whatever in various Countries outside the EU. I am naturally suspicious.”

“Janet” then says:

“Hello, 
Am in London right now but due to the nature of my work here in London I will not be able to post the item to my Business Partner Daughter in Nigeria as a New Year Gift. But I will pay for the postage and packing cost via FedEX. Get back to me with your paypal payment request now so that I can make the payment now and get the item posted out tomorrow Morning.”

Correspondence ends…..

Now whilst I have not lost the ‘phone, I have verified a usable email address to a bunch of scammers. I expect my spam volume to that address to increase dramatically. Never mind though, I’m not alone in losing out to the bad guys, and at least I haven’t lost any passwords in the process.

Still, I’m pretty pissed off.

Tags: , , , , , , ,
Posted in privacy, security, trivial musing | No Comments »

scroogle is having a problem

Sunday, July 4th, 2010

I posted a note about scroogle back in January. Scroogle offered an SSL interface to the google engine, and, moreover, didn’t lumber its users with google cookies and sundry other irritations. Since then, however, google themselves have started to offer an SSL interface and, coincidentally, scroogle seem to have started to have some problems.

If you visit the scroogle SSL interface, you get a redirect to a notice which explains why some changes made at google mean that scroogle can no longer work properly. Scroogle managed to get a workaround in place for a few days, but it seems that another google change has finally killed that too unless google can be convinced to help out – unlikely in my view. The scroogle redirect page (dated 1 July 2010) has the following line from Daniel Brandt:

“Thank you for your support during these past five years. Check back in a week or so; if we don’t hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers.”

That in itself is bad enough, but as a separate new posting explains, scroogle now seems to be the target of a botnet aimed at swamping its servers. As Brandt goes on to say:

“Google has a few hundred thousand servers, while Scroogle has six. They can put up with sites that spread malware, but our bandwidth is limited. Even if Google relents and the output=ie interface returns, this Scroogle malware problem could still be increasing at that point. Eventually it alone might shut down Scroogle.”

Sad. I hate to see the little guy lose out.

Tags: , , , ,
Posted in networks and networking, privacy, security, trivial musing | No Comments »

webDAV in lighttpd on debian

Wednesday, March 31st, 2010

I back up all my critical files to one of my slugs using rsync over ssh (and just because I am really cautious I back that slug up to another NAS). Most of the files I care about are the obvious photos of friends and family. I guess that most people these days will have large collections of jpeg files on their hard disks whereas previous generations (including myself I might add) would have old shoe boxes filled with photographs.

The old shoe box approach has much to recommend it. Not least the fact that anyone can open that box and browse the photo collection without having to think about user ids or passwords, or having to search for a some way of reading the medium holding the photograph. I sometimes worry about how future generations’ lives will be subtly impoverished by the loss of the serendipity of discovery of said old shoe box in the attic. Somehow the idea of the discovery of a box of old DVDs doesn’t feel as if it will give the discoverer the immediate sense of delight which can follow from opening a long forgotten photo album. Old photographs feel “real” to me in a way that digital images will never do. In fact the same problem affects other media these days. I am old enough (and sentimental enough) to have a stash of letters from friends and family past. These days I communicate almost exclusively via email and SMS. And I feel that I have lost some part of me when I lose some old messages in the transfer from one ‘phone to another or from one computing environment to another.

In order to preserve some of the more important photographs in my collection I print them and store them in old fashioned albums. But that still leaves me with a huge number of (often similar) photographs in digital form on my PC’s disk. As I said, I back those up regularly, but my wife pointed out to me that only I really know where those photos are, and moreover, they are on media which are password protected. What happens if I fall under a bus tomorrow? She can’t open the shoebox.

Now given that all the photos are on a networked device, it is trivially easy to give her acccess to those photos from her PC. But I don’t like samba, and NFS feels like overkill when all she wants is read-only access to a directory full of jpegs. The slug is already running a web server and her gnome desktop conveniently offers the capability to connect to a remote server over a variety of different protocols, including the rather simple option of WebDAV. So all I had to do was configure lighty on the slug to give her that access.

Here’s how:-

If not already installed, then run aptitude to install “lighttpd-mod-webdav”. If you want to use basic authenticated access then it may also be useful to install “apache2-utils” which will give you the apache htpasswd utility. The htpasswd authentication function uses unix crypt and is pretty weak, but we are really only concerned here with limiting browsing of the directory to local users on the local network, and if someone has access to my htpasswd file then I’ve got bigger problems than just worrying about them browsing my photos. There are other authentication mechanisms we can use in lighty if we really care – although I would argue that if you really want to protect access to a network resource you shouldn’t be providing web access in the first place.

To enable the webdav and auth modules, you can simply run “lighty-enable-mod webdav” and “lighty-enable-mod auth” (which actually just create symlinks from the relevant files in /etc/lighttpd/conf-available to /etc/lighttpd/conf-enabled directories) or you can activate the modules directly in /etc/lighttpd/lighttpd.conf or the appropriate virtual-host configuration file with the directives:

server.modules += ( “mod_auth” )
server.modules += ( “mod_webdav” )

lighty is pretty flexible and doesn’t really care where you activate the modules. The advantage of using the “lighty-enable-mod” approach however, is that it allows you to quickly change by running “lighty-disable-mod whatever” at a future date. The symlink will then be removed and so long as you remember to restart lighty, the module activation will cease.

Now to enable the webdav access to the directory in question we need to configure the virtual host along the following lines:

$HTTP["host"] == “slug” {
# turn off directory listing (assuming that it is on by default elsewhere)
server.dir-listing = “disable”

# turn on webdav on the directory we wish to share
$HTTP["url"] =~ “^/photos($|/)” {
webdav.activate = “enable”
webdav.is-readonly = “enable”
webdav.sqlite-db-name = “/var/run/lighttpd/lighttpd.webdav_lock.db”
auth.backend = “htpasswd”
auth.backend.htpasswd.userfile = “/etc/lighttpd/htpasswd”
auth.require = ( “” => ( “method” => “basic”,
“realm” => “photos”,
“require” => “valid-user” ) )
}

}

Note that the above configuration will allow any named user listed in the file /etc/lighttpd/htpasswd read only access to the “/photos” directory on the virtual host called “slug” if they can sucessfully authenticate. Note also, however, that because directory listing is turned off, it will not be possible for that user to access this directory with a web browser (which would be possible if listing were allowed). Happily however, the gnome desktop (“Connect to server” mechanism) will still permit authenticated access and once connected will provide full read only access to all files and subdirectories of “/photos” in the file browser (which is what we want). The “auth.backend” directive tells lighty that we are using the apache htpasswd authentication mechanism and the file can be found at the location specified (make sure this is outside webspace). the “auth.require” directives specify the authentication method (bear in mind that “basic” is clear text, albeit base64 encoded, so authenticatiion credentials can be trivially sniffed off the network); the “realm” is a string which will be displayed in the dialogue box presented to the user (though it has additional functions if digest authentication is used); “require” specifies which authenticated users are allowed access. This can be useful if you have a single htpasswd file for multiple virtual hosts (or directories) and you wish to limit access to certain users.

Passing authentication credentials in clear over a hostile network is not smart, so I would not expose this sort of configuration to the wider internet. However, the webDAV protocol supports ssl encryption so it would be easy to secure the above configuration by changing the setup to use lighty configured for ssl. I choose not to here because of the overhead that would impose on the slug when passing large photographic images – and I trust my home network……

Now given that we have specified a password file called “htpasswd” we need to create that file in the “/etc/lighttpd” configuration directory thusly:

httpasswd -cm /etc/lighttpd/htpasswd user

The -c switch to htpasswd creates the file if it does not already exist. When adding any subsequent users you should omit this switch or the file will be recreated (and thus overwritten). The “user” name adds the named user to the file. You will be prompted for the password for that user after running the command. As I mentioned above, by default the password will only be encrypted by crypt, the -m switch used forces htpasswd to use (the stronger) MD5 hash. Note that htpasswd also gives the option of SHA (-s switch) but this is not supported by lighty in htpasswd authentication. Make sure that the passwd file is owned by root and is not writeable by non root users (“chown root:root htpasswd; chmod 644 htpasswd” if necessary).

The above configuration also assumes that the files you wish to share are actually in the directory called “photos” in the web root of the virtual host called “slug”. In my case this is not true because I have all my backup files outside the webspace on the slug (for fairly obvious reasons). In order to share the files we simply need to provide a symlink to the relevant directory outside webspace.

Now whenever the backup is updated, my wife has access to the latest copy of all the photos in the shoebox. But let’s hope I don’t fall under a bus just yet.

Tags: , , , , , , , ,
Posted in linux and unix, networks and networking, tips, tricks and howtos, trivial musing | No Comments »

unplugged

Tuesday, March 30th, 2010

My earlier problems with the sheevaplug all seem to have stemmed from the fact that I had installed Lenny to SDHC cards. As I mentioned in my post of 7 March, I burned through two cards before eventually giving up and trying a new installation to USB disk. This seems to have fixed the problem and my plug is now stable. I had a series of problems with the SD cards I used (class 4 SDHC 8 GB cards) which may have been related to the quality of the cards I used. Firstly the root filesystem would often appear as readonly and the USB drive holding my apt-mirror (mounted as /home2) would similarly appear to be mounted read-only. This seemed to occur about every other day and suggested to me that the plug had seen a problem of some kind and rebooted. But of course since the filesystem was not writeable, there were no logs available to help my investigations.

I persevered for around two weeks during which time I completely rebuilt both the original SD card and another with Martin’s tarball, reflashed uboot with the latest from his site, and reset the uboot environment to the factory defaults before trying again. I also changed /etc/fstab to take out the “errors=remount-ro” entry against the root filesystem, and reduced the number of writes to the card by adding “noatime, commit=180″ in the hope that I could a) gain stability, and b) find out what was going wrong. No joy. I still came home to a plug with a /home2 that was either unmounted or completely unreadable or mounted RO. The disk checked out fine on another machine and I could find nothing obvious in the logs to suggest why the damned thing was failing in the first place. Martin’s site says that “USB support in u-boot is quite flaky”. My view is somewhat stronger than that, particularly when the plug boots from another device and then attaches a USB disk.

But I don’t give up easily. After getting nowhere with the SDHC card installation from Martin’s tarball, I reset the uboot environment on the plug to the factory default (again) and then ran a network installation of squeeze to a 1TB USB disk (following Martin’s howto). It took me two attempts (I hit the bug in the partitioner on the first installation) but I now have a stable plug running squeeze. It is worth noting here that I had to modify the uboot “bootcmd” environment variable to include a reset (as Martin suggests may be necessary) so that the plug will continue to retry after a boot failure until it eventually loads. The relevant line should read:

setenv bootcmd ‘setenv bootargs $(bootargs_console); run bootcmd_usb; bootm 0×00800000 0×01100000; reset’

The plug now boots successfully every second or third attempt. So far it has been up just over ten days now without any of the earlier problems recurring.

My experience appears not to be all that unusual. There has been some considerable discussion on the debian-arm list of late about problems with installation to SDHC cards. Most commentators conclude that wear levelling on the cards (particularly cheap ones) may not be very good. SD cards are sold formatted as FAT or FAT32 (depending on the capacity of the card). Modern journalling filesystems such as ext3 on linux result in much higher read/write rates and the quality of the cards becomes a much greater concern. Perhaps my cards just weren’t good enough.

Tags: , , , , , , ,
Posted in electronics, linux and unix, networks and networking, tips, tricks and howtos, trivial musing | No Comments »

plug instability

Sunday, March 7th, 2010

I’m still having a variety of problems with my sheevaplug. Not least of which is the fact that SDHC cards don’t seem to be the best choice of boot medium. I have had failures with two cards now and some searching of the various on-line fora suggests that I am not alone here. In particular, SD cards seem to suffer badly under the read/write load that is routine for an OS writing log files – let alone one running a file or web server. I have also had several failures with my external USB drive. It seems that the plug boots too quickly for the USB subsystem to initialise properly. This means that there is not enough time for the relevant device file (/dev/sda1 in my case) to appear before /etc/fstab is read to mount the drive. A posting on the plugcomputer.org forum suggested a useful workaround (essentially introducing a wait), but even that was only partially sucessful. Sometimes it worked, sometimes it didn’t. In fact, the USB drive still often fails after a random (and short) time and then remounts read-only. Attempts to then remount the drive manually (after a umount) result in failure with the error message “mount: special device /dev/sda1 does not exist”.

In my attempts to cure both the booting problems and the USB connection failures I have installed the latest uboot (3.4.27 with pingtoo patches linked to from Martin’s site) and updated my lenny kernel to Martin’s 2.6.32-2-kirkwood in the (vain as it turns out) hope that the latest software would help. Here I also discovered another annoying problem – installing the latest kernel does not result in a new kernel image, the plug still boots into the old kernel until you run “flash-kernel”. Fortunately this is reasonably well known and is covered in Martin’s troubleshooting page.

I will persevere for perhaps another week with the current plug configuration. If I can’t get a stable system though I will try installing to USB drive (perverse as that may seem) and changing the uboot to boot from that rather than the flaky SD card. Most on-line advice suggests that USB support in uboot is rather “immature”, but it can’t be any worse than the current setup. My thinking is that if I can introduce a delay in the boot process by uboot so that I can successfully boot from an external HDD, the drive connection might then be stable enough to be usable.

Of course I could be completely wrong.

Tags: , , , , , , ,
Posted in electronics, linux and unix, networks and networking, tips, tricks and howtos, trivial musing | No Comments »

from slug to plug

Sunday, February 28th, 2010

Well this took rather longer than expected. I intended to write about my latest toy much earlier than this, but several things got in the way – more of which later.

About three or four weeks ago I bought myself a new sheevaplug.

image of sheevaplug

The plug has been on sale in the US for some time, but UK shipping costs added significantly to $99 US retail price. Recently however, a UK supplier (Newit) has started stocking and selling the plugs over here – and at very good prices too. My plug arrived within three days of order and I can thoroughly recommend Newit. The owner, one Jason King no less (fans of 1970′s TV will recognise the name), kept me informed of progress from the time I placed the order to the time it was shipped. He even took the trouble to email me after shipping to check that I had received it OK. Nice touch, even if it was automated.

Looking much like a standard “wall wart” power supply typically attached to an external disk, the plug is actually quite chunky, but it will still fit comfortably in the palm of your hand. Inside that little box though there is enough computing power to make a slug owner more than happy. The processor is a 1.2 GHz Marvell Kirkwood ARM-compatible device and it is coupled with 512MB SDRAM and 512MB Flash memory. Compare that to the poor old slug’s 266 MHz processor and 32 MB of flash and you can see why I’d be interested – particularly since the plug can run debian (and Martin Michlmayr has again provided a tarball and instructions to help you out.

The plugs come in a variety of flavours, but all offer at least one USB 2.0 port, a mini usb serial port, gigabit ethernet and an SDHC slot. This means that debian (or another debian based OS such as Ubuntu) can be installed either to the internal flash or to one of the external storage media available. Newit ship the plugs in various configurations and will happily sell you a device fully prepared with debian (either Lenny or Squeeze according to your taste) on SD card to go with the standard Ubuntu 9.04 in flash. Personally I chose to install debian myself, so I bought the base model. (No, I’m not a cheapskate, I just prefer to play. Where’s the fun in buying stuff that “just works”?)

Given that Martin’s instructions suggest that installing to USB disk can be problematic, and that I have debian lenny on my slugs (and had a spare 4 Gig SDHC card lying around) I chose to use his tarball to install lenny to my SDHC card. Firstly I formatted the card (via a a USB mounted card reader) as below:

/dev/sdb1 512 Meg bootable
/dev/sdb2 2.25 Gig
/dev/sdb3 1024 Meg swap

(note that the plug will see these devices as “/dev/mmcblk0pX” when the card is loaded. The “/dev/sdbX” layout simply reflects the fact that I was using a USB mounted card reader on my PC. )

I then downloaded and installed Martin’s lenny tarball to the newly formatted card and as instructed edited the /etc/fstab to match my installation. Martin’s fstab file is below:

# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
# Boot from USB:
/dev/sda2 / ext2 errors=remount-ro 0 1
/dev/sda1 /boot ext2 defaults 0 1
/dev/sda3 none swap sw 0 0
# Boot from SD/MMC:
#/dev/mmcblk0p2 / ext2 errors=remount-ro 0 1
#/dev/mmcblk0p1 boot ext2 defaults 0 1
#/dev/mmcblk0p3 none swap sw 0 0

As you can see it defaults to assuming a USB attached device. You need to comment out the USB boot entries and uncomment the SD/MMC entries if. like me, you are intending to boot from SD card. At this stage I also edited “/etc/network/interfaces” to change the eth0 interface from dhcp to static (to suit my network) and I changed “/etc/resolv.conf” because the default includes references to cyrius.com and a local IP address for DNS.

Before we can boot from the SD card, we have to make a few changes to the uboot boot loader configuration to stop it using the default OS on internal flash (where the factory installed Ubuntu resides). Again, Martin’s instructions are helpful here but he points to the openplug.org wiki for instructions in setting up the necessary serial connection to the plug. On my PC (running Ubuntu 8.04 LTS) I got no ttyUSB devices by default and “modprobe usbserial” did not work but “modprobe ftdi_sio vendor=0x9e88 product=0x9e8f” did work for me.

Now open a TTY session using cu thusly “cu -s 115200 -l /dev/ttyUSB1” – don’t use putty on linux, it doesn’t allow cut and paste which can be very useful if you are following on-line instructions (of course it helps if you cut and paste the right instructions). I found that booting is too fast if you have to switch on the plug and then return to a keyboard so I recommend simply leaving the terminal session open and resetting the plug with a pin or paper clip. Hit any key to interrupt the boot session, then follow Martin’s instructions for editing the uboot environment.

My plug was running v 3.4.16 of uboot, so at first I used version 3.4.27 (downloaded from plugcomputer.org) and loaded that via tftp as described by Martin, But this turmed out to be a mistake because my plug failed to boot thereafter. I got the following error message via the serial console:

## Booting image at 00400000 …
Image Name: Debian kernel
Created: 2009-11-23 17:25:02 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1820320 Bytes = 1.7 MB
Load Address: 00008000
Entry Point: 00008000
Verifying Checksum … Bad Data CRC

Some searching suggested that the uboot image was probably the problem and that reverting to v3.4.19 would solve this. So I downloaded 3.4.19 from “vioan’s” post “#6 on: November 16, 2009, 03:21:34 PM” at the plugcomputer.org forum and reflashed the plug with that image. Success – my plug now booted into debian lenny. Tidy up, update the OS and add a normal user as recommended and we’re ready to go.

My plug was intended to replace the slug I was using as my local apt-mirror. That mirror is now fairly large because I have a mix of 32 and 64 bit ubuntus (of varying vintages) and 386 and ARM versions of debian. I therefore recycled an unused 500 gig lacie USB disk and mounted that as /home2 (originally as /home, but I soon changed that when I wanted to unmount it frequently and then lost my home directory….) Copying the apt-mirror (175 Gig) over the network from my old slug was clearly going to take forever – high speed networking is not the slug’s forte, so I mounted both the slug and the plug’s disks locally on my PC and copied the files over USB – much faster. It was here that I discovered why the old lacie disk (a “designed by porsche” aluminium coated beast) was lying idle. I’d forgotten that it sounded like a harrier jump jet on take off when in use. I put up with that for a week – just long enough to get me to a free weekend when I could rebuild the old slug (now used as just an NTP server and the webcam) to boot from a 4 gig USB stick so that I could recycle its disk onto the plug. I’ve just finished doing that.

One other problem I found with the plug which caused me much head scratching (and delayed my writing this as I noted above) was that it consistently failed to boot back into my debian install after a “reboot” or “shutdown -r” – I had to power cycle the device to get it to boot properly. I spent some time this weekend with the serial port connected before I noticed (using “printenv” at the uboot prompt) that I had mixed up the uboot environment variables printed on Martin’s site. I had actually copied part of the instructions for the USB boot variant instead of the correct ones for the SD card boot. Sometimes “cut and paste” can be a mistake.

Tags: , , , , , , ,
Posted in electronics, linux and unix, networks and networking, tips, tricks and howtos, trivial musing | No Comments »

isp shenanigans

Saturday, February 20th, 2010

I have recently been off-line. And I am less than happy about the reasons.

My ISP recently informed me that it was changing it’s back end provider from Entanet to Vispa. Like many small ISPs, my provider does not have any real infrastructure of its own, it simply repackages services provided by a wholesaler who does have the necessary infrastructure in a process commonly called “whitelabelling”. This whitelabel approach is particularly common amongst providers of webspace and it normally works fine. Amongst the smaller ISPs there are many who are simply Entanet resellers. And until recently Entanet had a good name for pretty solid service. Well not any more.

I had not noticed any particular problems and was slightly surprised to hear from my ISP that they were unhappy with the service they were getting from Entanet. Apparently there had been frequent network outages for many of their customers and so they had chosen a new provider and were notifying their customers of impending moves, Of course this would mean some local configuration changes so customers were advised in advance of those changes and the dates for action. Apart from preparing to change the ADSL login details on my router, in my case I also had to ensure that my SSH and other login details on various external services I have or use were modified to accept the new fixed IP address assigned to my router (I tend to lock down such services so that they only accept connections from my IP address, not foolproof I know, but it all helps).

In the migration advice letter, my ISP advised its customers to set up new direct debit arrangements for Vispa and cancel the existing ones to Entanet. That letter advised that any over or under charge either way during transition would be sorted out between the providers. So I did as I was advised and waited for the big day (approximately 10 days away). Big mistake.

About a week before the date of transition I found my web traffic intercepted and blocked by Entanet with the message “Your account has been blocked. Please contact your internet service provider”. This blockage only occurred on web traffic (my email collection over POP3S and IMAPS continued to work, as did ICMP echo requests and ssh connections out). This action actually pissed me off even more than I would have been if Entanet had completely cut my connection. It also, incidentally, betrayed the fact that they were using a transparent web proxy on the connection – not something that makes me very happy. But simply blocking web traffic was obviously designed to annoy me and make me contact my ISP and strongly suggests to me that Entanet were usnsure of their legal right to cut me off completely. Further, in my view, intercepting my web traffic in this way may actually have been illegal.

Interestingly, even http traffic aimed inbound to my ADSL line (where I run a webcam on one of my slugs) was similarly intercepted as is evidenced by this link from changedetection.com. Obviously, the imposition of the message from Entanet was picked up by changedetection as an actual change to that web page.

So I emailed Entanet and my ISP, pointing out that my contract was with them and not Entanet and told them to sort it out between themselves. I, as a customer, did not expect to be penalised simply because my ISP had decided to change its wholesaler. Meanwhile, I decided to bypass Entanet’s pathetic and hugely irritating web block by tunneling out to a proxy of my own. Of course I could have used my existing tor connection, but that is not always as fast as I would like, particularly at peak web usage hours, so I set up a new proxy on another of my VPSs using tinyproxy, listening on localhost 8118 (the same as privoxy on my tor node). I then set up an ssh listener on my local machine and set firefox to use that listener as its proxy – again, much as I had for tor. Bingo. Stuff you Entanet.

Unfortunately, it did not stop there. Entanet’s rather arrogant response to my email was to insist that I re-establish a direct debit with them for the few days remaining before the changeover (despite them having had my payment in advance for the month in question). No way, so I ignored this request only to find that Entanet then throttled my connection to 0.02 Mb/s – see the speedtest result below.

speedtest image

speedtest image

This sort of speed is just about usable for text only email, but is absolutely useless for much else. Now I had originally been given two separate dates for the changeover by my ISP, so in a fit of over enthusiastic optimism on my part, I tried to convince myself that the earlier (later corrected) date given was the correct one and so I reconfigured my router in the hope it would connect to Vispa, No deal. Worse, when I then tried to fall back to the (pitiful) Entanet connection, I found it blocked completely. I was thus without a connection for some four days (including a very long weekend).

So far my new connection looks good. But apart from my disgust with Entanet, I have not been overly impressed with the support I have received from my ISP during these problems. I’ll keep an eye on things – I may yet move of my own volition.

[Addendum] just by way of comparison, the test result below is what I expect my connection speed to look like. Test run at around 21.45 on Sunday 21 February 2010.

speedtest image

That’s a bit better. Note however that this test was direct from Vispa’s network rather than through my ssh tunnel.

Tags: , , ,
Posted in trivial musing | No Comments »

« Older Entries