Posts Tagged ‘ISP’

t-mobile resets its policy?

Thursday, January 12th, 2012

As I have mentioned in other posts here, I run my own mail server on one of my VMs. I do this for a variety of reasons, but the main one is that I like to control my own network destiny. Back in October last year I noticed an interesting change in my mail experience with my HTC mobile (actually my wife first noticed it and blamed me, assuming that I had “twiddled with something” as she put it). Heaven forfend.

My mail setup is postfix/dovecot with SASL authentication and TLS protecting the mail authentication exchange. My X509 certs are self generated (and so not signed by any CA). I pick up mail over IMAPS (when mobile) and POP3S (at home – for perverse reasons of history I like to actually download mail to my main desktop over POP3 and archive it to two separate NAS backups). I send via the standard SMTP port 25 but require authentication and protect the exchange with TLS.

My mail had been working fine ever since I set it up some years ago, but as I said, back in October my wife complained that she could no longer send email from her HTC mobile (we both use t-mobile as the network provider). She was at work at the time so away from my home network. Both our phones are setup to use use wifi for connectivity where it is available (as it is at home of course). When my wife complained I checked my phone and it could send and receive without problem. But when I switched wifi off, thus forcing the data connection though the mobile network, I got the same problem as my wife reported. On checking my mail server logs I read this:

postfix/smtpd[28089]: connect from unknown[149.254.186.120]
postfix/smtpd[28089]: warning: network_biopair_interop: error reading 11 bytes from the network: Connection reset by peer
postfix/smtpd[28089]: SSL_accept error from unknown[149.254.186.120]:-1
postfix/smtpd[28089]: lost connection after STARTTLS from unknown[149.254.186.120]
postfix/smtpd[28089]: disconnect from unknown[149.254.186.120]

(the ip address is one of t-mobile’s servers on their “TMUK-WBR-N2″ network)

Everything I could find about that sort of message suggested that the client was tearing down the connection because there was something wrong with the TLS handshake and it was not trusted. Checking earlier logs, I found that t-mobile’s address had apparently changed (to the address above) recently. So I assumed that some recent network change following the Orange/T-mobile merger had been badly managed and all would be well again as soon as the problem was spotted. Wrong. It persisted. So I had to investigate further. As part of my investigation of the error, I tried moving mail from port 25 to 587 (submission) because that sometimes gets around the problem of ISPs blocking, or otherwise interfering, with outbound connections from their networks to port 25, No deal. In fact it looked as if t-mobile were blocking all connections to port 587 (I assumed a whitelisting policy block, or again, a cockup).

So, the scenario was: mail works when connecting over wifi and using my domestic ISP’s network, but doesn’t when using t-mobile’s 3G network. Symptoms point to a lack of trust in the TLS handshake. Tentative conclusion? There is an SSL/TLS proxy somewhere in the mobile operator’s chain. That proxy sucessfully negotiates with our phones, but when it gets my self certified X509 cert from the server. it can’t authenticate it and decides that the connection is untrusted so tears it down. My server sees this as the client (my phone) tearing down the connection. [As it turns out, this conclusion was completely wrong, but hey].

I said in an email at the time to a friend whose advice I was seeking, “I suspect cockup rather than outright conspiracy, but if my telco is dumb enough to stick a MITM ssl proxy in my mail chain, they really ought to have thought about handling self signed certs a little better. Otherwise it sort of gives the game away.”

In response, he very sensibly suggested that I should run a sniffer on the server and check what was going on. At that time, I was busy doing something else so I didn’t. And because the problem was intermittent (and my wife stopped complaining) I never got around to properly investigating further. (I should explain that I rarely send mail from my mobile nowadays. I just read mail there and wait until I get home to a decent keyboard and can reply to whatever needs handling from there. My wife just gave up bothering to try).

I should have persisted because of course I wasn’t the only one to experience this problem.

Back in November, a member of the t-mobile discussion forum called “dpg” posted a message complaining that he could not connect to port 587 over t-mobile’s 3G network. In response, a member of the t-mobile forum team suggested that dpg might reconfigure his email so that it was relayed via t-mobile’s own SMTP server. Not unreasonably, dpg didn’t think this was an acceptable response – not least because he would then have to send his email in clear. He then posted again saying that “the TLS handshake fails when the mail client receives a TCP packet with the reset (RST) flag set.” (This is a bad thing (TM). Further, he posted again saying that he had set up his own mail server and repeated earlier tests so that he could see both ends of the connection. At the client side he posted mail from his laptop tethered to his phone which was connected to the t-mobile 3G network. By running sniffers at both ends of the connection he was able to prove to his own satisfaction that something in the t-mobile network was sending a RST and tearing down any connection when a STARTTLS was seen. Again, in a later post in response to one from another poster who apparently manages several mail servers and had been looking at the same issue for a client, dpg says:

“I must say I’m not too pleased to discover that T-Mobile may be snooping all traffic to check for SMTP messages. I have demonstrated that they may be doing this by running a SMTP server on a non-standard port and finding that they still sent TCP reset packets during TLS negotiation – so they must be examining all packets and not just those destined for TCP ports 25 and 587.

I’m also not that keen on T-Mobile spoofing/forging TCP resets. This is the sort of tactic resorted to by the Great Firewall of China (http://www.lightbluetouchpaper.org/2006/06/27/ignoring-the-great-firewall-of-china/) and also by Comcast back in 2007 (https://www.eff.org/wp/packet-forgery-isps-report-comcast-affair) until the US FCC told them to stop (http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-08-183A1.pdf).”

Then 9 days ago, dpg posted this message:

“I finally got to the bottom of this. I was contacted by T-Mobile technical support today and was told that they are now actively looking for and blocking any TLS-secured SMTP sessions. So, it is a deliberate policy after all, despite what the support staff have been saying on here, twitter and on 150. They told me it is something they have been rolling out over the last three months – which explains why it was intermittent and dependent on IP address and APN to begin with.

So, the only options for sending email over T-Mobile’s network are:
- unencrypted but authenticated SMTP (usually on port 25)
- SSL-encrypted SMTP (usually on port 465)
- unauthenticated and unencrypted email to smtp.t-email.co.uk

TLS-encrypted SMTP sessions are always blocked whether or not they are on the default port of 587.”

(As an aside, there is, of course, another alternative. You can ditch t-mobile as your provider and pick one which doesn’t use DPI to screw your connections. You pays your money….)

Following this, a new poster called “mickeyc” said this:

“I’ve been experiencing this exact same problem. I run my own mail server which has SSL on port 465 and also uses TLS on port 587. I used wireshark to confirm that the RST packets are being spoofed. This is the exact same technology used by “The Great Firewall of China”. I have two t-mobile sims. One is about a year old and doesn’t experience this problem (yet), one is a few weeks old and does.”

He went to say that he had also experienced problems with his OpenVPN connections and would be blogging about the problem (damned bloggers get everywhere) and sure enough, Mike Cardwell did so at grepular.com. That blog post is worth reading because it has an interesting set of comments and responses from Mike appended.

Mike’s post seems to have been picked up by a few others (El Reg has one, and as Mike himself has pointed out, boingboing.net has a particularly OTT post which seems to say that he is accusing t-mobile of something he clearly isn’t.

Finally, two days ago, dpg posted this:

“I’m pleased to report that T-Mobile is no longer blocking TLS-secured email on port 587. As a follow-up to an email exchange over the Christmas period I was contacted today to say that, contrary to what I had been told previously, it was never a deliberate policy to block TLS-secured outgoing email. There was a problem with some equipment after all, which was resolved yesterday.”

I tried again myself today. Initially, I got the same old symptoms (“lost connection after STARTTLS”) then I rebooted my ‘phone and lo and behold I could send email.

Like Mike, I tend to the cockup over conspiracy theory, it’s more likely for one thing. IANAL, but it seems to me that it would be in breach of RIPA part I, Unlawful Interception, for the telco to intercept my SMTP traffic in the way it seems to have been doing. That is not likely to be a deliberate act by a major UK mobile network provider.

But I’ll still keep an eye on things.

Tags: , , , , ,
Posted in network (in)security, networks and networking, privacy, security, trivial musing | No Comments »

isp shenanigans

Saturday, February 20th, 2010

I have recently been off-line. And I am less than happy about the reasons.

My ISP recently informed me that it was changing it’s back end provider from Entanet to Vispa. Like many small ISPs, my provider does not have any real infrastructure of its own, it simply repackages services provided by a wholesaler who does have the necessary infrastructure in a process commonly called “whitelabelling”. This whitelabel approach is particularly common amongst providers of webspace and it normally works fine. Amongst the smaller ISPs there are many who are simply Entanet resellers. And until recently Entanet had a good name for pretty solid service. Well not any more.

I had not noticed any particular problems and was slightly surprised to hear from my ISP that they were unhappy with the service they were getting from Entanet. Apparently there had been frequent network outages for many of their customers and so they had chosen a new provider and were notifying their customers of impending moves, Of course this would mean some local configuration changes so customers were advised in advance of those changes and the dates for action. Apart from preparing to change the ADSL login details on my router, in my case I also had to ensure that my SSH and other login details on various external services I have or use were modified to accept the new fixed IP address assigned to my router (I tend to lock down such services so that they only accept connections from my IP address, not foolproof I know, but it all helps).

In the migration advice letter, my ISP advised its customers to set up new direct debit arrangements for Vispa and cancel the existing ones to Entanet. That letter advised that any over or under charge either way during transition would be sorted out between the providers. So I did as I was advised and waited for the big day (approximately 10 days away). Big mistake.

About a week before the date of transition I found my web traffic intercepted and blocked by Entanet with the message “Your account has been blocked. Please contact your internet service provider”. This blockage only occurred on web traffic (my email collection over POP3S and IMAPS continued to work, as did ICMP echo requests and ssh connections out). This action actually pissed me off even more than I would have been if Entanet had completely cut my connection. It also, incidentally, betrayed the fact that they were using a transparent web proxy on the connection – not something that makes me very happy. But simply blocking web traffic was obviously designed to annoy me and make me contact my ISP and strongly suggests to me that Entanet were usnsure of their legal right to cut me off completely. Further, in my view, intercepting my web traffic in this way may actually have been illegal.

Interestingly, even http traffic aimed inbound to my ADSL line (where I run a webcam on one of my slugs) was similarly intercepted as is evidenced by this link from changedetection.com. Obviously, the imposition of the message from Entanet was picked up by changedetection as an actual change to that web page.

So I emailed Entanet and my ISP, pointing out that my contract was with them and not Entanet and told them to sort it out between themselves. I, as a customer, did not expect to be penalised simply because my ISP had decided to change its wholesaler. Meanwhile, I decided to bypass Entanet’s pathetic and hugely irritating web block by tunneling out to a proxy of my own. Of course I could have used my existing tor connection, but that is not always as fast as I would like, particularly at peak web usage hours, so I set up a new proxy on another of my VPSs using tinyproxy, listening on localhost 8118 (the same as privoxy on my tor node). I then set up an ssh listener on my local machine and set firefox to use that listener as its proxy – again, much as I had for tor. Bingo. Stuff you Entanet.

Unfortunately, it did not stop there. Entanet’s rather arrogant response to my email was to insist that I re-establish a direct debit with them for the few days remaining before the changeover (despite them having had my payment in advance for the month in question). No way, so I ignored this request only to find that Entanet then throttled my connection to 0.02 Mb/s – see the speedtest result below.

speedtest image

speedtest image

This sort of speed is just about usable for text only email, but is absolutely useless for much else. Now I had originally been given two separate dates for the changeover by my ISP, so in a fit of over enthusiastic optimism on my part, I tried to convince myself that the earlier (later corrected) date given was the correct one and so I reconfigured my router in the hope it would connect to Vispa, No deal. Worse, when I then tried to fall back to the (pitiful) Entanet connection, I found it blocked completely. I was thus without a connection for some four days (including a very long weekend).

So far my new connection looks good. But apart from my disgust with Entanet, I have not been overly impressed with the support I have received from my ISP during these problems. I’ll keep an eye on things – I may yet move of my own volition.

[Addendum] just by way of comparison, the test result below is what I expect my connection speed to look like. Test run at around 21.45 on Sunday 21 February 2010.

speedtest image

That’s a bit better. Note however that this test was direct from Vispa’s network rather than through my ssh tunnel.

Tags: , , ,
Posted in trivial musing | No Comments »

we’ve moved

Wednesday, September 9th, 2009

As I mentioned in the last post. I decided to move trivia from its old home on a shared hosting platform to my own VPS at bytemark. I also mentioned that this was proving trickier than it should – for no real good reason. However, the move is now complete and the blog is now completely under my control on my own VPS. So if anything goes wrong, I have only myself to blame.

So why did it take so long? Apart from the fact that I went on holiday immediately after the last post, the main reasons are twofold; firstly the difference in versions between that on my old host (2.2.1) and the current release (2.8.4) were sufficiently great to make the upgrade process trickier than it need have been; but secondly, and more importantly, my old provider’s DNS management process was less than helpful.

Before committing to the move, I naturally tested the installation and migration first on my new platform. This raised the problem of how I could install as “baldric.net” without clashing with the existing blog (I didn’t want to install under a different domain name for fairly obvious reasons). Changing my local DNS settings to point the domain name at my new IP address solved this problem (changing /etc/hosts would also have worked) but that meant that I could not have both old and new blogs on screen for comparison at the same time. Irritating, but not ultimately an insuperable problem. In moving to 2.8.4 I discovered that none of my (blogroll) links migrated properly and I had to recreate them all by hand. This took rather longer than I had anticipated, but it proved a useful exercise because I found some broken links in the process. They are currently still broken but at least I know that and I’ll fix them shortly. Because I use lighttpd and not the more usual apache I also had to address the problem of getting permalinks to work properly, but that didn’t prove too difficult – I’ll cover that in a separate post about wordpress on lighttpd.

Having got the new installation up and running to my satisfaction, I now wanted to point my domain name at the new blog. This is where I ran into some oddities in the way 1and1 set up their blog hosting and domain management. Ordinarily it is pretty easy to switch the A record for a 1and1 hosted domain (I have several) from the default to a new address. Not so if you have a blog hosted on that domain – the domain becomes “unmodifiable”. Technical support were initially not particularly helpful since they didn’t seem to understand my problem (and there were worrying echoes of my experiences with BT “support”). But this simply reaffirmed my belief that I was better off controlling my own destiny in future.

Eventually I was told that the only way I could unlock the domain to allow me to point to a new A record was to a) move the blog to a new domain (tricky if you don’t have one, and a pretty dumb idea anyway) or b) delete the blog (an even dumber idea if. like me, you are cautious enough to want to test the transition before committing). Eventually I decided to move the blog to a spare domain. I’ll delete it in the next week or so. Meanwhile, if you find an apparent duplicate of trivia on a completely different domain, you know why.

Tags: , , , , ,
Posted in linux and unix, trivial musing | No Comments »

leaving BT Broadband

Saturday, December 15th, 2007

My contract with BT has now expired and I am shortly to move my ADSL connection to one of the Entanet resellers (TitanADSL). All the Entanet resellers I have read about get good reviews. I picked TitanADSL because they offer additional webspace and mySQL databases on top of their broadband service. With luck my IP service will improve hugely (BT consistently throttle service at peak times) and I know that my “support” service will improve beyond recognition.

I know I shouldn’t have bothered, but I actually made the mistake of emailing BT Broadband “support” requesting a MAC (Migration Activation Code) so that I could get my new supply sorted. I received the response below. I cannot believe that I actually received an email from someone “trying to be part of the solution”. Needless to say I received no MAC so I phoned the number given on the the BT website and got the code over the phone in minutes.

——–
BT Email

Dear Sir / Madam,

Thank you for your e-mail dated 6/12/07 regarding your request for MAC code.

With regards to your email, I would like to inform you that I have to forward this matter to the relevant team for further assistance. Therefore, I would request you to kindly forward your account details, i.e. the customer account number and the telephone number in reply. We need this information for security reasons, as well as to access your account and assist you further.

I can assure you that on receipt of your account details we will assist you in an appropriate way and will make every possible endeavour to solve your concern as soon as possible.

I realise that I have not been able to resolve your concern immediately. I can assure you that I am trying my best to be a part of the solution and in the meantime I would like to thank you in anticipation of your continued patience and co-operation, and to assure you of our best intentions at all times.

Thank you for contacting BT.

Yours Sincerely,
eContact Customer Service

Tags: , ,
Posted in networks and networking, trivial musing | No Comments »

another update to correspondence with a corporation

Wednesday, January 17th, 2007

Since my last post at the end of last year I have been testing my ST780 with a variety of alternative VOIP providers whilst at the same time trying to get BT to sort out my connection. I also lodged a formal complaint about the appalling level of technical support with the BT complaints department on 30 December.

The complaints department initially responded to me on 4 January with an acknowledgement and a comment that I could expect a fuller reply in 24 hours. On the 8th of January I received the following gem:

“BT Broadband – Complaint Management Team

Dear Sir

Thank you for your e-mail regarding the problems you are experiencing with your BT Broadband service. Please accept my apologies for the inconvenience this has caused you.

Unfortunately we are unable to assist with technical issues, we have however passed your email to our technical support team, who will be in contact with you in the next 3 to 4 working days to work towards a satisfactory resolution. Should you wish to contact the technical support desk please call 0845 600 7030.

I would again like to apologise for the problems you have experienced. I do hope this information will be of assistance to you.

Kind regards
BT Broadband – Complaint Management Team”

Since that date I have heard nothing – though I have now received my shiny new hub (which I do not intend using).

Now since the substance of my complaint was that the technical support department was neither technical nor supportive I have decided that it is futile to continue down this road and I will simply escalate my complaint (on paper) to the Customer Relations Manager.

Meanwhile, just to prove that there is nothing wrong with my ST780 router, as I mentioned above, I have been experimenting with alternative VOIP providers and have now signed up with Sipgate. Sipgate offers free VOIP services within its own network and with peer networks such as FWD. It only charges for its gateway out to the PSTN. But its charges are very reasonable indeed. Sipgate also offers a rather neat opportunity to gain a UK geographic based telephone number for no additional charge. During my testing (for free) I could successfully dial in to my new Sipgate number from the PSTN and mobile networks but initially could not dial out to the Sipgate test number. Given the problems I have with BT I contacted Sipgate support who very generously credited my account with a small test sum so that I could check outbound connectivity to the PSTN. It worked fine so I have now signed up to Sipgate’s services.

Now compare this attitude and response from a company with whom I had no contractual relationship and had paid no money with that woeful response from BT to whom I pay a very considerable sum of money each month.

Tags: , , , ,
Posted in networks and networking, trivial musing | No Comments »

update to correspondence with a corporation

Sunday, December 31st, 2006

BT support finally called me back again today (two days late, but hey) and again attempted to transfer me immediately to the Broadband Talk support department. Before I allowed them to do so, however, I made certain that the person I was talking to fully understood my problem. I believe she did. But she confirmed that there was absolutely nothing that she or anyone else in the broadband support department could do to help me – the problem lay with another department. Despite my protestation that we had been around this particular tree before I allowed myself to be transferred.

As before, the guy I was transferred to at Broadband Talk support had no idea what my problem was and I had to explain yet again, why I was attempting to get support. (As an aside, I find this lack of joined up support apalling. The recipient of the support call should have the full details of the caller’s problem on-screen when he or she takes the call – particularly in cases such as this where the support organisation has actually placed the call). As soon as he realised I was not using a Home Hub, the problem became clear to him. My router is at fault.

Actually, no, my router is fine. It works perfectly for all other services and actually works fine for VOIP services on FWD. But no, BT is correct, I am wrong. To be fair to this particular support person, he offered to provide me with a replacement Hub (an offer I eventually accepted, if only so that I can check its configuration in detail to compare against my ST780).

Maybe I’m overly paranoid, but I get the distinct impression that BT has decided to lock its services to the Hub rather than allow people to use alternative products on its network. I actually prefer to have something that I control, rather than something that BT control. When my new Hub arrrives I’ll dump its configuration and try to figure out how BT are locking out my ST.

Tags: , , ,
Posted in linux and unix, networks and networking, trivial musing | No Comments »

correspondence with a corporation

Friday, December 29th, 2006

Recently I have been experiencing a small problem with my BT broadband connection. I should point out that in general my experience wth BT’s broadband offering is very good. Whilst not the cheapest around, the quality and reliability of the connection are better than I have heard reported from friends and colleagues with other ISPs. But for the past two weeks I have been having great difficulty in getting the support department to fix (or even recognise) a problem I am having with my VOIP connection.

I upgraded to BT Total Broadband (up to 8Mbps) from my previous 2Mbps BT contract for two reasons – firstly the additional bandwidth obviously, but secondly because it offered free VOIP calls after 18.00 and at weekends. Thus I would be getting a second phone line for no additional cost. Indeed, given that the contract I changed to was actually cheaper than my existing BT contract, I was initially quite pleased.

Too soon.

The BT Total Broadband contract includes a DSL router made for BT by Thomson) and branded by BT as the “Home hub”. That hub offers VOIP connectivity and WiFi access as well as wired ethernet connectivity to the net. Despite the fact that BT have heavily customised the standard Thomson Speedtouch interface, it is still an attractive package. However, I am not the standard user that BT intends to support. For a start I use Linux, not Windows. This normally gives support departments problems.

Having signed up, taken delivery and configured my Hub all was well until I opted to include what BT calls its “softphone” package as well. This is a software only SIP package which is supposed to allow PC users to make VOIP calls from the PC in addition to using a standard phone (or DECT phone) attached to the hub. This can be useful for laptop users for example who may be using their machines in rooms without immediate access to the telephone handset. Obviously, being a Linux user, the BT softphone is useless to me, so I downloaded some potential alternatives – linphone, wengophone and x-lite to try. Unfortunately for me, my Hub threw a fit when I fired up the softphones. Indeed, after attempting to use linphone, the router rebooted, and rebooted, and rebooted, and rebooted. Eventually I tried a hardware reset to get back to factory settings, but even that failed and the hub now just broadcasts BOOTP requests over the ethernet connection.

Having fritzed my hub I went back to using my D-Link router until I could source a replacement. I soon took delivery of a Thomson Speedtouch ST780WL (big brother to the hub). I configured this in the same way as the hub in the expectation that all would be well. My mistake. Whilst, indeed, all is well with my standard internet connectivity, I cannot get the VOIP service to work. In desperation (trust me) I contacted BT Broadband support. That started a thread of correspondence which has lasted so far for two weeks and has descended to the level of farce. No-one , but no-one in BT support seems capable of reading and understanding my request. My experience is shown at Problems with BT Broadband I have anonymised the relevant contact details for obvious reasons.

Tags: , , ,
Posted in linux and unix, networks and networking, trivial musing | No Comments »