have I been pwned?

Well, I don’t think so. But for a while I was not entirely sure.

Following the move last November of trivia from a VM on UK2’s datacentre in London to our new home on a faster VM on ITLDC’s network I have been making a variety of minor changes and doing some essential housework. One of the biggest changes of course (fortunately for me as it turns out) was the complete separation of my two main services (mail and web) onto different VMs in different countries. My mailserver is now housed in Nuremburg where I have made some additional changes (for example I now run opendkim on it). This VM in Prague now houses just my webserver and of course is home to this blog.

Following the configuration changes which I noted in my last post, I spent a short while checking my web server logs – particularly the error log. That log shows a variety of messages such as:

SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
SSL: 1 error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low
SSL: 1 error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher

which indicate browsers or other clients attempting to connect using either protocols I no longer support (such as ssl3) or TLS versions lower than I support server side. This is expected behaviour and the frequency of such log entries should decline over time as clients out there catch up with current acceptable security standards. I know from long experience that there are still a huge number of old, outdated browsers still in use – possibly on equally old and outdated platforms such as Android 4, or Windows XP (yes, it still exists). As a cross check I started looking through my access logs and sure enough I found user agent strings like:

“Mozilla/5.0 (Linux; U; Android 4.4.2; en-gb; SM-T310 Build/KOT49H)”

(almost certainly the default broswer on an old Android tablet) and

“Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.54 [en]”

(probably Opera 7 on Windows XP).

So, no real surprise then that clients like that should have problems negotiating a secure connection with my server. However, this is where things started to look a little weird.

As I was scanning through my access logs I noticed entries like the following (client side IP addresses deliberately obfuscated with RFC1918 entries) :

192.168.1.1 microsoft-hub-us.com – [23/Dec/2019:19:27:52 +0100] “GET / HTTP/1.1” 200 169284 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50728)”
192.168.10.10 microsoft-hub-us.com – [23/Dec/2019:21:20:52 +0100] “GET / HTTP/1.1” 200 155411 “-” “Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)”
192.168.1.15 microsoft-hub-us.com – [23/Dec/2019:21:32:17 +0100] “GET / HTTP/1.1” 200 169272 “-” “Mozilla/5.0 (Linux; U; Android 4.1.2; ja-jp; SC-06D Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30”
192.168.2.20 microsoft-hub-us.com – [24/Dec/2019:15:32:47 +0100] “GET / HTTP/1.1” 200 169257 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/67.0.3396.99 Safari/537.36”

Now that says that the client connecting from the address in the first field is asking for the root of a webserver called “microsoft-hub-us.com”. I don’t own that domain, I don’t host that domain and the only way that sort of entry could possibly appear in my logs is if someone, somewhere who /does/ own that domain has made a DNS A record pointing to my IP address. Well, actually there is another scenario. It is entirely possible that someone has made a local DNS entry (such as in a local hosts file or a local DNS server using say, DNSMasq or Unbound, pointing to my IP address. I do exactly that sort of thing myself when I move webs between servers so that I can test the entry on the new server before switching my DNS. However, given the sheer number of the log entries (in the tens of hundreds!) from multiple different source addresses it seemed to me unlikely that this latter scenario is accurate. So, someone has a DNS entry pointing to me that I don’t know about.

Having found one odd host name I did a quick scan for others (awk { print $2 } accesslogfiles | grep -v mydomains) and found around fifteen more. Fortunately, with the exception of just one other domain name, none of the others (for which there were mercifully few connections) pointed to my address at the time I checked the DNS (late January). I assumed that those domains were also potentially hostile, and had now moved elsewhere but some of course could just have been accidents – it can happen (be careful of ascribing to malice that which could be simple stupidity).

I decided to concentrate on the main domain appearing in my logs and did a bit more research.

Firstly, the DNS:

mick@shed ~ $ dig +ttlunits -t a microsoft-hub-us.com

;; ANSWER SECTION:
microsoft-hub-us.com. 10m IN A 195.123.246.12

So there is an A record pointing to me – and it has a suspiciously low TTL value (meaning the owner can change it quickly).

What about the nameserver(s)?

mick@shed ~ $ dig +ttlunits -t ns microsoft-hub-us.com

;; ANSWER SECTION:
microsoft-hub-us.com. 10m IN NS a.dnspod.com.
microsoft-hub-us.com. 10m IN NS b.dnspod.com.
microsoft-hub-us.com. 10m IN NS c.dnspod.com.

The standard TTL value for an A record is about 1 day (or longer) so that nameservers can cache the answer to the question: where is “baldric.net”? for a reasonable length of time before having to ask again. Most people would only use a very short TTL (here it is 10 minutes) if they wanted to be able to move the domain name queried to a new host very quickly. There are legitimate reasons for this. For example if you manage a server which you know you are going to move to a new address shortly and you wish to minimise the lag in the DNS. However, “Bad Guys” (TM) are known to do this sort of thing when they point to compromised hosts on the net. Said “Bad Guys” will also often use an obviously spoofed domain (this one is meant to look like a genuine Microsoft domain for the MS Hub) in phishing attacks.

Conclusion? This looks bad.

What about the whois record?

That shows the domain to have been registered on the 6th of November last year. About three weeks before I was given the IP address.

mick@shed whois microsoft-hub-us.com

Domain name: microsoft-hub-us.com
Registry Domain ID:
Registrar WHOIS Server: whois.eranet.com
Registrar URL: http://www.eranet.com
Updated Date: 2019-11-06T00:00:00+08:00
Creation Date: 2019-11-06T22:52:03.0000Z
Registrar Registration Expiration Date: 2020-11-06T00:00:00+08:00
Registrar: ERANET INTERNATIONAL LIMITED
Registrar IANA ID: 1868

Interestingly though, a current whois lookup gives:

mick@shed ~ $ whois microsoft-hub-us.com

Domain Name: MICROSOFT-HUB-US.COM
Registry Domain ID: 2452061049_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.eranet.com
Registrar URL: http://www.eranet.com
Updated Date: 2020-02-24T02:19:45Z
Creation Date: 2019-11-06T14:52:03Z
Registry Expiry Date: 2020-11-06T14:52:03Z
Registrar: Eranet International Limited
Registrar IANA ID: 1868

So the record was last updated on the 24th of this February. And sure enough, that domain name disappeared from the DNS records at around 1.30 GMT on 24 February. Note – it disappeared – it did not get pointed to a holding address (such as 127.0.0.1). But I’m getting ahead of myself here so let’s take a step back again.

Have I been pwned? And do I host malware?

Next step is some research on the domain name. A search for “microsoft-hub-us.com” and “malware” turns up:

Firstly, joesandbox which, sure enough, shows that domain name on my IP address is dropping malware. Ouch. Not good. Not good at all. But wait. The submission time for that analysis (top right of the full page) is shown as 08.11.2019 14:08:03 – only a couple of days /after/ the domain was registered and again some three weeks /before/ I was given the IP address.

Secondly, also at joesandbox he shows that my IP address was hosting another set of malware. Also not good, but again at a date before I had the IP address (Submission Time: 08.11.2019 17:56:13)

Thirdly, again at joesandbox, there is a very detailed, and scary, analysis of the behaviour of a downloaded file “contract1.doc” taken from the spoofed domain on 8 November last year. That analysis is here and a copy is shown below:

The behaviour graph in that analysis, shown in the next image, shows how the dropper works and sure enough, my IP address is implicated. But again that analysis dates to before I inherited the IP address.

In the HTTPS packet section of the Network Behaviour analysis (shown in the next image below) it says that the domain originally had its own Lets-Encrypt certificate, valid from Friday November 8 2019 to Thursday February 6 2020. That in itself is interesting because it means that from the date I moved trivia (5 December last year) to that IP address with my own Lets-encrypt certificate covering my domains (and ONLY my domains) all future requests hitting my server with an invalid host name would get a big scary “This Connection is Untrusted” browser warning. But of course I know from my logs that almost all of those warnings were ignored.

Finally, in the “Domains” section of the analysis there is a link to Virustotal so that is the next port of call.

The image below, taken from VirusTotal gives us an overview of a scan from three months ago which shows that 7 engines out of a total of 76 used, recorded the domain as malicious. I’m not sure whether that is good, or bad. If, as I now believe to be the case, that domain is/was a source of windows malware then I would have expected a much higher percentage of positives. But no matter at this stage.

The “relations” section of the analysis (in the image below) shows the results of the scans for various URLS on the domain and give a worrying result of positives for dates when I /did/ have the IP address in question. Fortunately however, a click on the links (for example “https://microsoft-hub-us.com/download.php” at 9/12/2019) gives the result that 2 months ago the URL gave a 404 not found. (As it should)

So, whilst the document originally at that URL registered as malware on a variety of tests, when the link was last tested by VirusTotal (9 December 2019) it was no longer found. I should add here that I have run a recursive find on my webserver for all the documents listed by a variety of analysts out there, and additionally for any “.doc” or “.exe” or “.xls” files and come up blank. So I am reasonably confident (he said!) that the site is clean.

The “details” section of the VirusTotal analysis (below) gives us the DNS records for the domain, together with the the HTTPS certificate seen when the domain was last checked (which is now mine, and not the original spoof microsoft certificate).

That same page gives us the results of a google search for the domain name as below:

About 5 results (0.20 seconds)

Sort by: Relevance

Kyle Ehmke on Twitter: “Most likely TA505 domain box-en-au[.]com …
twitter.com
19 Nov 2019 … This one is calling out to an older site: microsoft-hub-us[.]com. I have to imagine a wave of new docs will pop up soon. 1 reply 0 retweets 2 likes.

AS204957 – LAYER6, UA – urlscan.io
urlscan.io
www.ram6.ac.th, 2 days ago, 3 MB, 47, 9, 6. microsoft-hub-us.com, 2 days ago, 5 MB, 91, 3, 3. xurl.es/12l4x, 3 days ago, 54 KB, 30, 4, 2. microsoft-hub-us.com …

The Blacklist from UT1 bad Recipe # # This recipe demonstrates …
www.pluckeye.net
File Format: text/plain
17 Nov 2016 … … microsoft-cnd-en.com deny microsoft-home-en.com deny microsoft-hub-us.com deny microsoft-live-us.com deny microsoftoffice365box.com …

Ransomware Clop : une communication officielle trop tardive ?
www.lemagit.fr
25 nov. 2019 … … et évoqué publiquement sharefile-cnd[.]com, ms-home-live[.]com, box-en-au[.] com, box-en[.]com, microsoft-hub-us[.]com, microsoft-live-us[.] …

『男性は2019年6月、他の者と共謀してゆうちょ銀行のネットバンキングに …
benzaiten.dyndns.org
2019年12月17日 … 2019年11月07日22時15分59秒 RT @kyleehmke: Possible TA505 domain microsoft-hub-us[.]com was registered on 11/6. Less confidence in …

So, onwards to Kyle Ehmke who is a researcher for Threat Connect. His tweets of 7 and 8 November last year say:

Kyle Ehmke
@kyleehmke
·
7 Nov 2019
Possible TA505 domain microsoft-hub-us[.]com was registered on 11/6. Less confidence in that association though as the domain is not currently hosted.

and

Kyle Ehmke
@kyleehmke
·
8 Nov 2019
The microsoft-hub-us[.]com domain is now hosted at 195.123.246[.]12.

That hosting at that address cannot have lasted long, because I was allocated the IP address along with my new Debian VM on 27 November last year. But I know from later analysis that the A record for that domain name continued to point to my address right up until 24 February this year. Kyle refers to the threat actor as “TA505“, known as an active and prolific attacker operating in the financial sphere – i.e. a criminal group motivated by money (rather than politics). On 19 November last, Kyle posted again on Twitter that:

“Another most likely TA505 domain registered at essentially the same time as box-en-au[.]com: microsoft-store-en[.]com. Currently hosted at 103.199.16[.]197.”

to which Kyle Eaton responded:

“Nice find! Seems to me, when a new site is spun up they’ll send out an older static doc for a while before we start getting new files. This one is calling out to an older site: microsoft-hub-us[.]com. I have to imagine a wave of new docs will pop up soon.”

Searches for TA505 on Mitre give us the information that:

“TA505 is a financially motivated threat group that has been active since at least 2014. The group is known for frequently changing malware and driving global trends in criminal malware distribution.”

The Mitre page about the group lists some 15 different attack techniques used and 5 different pieces of malware. Mitre also reference Proofpoint analyses of the group going back several years. A quick search on the Proofpoint site gives us a list of 27 separate postings about the group. Their profile of TA505, dating from September 2017, describes the group as:

“One of the more prolific actors that we track – referred to as TA505 – is responsible for the largest malicious spam campaigns we have ever observed, distributing instances of the Dridex banking Trojan, Locky ransomware, Jaff ransomware, The Trick banking Trojan, and several others in very high volumes.”

That profile gives an interesting timeline of activity attributed to TA505 going back to June 2014. So these guys have around for some time, they are well established, well organised and (apparently) quite successful. If I /have/ been pwned, at least it was done by a professional group……

Finally, urlscan.io is listed as having scanned the domain on December 23 2019. That analysis, given in the image below, shows the front page of my blog as it looked at the time with my “Welcome to Prague” post at the top.

Reassuringly, moreover, the ioscan analysis shows the website as “clean”. The historic list of scans given by ioscan (and shown below) detail two failed connects three months ago, four successful connects to my server pipe.rlogin.net, and a final failed connect attempt one hour ago (from the time of finishing writing this).

It is worth noting at this stage that the reason connections to the spoofed microft domain resulted in delivery of my blog is because (I confess) I had been dumb in my web server configuration. The server software I use (lighttpd) has a very simple virtual hosts configuration system. That mechanism allows you to set whatever host name you wish to be served depending upon the host name requested. So if you have a virtual host called “something.com” and another called “somethingelse.net” you merely need to tell the webserver to deliver the appropriate pages from the directories called “/var/www/pages/something.com” or “/var/www/pages/somethingelse.net” (or wherever you configure your web root to be). But what if someone connects just to the IP address and not a virtual host name? Here is where I was dumb. Lightty allows you to set a “default” virtual host and serve that in such a case. I had set “baldric.net” as my default. Thus anyone coming in and asking for a domain that I don’t host would get my blog. Stupid. Very stupid on reflection. And as soon as I realised that was what was happening (and why my logs were full of crud) I changed it so that the default went to an empty directory with a blank index file. I have actually improved that now by changing the default to give a “403 Forbidden” response. Better, and more logical methinks.

Now, as I was documenting all this (for just this sort of blog post) I received an email from my hosting provider (ITLDC) saying:

Ticket no: “blah”

195.123.246.12/32 is listed on the Spamhaus Block List – SBL
195.123.246.12/32 is listed on the Spamhaus Botnet Controller List – BCL
2020-02-23 12:30:36 GMT | itldc.com
TA505 botnet controller @195.123.246.12

and telling me that accordingly they had shut my VM down. (For which I cannot blame them. In their position I would do exactly the same.)

Bugger – and this is where I am exceptionally grateful that I had separated my blog from my mail. I cannot afford to have my email server blacklisted by spamhaus, correct or not. It takes a long time to gain a clean record for a mail server. One bad listing can see you blocked by multiple other mail providers and things then start to cascade out of control. I can afford to lose my blog for a while, but not my mail reputation.

I responded to the ticket explaining what I had found myself and asking that they investigate further. I also offered copies of all my log extracts showing what I had found so far together with whatever further assistance they might need. Unfortunately, this happened at a weekend and my ticket had to be escalated to second line advanced support, who only worked monday to friday.

It was long weekend.

On the Monday after the weekend we corresponded further on the ticket and by about lunchtime I got the good news that whilst Spamhaus is a trusted source of abuse reports and, as is right and proper a responsible ISP will take appropriate steps to prevent damage to their own or others’ networks following a Spamhaus alert, in this case the report turned out to be a false alarm and I could have my VM back.

Even better news was that they offered to allocate me a new IP address – which I happily accepted. As you can see (because you are reading this) we are back up on that new address and all looks good.

Conclusion then? I probably have /not/ been pwned at trivia. The most likely scenario seems to be that a previous user of that IP address had been compromised, or, given that the TA505 mob seem to have gone to the trouble (and had been able) to get their own, valid, Lets-encrypt certificate for the spoof domain, that group itself rented a VM on that address. My money is on a root compromise of a previous owner.

My immense gratitude to the support team at ITLDC and my particular thanks to Dmitry in that support team for taking my problem seriously, investigating it appropriately and coming up with a satisfactory outcome. That kind of service would be exceptional even if I were paying them ten times what I actually pay for my VMs. Given how little I do spend with them it is nothing short of amazing. I can think of several hosting providers who would happily throw you under a bus following a spamhaus report rather than spend time supporting you.

So, my thanks again to Dmitry and his team.

Go buy some VMs from them. They are excellent.