Archive for the ‘tips, tricks and howtos’ Category

moxie’s proxy

Sunday, January 22nd, 2012

Moxie Marlinspike, a security researcher probably best known for his SSL proxy tool, likes google even less than I do. His googlesharing website says:

“Google thrives where privacy does not. If you’re like most internet users, Google knows more about you than you might be comfortable with. Whether you were logged in to a Google account or not, they know everything you’ve ever searched for, what search results you clicked on, what news you read, and every place you’ve ever gotten directions to. Most of the time, thanks to things like Google Analytics, they even know which websites you visited that you didn’t reach through Google. If you use Gmail, they know the content of every email you’ve ever sent or received, whether you’ve deleted it or not.

They know who your friends are, where you live, where you work, and where you spend your free time. They know about your health, your love life, and your political leanings. These days they are even branching out into collecting your realtime GPS location and your DNS lookups. In short, not only do they know a lot about what you’re doing, they also have significant insight into what you’re thinking.”

His solution to this problem was interesting. He came up with the idea of a proxy system which would intercept all google queries, strip off identifying material (such as cookies and UserAgent strings and other HTTP headers) substitute new identifiers and mix the requests up with those from other users before forwarding to google. Implementation depended upon a Firefox addon (nothing for other browsers) which identified google queries and forwarded them to the proxy. All other traffic was untouched.

image of googlesharing proxy

I stopped using google (except via scoogle) some time ago, and when Moxie’s new proxy first surfaced I thought it interesting but susceptible to the same problem I discussed in mid 2009 when writing about Hal Roberts’ experience of GIFC – all you are doing is shifting knowledge of your searches from google to a new intermediary. However, Moxie later addressed this problem with the release of version 0.20 of his addon so I thought I’d take another look at it. Unfortunately the addon won’t work with FF 9 (which I am using). Moxie’s proxy is not the only one out there however. Because he released the code under an open source licence, others have picked it up. I found one at gs.netsend.nl. They also provide an updated FF addon which will work with versions up to 15 (i.e. probably around next wednesday given the speed with which Mozilla is currently shipping new FF releases).

Once the addon is installed, it gives you two proxy options in the preferences settings – one is the original proxy.googlesharing.net, the other is gs.netsend.nl itself. In testing I found that the original googlesharing proxy seemed to be off-line, but when using the netsend.nl proxy I was reassured to see the message “Search results anonymized by GoogleSharing” added to the google homepage. I was even more reassured that my sniffer showed a connection to vps1101.pcextreme.nl on 31.21.98.201 and not to any known google network.

So, will I use it? Maybe. But the proxy mechanism seems to be unreliable. In many tests, the proxy connection seemed to be bypassed and the connection was obviously made direct to google (as evidenced by my sniffer). I think this failure is doubly unfortunate because it does not fail safe (i.e. the connection does not simply fail with an error message, it passes you direct through to google). This could lead the unwary to think that they are protected when in fact they are not.

I prefer not to use google at all. And in those cases where I do want to compare results with another search engine I prefer to do so via tor. But it is one more option in my toolkit if used carefully. And if using it pisses off google, then it is worth it occasionally.

tails in a spin

Thursday, January 12th, 2012

When I first tested running a tails mirror on one of my VMs, the traffic level reported by vnstat ran at around 20-30 GiB per day. I figured I could live with that because it meant that my total monthly traffic would be unlikely to exceed my monthly 1TB allowance. However, when I checked the stats on that server last week (around the 9th of Jan) I found that I was shipping out around 150 GiB per day and vnstat was predicting a monthly total of close to 3 TB. As the tails admins said when I told them that I would have to shut off the mirror on that VM while I sorted something, “Ooops”. Ooops indeed. I couldn’t chance a massive bill for exceeding my bandwidth allowance by quite that much. The actual stats for 4, 5, 6, 7, 8 and 9 January before I pulled the plug were: 34.23 GiB, 69.14 GiB, 178.31 GiB, 131.68 GiB, 99.05 GiB and 133.27 Gib. It turns out that tails 0.10 was released on 4 January and I hadn’t been prepared. A lesson learned.

Having shut down and had the DNS round robin amended, I attended to finding some way of throttling my traffic so that I could live within my allowance whilst still providing a useful mirror. I scratched my head for a while before stumbling on the obvious, I should be throttling at application level. (Sometimes I find that I miss simple answers because I am looking for complicated ones).

I started out by assuming that I should be using tc and iptables mangling, or something like the userspace tool trickle, all of which looked horribly more complicated than the approach taken by tor (which allows you to simply set the acceptable bandwidth rate to some limit, plus set an accounting period maximum of some total transfer limit per day/week whatever). And of course it turns out that my webserver (lighttpd) allows something similar. Just set the server limit to some chosen max transfer rate and, if necessary, also impose a per IP max rate. The magic configuration file options are:

# limit server throughput to 300 kbytes/sec (~3000 kbits/sec)
server.kbytes-per-second = 300
# and limit individual connections to 50 kbytes (~500 kbits/sec)
connection.kbytes-per-second = 50

I tested this by pulling a copy of the tails iso from one of my other VMs which has a high bandwidth connection and got acceptable (and expected) results. So now I can go back on-line later this month safe in the knowledge that I’m not going to blow all my bandwidth in one week.

tunnelling X over ssh

Monday, December 19th, 2011

OK, yes, I know there are probably already a gazillion web pages on the ‘net explaining exactly how to do this, but I got caught out by a silly gotcha when I tried to do this a couple of days ago, so I thought I’d post a note.

Firstly, X is not exactly a secure protocol, nor is it easy to filter at NAT firewalls, so the ability to tunnel it over ssh is hugely welcome. In fact, ssh can be used to tunnel practically any other protocol you care to name, so it should be your first port of call should you wish to connect to a remote system using an insecure protocol. (I use it to wrap rsync for example).

I don’t run X on my VMs (there is no need, they don’t run desktop software) and I had not previously seen the need to run X based graphical programs on those servers. However, a couple of days ago I thought it would be really useful to run etherape on one particular remote server so that I could watch the traffic patterns. Normally I use iptraf (which is ncurses based) when I want to monitor network traffic in real time, but etherape is pretty cool and gives a nice graphical view of your network connections. But it runs on an X based gui.

So. I changed the remote server’s sshd_config to enable X forwarding (“X11Forwarding no” becomes “X11Forwarding yes”) and restarted sshd. On my desktop I similarly changed my local ssh_config file to allow X forwarding (“ForwardX11 no” becomes “ForwardX11 yes”) to obviate the need to use the -X switch on the command line. I then installed etherape on the remote server and fired it up only to get the message “Error: no display specified”. Sure enough “echo $DISPLAY” showed nothing. But I had thought (and everything I had read confirmed) that ssh should take care of setting the appropriate display when X11 forwarding was set.

So I then tried setting a display manually (export DISPLAY=localhost:10.0 on the remote server) and then got the response “Error: cannot open display: localhost:10.0″. So, still no deal. I spent some time scratching my head (and reading man pages) and sent off a query to my local Linux User group in parallel asking for advice. They were gentle with me.

The first, and rapid, response, said:

On the server:

sudo apt-get install xauth

Then disconnect and reconnect the client.

Jobs a good un.

Thank you Brett.

So the moral is, make sure that you have X authorisation working properly on the remote system (check for the existence of $HOME/.Xauthority) if you experience the same symptoms I did.

webcam mark II

Tuesday, September 27th, 2011

Upgrading the slugs to squeeze killed the webcam. At first I thought that squeeze was missing the necessary gspca drivers, but no, a quick look in /dev revealed an entry for video0 and “lsmod” reported “gspca_zc3xx” loaded correctly. This is a different driver to that which my camera loaded in lenny (spca5xx) but a quick search around the web confirmed that the camera should be happy with the new driver. So clearly the webcam program itself was at fault (confirmed by checking the output when running the program by hand). Time to find an alternative.

A scan of the debian repositories turned up a bunch of possibilities, some of which I had looked at in the past when first installing the camera. I eventually plumped for fswebcam because it was quick and easy to install and configure, it seemed to be actively supported (unlike Gerd Knorr’s old program) and it didn’t need a GUI. More to the point, it works with my old Logitech camera.

The program can be run from the command line with option switches or configured to read its options from a config file. And despite being advertised as a tiny, it has quite a rich feature set. It can even perform simple manipulation of the captured image, such as resizing, averaging multiple frames or overlaying a caption on the image. Nifty.

My current config file is shown below as an example. I may change this because I’m not quite happy with the brightness/contrast mixture and I’m still playing with the options available. Still, a recommended package.

# /etc/fswebcam.conf
#
# config file for fswebcam utility
#
# device and input are defaults
device /dev/video0
input 0
# log messages – only use this is debugging setup
# log /var/log/fswebcam.log
# repeat image capture every 30 seconds
loop 30
# set number of frames to skip if the camera sends bad frames on startup.
# Note – this is only necessary when using the camera for video capture
# skip 10
# background the webcam process
background
# set resolution of image (default is 384 x 288)
resolution 320X240
# set the palette to use in the output format (here JPEG) and the jpeg quality
# (my camera doesn’t like this option so it is commented out)
# palette JPEG
jpeg 95
# set some additional controls
# (use “fswebcam -d v4l2:/dev/video0 –list-controls” to see options)
set brightness=55%
set contrast=40%
# place the banner at the bottom of the image (default)
bottom-banner
# and colour it black
banner-colour #000000
# set the font to use in the banner title and timestamp
font /usr/share/fonts/truetype/msttcorefonts/arialbd.ttf
# set the title and timestamp to display in the banner – (timestamp uses strftime format)
title “Webcam on the Slug”
timestamp “%d %B %Y at %H:%M:%S”
# where we save the image
save /home/web/webcam.baldric.net/images/webcam.jpeg
#
# end

squeezing the slugs

Monday, September 26th, 2011

Debian 6 (squeeze) has been the current stable version since February 2011. The latest version (6.02) was released in late June. I have put off updating my slugs from lenny (old stable) for a while because I wanted to see how others faired before committing myself. Indeed, initial reports on the debian arm list indicated that the upgrade could be problematic. Worse, a completely clean install of squeeze turns out to be impossible because the debian installer uses more memory than is physically available on the slugs. So the only way to go, even for a clean new installation, is to install lenny first, then upgrade.

Given that both my slugs are operational, and are now an integral part of my network, I decided to invest in a new one as a development machine to test the upgrade rather than risk fritzing a perfectly good setup. (Back in the day I would have been happy to “fix it ’till it broke”, but these days I don’t really need to experiment that much and I’d rather keep a working system, well, working).

Second hand slugs go for around £25 on ebay, and there are still plenty about, so I bid for one that had only about a day to go and was successful. Unfortunately, when it turned up I found that the power supply was fsckd and so I had to switch off one of my operational slugs in order to test the new one. Happily it appeared to boot up OK so I fired off a disgruntled email to the seller and then ordered a new PSU. The seller claimed that it “worked OK when I boxed it” and didn’t offer to pay for the replacement PSU so I wasn’t too happy with him. I became even less happy when the new PSU arrived and I booted up the slug in preparation for reconfiguration to match my network before installing debian.

The debian installation process is handled via an SSH shell. You need to know the address of the slug in order to connect and install. The installer also needs the addresses of a local DNS server and the default route to the outside world (so it can find the servers containing the installation packages). Now the default, factory settings, for slugs includes a fixed IP address of 192.168.1.77. If this does not match your requirements, it must be changed before reflashing with debian. Guess what? The default address didn’t work, so the previous owner must have reconfigured the slug to match his network and he had not bothered to reset to factory default before selling. Nor had he been considerate enough to let me know the new configuration. Needless to say I won’t be buying anything else from him. Nor did he get decent feedback.

I couldn’t reconfigure the new slug until I could connect so I needed to find out what address it was using. A quick nmap scan of the 192.168.1.0/24 netblock showed that it wasn’t even on the default network range so I fired up wireshark and etherape on one of my machines in the hope of catching the slug arping and getting the address from the request. In the event, etherape proved to be quicker (and easier) in providing the answer since the slug quickly popped up and disclosed its IP address as 192.168.2.10. Adding a route to the 192.168.2.0/24 net then allowed me to finally connect and reconfigure the new beast to suit my network. I then rebooted and started a fresh installation of lenny (as previously described in one of my earlier posts). About four hours later I had a nice new clean slug running lenny.

However, since the main purpose of getting the new slug was to allow me to test the upgrade to squeeze in safety I really needed to make it look like my main operational slug. That slug is my DNS and DHCP server, and primary rsync backup for my desktops. It also runs a webserver. Like most (lazy) sysadmins my system documentation tends to lag somewhat behind reality so I can’t rely on the various readme files I routinely create on my boxes to be completely up to date (or even accurate). Fortunately for me though, debian provides a neat way of snapshotting installed packages on a system. You can then use this snaphot to create a mirror of that system which will include all the same packages. Here’s how:

On the source system:

dpkg –get-selections | grep -v deinstall > packages.txt

This lists all active packages, except those deinstalled, and sticks the list in a text file.

Now copy that file to the target system, ensure that the target system’s “sources.list” file matches that on the source, and then run:

dpkg –clear-selections
dpkg –set-selections < packages.txt
apt-get dselect-upgrade

This will download and install all the packages necessary to get the target system matching the source.

All that is now left to do is copy across any relevant configuration files so that the two systems fully match and then rebooot the target to check that everything looks OK.

New slug now finally matching old slug it was time to upgrade to squeeze. Martin Michalmayr’s excellent website documents the upgrade process in meticulous detail. The key points to note here are his recommendation that you read the release notes for debian 6.0. In particular, note and follow the chapter on upgrades from debian 5.0 before attempting an actual upgrade. One of the main differences between 5.0 and 6.0 is the use of UUIDs to reference disks. In my case this meant changing my /etc/fstab from this:

# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/sda2 / ext3 errors=remount-ro 0 1
/dev/sda1 /boot ext2 defaults 0 2
/dev/sda5 none swap sw 0 0

to this:

# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
UUID=db57451a-e3e5-4d8a-95b9-494c48bb5e8d / ext3 errors=remount-ro 0 1
UUID=022bc211-1c52-4848-9ee1-e211e72b28e4 /boot ext2 defaults 0 2
/dev/sda5 none swap sw 0 0

Before finally starting the upgrade I opened two separate SSH sessions to the slug. In one I ran the upgrade process as below:

first a partial upgrade as recommended at Section 4.4.4, “Minimal system upgrade”

apt-get update
apt-get upgrade

then install the required linux kernel image and udev as outlined in Section 4.4.5. “Upgrading the kernel and udev”

apt-get install linux-image-2.6-ixp4xx
apt-get install udev

followed by

reboot

and

apt-get upgrade
apt-get dist-upgrade

to complete the system upgrade.

Now here is where the second SSH session is most useful. The final upgrade and distribution upgrade installs the file indexing package “apt-xapian-index”. Correspondents on the debian arm list have noted that this package consumes more memory than is available on the poor old slug and it starts swapping itself to death. The process must be killed immediately and the package removed. If you leave it too long after the upgrade has completed you will find it impossible to log in until the initial indexation has completed (in excess of 24 hours or more has been reported) because the system is too busy. I ran “top” in the second shell during the dist-upgrade process and kept an eye on the load averages. As soon as they started climbing above 3 I knew that it was time to watch out for the apt-xapian-indexer and kill it. Once the system load returned to normal I was then able to finalise the upgrade with:

apt-get purge apt-xapian-index
apt-get autoremove

to remove the offending indexer and clean up any residual unneeded packages. A final reboot to check all was well was sufficient to convince me that it was safe to upgrade my two operational slugs using the same process. Testing the upgrade on the new slug in the way I did also meant that I now had a backup slug configured exactly like my main DNS server but running squeeze. Any failure on the remaining upgrade would not then be critical.

I must be getting old. I never used to be this cautious.

wordpress setup

Monday, January 24th, 2011

I have just added a couple of new plugins to this blog and tidied up some old cruft that I had been meaning to get around to for a while. One of the plugins I have added is a really rather good statistcs tool called Counterize II. It provides a very quick (and impressively comprehensive) set of stats about page hits, browser types, referring URLs etc. all readily accessible from the wordpress admin dashboard – so no need to trawl through web logs to find out where your visitors are coming from or which is your most popular post or page. Thoroughly recommended.

In the process of searching for such a plugin I also came across Angie Bowen’s posting about the top ten things she always does when setting up a new blog. Whilst I don’t agree with all her recommendations (identifying the ones I disagree with is left as an exercise for the class) I think this is a very handy aide-memoire. Her pointers about blog security and discussion settings in particular are sensible and worth reading.

a graphical web of trust

Sunday, September 12th, 2010

I recently stumbled upon sig2dot, a gpg/pgp keyring graph generator. In fact this seems to have been around for some time, but I’d never come across it before. It can be used to generate a graph of all of the signature relationships in a GPG/PGP keyring, and, like other visualisation tools, this graphical image producing program can give new insight into relationships between objects.

The sig2dot program itself is available in the debian/ubuntu repositories in the package called “signing-party”. But unless you want to install a shed load of other unnecessary cruft along with it (exim? for god’s sake, why?), I recommend you simply pull the perl code direct from the author’s site. Along with the sig2dot program itself, you will need “neato” from the graphviz package and “convert” from the wondrous imagemagick package suite. If you don’t already have those installed then it is pretty safe to pull them from your distro’s package repository.

That done, try the following:

first create an ascii graphviz dot file ready for neato

$ gpg –list-sigs –keyring ~/.gnupg./pubring.gpg | sig2dot.pl > ~/.gnupg/pubring.dot

now convert to a postscipt file

$ neato -Tps ~/.gnupg/pubring.dot > ~/.gnupg/pubring.ps

before using imagmagick to convert to a png graphic

$ convert ~/.gnupg/pubring.ps ~/.gnupg/pubring.png

Those of you with gpg keyrings may wish to try it out (and no. I’m not going to show you mine).

update to autossh – or how ServerAliveInterval makes this unnecessary

Friday, August 27th, 2010

I had a couple of comments on my earlier post about autossh which suggested that I should look at alternative mechanisms for keeping my ssh tunnel up. Rob in particular suggested that setting “ServerAliveInterval” should work. Oddly I had tried this in the past whilst trying out various configuration options and I swear it didn’t work for me. But since the autossh mechanism felt inelegant I thought I’d revisit my ssh_config file as Rob suggested. And indeed setting ServerAliveInterval to 300 (i.e. 5 minutes) solved my tunnel drop problem. I’d guess that other intervals of less than 1 hour would equally work but I haven’t checked.

I have no idea why my earlier experiments failed.

autossh – or how to use tor through a central ssh proxy

Sunday, August 1st, 2010

Since I first set up a remote tor node on a VPS about this time last year, I have played about with various configurations (and used different providers) but I have now settled on using two high bandwidth servers on different networks. One (at daily.co.uk) allows 750 Gig of traffic per month, the other (a new player on the block called ThrustVPS) allows 1000 Gig of traffic. These limits are remarkably generous given the low prices I pay (the 1000 Gig server cost me £59.42, inc VAT, for a year. OK, that was a special offer, but they are still good value at full price) and they allow me to provide two reasonably fast exit servers to the tor network. Both suppliers know that I am running tor nodes and are relaxed about that. Some suppliers are less so.

I fund fast exit nodes as a way of paying something back to the community – but as I have pointed out before, they also allow me to have a permanent entry point to the tor network which I can tunnel to over ssh, thus protecting my own tor usage from snoopers. For some time I used a configuration based on tyranix’s notes documented on the tor project wiki. But eventually I found that to be rather limiting because it meant that I had to remember to run an ssh listener on each machine I used around the house (my laptop, netbook, two desktops, my wife’s machine etc) and to configure the browser settings as necessary. Then I hit upon the notion of centralising the ssh listener on one machine (I used the plug) in a sort of ssh proxy configuration. This meant that I only had to configure the local browsers to use the central ssh listener as a proxy and everything else could be left untouched. It also has the distinct advantage that my wife no longer has to worry about anything more complex than switching browser when she wants to use tor.

But I hit a snag when initially setting up ssh on the plug. For some reason (which I have never successfully bottomed out) the ssh process dies after an hour of inactivity. This is not helpful. Enter autossh. Using autossh means that the listener is restarted automagically whenever it dies so I can be confident that my proxy will always be there when I need it.

Here’s the command used on the plug to fire up the proxy:

autossh -M 0 -N -C -g -l user -f -L 192.168.57.200:8000:127.0.0.1:8118 tornode

That says:

- M 0 – turn off monitoring so that autossh will only restart ssh when it dies.
- N – do not execute a command at the remote end (i.e. we are simply tunneling)
- C – compress traffic
- g – allow remote hosts to connect to this listener (I limit this to the local network through iptables on the plug)
- l user – login as this user at the remote end
- f – background the process
- L 192.168.57.200:8000 – listen on port 8000 on the given IP address (rather than the more usual localhost address)
- 127.0.0.1:8118 tornode – and forward the traffic to localhost port 8118 on the remote machine called tornode

Of course “tornode” must be running ssh on a port reachable by the proxy. Again, I use iptables on tornode to limit ssh connections to my fixed IP address – don’t want random bad guys knocking on the door.

Now on tornode I have polipo listening on port 8118 on localhost. I used to use privoxy for this, but I have found polipo to be much faster, and speed matters when you are using tor. My polipo configuration forwards its traffic to the tor socks listener on localhost 9050. I also disabled the local polipo cache (diskCacheRoot = “”) because leaving it enabled means that the cache (by default /var/cache/polipo) directory will contain a copy of your browsed destinations in an easily identifable form – not smart if you really want your browsing to be anonymous (besides, my wife deserves as much privacy as do I).

The final bit of configuration needed is simple. Set your chosen browser to use the proxy on port 8000 on address 192.168.57.200. Since I use firefox for most of my browsing, I simply use opera for tor and I have that browser stripped to its basics and locked down as much as possible. Using tor is then simply a matter of firing up opera in place of firefox. This means that I always know when I am using tor or not (and just to reassure myself, the opera homepage is the torcheck page).

You can’t be too careful.

email address images

Monday, May 3rd, 2010

Adding valid email addresses to web sites is almost always a bad idea these days. Automated ‘bots routinely scan web servers and harvest email addresses for sale to spammers and scammers. And in some cases, email addresses harvested from commercial web sites can be used in targetted social engineering attacks. So, posting your email address to a website in a way which is useful to human being, but not to a ‘bot has to be a “good thing” (TM). One way of doing so is to use an image of an address rather than text itself. Of course this has the disadvantage that the address will not be immediately usable by client email software (unless, of course you defeat the object of the exercise by adding an html “mailto” tag to the image) but it should be no big deal for someone who wants to contact you to write the address down.

There are a number of web sites which offer a (free) service which allows you to plug in an email address and then download an image generated from that address. However, I can’t get over the suspicion that this would be an ideal way to actually harvest valid email addresses, moreover addresses which you could be pretty certain the users did not want exposed to spammers. Call me paranoid, but I prefer to control my own privacy.

There are also a number of web sites (and blog entries) describing how to use netpbm tools to create an image from text – one of the better ones (despite its idiosyncratic look) is at robsworld. But in fact it is pretty easy to do this in gimp. Take a look at the address below:

This was created as follows:

open gimp and create a new file with a 640×480 template (actually any template will do);
select the text tool and choose a suitable font size, colour etc;
enter the text of the address in the new file;
select image -> autocrop image;
select layer -> Transparency -> Colour to Alpha;
select from white (the background colour) to alpha;
select save-as and use the file extension .png – you will be prompted to export as png.

Now add the image to your web site.