Using the Google Calendar API from your web site with PHP

February 22nd, 2024

This post is mainly to remind myself how to do this when I inevitably forget in a year or two and want to integrate data from a Google Calendar into a project. There are quite a few steps, but it’s pretty straightforward once you know what to do. As with most programming, there are many other ways of acheiving the same thing, but I won’t be exploring any of them, other than the one that worked for me. All the information here is already out there, but it’s broken into pieces – I couldn’t find any posts showing how to do the whole process from start to finish. So I wrote this.

This presumes you have a Google account and a calendar set up on Google Calendar with events on it which you want to read from a PHP application on a web site. You also need to have a shell account on your server1.

You will see how to set up a project containing a service account, how to set up authorisation for the service account to retrieve the Calendar data for your web page, and how to set up your web site using PHP to access your calendar using the service account.

Setting up the service account

The service account is like a robot user that accesses Google services on behalf of your web site. It has an email address and an ID, and can log in to Google services using a public/private key pair. Your service account has to belong to a project, and each project can contain multiple service accounts. You can have up to 12 free projects, so it’s probably best to create a new one for your calendar data slurper.

  1. Go to https://console.cloud.google.com and log in if necessary. Click on the 3-dot logo / pull down menu at the top to open the “Select a project” box. Click on “Add new project” or, if you already have one you are going to use, select it here and go to step 3.
  2. Give your project a name and see if the “Project ID” is to your liking. I called mine “Testy test”. Click “Create”.
  3. From here on I’m calling the project “Testy test”. You might want to call yours something a bit less stupid.
  4. If the page doesn’t say “Welcome. You are working in Testy test.”, select the project, either from the notifications drop down, or by clicking the 3-dot logo.
  5. Click on the “IAM & Admin” quick access button or select it from the navigation menu on the left. The page will show you as being the principle for the project “Testy test” but not much else. Click on “Service Accounts” on the left navigation pane to bring up an empty list of service accounts belonging to the project.
  6. Click “+ CREATE SERVICE ACCOUNT” to go to the next page, which has 3 steps to creating it. First, give your service account a name and description:
  7. Make a note of the email address. You will need it later when you share the calendar with your service account. Click “CREATE AND CONTINUE”.
  8. Skip the next two optional steps.
  9. You will now be back at the service account list, with your newly created account showing:
  10. You now want to set up authorisation for your service account by creating keys so it can access APIs. Click on the “Actions” dots and choose “Manage keys”. You will go to a page with an empty list of keys for that service.
  11. Click on “ADD KEY” and choose “Create new key”.
  12. Make sure that “JSON” is selected and choose “CREATE”. Your browser will automatically download a file containing your private key. Upload this file to somewhere safe on your server. This is your private key and has to be accessible to your PHP script but must not be kept anywhere accessible by your web server. On a Linux system, keep it somewhere off your home directory, not your web root (usually public_html) directory. This is really important, so much so that the word “not” is not only bold, but red as well. You have to keep this key private.
  13. You now need to enable the Calendar API for your project. Click on “APIs & Services” in the quick links box of the “Welcome” page or the left menu. Click on “+ ENABLE APIS AND SERVICES”.
  14. Do a search for “Calendar” and click on the result that says “Google Calendar API”.
  15. Click on the “Enable” button and you will be taken to the entry for the Calendar API off the “Enabled APIs and Services” page, showing stats for that API.

Your service account is now ready to go.

Setting up your web server with the PHP for using the API

You now need to download the PHP scripts to for use with the Google API. Well, you don’t NEED to, you could write it all yourself, and there is information out there on how to do it. But anyone sane would just use the scripts that Google provide for free.

  1. The easiest way to install the scripts you need for PHP is to use Composer. This is an installer which works in a similar way to Apt. Follow the instructions on this page to install it.
  2. Install the Google API PHP files by using this command in your web site’s root directory:
    composer require google/apiclient:^2.15.0

Now you are ready to start using the calendar API.

Allowing the service account to access your calendar

Before you can access the calendar from your PHP pages, you need to share it with your service account.

  1. Start Calendar in a web browser and click on the burger of the calender you want to use in the “My calendars” section and choose “Settings”:
  2. Scroll down to the “Share with specific people or groups” section and click “Add people and groups”.
  3. Remember part 7 of setting up the service account? Where I said make a note of the email address? Yup. That’s what you put in the “Add email or name” box. Make sure the “See all event details” is chosen and then click “Send”. If you want your PHP script to be able to alter the calendar you need to choose another option that allows it. Only grant permissions that are necessary.
  4. Scroll down to the “Integrate calendar” section and make a note of the Calendar ID. It looks something like “qhhbdvqi5dom44arse60oav68k@group.calendar.google.com”.
  5. It’s at this point that I wish I had read the documentation a bit more closely and seen this:

Note: Sharing a calendar with a user no longer automatically inserts the calendar into their CalendarList. If you want the user to see and interact with the shared calendar, you need to call the CalendarList: insert() method.

Read that again. It’s important. I spent literally hours trying to find out why the API couldn’t see the calendar. Hours wasted because I didn’t read a paragraph of text. Anyway I’m not bitter, as you can tell.

Hitting the PHP

There doesn’t seem to be a way to insert a calendar into the service account’s calendar list from the admin console, so you need to run the following code on your server. Download it here – right click on that link and choose “Save link as…”

<?php
require_once __DIR__.'/vendor/autoload.php';

if ($argc < 2) {
    echo "Supply the calendar name as an argument\n";
    exit;
}

$calendarId = $argv[1];

$client = new Google_Client();
$client->setAuthConfig('/path/to/credentials.json');

$client->setScopes('https://www.googleapis.com/auth/calendar');
$client->setApplicationName("My Calendar");

$service = new Google_Service_Calendar($client); 

$calendarListEntry = new Google_Service_Calendar_CalendarListEntry();
$calendarListEntry->setId($calendarId);

$service->calendarList->insert($calendarListEntry);

$calendarList = $service->calendarList->listCalendarList();

while(true) {
  foreach ($calendarList->getItems() as $calendarListEntry) {
    echo $calendarListEntry->getSummary() . "\n";
  }
  $pageToken = $calendarList->getNextPageToken();
  if ($pageToken) {
    $optParams = array('pageToken' => $pageToken);
    $calendarList = $service->calendarList->listCalendarList($optParams);
  } else {
    break;
  }
}
?>

Edit the highlighted parts with your path to the keys file (step 11 of setting up the service account) and change the application name if you want to. Then run the script from the command line with:

naich:~$ php add_calendar.php qhhbdvqi5dom44arse60oav68k@group.calendar.google.com

Obviously change the calendar ID to the one you want to use (step 4 of allowing the service access to your calendar). If all goes well you should see the name of the calendar you have added along with the other calendars (if any) that have been added to that service account already. If not you will see lines of error messages. Make sure you have followed all the steps in “Allowing the service to access your calendar”.

Your service account is now ready for your scripts to use.

Getting started

https://developers.google.com/calendar has information about using the calendar API and the examples (e.g. in this tutorial) usually have PHP versions. The examples assume you have already set up a service in your PHP script – something like this:

require_once __DIR__.'/vendor/autoload.php';

$calendarId = "qhhbdvqi5dom44arse60oav68k@group.calendar.google.com";

$client = new Google_Client();
$client->setAuthConfig('/path/to/credentials.json');

$client->setScopes('https://www.googleapis.com/auth/calendar');
$client->setApplicationName("Calendar");

$service = new Google_Service_Calendar($client);

There is a list of Google_Service_Calendar methods which is confusing as hell to me. If you use the links on the left with “_Resource” at the end you get a list of functions for that class. So, for example, the Google_Service_Calendar_Events_Resource page shows how to get a list of events for a calendar. The code would be:

$events = $service->events->listEvents($calendarId);

Follow the link in the “Returns” section to see how to use the $events class. Something like:

  foreach ($events->getItems() as $event) {
    $name = $event->getSummary();
    $startDate = $event->getStart()->getDate();
    $endDate = $event->getEnd()->getDate();

And so on. Basically you need to do a lot of reading of documents, which is where I’ll leave you now.

Good luck!

  1. I think that in theory you could do all this on a hosted account, but it would not be straightforward to keep the private key secure if you can only access space that is readable by the web server. You would also have to install the Google PHP APIs manually. ↩︎

About: beaks.live – the software

April 17th, 2023

This is the bird box that is shown at beaks.live. It is on the side of a house in Cambourne, about 8 miles west of Cambridge, in the UK.

Right from the start, the plan was to get it working roughly and quickly and then improve it until it was the best I could do with the crap hardware – this being a £11 webcam connected via USB to a Raspberry Pi 4, which also drives transistors to work the cheapest infra-red LEDs I could find.

Having messed around with RTMP (no one uses it any more) and HLS (I’ll be fucked if I can get it to work) for streaming, I eventually ended up with this system:

The Raspberry Pi takes care of the camera and lighting, uploading the video to the server (a VDS hosted with Mythic Beasts), which does all the heavy lifting of looking for motion and streaming live footage to the many dozens of viewers who are eager to catch a glimpse of beak.

Did I mention the camera is crap? The automatic exposure sets itself to some random level and occasionally flashes up and down twice a second, apparently to relieve the boredom. So the R-Pi has to sort out the exposure, and luckily, you can set most of the camera settings manually via USB. Every 10 minutes the Pi records 5 seconds of video, takes 5 frames and averages the light level on each of them. It then sets the exposure, gamma, and LED levels* depending on whether it needs to be lighter or darker. Or it just leaves things as they are if it’s all hunky dory.

* the LEDs are so dim I just leave them all on all the time now.

It records 5 minutes of video at a time, using FFmpeg (with some video tweaking and normalisation to make the crap camera’s video a bit nicer), which is then uploaded to the server. Funny story – I originally set up the Pi’s exposure setting software so it calculated the camera’s exposure settings from this video – this video which has been normalised. So whatever is coming out of the camera, FFmpeg “fixes” it, and then exposure setting software thinks everything is hunky dory, despite the exposure being so wrong the video is just noise. This is why it records 5 seconds of unfixed video separately to check the exposure. A couple of months later I had forgotten this, and had the brilliant idea of using samples from the 5 minute feed rather than doing a separate 5 second one. I thought the camera had died, until I remembered the normalisation and why I didn’t do it like that originally. I look forward to doing the same thing again in July, September, November, etc.

Incidentally, all this software is written in a mixture of Python and Bash scripting because I am a masochistic lunatic. I love Bash – it’s just mad, with random shit like functions looking like “function my_function () { …” where the ()’s do nothing because you can’t put anything inside them – they are purely decorative.

But I digress. The server has the latest video uploaded to it. It keeps the last 4 uploads so there is 20 minutes of buffer. It deletes the oldest one once it has been processed for motion detection. There is a watchdog timer on the server and the Pi will only upload a video if it’s been updated recently enough. This is to stop the server being filled up with files if it reboots and the processing stops or something. Each 5 minutes is about 100MB.

The motion detection is done with DVR-Scan and hits are processed to generate thumbnails and a static web page. Anything less than 30 seconds long is discarded to get rid of most of the dross. Videos older than 25 hours are deleted so there’s a rolling list of videos.

The live page is also static and uses video.js for the player. The current 5 minute chunk location is obtained using an XMLHttpRequest, then the video loaded with JS. When it gets to the end, the JS gets the next section and plays it with a minor blip for the viewer.

The “live” video is actually always 10-15 minutes in the past because it takes 5 minutes to record a chunk before it’s uploaded and then the server tells the player to play the previously uploaded one so you don’t start watching one that’s still uploading.

It’s a bit like the HLS streaming system, except there’s hideous latency and mine works. If you want to mess it up, right click and choose “show all controls” and then slide the slider to the end. I’ve no idea why I’ve told you that.

About: beaks.live – the hardware

April 13th, 2023

This is the bird box that is shown at beaks.live. It is on the side of a house in Cambourne, about 8 miles west of Cambridge, in the UK.

When I put a camera in this bird box last year, I was not optimistic. Expecting to capture nothing more than the inside of an empty box, there didn’t seem much point in spending any significant sum of money on a camera. I decided to see how well I could get it working for how little money.

Two cameras for £21.66 doesn’t scream quality, but they are able to manually focus down to a few cm. Being cheap and nasty also means they won’t have an infra-red filter on the lens, which means I can illuminate the box at night with a light the beaks can’t see.

I picked one and sawed off the mounting at the bottom, knocked up a 3d printed housing to fit it in the apex of the bird box roof, and fitted some cheap Ebay IR LEDs.

A mess of wires being put into the 3d printed camera mount.
Cheap and nasty does it every time

This is the camera and LED housing mounted in the bird box:

Looking up into the box with the mounting fitted.
Looking upwards into the roof of the box

On the outside is the 3d printed box which holds the interface to the cable that goes into the house and the drivers for the LEDs. I actually had a proper PCB made with a D/A for the microphone but I never wired it up because I’m lazy. That’s why there is no sound. Sorry.

The interface box with unused D/A.

The LED controls and USB for the camera share a length of CAT-5 cable into the house, where they plug into the Raspberry Pi, which has an ethernet connection to the router.

And that’s the hardware. Total cost probably around £75, including custom made PCBs, which were ridiculously cheap. I mean like stupidly cheap – around £5 for 5 PCBs, including delivery from China. Anything clever is done in software, including stuff to improve the performance of the (frankly substandard) parts I used. Next year I’ll replace it with decent kit, including a camera that isn’t shit.

Coming up next… The software

Ever wondered why plumbers are paid so much?

November 25th, 2022
Standard home brewing conditions

It’s a difficult job that combines working in horrible conditions with the need for multiple skill sets. But the main reason plumbers are well paid is because they know the arcane secrets of plumbing fittings. It is dangerous, forbidden knowledge, some of which I am about to share. Strap in. We are going through the looking glass…

Update: Thanks to all the good people on Hacker News for their input, from which I’ve learned a lot. I should stress that any following advice is not from a professional plumber and is purely from my own experience as an idiot making a low pressure beer handling system. It should not be read as the proper way to do anything, especially if you are working on pressurised systems and definitely totally 100% not with gas fittings. Get someone in to do that, you lunatic. Seriously. Don’t mess with gas.

This is posted from the perspective of a home brewer, so it’s just a small subset of the world of plumbing fittings – we use mainly stainless steel fittings for sanitary reasons.

The first thing to remember is this:

Nothing makes sense

In the UK, stainless steel fittings usually screw into each other, using a standard thread called “BSP” – British Standard Pipe thread. You will see 1/4 BSP, 1/2 BSP etc. A common size is 1/2 BSP – the “1/2″ is, of course 1/2 an inch. So which part of the thread is 1/2 an inch? None of it. So it’s the diameter of the pipe? Nope. The pipe’s diameter is about 3/4”. The 1/2″ refers to the inside diameter of some random cast iron pipe the fittings were originally made for. This type of pipe has probably not been used since 1834 when Isambard Kingdom Brunel rigged his privy to flush as a birthday present to his wife. Nothing in a 1/2″ BSP fitting measures half an inch – not even the inside diameter of the pipe, because modern pipes have thinner walls.

So to recap, the “BSP” measurement is a standard for the outside pipe diameter and threads and gets its number from the inside diameter of a pipe that doesn’t exist. I guess we should just be grateful that a larger number means a larger pipe – I’m looking at you, Standard Wire Gauge.

But let’s not get downhearted, it is a standard after all. At least any 1/2 BSP thread fits any other 1/2 BSP thread. It must do, right? Oh dear god no. There are two types of BSP thread and the first fits in the second but the second doesn’t fit in the first.

You can have either a tapered thread or straight thread on either the female or the male part of the fitting. Note that the picture shows the straight thread having an O ring. Nah. That’s way too easy, so we don’t do that – more later. Anyway, obviously a tapered male in a tapered female is fine, and you get a nice tight fit. Now imagine putting a tapered male in a straight female. That too is fine; the bottom few threads don’t fit properly, but the top ones do and there is enough contact to make a seal. But putting a straight male in a tapered female does not work. It will leak and you will be sad. The thread of the straight male hits the bottom of the tapered thread while the top is loose, so there is hardly any contact area.

So you need to make sure you are getting your tapers and straights correct, and naturally no one goes to any effort to tell you what you are buying. In theory “BSPT” means BSP Tapered and “BSPP” means BSP Parallel but hardly anyone uses these terms because that would make it too easy. Everyone just calls them “BSP” so there is no way to know if they are parallel or tapered, unless it’s actually stated somewhere in the description, which it usually isn’t. Some people even call straight ones “BSPT” because they think the “T” stands for “thread”. Marvellous. That really fucking helps, thanks.

Given that using a tapered female fitting means 50% of the male fittings don’t actually fit, then they must be rare, right? Nope. They are everywhere and you have to use your telekinesis because no retailer ever bloody tells you if they are tapered or straight. You might be thinking “what is the point in making tapered female fittings if you can fit both straight and tapered males in straight ones, which are easier to make?” The answer is simple – they hate you and they hate me. Or possibly tighter coupling or something, but I suspect it’s just plain spite. When buying male fittings, it’s best to always get tapered ones so they fit in either. I mean why do they even make straight ones? The only possible reason is malice. Or they are cheaper. It’s malice though.

It would be useful if manufacturers indicated the fitting type with some sort of mark, so of course they don’t. You have to squint at it and guess.

This is not a problem for plumbers, who have to have every type of fitting on the planet rattling around in the back of the van. If they buy 50 flanged wibblers with tapered threads by mistake, they can just buy 50 straight ones from somewhere else and they will all be used eventually. You and I end up with a box full of unwanted fittings, but it’s useful to have spares, I suppose.

There are other types of fitting – “G” (as in G 1/2) and “NPT” (as in 1/2 NPT) . You might see “G 1/2” used with metric push-fit connectors. Yeah, that’s BSP as well. G 1/2 is 1/2 BSP because it’s not confusing enough to just have one name for a standard that doesn’t always fit itself. At least they kept the numerical part of the name – which doesn’t actually match any dimension of the fitting,

Then there’s NPT. NPT threads are different from BSP, which actually comes as something of a relief at this point. Naturally, they aren’t different enough to be immediately obvious because that would spoil the fun. Oh yes, NPT is identical to BSP, with the only difference being angle of the valleys in the thread and that the threads are pointed. This means that despite looking the same, NPT is not compatible with BSP. Because fuck you, that’s why. Luckily, you don’t see many NPT threaded fittings in the UK and they are only sold by genuinely evil retailers.

Let’s dip briefly into the sane world of metric fittings. Ahhh… 15mm compression fittings make sense, with their sensible millimetres sensibly representing the actual diameter of the pipe. Except that… Sorry… The thread on a METRIC 15mm compression fitting is not metric, it’s 1/2 BSPP! You cannot escape the lunacy in a metric lifeboat. Actually, best not to complain because it’s quite useful that you can do things like bodge cheapo 15mm isolating valves into your 1/2 BSP pipework.

Unless it’s tapered.

Using the bloody things

Given the ubiquity of BSP threads in stainless fittings, you might think there is some sort of advantage in using this utterly psychotic standard. It must give nice leak-tight results and be easy to use? No. It’ll drip like a fucked fridge and you don’t know what angle the joint will be when it’s tightened up. On the plus side, you get a genuine feeling of achievement when you get a nice looking leak-free system. On the minus side, everything else.

Remember that picture up there showing tapered and straight threads? Remember the one on the right says it uses an O-ring? Bullshit. It’s a fairy story that plumbers tell their children (probably). In the real world, there is no flange on your typical straight fitting. Look.

A 90 degree 1/2 BSP coupling, yesterday

Where does the O-ring go, eh? Eh? EH??? No, we use PTFE tape, and it sucks. For what it’s worth, I tightly wrap the tape 10 times round the male thread and get an enraged mountain gorilla to tighten it up. Or a bloody great spanner if there are no nearby gorillas. This means, of course that it ends up with the other end pointing in a random direction. If you want your pipework to look like a Windows XP screensaver, then that’s fine. For those of us with an ounce of pride left and some vestigial will to live, there are 2 choices:

  1. Before going full gorilla on the joint, there is a small window of tightness where the thread hasn’t bottomed out, but it will still be leak proof. If your desired angle is in that window, then the gods of plumbing have smiled on you this day. Good luck finding it. Plumbers, of course, have an extra sense to us mortals and can find it easily.
  2. You use a union. This lets you connect two BSP fittings without screwing them together. Each thread is screwed in to a round flange and the two flanges push together with a PTFE washer between them. A large nut clamps the two halves tightly.
The state of the union.

This is especially useful when you are assembling the final mess.

I hope this is useful to you and good luck in your plumbing. Final comment: Don’t bother using “Dope” or “Rectorseal” or whatever the name of the compound that is supposed to magically produce a leak-free result in seconds. Just buy cheap PTFE tape, and lots and lots of it. 12 rolls might see you through a medium sized project.

Conclusion.

Fucking hell.

Chill out man

November 24th, 2022

The journey of creating a proper brewing setup begins (and continues) with various pipes popping off and blasting your face with water, cartoon-style, before you eventually end up with something that works and doesn’t fill your garage with water.

The horrible mess in that picture is what I actually tried to use for my first brew with a plate chiller. It did such a terrible job that it prompted me to spend 3 months doing it properly. If your set up looks anything like that you might want to read on.

The setup shown here is pretty typical from what I’ve seen on YT videos. It takes the wort from the outlet at the bottom, runs it through the chiller, on to the pump, and then out to the whirlpool outlet at the top. Other than just looking plain nasty, there are a number of things wrong with it:

  • Each silicon pipe held on with jubilee clip is a time bomb, waiting for you to forget to tighten it up.
  • The chiller is sitting on its back, which is inefficient, and means you end up with a chiller full of wort at the end of the brew.
  • The pump is higher than the chiller. It’s not self-priming so you have to fill up the chiller first before you can start pumping.
  • Bump into the table and all that stuff goes on the floor, probably pulling off a pipe or two before putting a dent in your chiller.
  • You have to disconnect and reattach pipes when you want to change the configuration, like if you want to pump out to a conical fermenter. Each disconnect is another splash of wort on the floor. Forget to turn a tap off first and it’s a splash and a gush.
  • Cleaning it will be a pain because of all those silicon pipes flobbling everywhere. Poke them in a bucket to circulate cleaning solution and as soon as your back is turned at least one will flibbit out and start spraying everywhere.
  • “Flobbling” and “flibbit” are apparently already words in the Urban Dictionary with utterly disgusting meanings.

So, enough wibbling, here is my solution:

The Desplashinator 3000000

If you are thinking “that still looks pretty shit”, then you would be right, but it works better than it looks. From whirlpool to chilling to pumping out to cleaning, you don’t need to keep disconnecting stuff, thanks to all those valves. This is a diagram of what it is:

More valves than a 1945 radio

Before we get into how it works, let’s talk valves. It uses two different types: expensive 3 part ball valves and cheap as chips 15mm compression fitting valves. Did you know that the thread on a 15mm compression fitting is 1/2 BSP? It fits into a stainless 1/2 BSP female thread (as long as it’s not tapered), and that can lead to all sorts of fun and bodgery, as long as you use enough PTFE tape. I might do a separate post on plumbing fittings because they are a nightmare of unexpected incompatibility and unexpected compatibility.

Anyway, basically – expensive 3 part fittings are for passing wort through so they can be disassembled, and cheapo 15mm ball valves are for flushing with water and cleaning. Wort never goes through the 15mm ones, so they don’t need to be taken apart when you do a deep clean. Almost all the other parts are stainless 1/2 BSP fittings, which aren’t cheap but are pretty bullet proof.

Other features are:

  • The pump is at the bottom for easier priming.
  • The chiller is mounted vertically for more efficiency and easier draining.
  • Draining and cleaning ports have Hozelock connectors on them for cheap and easy connecting to hosepipes.
  • It’s all mounted on a sturdy wooden frame so bits don’t fall over.
  • There is a dedicated line to the FV so no fiddling with pipes. It’s essentially a sealed system to stop contamination.
  • You can whirlpool without the chiller being inline, so it doesn’t get clogged up with bits of hops before the hop cone forms.

How to use

So your boil is going to finish soon. First thing is to sterilise the equipment by running the boiling wort through it. Don’t start the cooling water yet. Open V1, V2 and V4 and wait for the pump to fill up and turn it on. You can speed up the process by pulsing the pump to shake the bubbles out.

Leave it running for a minute or two to get the hops whirlpooled towards the centre of the boiler. Then slowly open up V3 to start running the wort through the chiller. Do it slowly because it’s full of air and you’ll get big bloikking bubbles in your boiling wort. When the bubbles have stopped close V4 so it’s just circulating through the chiller.

Give it a few more minutes and then turn on the cooling water. Start off with a high flow rate and turn it down as the wort cools.

When you are down to temperature, close V1 and open V5 and pump her out! You will end up with some wort left in the chiller and pump but it’s only a few 100 ml, so I don’t worry about it.

How to clean

The Desplashinator 3000000 makes cleaning fun! If you are weird. I find it fun and I’m pretty weird. Just connect a hose to the drain valve connector, then open up all the 3 piece valves and the drain valve to empty the system – I empty it down a drain in the floor. It has to be lower than the system, obviously.

Stick a hosepipe connected to mains water on the source valve connector, close V1, V2, V3, and V5 and leave V4 open. This flushes the pump out backwards to get rid of bits of hop that might have caught in it. Then close V4 and open V3 to flush the chiller out backwards. You could probably do away with the “Aux” valve and connector because I never use it.

Clean out the boiler, put 5 litres or so of cleaner in it and reconnect it back to the chiller system. Then leave it circulating for a while before flushing with clean water. You can fill up the boiler through the source valve and V1 and flush with that out of the drain valve.

Conclusion

This is so much better than having it on a table and connecting everything with silicon tubing. It was a lot of effort to build, but totally worth it.

Next up: instructions for how to build one.

Spicy Potatoes!

October 31st, 2021
Some spicy spuds, yesterday.

The Spicy Spud is a unique Cambridge phenomenon which probably occurs in other places too. Every fish and chip or kebab take away in Cambridge seems to do them – except the ones that don’t. There is no standard recipe or supplier, so each shop has their own version. This recipe is my attempt to recreate the spicy potato of my youth. Best eaten with huge gobs of chip shop mayonnaise – the nastiest, gloopiest, vinegarest, stuff available – after several pints of IPA and a terrible band at the Sea Cadet Hut. The Spicy Potato is heaven in a greasy cardboard box.

It is basically a potato chunk, covered in a batter and rolled in breadcrumbs. The only spice it actually contains is pepper but various salts are used to give it extra flavour. Adjust the ingredients to your taste until you are taken back to the halcyon days of The Destructors belting out “Sewage Worker” while you jump up and down and try not to spill your 70p pint.

Ingredients:

  • 3 or 4 large potatoes. I use baking potatoes, chopped into roughly 30mm cubes.

For the batter:

  • 50g flour
  • 2 tsp onion salt
  • ½ tsp garlic granules
  • ½ tsp celery salt
  • 85ml milk

For the coating:

  • 6 Tbsp breadcrumbs
  • 6 Tbsp flour
  • 1 Tbsp coarse ground black pepper

Put the potatoes in boiling water, bring them back to the boil and simmer for 5 minutes. Drain well and make sure they are dry. Let them cool down for a bit. Don’t cook them until they are soft – we don’t want mashed potato.

Heat a pan of sunflower oil (about 40mm deep) to 140C. Pre-heat your oven to 180C.

Make the batter by mixing the ingredients together and smushing out the lumps. Coat the potatoes with the batter and put on a rack to drain the excess. You don’t want too much batter on them or you end up with huge lumps of coating. This can also happen if the batter is too thick.

Mix the coating ingredients and roll the battered pots in it to get a nice even coating. Be careful when handling them because it’s easy to scrape the coating off at this stage. Like Luke Skywalker, I use the forks to move them around.

Put them in the oil in batches of 4 or 5. Cook them for 30 seconds and whip them out again. All you are doing is solidifying the batter and infusing some oil into the coating. The actual cooking happens in the oven.

Put them on a baking tray and put them in the oven for 25 minutes, turning at 15 minutes. For an authentic non-crispy coating, cover them with foil after 15 minutes and cook for another 20 minutes.

Enjoy your spicy potatoes with the cheapest, most horriblest mayonnaise you can find.

Abusing Public WiFi Access Point Protocols for Fun and Beer Measurement (Raspberry Pi)

June 9th, 2020

This is a little sub-project of what I’ve been working on recently – a hideously over-engineered Raspberry Pi-based system to measure the amount of beer left in the kegs in my keezer.

Normally I would simply set up a web server on the Pi and have it on the home network, so I could see the levels remotely. The problem is that the routers are all inside the house and the Pi is in the garage, invisible to them all thanks to the 2 external walls between them. I needed some way to read out the beer levels on my phone – after all, walking up to something and looking at the level gauge is so last millennium.

So – Bluetooth or some sort of ad-hoc Wifi thing? I like to re-use stuff I’ve got lying around in drawers, so the solution seemed to be an old WiFi dongle that was gathering dust. And Bluetooth is awful. Setting up a Pi as an access point is fairly well covered on the internets, but this is a bit different in that we don’t want to forward traffic onto our network like an access point – not that it could connect anyway, being out of range. I also didn’t want to install a web server on the Pi. It’s only a Pi 1 model B, so sticking Apache and PHP on it might be asking a bit much – especially when you can do it all with one command and a small BASH script.

So the cunning plan was to take advantage of a feature of public access points – the ones that show you a registration page for you to fill in with fake info.

When you connect to a public WiFi hotspot your device tries to load a page on the internet using non-SSL http. It might be any page (captive.apple.com/ seems to be popular), but it will be a web page that the device knows should exist and if it loads, your device knows the internet is working.

A public access point intercepts the page request and, rather than forwarding it, sends a 30x redirect HTTP response back to the device – basically hijacking the request and spoofing the reply. Your device then loads up the page it has been redirected to and displays it as a sign-in page.

It is this mechanism that I used to show the keg levels on any phone, just by connecting to the Wifi. This is how to do it if you want to do something similar. I’m assuming you SSH on to a Pi connected with an ethernet cable to your network, and you have a Wifi dongle hanging out of its USB port. In all likelihood they will be eth0 and wlan0 respectively, so I’ll use them.

wlan0 is going to use a different range of IP addresses from the ones used by eth0, so edit /etc/dhcpcd.conf to manually assign an IP address to the wlan0 interface. Add this at the bottom (comment out any existing definition for wlan0):

interface wlan0
    static ip_address=192.168.4.1/24
    nohook wpa_supplicant

Next we need to install hostapd to run the hotspot and dnsmasq to sort out assigning IP addresses to devices that connect.

sudo apt-get install hostapd
sudo apt-get install dnsmasq
sudo systemctl stop hostapd
sudo systemctl stop dnsmasq

The second two commands disable the services we just installed so we can edit config files before starting them again.

Create the file /etc/dnsmasq.conf and put this in it:

interface=wlan0      # Usually wlan0
dhcp-range=192.168.4.2,192.168.4.20,255.255.255.0,24h
address=/#/192.168.4.1

This tells dnsmasq to assign the range 192.168.4.2 – 192.168.4.20 with a netmask of 255.255.255.0 and a lease time of 24 hours. The third line tells it to return the server address for all domain lookups that aren’t in /etc/hosts, i.e. all of them. When dnsmasq restarts it will look at this file and load up the config information.

Now to set up hostapd. Create /etc/hostapd/hostapd.conf and put this in it:

interface=wlan0
driver=nl80211
ssid=Your SSID here
hw_mode=g
channel=7
wmm_enabled=0
macaddr_acl=0
ignore_broadcast_ssid=0

It’s pretty obvious what is happening there, other than some of the technical bits; wmm_enabled is something to do with packets (no idea what, though), macaddr_acl tells it to whitelist all connections and ignore_broadcast_ssid tells it to broadcast the SSID – set it to 1 to hide it. There is no WPA password or setup, obviously. Change the SSID to something hilarious.

Now you need to tell hostapd where to find the config file when it starts. Edit /etc/default/hostapd and add (or uncomment and edit) the line:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

We have now set up our access point. Start dnsmasq and hostapd again:

sudo systemctl start hostapd
sudo systemctl start dnsmasq

If there are no errors, your AP should show up in the list of APs on your phone, laptop etc. Try connecting to it – it should connect but you won’t be able to see the internet because there is no forwarding. One thing you can still do however, is connect to SSH on the Pi. You really don’t want any ports other than 80 visible from an unsecured AP. We’ll use iptables to set up a firewall and do the test page hijacking.

sudo iptables -A INPUT -p tcp -i wlan0 --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -i wlan0 --dport 53 -j ACCEPT
sudo iptables -A INPUT -p tcp -i wlan0 -j DROP
sudo iptables -t nat -A PREROUTING -p tcp -i wlan0 --dport 80 -j DNAT --to-destination 192.168.4.1:80

The first two tell iptables to allow through connections on port 80 (HTTP) and 53 (DNS), the second tells iptables to drop all other TCP connections from wlan0. The third redirects any connection with a destination port 80 (regardless of the IP address) to the Pi at IP address 192.168.4.1, port 80, for our server to handle. If you are a bit confused about how iptables work, this flowchart will either clear things up or make it more confusing. Basically there are 4 tables – filter (default if no -t switch), nat, mangle and raw which each contain “chains” such as INPUT which are the instructions on how to route traffic. It’s a vast subject and I learned just enough to work out the 3 lines above. There are other guides that go into more details.

One thing to do at this point is make it so that the iptables configuration is not lost when the system is rebooted. This command saves it to a file:

sudo iptables-save >/etc/iptables.ipv4.nat

To reload the configuration on boot put this in /etc/rc.local

iptables-restore < /etc/iptables.ipv4.nat

So, moving on to the web server. I’m using socat and a bash script. socat is one of those amazing Linux tools that is impossible to explain to a layperson. “What it does is, it takes data from one place and puts it in another but it’s more complicated than that…” and so on. Best just to tell them it’s the computer equivalent of magic, before their eyes glaze over and they start thinking about feigning an illness in order to escape. We are going to use it to pipe data from an internet port to a script and back again. Incoming text from port 80 is sent to the script on stdin and anything written to stdout gets sent back to the port. It’s easy enough to set up with this command:

sudo socat TCP4-LISTEN:80,reuseaddr,fork EXEC:"/home/your_path_here/server.sh >/dev/null" 2>/dev/null &

Obviously change “your_path_here” to where you are doing all this stuff and put this line in /etc/rc.local if you want it to start automatically on boot. The command tells socat to listen on port 80 and then fork off the script when there is a connection. The script referred to as /home/your_path_here/server.sh is this:

#!/bin/bash

PAGE_NAME="kegs"
FOUND_URL="http://1.1.1.1/$PAGE_NAME"

request=""
while read -r  -t 5 line; do
  if [[ ! -z "${line:-}" && $line == *[^[:cntrl:]]* ]]; then
    if [[ ${line:0:4} == "GET " ]]; then
      request=$(expr "$line" : 'GET /\(.*\) HTTP.*')
    fi
  else
    break
  fi
done

if [[ "$request" == "$PAGE_NAME" ]]; then
  printf "HTTP/1.1 200 OK\n"
  printf "Content-Type: text/html\n\n"
  cat index.html	# Show this as a registration page.
else
  printf "HTTP/1.1 302 Found\n"
  printf "Location: $FOUND_URL\n"
  printf "Content-Type: text/html\n\n"
  printf "Redirect to <a href=\"$FOUND_URL\">$FOUND_URL</a>\n"
fi

That’s pretty dinky for a web server, huh? Don’t forget to change permissions of server.sh with chmod 755 server.sh. Rename PAGE_NAME and FOUND_URL to whatever you want. Note that because we are grabbing all port 80 traffic coming in on wlan0, it doesn’t matter what you put for an IP address – it’ll all go to our server. The first block of code reads the HTTP request coming from the device, which will be saying something along the lines of:

GET / HTTP/1.1
Host: captive.apple.com
Accept: image/gif, image/jpeg, */*
... and so on

The script ignores everything except the GET /… part, from which it extracts the page name, if any. It won’t match (unless the test page is called “/kegs” – unlikely), so it will respond with the redirect code 302, to send the device to “/kegs”. The device sees the redirect, thinks it’s for a registration page and loads 1.1.1.1/kegs. This time the script sees that /kegs has been requested, sends a 200 OK code and the contents of index.html, which the device displays. My beer measurement system generates index.html as a page showing how much is left in each keg.

As a useful tool with which to quickly see the levels of my kegs without any fuss, this is rubbish, quite frankly. But then the whole raspberry-pi-based-keg-measurement thing could be replaced with cheap mechanical bathroom scales, so I might as well go all in on the pointless technology.

Updated 10/6/2020 : Improved the firewall rules.
Updated 18/2/2021 : Improved DNS rules.

Taming the PiFM Transmitter (Part 2)

March 25th, 2018

In part one of this guide it became clear that a Raspberry Pi with a 700 mm long wire on pin 7, running a variant of the PiFM software is an easy way to make a nuisance of yourself. We might not be broadcasting kilowatts of power and realistically, you are not going to be knocking planes out of the sky, but the Pi is a dirty old man when it comes to broadcasting and we need to clean up its act.

The obvious way to do that is to put a filter between the Pi’s output and the aerial. If the design considerations and analysis of the filter’s performance don’t interest you, skip to the end for circuit diagrams, construction instruction and purchase info (possibly).

As always, these posts are for educational use only. Do not use your Pi as a transmitter unless it is legal for you to do so, which is highly unlikely. Using a filter will not make it any less illegal for you to use your Pi as a transmitter.  Always brush your teeth before bedtime and be nice to people.

To recap, this is typically the sort of thing that comes out of your Pi when you use it as an FM transmitter:

There’s a lovely spike around the 144 MHz mark, which is the amateur radio 2m band. There are probably not many radio hams near me that like the sort of music I listen to. Come to think of it, some times I’m not sure I do either. In general, it’s a broad splattering of crap all over the spectrum. And the Pi’s transmitted output is just as bad, ho ho ho. Ahem. This is the sort of thing we need:

You might notice R1 there. The GPIO pins are not designed to drive inductive or capacitive loads, so we need to make the filter input a bit more friendly. The easiest way is to put a resistor between the Pi and the filter’s inductor. I’ve tried it and it works, but it does reduce the range of the transmission. If you want to try it without R1, don’t blame me if you fry your Pi. There’s about 5 dB loss with this design, which might be fine for you. For me, it reduced the range just enough that the signal was fading out if I stood in the wrong part of the kitchen. The solution was either to avoid using the fridge or to amplify the output a bit.

I’m not an expert with RF circuits (although I probably know more than you), so I used the interwebs to find a design that would

  • Be cheap
  • Work on a 5V supply
  • Not require any fine tuning
  • Be cheap
  • Be easy to make
  • Not have any expensive components

You can probably tell what my priorities were. This was the prototype:

Those with a keen eye have probably already spotted that it looks shit. Bear in mind that it’s already been bodged around a bit, and it looked worse than that by the time I’d finished experimenting with the poor thing. It is a single stage class C amplifier with a low pass filter on the output. The 2N4427 transistor is old and cheap; I bought 5 from China for about £3. Everything else (apart from the variable capacitors) is bog standard and the coils are easy to wind. The variable capacitors are stupidly expensive – there is about £20 worth of them in that photo, so they had to be replaced with fixed ones that cost pennies.

The end result was this circuit:

Pi Hat Filter – click to enlarge.

It’s cheap, simple and it works quite well. This is the finished hat installed and working:

This is the output with the filter hat on:

Out of band signals are attenuated by at least 20 dB, which means they are 1/100th the power of when it was hatless. There is even a little bit of gain at our broadcast frequency, which also amplifies the in-band harmonics, unfortunately. It’s not exactly BBC quality but it should stop you annoying the neighbours. If you want to get the absolute maximum performance out of the filter, use 5-95pF variable capacitors instead of C7, C8, C13 and C16 and keep tweaking them until it becomes apparent that you aren’t really having any effect.

The design files are here.  If you would be interested in a kit of parts or a ready made hat, leave a note in the comments and I’ll look into it.

I’ll leave you with a comparison of the filtered (orange) Vs. unfiltered (blue) Pi:

Good, eh?

Taming the PiFM Transmitter (Part 1)

March 17th, 2018

One of the million things you can do with a Raspberry Pi is using it as an FM radio transmitter. It is stupidly easy; you just attach a 700 mm long wire to pin 7 and install one of the many variants of the original program which was hacked together at a code club meeting. And now you are a radio pirate. This guide is aimed at those who can do the above but don’t know why it might be a bad idea. If you already know about harmonics or know that they are bad but  don’t care why, you can skip to the next post which is about adding a filter to your Pi.

At this point it is traditional to say that broadcasting without a license is illegal in most countries and doing something like using your Pi to stream internet radio stations to the analogue radio in your kitchen is wrong and makes Eben Upton cry.

One thing you will often hear in discussions about Pi radio is that lots of unwanted harmonics are produced on frequencies other than the one you are broadcasting on. These can interfere with legitimate users of that frequency; for example a passing pilot might not want to hear the Crisp Biscuit Breakbeat remix of Josh Wink’s Higher State of Consciousness instead of the control tower telling her where to land. You might think that anything broadcast from your little Pi won’t be powerful enough to interfere with a professional communications system, and you are very probably right. But what exactly is coming from our Pi’s aerial? This is what we would like to see:

What we would like to see coming out of our Pi. Click for a better view.

Along the X axis are all the frequencies from 50 MHz to 350 MHz. The Y axis shows the level of the signal at that frequency in dB. If you just clicked on the “dB” link and are none the wiser, the upshot is that the dB makes it easy to compare the relative powers of signals. For example, a +3 dB difference is twice the power and a -3 dB difference is a half the power. So +6 dB means the signal is quadruple the power of whatever you are comparing it to – it’s the same as a +3 dB doubling and another +3 dB doubling. As the dBs go up in linear fashion, the corresponding power goes up exponentially: +3 dB, +6 dB, +9 dB corresponds to x 2 power, x 4 power and x 8 power.

The graph shows the main peak of our transmission at 107.3 MHz, at about +11 dB, a harmonic at 214.6 MHz at -25 dB and another at 321.9 MHz at -30 dB. These unwanted harmonics are bad, but their levels are a lot lower than the main broadcast frequency. If the harmonic’s signal level is -36 dB compared to the main one, it means it’s about 4000 times weaker and we don’t really need to worry about it, given that the main signal only goes a hundred metres or so. The other unwanted signal is even lower: -41 dB compared to the main signal so I’m not even going to bother to work out the exact value because it is so small. OK, it’s 12,500 times less than the main signal. That won’t even make it out of the room.

So that’s our ideal transmitter, with a nice strong signal on the frequency we want and weaker signals on the frequencies of the harmonics we don’t. How does this compare to the actual signal from a Pi? Take a look at this little beauty – it is what’s coming out of my my Pi when it’s broadcasting on 107.3 MHz:

Actual output from a Pi. Barf.

It’s spewing crap from 50 MHz to 800 MHz and probably beyond. One of the harmonics is actually more powerful than the frequency we want to transmit on. The signals coming out of that aerial are dirtier than a dog in a field of incontinent cows [todo: change this to something more tasteful]. The neighbours are probably wondering why old skool breakbeat trance music is coming out of their hoover.

It gets worse though. The harmonics coming from the Pi change in number, size and position as you change the frequency. Broadcasting on exactly 100 MHz actually produces a graph somewhat similar to the first one, but drop the frequency by 1 MHz to 99 MHz and you get this:

99 MHz – Craptastic!

Now that is quite pretty, but it did genuinely start making lines go down my monitor and a hum come out of my speakers. God knows what it was doing to the output electronics in the poor Pi.

So, before doing anything else, you need to pick a frequency that won’t make music come out of your granddad’s fillings. The cleanest one is 100 MHz, but where I live there’s already a commercial station there, so that was out for me – my little Pi couldn’t compete with the big boys. You don’t want to pick one that’s close to 100 MHz either because they seem to be the dirtiest. Trial and error, picking the gaps between existing stations, seems to be the way to do it. Ideally, you want a spectrum analyser like the one I used to make the graphs, and surprisingly, you can get the hardware for about £20, and use something like this to turn it into a spectrum analyser. At the very least, you could pick a frequency and see how many times your broadcast appears when you tune your radio up and down. As a general rule, the fewer times the better.

But however carefully you choose your frequency, your Pi will still be broadcasting all over the spectrum and possibly making someone near you very angry. You can improve things by putting a filter on its output, and I can show you how in the next post.