I’ve been making musical visualisations based on data (mostly Cube Accounts data, but also some other datasets) - the challenge being to take some fixed points and change things like speed, synth type, key to make them sound niceish, and then make something representative appear on-screen. I settled on clientside Javascript for this after some experiments with Python - you can see a bunch of the better results in the blog post above. The challenge is that making small changes is time consuming, and I never really get into the flow of making the thing sound and look good because I have to keep stopping and changing the code.

I’ve had a visual tool in mind for a while and Makevember was an opportunity to take some steps towards it.

The idea is to take some suitable parameters and add and remove them to change the musical character of a fixed set of notes. The fixed set is a printout of some data as little squares spread out on an A4 sheet. This is calibrated to fit roughly in a square in the centre of the paper and be printable at A4 at 72dpi.

I then use a Raspberry Pi camera to read the little squares and translate them to midi notes based on a key. This is all based on pixels….round tripping is a bit approximate and untested, but it’s more or less right.

Then I add lego bricks of different colours to change the parameters - in this case the defaults are slow tempo, beepy synth and key of C, and adding various coloured bricks flips that to a fast tempo, crunchy synth and key of C#.

I’m not sure if it conceptually quite works, but I quite like where it’s gone. It’s inspired by WhiteboardTechno by Richard and this work by Martin. And in the back of my head is Oramics, though this is only superficially related.

From a technical perspective I learned a lot, and I like the final noise - but I don’t think there’s enough capacity in the variables to make enough of an interesting piece of music - and maybe this is connected with the lack of randomness or serendipity in the process. But perhaps it’s a first step.

I used OpenCV in Python to start with and then got frustrated by the lack of simple audio libraries in Python, so moved to a headless browser and OpenCV in client side Javascript (with Tone.js for the music). This will make it easier to eventually spit out the code for a finished piece, too.

Here’s the result so far (it picks up a bit at the end):

You may well ask - why not use a camera from a laptop instead of a headless browser and Chrome’s dev tools? I had in mind an appliance, thinking maybe I can scale it up and down, but still be able to test it on a laptop. Apart from a few teething issues, headless Chromium on a Pi 4 works very well (the main problem was that the windowing layer now shows a dialogue for you to accept the browser’s use of a camera, which of course doesn’t show up if you are using it headlessly; solution was to use it once with a screen to get the permissions remembered - but there’s probably another way).

I’ve stashed the code here.

I got a load of useful hints from Sarah on Mastodon (thank you!)

Acoupi is a framework for classifiers running on-device - it can use birdnet or batnet machine learning models, maybe others? The docs are pretty good but I want to preserve a few more details for my setup.

I started with bookworm 64bit lite installed using the Raspberry Pi imager. I set up my audiomoth using these instructions and attached it to the Pi via USB.

Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

Then, install acoupi -

curl -sSL https://github.com/acoupi/acoupi/raw/main/scripts/setup.sh | bash

Install pip3

sudo apt install python3-pip

Install acoupi-batdetect

pip3 install acoupi_batdetect2 --break-system-packages

Acoupi can send detections to an MQTT server. I want to run that server on the device itself because I want it to be self-contained and it's an easy way to get at the data.

So, install mosquitto

sudo apt install mosquitto mosquitto-clients

check it's running:

ps ax | grep mos

4032       Ss     0:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

A handy little pair of commands to check mqtt is working are -

mosquitto_pub -h localhost -p 1883 -m 'hello' -t 'acoupi'   
mosquitto_sub -v -h localhost -p 1883 -t '#'

Set up acoupi-batdetect2 -

acoupi setup --program acoupi_batdetect2.program

Here are my answers. Note that it can take 15-30 secs to process a 3 second file sometimes, so if you record every 10 seconds they will build up. 20 seconds seems ok.

Collecting program files. It will take a minute or so, be patient...
Would you like to set timezone='Europe/London'? [Y/n]: 
[at this point the pi farts out a load of stuff about missing audio things]
Available audio devices:
Index  Name                                    Channels  Sample Rate
[ 1]   384kHz AudioMoth USB Microphone         1         384000.0  
Select an audio device index (1): 1
Info of selected audio device:
index                = 1
name                 = 384kHz AudioMoth USB Microphone
max channels         = 1
default sample rate  = 384000.0
Select the number of audio channels (1) [1]: 1
Select the samplerate. The default samplerate is recommended but your device might support other sampling rates. [384000.0]: 
Would you like to set recording? [y/N]: y
Would you like to set recording.duration=3? [Y/n]: 
Would you like to set recording.interval=20? [Y/n]: 
Would you like to set recording.schedule_start=datetime.time(19, 0)? [Y/n]: 
Would you like to set recording.schedule_end=datetime.time(7, 0)? [Y/n]: 
Would you like to set paths? [y/N]: y
Would you like to set paths.tmp_audio=PosixPath('/run/shm')? [Y/n]: 
Would you like to set paths.recordings=PosixPath('/home/pi/storages/recordings')? [Y/n]: 
Would you like to set paths.db_metadata=PosixPath('/home/pi/storages/metadata.db')? [Y/n]: 
Would you like to set messaging.messages_db=PosixPath('/home/pi/storages/messages.db')? [Y/n]: 
Would you like to set messaging.message_send_interval=120? [Y/n]: 
Would you like to set messaging.heartbeat_interval=3600? [Y/n]: 
Would you like to set http? [y/N]: 
Would you like to set mqtt? [y/N]: y
Please provide a value for messaging.mqtt.host.: localhost
Please provide a value for messaging.mqtt.username.: none
Please provide a value for messaging.mqtt.password.: none
Would you like to set messaging.mqtt.topic='acoupi'? [Y/n]: 
Would you like to set messaging.mqtt.port=1884? [Y/n]: n
Please provide a value for messaging.mqtt.port. [1884]: 1883
Would you like to set messaging.mqtt.timeout=5? [Y/n]: 
Would you like to set detections? [y/N]: y
Would you like to set detections.threshold=0.2? [Y/n]: 
Would you like to set model? [y/N]: Y
Would you like to set model.detection_threshold=0.4? [Y/n]: 
Would you like to set saving_filters? [y/N]: 
Would you like to set saving_managers? [y/N]: 
Would you like to set summariser_config? [y/N]:

(Note that it won't accept blank/no username / password for the mosquitto server. I edited the file after as I'm being lazy and not configuring mosquitto except for defaults - that explains 1883 vs 1884 as the port too.)

So after the 'none' mqtt u/p edit my config is now

cat ~/.acoupi/config/program.json 

{
  "timezone": "Europe/London",
  "microphone": {
    "device_name": "384kHz AudioMoth USB Microphone",
    "samplerate": 384000,
   "audio_channels": 1
 },
 "recording": {
   "duration": 3,
   "interval": 20,
   "chunksize": 8192,
   "schedule_start": "19:00:00",
   "schedule_end": "07:00:00"
 },
 "paths": {
  "tmp_audio": "/run/shm",
   "recordings": "/home/pi/storages/recordings",
   "db_metadata": "/home/pi/storages/metadata.db"
 },
 "messaging": {
   "messages_db": "/home/pi/storages/messages.db",
   "message_send_interval": 120,
   "heartbeat_interval": 3600,
   "http": null,
   "mqtt": {
     "host": "localhost",
     "username": "",
     "password": "",
     "topic": "acoupi",
     "port": 1883,
     "timeout": 5
  }
 },
 "detections": {
   "threshold": 0.4
 },
 "model": {
   "detection_threshold": 0.6
 },
 "saving_filters": {
   "starttime": "19:00:00",
   "endtime": "07:00:00",
  "before_dawndusk_duration": 0,
  "after_dawndusk_duration": 0,
  "frequency_duration": 0,
  "frequency_interval": 0,
  "saving_threshold": 0.3
 },
 "saving_managers": {
  "true_dir": "bats",
  "false_dir": "no_bats",
  "timeformat": "%Y%m%d_%H%M%S",
  "bat_threshold": 0.5
 },
 "summariser_config": {
  "interval": 3600,
  "low_band_threshold": 0,
  "mid_band_threshold": 0,
  "high_band_threshold": 0
 }
}

At that point you could do a test by setting it up like this (with your gps coordinates):

acoupi deployment start

Enter the name of the deployment: bats
Enter the latitude of the deployment: 51.45
Enter the longitude of the deployment: -2.59

and then doing this:

mosquitto_sub -v -h localhost -p 1883 -t '#'

to see the bat data come in, maybe also with

tail -f ~/.acoupi/log/*

The deployment will restart when you restart the device.

Of course proving that any of this is detecting real things is a bit tricky, particularly for bats. I tried installing acoupi-birdnet as at least I could hear what it was recording. It was all pretty similar to install and seemed to work well (although interestingly it includes gunshots, human speech, wolves, power tools and dogs as well as birds, and it seems very certain about some of those, and there are no wolves or gunshots round here).

With batnet I also managed to see (with my eyes) and detect (with the classifier) a Pipistrelle, and had a play with changing the audio recording so I could hear it, like this:

sox 20250528_214125.wav 20250528_214125_warp2.wav pitch -6000

(from here), and making a spectrogram

sox 20250528_214125.wav -n rate 384.0k spectrogram -l -m -X 160 -z 95 -Z 0 -r -Y 257 -o spectro.png

A participant in a planned hack in this area suggested running the audio though this bioacoustic pipeline to test it. Comments in this thread were really helpful on the mysterious Nyctalus I kept finding (preprint article).

As well as getting the data out I want to additionally display it on a little screen I have, displayHatMini. So -

pip3 install displayhatmini --break-system-packages

enable spi:
sudo raspi-config nonint do_spi 0
cat mqtt_display.py

#!/usr/bin/env python3
import time
import paho.mqtt.client as mqtt
from paho.mqtt.client import CallbackAPIVersion
import random
from displayhatmini import DisplayHATMini
import json

try:
    from PIL import Image, ImageDraw, ImageFont
except ImportError:
  print("""This example requires PIL/Pillow, try:
sudo apt install python3-pil
""")

width = DisplayHATMini.WIDTH
height = DisplayHATMini.HEIGHT
buffer = Image.new("RGB", (width, height))
draw = ImageDraw.Draw(buffer)
displayhatmini = DisplayHATMini(buffer)

draw.rectangle((0, 0, width, height), (0, 0, 0))
displayhatmini.display()


def text(draw, text, position, size, color):
    #fnt = ImageFont.load_default()
    fnt = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", 28, encoding="unic")
    draw.text(position, text, font=fnt, fill=color)

client = mqtt.Client(CallbackAPIVersion.VERSION2,
                     client_id="test-client")

def on_connect(client, userdata, flags, reason_code, properties):
  print(f"Connected: reason_code={reason_code}, properties={properties}")
  client.subscribe("acoupi", qos=1)

"""
data is like this {'tag': {'key': 'species', 'value': 'Pipistrellus pipistrellus'}, 'confidence_score': 0.526}
"""

def on_message(client, userdata, msg):
  print(f"{msg.topic}: {msg.payload.decode()}")
  draw.rectangle((0, 0, width, height), (0, 0, 0))
  displayhatmini.display()
  x = msg.payload.decode()
  y = json.loads(x)
  if(y["detections"]):
    species = y["detections"][0]["tags"][0]["tag"]["value"]
    score = y["detections"][0]["tags"][0]["confidence_score"]
    dt = y["recording"]["created_on"]
    print("species",species)

    text(draw, species, (25, 25), 15, (255, 255, 255))
    text(draw, str(score), (25, 100), 15, (255, 255, 255))
    text(draw, str(dt), (25, 175), 15, (255, 255, 255))
    displayhatmini.display()

client.on_connect = on_connect
client.on_message = on_message
client.connect("localhost",
               port=1883,
               keepalive=60)
client.loop_forever()

For completeness

cat display.service

[Unit]
Description=display

[Service]
Type=simple
WorkingDirectory=/home/pi/
ExecStart=/usr/bin/python3 mqtt_display.py
Restart=on-failure
StandardOutput=syslog
SyslogIdentifier=display
Type=idle
User=pi

[Install]
WantedBy=multi-user.target

sudo cp display.service /etc/systemd/system/
sudo systemctl enable display.service 
sudo systemctl start display.service 
sudo systemctl status display.service 

Even more for completeness - I find that sometimes if I get a power failure or for other undetermined reasons the message.db and metadata.db seems to lose data or maybe get corrupted. So here's a tiny python script to log it all as well.

import paho.mqtt.client as mqtt #import the client1
import time
import json
import traceback

def on_message(client, userdata, message):
    decoded_message=str(message.payload.decode("utf-8"))
    msg=json.loads(decoded_message)
    print("message received " ,decoded_message)
    print("message topic=",message.topic)
    try:
      with open("message_log.txt", "a") as myfile:
         myfile.write(decoded_message)
         myfile.write('\n')
    except Exception as e: 
      print(e)
      print("Exception")
      traceback.print_exc()

broker_address="127.0.0.1"

print("creating new instance")
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION1, "P1") 

client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(broker_address) #connect to broker
print("Subscribing to topic","acoupi")
client.subscribe("acoupi")

client.loop_forever()

Yesterday I gave a 5 min talk at Dorkbot Bristol "#mayke special" about what Mayke is, and a little bit about why we've been doing it. Here's a slightly expanded version.

Ma(y)ke Manifesto
------------------------------
Grasp the means of production
 and
Make something, big or tiny, every day in May
unless
 You want to slack off for the day.
There is no fayle, only mayke

A few of us are doing #mayke this year, sharing on mastodon and discord (because of the "slack off" clause, you absolutely can start now even though we're nearly half way through May. Join us!)

I think I invented mayke? maybe? I had a sudden doubt as I said it yesterday - I definitely remember talking to Richard about it - but in any case it has a lot of history and precedent so "inventing" is rather a strong word.

Jason Marc Taylor's year long project

Sometime in 2017 I came across Jason's site somehow and just thought - what an amazing idea! Jason made a thing every day for an entire year.

"So it's a silly idea but a challenge for me to make an object a day for the whole of 2012…. I'm going to try and transform everyday objects in someway (cause that's what I do) then post the results, I'm not saying they will be any good but hopefully they will spark my imagination in some way…so as I write this there is one day left. Enjoy 2012, make it great!"

His things are lovely, funny things made of household objects. I thought it was a great idea and so for a few months in 2017 I had a go at making a thing a day and putting it on tumblr (I've since rescued it from tumblr and put them here). My job at the time didn't feel very creative so that was part of why I wanted to do it then. Quite a few of the things I made were a bit shit, but I was making lots of things and some I was quite proud of.

This time coincided with me spending a lot of time at Bristol Hackspace. When I was stuck, there were people to help and lots of bits and pieces to make things with, and the general spirit of "half a thing is ok".

#makevember

In 2018 there was #makevember. Dominic Morrow's idea, inspired by Inktober and much more fun than doing it alone. We all talked about it on twitter when twitter was fun. Robert made a bot that retweeted #makevember and so we actually saw each others' work.

Dominic thought very carefully about what #makevember was, and how to balance the momentum of doing it every day with the pressure of having to do it every day:

Unfortunately I don't have very good pictures for this - I'm not on twitter any more, but here's a few I found (check out that lovely automata!)

#mayke now

As I was speaking yesterday I thought - for the first time - but why do it? It seemed such an obvious thing to me to do once it was suggested, I didn't question it. But there are some interesting effects of doing it, if you're thinking about joining in. It depends a bit on your approach.

My favourite way of doing it is just to see where life takes me, very much the makevember idea, to not think about it too much and just see what I can make with what I have around me. This has a lovely side effect that everything becomes a potential material for making, you kind of see the world differently. Zool has been using Concretedog's idea of "unconscious making" for this which I love.

But that's quite hard - you really need to be up for it. And I have a lot of unfinished things and things I want to start. So this time I've been doing a few multi-day experiments towards those. Barney did this for makevember one time ("Mechanical sequencer") and Oliver and Martin are doing it now and coming up with some ace stuff (web control of 3D printing in real time and a manga translation device respectively). Mine (so far) are "Spring simulator" and "Cube accounts music (cont.)", documented here. The benefit here is to start stuff or finish it, and keep going. Interim steps in public are great with a supportive community.

Another thing you can do is explore something, maybe a technique or technology with no particular aim. Tiff's been inventing "shonkoprinting" and Richard often does this kind of experimentation too.

I think it's a way to not procrastinate, to do something, maybe learn something, and keep your creative muscles exercised. Good things I think. Join us! Go on!

[later]

I realised this is related to a book I have been reading, Meditation for Mortals by Oliver Burkeman

The answer is: you just do them. You pick something you genuinely care about, and then, for at least a few minutes - a quarter of an hour, say - you do some of it. Today. It really is that simple.

]]>

This is a version of a talk I gave at Dorkbot in May 2024. It was pretty funny because it had lots of pictures of Cilla Black, because my brain always says "Lorra lorra" in her voice when I think of LoRa. But I don't know how she would actually feel about it so this is MoRe SeRious.

At that point I'd just left the BBC, I like puzzles and was looking for interesting mini-projects. I love Bristol Wireless' ethos, and love the idea of tiny long distance packets of data, so maybe LoRa(WAN) would be fun?

However, I'm a self taught programmer, more comfortable at the Javascript or Python levels of the stack. I rarely RTFM 😬. Maybe bad a idea? -> ⚠️❌

So the rest of this is -
* A little bit about LoRa(WAN) and what it is
* The three main faffs I've discovered (so far)
* A few ideas of what we could do with it
* Links and tips

This is from the point of view of someone starting out, who found it all a bit tricky but got there in the end. We now have a working gateway up and running in Bedminster 🎉

Some definitions

LoRaWAN: ITU[*]-standardised protocol and network architecture for LoRa
[*]International Telecommunications Union

LoRa: proprietary radio communication technique for sending a few bytes[*] of data to other similarly equipped devices
[*]0.3 kbit/s - 27 kbit/s according to wikipedia

The Things Network (TTN): mechanism to route your bytes anywhere and do something with them

Semtech: company that make all the LoRa Radio chips and own the LoRa patent

Some examples of usage

Node to node (device to device) - Usually ESP32 + LoRa radio. The nodes talk to each other directly without intermediary. Meshtastic uses this to pass messages on, but often you'd just be getting a pair to talk to each other. This is reasonably easy to get going.

+--------+          +--------+
|Node    |<-------> |Node    |      
+--------+          +--------+

Node to Gateway to The Things Network typically ESP32 -> Raspberry Pi hat or dedicated gateway.

The Things Network's main business model seems to be running private LoRaWAN networks for companies. They also provide certified training and a conference. They also run a free, open network of gateways that anyone can add to or use. Here's a map of their gateways.

In this usage the node talks to the nearest gateway, even if it's not yours, and gets funneled to your part of the Things Network (TTN) where you can retrieve it. On the free tier it's not stored, so you need to get it and do something with it, e.g. via MQTT. This gives you better range but means you are restricted in what you can send and how often by fair use - and you need a library that supports TTN.

                                        +---+       	 
                 +-------+           	| T |       	 
+--------+       |Gateway|              | T |     +------+
|Node    | ----> |in     + -----------> | N | --> |MQTT  |
+--------+       |range  |           	|   |     |etc   |
                 +-------+ +-------+ 	+---+     +------+
 +--------+                |Gateway|      ^          	 
 |Node    | -------------->|in     + -----+          	 
 +--------+                |range  |                 	 
                           +-------+                  	

TTN run the servers. These apparently can also be deployed on AWS - and an open source version of the server is available. I've not tried doing that - all my experiments have been via TTN's free tier with their servers.

The Faffs

Faff #1: Nodes
✅ It's reasonably easy to get device-to-device communication working.
✅ There are quite a few Arduino-IDE-compatible libraries and quite a few devices available
But!
⚠️ Get the wrong device and you're in a world of pain (Heltec, I'm looking at you)
✅  Unless! You use the RadioLib library
⚠️  Although! the RadioLib library doesn't actually have a simple device-to-device example

Faff #2: Gateways
⚠️The iC880A hat seemed to work but we couldn't see how to debug it or see packets
✅ The LR1302 hat worked fine (apart from a few weird bugs in the docs and also the spectral scan doesn't work)
⚠️The TTN indoor gateway worked seamlessly but the docs were super-confusing 

Faff #3: TTN
The mental model is a little confusing. You can have gateways and applications and these are separate. But the main issue is device setup.
All I can say here is:
* Find a library that supports LoRaWAN specification 1.1.0
* The rest is a world of pain
* I have never found a good explanation of what to put for regional parameters / technical specifications

If you are testing, just use all zeros for the EUI. You can also just make the number up, or some devices will generate it for you. Let the site autogenerate all the other numbers. 
You may be tempted to use v1.0.0 because it has fewer strange numbers to add DO NOT DO THIS.

Watch out for LSB/MSB (byte order) and hex, uint_32t - RadioLib is clear what you need here, many libraries are not

Worth it?

Hopefully
We could have lots of tiny, low-power devices dotted around Bristol, sending or receiving little pieces of information, all within community control. Like -

Allotment moisture sensors
Bird or animal detectors
Docks water quality sensors 
Bike path usage
Twinkly LED installations (maybe)
other things?

Tips and links

Use RadioLib for node devices
Use the LR1302 hat for a gateway
Use LilyGo devices for nodes
Use LoRaWAN protocol version 1.1.0

]]>

Some quick notes in case I ever have to do it again. A WARNING and REMINDER: this took 7 days for a 2TB disk.

One of my backup drives died and wouldn't mount. Really I was using it as an archive as I keep running out of disk space. This is probably a bad idea. Anyway I have rescued it.

My initial thought was using dd but this just kept failing (on unmounted disks; 5 was my new one, 4 the old one, identified using Disk Utility on my mac or diskutil list)

sudo dd bs=4M if=/dev/disk5s2 of=/dev/disk4s1
dd: /dev/disk5s2: Input/output error

A bit of searching round I found a shit article on ddrescue but that led me to the excellent manual page. The first example on there worked for me as a template ("Example 1: Fully automatic rescue of a whole disc with two ext2 partitions").

I used brew install ddrescue then

sudo ddrescue -f -r3 /dev/disk4 /dev/disk5 mapfile

[5 passes for Copying non-tried blocks, 2 for Trimming failed blocks, 3 for Retrying bad sectors]

rescued:    2000 GB,   bad areas:         14,       run time:  7d  1h 14m

then

sudo fsck_hfs -fy -l /dev/disk5s2

** The volume LaCie appears to be OK.
Hooray!

]]>

This is my talk from the Pervasive Media Studio Lunchtime Talk on 24 Jan 2025. For more on the importance of shonky prototypes, here's my previous talk, "Poking Holes in Reality with prototyping".

Thanks to the PM studio and Lawottim for hosting me. There's a video version on Youtube here. Thanks to everyone who reviewed it for me.

This is a talk about a little bit of my own history, some of the really great projects I've worked on, a recommendation for certain type of approach to projects, and a call to action to consider a missing piece of the problem. It's very much my personal view. My ex-colleagues may percieve things differently!

Here's me nearly 30 years ago. Not Mr Happy. fed up with my PhD, disappointed in the whole subject. I had not found my tribe; phds are a lonely business and philosophy of economics was an outlier in Bristol's economics department, which was mostly about mathematical modelling, auction theory and markets.

By showing me 'view source' in html - something he did to multiple people, with similar effects - my housemate Dan Brickley sent me on a journey to confirmed geek.

I forged myelf a career in technology, taught myself how to program, worked on the Semantic Web at the wonderful ILRT with Dan at Bristol University, then in a long-forgotten peer-to-peer web TV startup ('Joost').

In March 2024 I left BBC R&D (IRFS and Audiences teams) after nearly 15 years, where by the end I co-led a small team tasked with finding things the BBC could do in the future, with specific reference to young people, who …basically don't use the BBC right now. 

All this time later, through working in research, a startup and then more research, technology has served me very well, and yet - technology as something that now pervades everything we do has not served us well. In fact I find myself drawn away from technology and towards things that are real, physical and tangible, have no screens, and allow people to stop, think, and form their own ideas.

[Image from Buttystock "Burning crisp sandwich on a plate on a log" by mytalkingfist & moosecream]

One of the reasons I was interested in Economics to start with was that I was intrigued by the way new things get invented and made. It drew me to Economics, and yet it's not something Economics was (maybe is, I'm long out of that game) equipped to answer.

But I wanted to know, how did a new sandwich get invented? 

Actually the closest shop to my office at the UOB was a shop where you could choose your sandwich fillings and combine them. Having tried a few different things, I "invented" one (stilton and walnut, better, no salad, salt and pepper) which I still regard as the optimal sandwich. There the amateur sandwich maker, me, could invent a sandwich with the help of the expert, covering a range of options with many possible combinations.

So that's one way.

And the "expert-amateur [sandwich] discovery problem" seems to me to be an extremely important one.

Because people are experts in themselves and their own lives and we - you - are experts in other things (combining sandwich fillings, Javascript, Raspberry Pi audio streaming, ESP32s, whatever it is). So we need to try and combine those areas of expertise if we are to make something useful to people.

Here's my philosophy of R&D.

If we think about the possible space of, let's say, lunchtime foods, then we should start in places we might not even have thought of (by asking the people who are experts in their own lives) and then burrowing around the space around those places using our expertise. So we'll find out about maybe an interesting new Thai-German crossover place that people are going to and then invent things like basil and chili pretzels or green curry schnitzel sandwiches on dark bread and then test these and end up somewhere hopefully (and testably) popular.

So we're interacting with the people who are experts in their own lives at least twice: once to find out what they are already doing, and once to test our ideas. with them. Actually we probably ought to do the testing bit multiple times. And our expertise comes in when we're burrowing around the new space we have found.

This is a version of the 'double diamond' or as we used to call it - three onions. Essentially you are being extremely expansive and thinking of as many ideas as you can, and then filtering the ideas back down to the best ones - repeatedly. The difference here is I'm saying more about where we start each onion.

Now, I've (fairly!) recently left BBC R&D so it's still weighing on my mind: what we did that was great, why it was great, and why, ultimately we didn't manage to change a thing.

When I joined BBC R&D, with a few honourable exceptions, there were few interactions with "end users" and if there were, they tended to be "user testing" at the end of a project.

This always baffled me. Don't you want to know if the thing you want is actually wanted?

Well actually, on reflection, no.

This is a case where - to be really honest - no-one wants to hear that what they have invented is not wanted.

Research projects are in some sense about power, authority and who decides what gets made, and this matters, because while the power structures are in place, favoured projects will continue, and crowd out more interesting, relevant and useful ones.

So this is a story about a struggle to place some structures in place that would defeat - or at least prove valiant opponents to - the existing power structures in the organisation. But what we thought we were doing was implementing something that was already happening - but doing it in a more effective way.

I was part of this attempt to solve the expert-amateur discovery problem in R&D: how do you give amateurs - perhaps your future users or some people who need something - the tools to contribute meaningfully to some technical enterprise?

Solving the expert-amateur discovery problem is important for any organisation that wants to remain relevant as technologies, tastes, opinions change over time. Not just for lunch.

Here's an example of what I mean.

We dipped a toe in the water of understanding what people wanted by making these simple postcards - the project was about radio, we wondered what people wanted from a radio and this avoided the "blank page" problem - and most people could engage with a bit of drawing about their lives.

But one thing we discovered - obvious in retrospect but startling at the time - was that young people just didn't use physical radios. This little postcard process just didn't work for them because they didn't use them - if anything they were annoyed by their parents using them. So getting them to imagine what their ideal physical radio would do didn't work.

So this is an example of a technique which by its failure reveals that the whole sandwich ecosystem has been disrupted by, I dunno, the falafel upstarts.  Lots of people are still buying sandwiches, but the falafel movement has revealed a major change in lunch habits.

Not sure how far I can take this analogy. But the point is, we woudn't have realised this without going out and asking humans about their lives in an inventive fashion, well before we started to actually prototype anything. Ultimately in this case it allowed us to broaden out our remit from sandwiches to lunch, or more non-metaphorically, from radio to the role of audio in peoples' lives, and respond to changes within society that we hadn't realised were there.

Why is it important?

• If you care about the quality and usefulness of the work you are doing, OR
• you care that your work improves the world OR
• you think it's important for fairness that people are involved in things that affect their lives OR
• you want to find something new in a crowded market

or some combination, then this is a good way of going about it. And we wanted to do all of these things. 

People are experts in their own lives but you need ways to help them express what they think, want and need. Often people will be too shy, or just talk about things they have seen, or films or TV or something they heard on a podcast. You need to get them to reflect on their lives, and think, feel, and imagine the future.

One of our areas of expertise of our group in R&D was in designing techniques to help people do this. And many of them centred around prototypes.

[Image from "What do prototypes prototype?", Houde and Hill]

But perhaps not prototypes as you might be thinking of them.

I'm not talking about technology demonstrators or tests, or alpha websites.

I'm talking about things that elicit feelings and thinking from people about their lives, with respect to some technology.

The experimentation we did suggests that although something real-looking can help people imagine the role it would play in their lives, perhaps counter-intuitively, something unlike the real thing but which captures their imagination can be much more useful (and also, usually, cheaper)

One technique we used draws on "thingness", (Richard Sewell's term) - if you can get people to interact with the right kind of thing - a physical thing ideally, and one which embodies one idea - then you can get great feedback and attention, authentic and genuine involvement in imagining what it would be like if this thing were real. And then you can apply those insights to digital objects or sandwiches or whatever you need to.

Here's an Archers Avoider. The Archers, as I'm sure you all know, is an extremely long-running audio soap opera on Radio 4. Many many people love the Archers. Many many people also hate it.

The Archers Avoider just does one thing - you press a button on the top and it avoids the Archers for you. 

This enabled us to have a variety of enlightening conversations about what you want to avoid on the radio, what stops you listening, what radio doesn't allow you to do, and what you do when other people just won't stop listening to the Archers when you are in the room. 

Of course, if the BBC ever did make an Archers Avoider (which obviously it never would) it would be a button on the app, but the physical embodiment of that button helped people imagine what it might be like to react and behave in that world It did this because it was both silly and funny and single-minded. 

For balance, here's an "Exclusively Archers" version that we also made, which switched itself on when the Archers came on and off again afterwards. also just does one (opposite) thing too.

Physicality and thingness get you a long way in getting over the expert-amateur barrier. Physical objects avoid the flattening, boringness of screens and also the distractableness of phones and laptops - and so can persuade people to give you their attention and imagination. 

You'll also notice quite a few of these things are also quite silly.

Thingness - singluarity of purpose - helps with testability by decomposing the problem.

Physical things summon peoples' imagination and attention

But humour and silliness defuse one of the most significant problems in this relationship - power.

In a corporate environment it's not really the done thing to do silly or funny things - and yet, it's the best way to give yourself and others permission to say what isn't being said. Plus it's (obviously) a lot more fun.

And when it comes to working with people where there's a big power differential, it lets them know that you are not intransigent and you don't know everything.

And here we come to one of my favourite projects - and one that proves that the expert-amateur power differential can be overcome even when it's huge. My friend Jasmine Cox and the talented Marvin McKenzie worked with 8 and 9 year olds to invent a thing for the future. We had to get past our "boffin" status and make sure that the kids understood that their ideas were valuable - while trying to understand their lives and not just have them talk about Wall-E or something they had done in class.

Silly and funny is a lot easier with kids of course.

But power differentials appear within the "expert" realm too and silliness can work there too.

After I moaned about the inability of R&D to stop projects, Richard and I made catwigs (github), a tool to critique a project - in a funny way - in several dimensions.

Here, humour is used as a power-reset button within a group working on a project, allowing people to speak out where they might not, 

This was quite a powerful lesson for me, that turn-taking and explict rules for speaking can help enormously when people are hesitant to speak up, often those with actual hands-on knowledge of a project, and especially if they are young, shy and geeky, or have not been listened to before.

And it leads to all sorts of questions about projects

• who decides when to start and stop them?
• is anyone going to give up their power?

For any of these techniques to be effective, you've got to want to know more and accept that you don't know everything, that being great with say, node.js or immersive video or finance or management doesn't make you an expert on what these particular young people want from TV, for example.

And what about the "burrowing" part, the part where the experts take the new position in thingspace and think of things to make inspired by it? In some ways this is the exciting part, and the most fun part. We've done it collaboratively with potential end users, but also created various techniques to explore as much of that space as possible. 

Analogies are great for this ("what if TV was a dog? milk?" [another Richard Sewell special]) and processes and tools that take examples from the physical world and then use that to think about the problem ("what is it about IKEA that makes discovery of new things interesting?") [from a hopefully forthcoming article by Mathieu Triay and Andrew Wood]. 

Physicality comes in here too. Acting things out is a brilliant way of imaginatively placing yourself in a situation where you might use something. Even imagining examples from the physical world can help.  

[I didn't mention this in my talk but there are lots of existing techniques for this sort of thing - the term to look for is "design thinking"]

But the key thing is not to flatten or ignore or fail to develop ideas before they have had some space to flourish and effloresce. And this means - as with the catwigs example - giving all individuals space to speak and experiment. It also means that diverse, differently skilled, teams with different life experiences enable you to cover more of that potential space (there's plenty of evidence (archive) for this too).

Using these prototyping principles (and many custom techniques that we invented) in various combinations, we made some lovely things that people wanted. For example:

Orbit music a music browser purely about discovery, that helps you find genuinely new things to listen to in a beautiful and enjoyable way.

I didn't make this - it was mostly down to the brilliant Mathieu Triay - but I helped with the research. This was a project designed by my friend Tristan Ferne. It was about the BBC's incredible music resource, BBC Introducing, a place for new and emerging artists to get their work heard on the BBC.

We spent a good deal of time experimenting to try and understand how best to collaborate with our target audience of 18-24s.  In the end we had a really good process that treated them almost as clients. We interviewed them about their lives, showed them a wide range of possible thing-like prototypes based on those interviews, and then generated even more prototypes based on what they told us.

The second project I want to talk about is Tellybox - in the end we made four ways to help you make the best of the time you have to watch something with someone.

This project started with user research about what was difficult about watching TV, which turned out to be choosing what to watch. TV watching is a low stress, low effort way of hanging out with others, but this is completely ruined if you spend the first 30 mins of a limited time arguing about what to watch or scrolling through a TV app, getting more and more bored, irritated and despondent [it me].

We made and tested drawings, some very silly design fictions, nine working phyiscal prototypes, and then figured out a way to turn them into four digital objects which we could also test. The results included:

• a game where one player chooses five programmes, the next two, and the next one - and then you are all / both invested in whatever you both choose
• a randomiser that picks for you
• something that allows you to select programmes that fit the time you have (this would be trivially easy to implement, at least from a data point of view!)
• a voice based app that allowed you to vaguely select a kind of programme to watch, for vibe-based watching

The final project I want to talk about is BBC Together - which was less thing-like and physical object based, because it was made during Covid. Like a lot of people at the time, we wanted to make something that would help, and we already had the synchronisation technology.

So we built a way to watch TV with loved ones at precisely the same time when you are apart from them. And here's a lesson I learned from this about evaluation and testing that's also a clue about power: 

You can't do what people want you to do.

You can't show that you have the ideas, best structure, the best techniques, by getting large numbers of people using it, or by getting great qualitative feedback, because the counterfactual is what exists already - which at the BBC, already has more people using it than you could possibly have for a prototype.   

BBC Together had many thousands of users and very high ratings. But that wasn't enough to change peoples' minds.

So blah blah yes Libby is great at this stuff, sees things differently, is very employable and so on. And, yes, yes I am. But

The structures and norms we put in place around listening to potential end users, getting them to imagine futures, working in multidisciplinary teams, listening to our colleagues, speaking truth to power, making all this part of projects, showing everyone the results over and over - long term, didn't even change R&D, never mind the BBC.

And that's where it all falls down of course 😩

Musing on this, as one does after nearly 15 years trying and failing to do something, I recently came upon a rather brilliant page of books that should exist but don't. And this specific one chimed with me - 

"This book is for people who have been managing research projects long enough to become frustrated and furious by institutional inefficiency, inertia, and hypocrisy. Drawing on more than a century of community organizing, it explains how to figure out where power actually resides, how to build support for change, how to get into a position to make those changes…this book focuses on the practical "how" of being the right person in the right room on the right day to make things better a little bit at a time."

That seems like a very good expression of the remaining part of the problem. Flogging my analogy to death, we know how to find out how lunching is evolving, and we know how to invent different kinds of lunches, and both of those are really valuable to lunch-providing - or even any sustinance-providing organsations, if they are willing to listen. But what we don't know - never knew because we took people at face value when they said they were amenable to change - is how to get them to listen and then act.

We were trying to build new ruts, a new culture - but it didn't take, perhaps because we thought that it would be obvious how great the techniques were by the cool things we came up with. And in R&D you expect to fail, multiple times (but not all the time).

Mathieu made a very interesting point while critiquing these slides for me: in fact people did, eventually listen to us - but they didn't - or couldn't - act on the recommendations we gave them, even when we presented them with existential threats to the organisation. It was just too difficult for them to do so. Inertia, conservatism (small c) and technical debt were all reasons not to act.

"It's easier to drop it…until time and external forces have eroded enough of a path that there isn't a choice left and everyone falls all at once in the new ravine"

I'm interested in how we might accelerate this falling into the ravine -  I think this quote from Cory Dotorow is probably key - 

"Any time you see a group of people who've tried over some long timescale to do something, without success, it's *possible* that their failure is down to having bad tactics, but it's *far* more likely that they've failed because they just aren't powerful enough on their own to save the owls, or the planet, or their would-be ethnostate or whatever they're agitating for."

Change comes from coalitions of groups with the same interest, so if we think we have some great, successful, fair ways of working, then we need to build coalitions to make them the new norms, the new ruts, the new ravine.

This is important because these techniques and attitudes can - if actually acted upon - lead us to technologies that are closer to what people actually want and need.

What I'd love to know from you:

• have you tried techniques like this? 
• have you succeeded? 
• how did you know you had succeeded?
• Have you changed institutional minds? how??

I'm off to build some coalitions I think! 

I'm very interested in any thoughts and questions you have.

]]>

Everyone seems to have one of these - hope mine is useful to someone - I visited in November 2024.

1. If you have backups running on your phone and a non-Japan plan, disable roaming before you get there or it'll cost you. We got an eSim (I used Sakura) for £35 for 2 weeks for unlimited data. Internet works in tunnels and on the underground lines. Wheeee!
2. If you have an iPhone (didn't test Android) you can get a Suica IC card on your phone by just doing "add card" in Wallet. That means you can use the Tokyo metro and trains, some buses, and some vending machines too by tapping in - without needing to use cash or calculate how much a journey will be, which is massively useful.
3. You can also top these up using Apple Pay
4. Lots of places - including some corner shop 7-11 type places - took physical contactless UK cards. Some took virtual cards (like Apple Pay). Especially in Tokyo.
5. Cash was sometimes the only option outside Tokyo / touristy places. But you can get cash from most 7-11 type places. And a tiny, lovely, Sushi place in Yudanaka (Isamizushi) took contactless.
6. USB A or normal Japanese plugs (same as USA) were common on long distance trains but not USB C
7. You have to self-certify as being old enough to drink alcohol (by clicking 'yes') on the machines (which are supervised by people) in 7-11 type shops.
8. Navitime is great for route planning and includes Japan rail pass options
9. Japan Rail pass doesn't work on metro lines or some rail lines, including the one you are most likely to wander into at Haneda Airport. If you've got an IC card though, it's really cheap and you don't need to worry.
10. If you have a rail pass, the rail pass site is good for planning routes and booking reservations for Skinkansen, but you have to pick up the reservations from a physical machine in a station.
11. Getting anywhere in Tokyo takes ages even though the transport is incredibly good and has destinations, announcements, maps etc in English, Chinese and Korean as well as Japanese. It's just very big, very busy, and stations always have multiple exits.
12. There are loads of great public toilets everywhere
13. Lots of art galleries and museums are closed on Mondays in Tokyo
14. All railway stations have their own little tune that plays as the doors shut
15. Reddit was good for questions and tips
16. There's quite a bit of craft beer in Japan. There's a really really nice, friendly beer place in Shimbashi ("Dry Dock", try the chicken) and another in Asakusa "The Day" (with hotdogs).
17. Quite a lot of people speak English, especially in Tokyo.
18. There are some large supermarkets, but they are quite hard to find.
19. OMG the plastic

]]>

I have a server hosted at Mythic Beasts and I'd run out of disk space. They have instructions but I'm not experienced enough to be able to follow them as is, though I broadly followed it to make sure I was roughly on track.

Here's what I did so I can do it again. I do not necessarily think I did it in the optimal or correct way, but it did work. YMMV.

> df
...
/dev/mapper/nicecupoftea2--vg-root 156193244 135604120  12628756  92% /
...

> sudo lsblk

...
sr0                           11:0    1   667M  0 rom  
vda                          252:0    0   200G  0 disk 
├─vda1                       252:1    0   487M  0 part /boot
├─vda2                       252:2    0     1K  0 part 
└─vda5                       252:5    0 159.5G  0 part 
  ├─nicecupoftea2--vg-root   253:0    0 151.5G  0 lvm  /
  └─nicecupoftea2--vg-swap_1 253:1    0     8G  0 lvm  [SWAP]


> sudo parted /dev/vda

(parted) print free
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type      File system  Flags
        32.3kB  1049kB  1016kB            Free Space
 1      1049kB  512MB   511MB   primary   ext2         boot
        512MB   513MB   1048kB            Free Space
 2      513MB   172GB   171GB   extended
 5      513MB   172GB   171GB   logical                lvm
        172GB   215GB   43.0GB            Free Space

(parted) mkpart 
Partition type?  primary/logical? primary                                 
File system type?  [ext2]?                                                
Start? 173GB
End? 215GB
(parted) print                                                            
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size    Type      File system  Flags
 1      1049kB  512MB  511MB   primary   ext2         boot
 2      513MB   172GB  171GB   extended
 5      513MB   172GB  171GB   logical                lvm
 3      173GB   215GB  41.7GB  primary   ext2         lba

> sudo fdisk -l

Device     Boot     Start       End   Sectors   Size Id Type
/dev/vda1  *         2048    999423    997376   487M 83 Linux
/dev/vda2         1001470 335542271 334540802 159.5G  5 Extended
/dev/vda3       337891328 419430399  81539072  38.9G 83 Linux
/dev/vda5         1001472 335542271 334540800 159.5G 8e Linux LVM

> sudo pvcreate /dev/vda3
Physical volume "/dev/vda3" successfully created.

> sudo vgdisplay
  --- Volume group ---
  VG Name               nicecupoftea2-vg
  System ID             
  Format                lvm2
...

> sudo vgextend nicecupoftea2-vg /dev/vda3
  Volume group "nicecupoftea2-vg" successfully extended

> sudo lvextend -l +100%FREE nicecupoftea2-vg/root
Filesystem at /dev/mapper/nicecupoftea2--vg-root is mounted on /; online resizing required
old_desc_blocks = 10, new_desc_blocks = 12
The filesystem on /dev/mapper/nicecupoftea2--vg-root is now 49911808 (4k) blocks long.

> df
Filesystem                         1K-blocks      Used Available ...
/dev/mapper/nicecupoftea2--vg-root 196321316 135604972  51129744  73% /
...

Various links

How to resize a logical volume with 5 simple LVM commands guide helped and so did Creating and managing partitions in Linux with parted.

Overview of Logical Volume Manager (LVM) was handy, and How to Increase the size of a Linux LVM by expanding the virtual machine disk (steps to take, what the hex things mean).

I didn't know whether to use ext4 or what so I went with the default/existing, ext2 (some info). I didn't know if it should be logical or physical, my conclusion was that it didn't matter for my purposes.

]]>