UK Linux User Groups

So I arranged to take my car in Saturday to have the bit-of-plastic-on-the-driver’s-door fitted. I turn up, chap at the desk expected me and was very helpful.

Him> Ahh yes, part’s right here, shouldn’t take a mo to fit, I’ll just get some tools.

Me> Super thanks.

<fx>fiddling with the door happens</fx>

Him> Just going to make a phonecall, this isn’t looking how I expected

Me> It doesn’t look like a door with a well understood piece of plastic missing, I didn’t say.

Me> Uhh, sure.

<fx>phone call happens</fx>

Him> Sorry, but it looks like I can’t fit this part.

Me> You can’t fit this part.

Him> No, I think the inside door panel needs to be removed.

Me> The inside door panel needs to be removed, you think.

Him> Yes, and I’m not qualified.

Me> You’re not qualified. Super. Thanks.

I still have a hole in my door.

Meanwhile, 75cm toward the rear of my car….

You may recall that Renault Romford mistakenly did work I didn’t ask them to do, then insisted I pay for said work or I couldn’t have my car back.  I decided to take them up on their offer to simply undo the work and put back the faulty door mechanism and then give me a full refund.

Me> I’ve decided to take you up on your offer to simply undo the work and put back the faulty door mechanism and then give me a full refund.

Renault Romford> We can’t, we’ve thrown the old parts away.

Me> You’ve thrown the old parts away. Can I have a refund?

Renault Romford> I’ll need to call customer services for you….

I’ve had no call from Renault UK Customer services today, I left messages, Yvonne’s been a bit busy.

Still, nice to know I’m dealing with a reputable professional company, and not some east-end railway-arch crook.

Section 47 of the Government Response to the Science and Technology Committee's Evidence Check on homeopathy:
We note the Committee’s view that allowing for the provision of homeopathy may risk seeming to endorse it, and we will keep the position under review. However, we do not believe that this risk amounts to a risk to patient trust, choice or safety, nor do we believe that the risk is significant enough for the Department to take the unusual step of removing PCTs’ flexibility to make their own decisions. We believe that providing appropriate information for commissioners, clinicians and the public, and ensuring a strong ethical code for clinicians, remain the most effective ways to ensure quality outcomes, patient satisfaction and the appropriate use of NHS funding.
So basically no change. Our new government is just as capable of ignoring scientific evidence as the old one. And the NHS will continue to squander millions on sugar pills.
I expect I'll come back and fill this in with some more detail when I've calmed down a little.

Here are the slides from my talk at Europython 2010, Advanced Django ORM Techniques.

Advanced Django ORM techniques

The talk is mainly a summary of the query optimisation tricks I've previously talked about on this blog, although I did begin by explaining briefly how models, fields and relationships work behind the scenes - I'll write some of that up here at some point.

I'll also be posting a longer review of Europython here, hopefully in the next few days.

I added support for creating new LUKS disk images and managing existing key slots.

Two people asked me in about as many days if libguestfs supports encrypted VMs, so with the help of LUKS I added this feature.

A typical session in guestfish looks like this:

$ guestfish --ro -a encrypted.img ><fs> run ><fs> list-devices /dev/vda ><fs> list-partitions /dev/vda1 /dev/vda2 ><fs> vfs-type /dev/vda2 crypto_LUKS ><fs> luks-open /dev/vda2 luksdev Enter key or passphrase ("key"): ><fs> vgscan ><fs> vg-activate-all true ><fs> vgs vg_f13x64encrypted ><fs> lvs /dev/vg_f13x64encrypted/lv_root /dev/vg_f13x64encrypted/lv_swap ><fs> mount /dev/vg_f13x64encrypted/lv_root / ><fs> ll / total 132 dr-xr-xr-x. 24 root root 4096 Jul 21 12:01 . dr-xr-xr-x 20 root root 0 Jul 21 20:06 .. drwx------. 3 root root 4096 Jul 21 11:59 .dbus drwx------. 2 root root 4096 Jul 21 12:00 .pulse -rw-------. 1 root root 256 Jul 21 12:00 .pulse-cookie dr-xr-xr-x. 2 root root 4096 May 13 03:03 bin

Since it’s a little clumsy to use in guestfish at the moment, I hope we can add some convenience commands in a future release.

PS. WTF are dbus and pulseaudio doing creating those files and directories under /?

My nearly-four-years-old Renault Clio was due a bit of tender loving care.  The rear door lock had stopped being able to lock the doors, a small piece of plastic fell off the front driver’s side door and, the one that tipped it,  the driver side wing mirror was shattered; someone hit it in a car park – thanks!

I booked my car in to Renault Romford, dropped it in before work, and waited for a call back letting me know what everything was going to cost.  The call arrived and the suggested shopping list was a little longer than I expected.

• Rear engine mount starting to split – £103.00
• Front disks and pads should be replaced – £245.60
• Coolant and brake fluid drain and refill – £49.99
• Air conditioning service – £79.99
• Replacing the bit of plastic that fell off – £7.76
• Replacing the mirror – £27.87
• Replacing the rear door handle – £365.68 – wait, what?

I totally accept that most of the stuff in that list falls under the categories of “normal wear and tear” and “you broke it”, but things like a door handle should reasonably last the lifetime of the car.  It’s not like it’s even the most commonly used door.  Had this been the driver’s door, I might have grumbled a bit but understood.

I asked Renault Romford to replace the glass, replace the bit of plastic and service the aircon, explaining that I would sort out the brakes etc next month and would call Renault UK Customer services to talk to them about the rear door handle.

Renault UK customers service have still not contacted me, despite Tweets, emails and a telephone message asking them to do so.

I got a call yesterday saying my car was ready to be picked up, and that I owed them some £450.00.  I was confused.  They’d changed the rear door lock, despite me very explicitly saying that I was going to talk to Renault customer services about it.  Opinions were exchanged, we agreed this was most likely a misunderstanding.

I went to pick up my car this morning.  The car would not be released to me unless I paid in full.  In other words, I have paid hundreds of pounds for parts and work I didn’t ask to be done.

Insult to injury?  Hell, yes.  The didn’t replace the tiny piece of plastic that fell off the front door, one of the three things I did very specifically ask them to fix.

THERE’S A HOLE IN MY DOOR, DEAR RENAULT.   And I’m hundreds of pounds out of pocket.

We need to fix virt-rescue so that ^C works. It needs some Perl hacking though.

In the meantime, Ray Strode just pointed out that this works:

><rescue> setsid bash < /dev/ttyS0

Want to try out the latest development version of libguestfs but don’t want to break your Fedora 13 machine by installing Rawhide? Justin Forbes has added libguestfs to the Virtualization Preview repository for F13. If you add this repository you’ll be able to explore new virt features (not just libguestfs) before F14 is released.

(Thanks Justin!)

Software Freedom Day is on the 18th September. Software Freedom Day in the UK. Some ideas on what to do to celebrate free software.

(Thanks Michael Dorrington)

Rumors of my disappearance into a void are somewhat overrated.

I was invited to talk at the London Java Community on DevOps, I decided for that community it'd be good to have some examples with some code in. Fortunately my previous employers have open-sourced some of the frameworks used including their version of feature switches and a handy log configuration servlet.

It was good to catch up with old colleagues from ThoughtWorks, the London DevOps community and also meet new people.

The slides and talk are my own work, and do not represent the opinions of my employer...


Dev ops ljc 2010View more presentations from Paul Nasrat.

After someone asked me a question about “balloons” (in the virtualization sense) today, I noticed that there is not very much documentation around. This post explains what the KVM virtio_balloon driver is all about.

First of all, what is a balloon driver if you’ve never even heard of the concept? It’s a way to give or take RAM from a guest. (In theory at least), if your guest needs more RAM, you can use the balloon driver to give it more RAM. Or if the host needs to take RAM away from guests, it can do so. All of this is done without needing to pause or reboot the guest.

You might think that this would work as a RAM “hot add” feature, rather like hot adding disks to a guest. Although RAM hot add would (IMHO) be much better, currently this is not how ballooning works.

What we have is a kernel driver inside the guest called virtio_balloon. This driver acts like a kind of weird process, either expanding its own memory usage or shrinking down to nearly nothing, as in the diagrams below:

When the balloon driver expands, normal applications running in the guest suddenly have a lot less memory and the guest does the usual things it does when there’s not much memory, including swapping stuff out and starting up the OOM killer. (The balloon itself is non-swappable and un-killable in case you were wondering).

So what’s the point of a kernel driver which wastes memory? There are two points: Firstly, the driver communicates with the host (over the virtio channel), and the host gives it instructions (“expand to this size”, “shrink down now”). The guest cooperates, but doesn’t directly control the balloon.

Secondly, memory pages in the balloon are unmapped from the guest and handed back to the host, so the host can hand them out to other guests. It’s like the guest’s memory has a chunk missing from it:

Libvirt has two settings you can control called currentMemory and maxMemory (“memory” in the libvirt XML):

maxMemory (or just <memory>) is the memory allocated at boot time to a guest. KVM and Xen guests currently cannot exceed this. currentMemory controls what memory you’re requesting to give to the guest’s applications. The balloon fills the rest of the memory and gives it back to the host for the host to use elsewhere.

You can adjust this manually for your guests, either by editing the XML, or by using the virsh setmem command.

I had to build a fresh set of rpms for the area_cli tools, and decided it was also a good time to rebase some of my local personal build tools to rhel6beta2 since its there and I need to do some tests with it anyway.

Firstly, there are a couple of cool things with rhel6 beta2 - rpmdevtools is included by default, which helps gets things started and helps a bit in managing spec files etc.

What isnt so cool is that the newer rpm in el6beta creates packages which then cause issues with some of the older CentOS releases. eg. this areca_cli package built on el6beta2/x86_64 for target i686 caused this to happen with yum:

Downloading Packages: areca_cli-1.83_091103-1.e 100% |=========================| 493 kB 00:00 Running rpm_check_debug ERROR with rpm_check_debug vs depsolve: rpmlib(FileDigests) is needed by areca_cli rpmlib(PayloadIsXz) is needed by areca_cli Complete! (1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Which causes the package to not install. This is on a CentOS-5.2 machine. And I need to maintain that at CentOS-5.2 due to various issues ( no, machine is not available on the internet, and only hosts a single app in production ). Trying the install on a CentOS-5.3 machine I get the exact same issue. Upgrading rpm to 4.4.2.3-18.el5 ( the version in 5.4 and 5.5 ) makes no difference.

On the other hand, builds run on CentOS-5 have no problems installing on EL6Beta. So for the time being it looks like all buildhosts and all build stuff will need to stay on EL5. Specially since that allows you to target CentOS-3 and CentOS-4 as well.

- KB

Note: yeah, I see that url pointing at bugzilla isnt idea. I'll look into plumbing in a bugs.centos.org reference instead.

Original post.

Earlier this year, I created a petition on the (then) government's petition web site. The petition called for the government to fully implement the recommendations of the House of Commons Science and Technology Committee's Evidence Check on Homeopathy - basically calling on the government to stop wasting money on homeopathy.
The petition was due to be open for signatures for a year. In retrospect, that was probably a mistake as no matter who was in government, they would have made up their mind about the issue long before the petition closed.
But since the general election everything has changed. All of the petitions were closed to new signatures during the election campaign and they didn't re-open once the new government was in place. Instead the web site explained that the new government was considering the best way to proceed with the site. The front page of the site now says:
With a new Government in place a review is taking place of online services, including e-petitions. We are committed to improving the e-petitions process and are looking at ways of ensuring that it functions as part of a cohesive approach to public debate and transparent government. A full announcement on how we plan to use these and other services across Government will be made as soon as this important work is completed.It goes on to say:
Existing e-petitions, submitted to the previous administration, will not be carried forward to the new administration as part of this process. E-petitions that were live at the time of the election announcement on 6 April, when the e-petitions system was suspended, will therefore not be reopened for signatures. We are issuing responses to petitions that had exceeded the 500 signatures threshold as of 6 April 2010 and these can be viewed on the HMG e-petitions responses page.
So my petition has been closed. In the three or four months that it was open, over 1,600 people signed it. That means that we can expect some kind of response from the government, although it's not there yet and there's no indication of when we will receive it.
Thanks to everyone who signed the petition. Perhaps in this new cuts-driven regime removing finding for magic water on the NHS is an obvious way to save a few million quid.

Update: This git repository contains the source files for the video.

Or download from here (MPEG-2 format)

I hit a number of problems:

1. I lost my nice microphone and had to use a cheap USB headset. As a result the sound quality is pretty bad and there is also some popping.
2. The MJPEG converter in Blender isn’t able to include the sound track. (The AVI container should support adding a sound track, it just seems Blender can’t do it). So I had to use FFMPEG, and that only supports the proprietary MPEG-2 output format actually supports other proprietary formats. I reencoded the video has H.264 which looks a bit better. I will have to try and make a Ogg Theora version somehow.
3. YouTube has wrecked the video somehow. Obviously transcoding from the MPEG-2 upload to whatever YouTube uses wasn’t a good idea.
4. There are some less serious editing and timing issues with the recording and voiceover.

Click to view video (MJPEG format) (deleted it — see next posting)

All that is left is to add the voiceover.

When the fedorahosted.org virt tools repository has been set up (hopefully before the end of this week) I’ll be publishing the source for all of this.

I think I’m getting the hang of Blender now. Click here to view the video title sequence (MJPEG format — should play everywhere)

Below is a still from the sequence:

Being part of an oncall rota is pretty much a certainty if you work in the systems or operations team at any IT orientated company. Stuff needs to be working 24 hours a day, 7 days a week and therefore you need someone available to apply duct tape and staples when things go wrong out of normal working hours. It continually amazes me, therefore, that so many companies get the management and organisation of their rota badly wrong, sometimes to the point of it being a significant factor in staff moving on.

As someone who’s participated in oncall rotas at all levels, here’s how I think one should be organised:

• Sysadmins should only participate in the rota after they have completed their probationary period. Think of the goal of the probation period as being getting someone up to speed so they can participate.
• Each oncall shift should last one week, and rotate at 2.00pm on Tuesday. If you run a late shift, i.e. 11.00am to 8.00pm, make the person on that shift also be oncall. Get all the unsociable hours out of the way in one lump.
• The bare minimum gap between oncall shifts should be 5 weeks. If your rota is shorter than this, then you don’t have enough qualified staff to cope with holidays, illness, paternity/maternity leave etc anyway.
• The oncall shift should be mapped at least 3 months into the future. Staff should be free to swap oncall weeks providing that no-one is ever oncall two weeks in a row, and that all swaps are cleared with their line manager.
• Issue the oncaller with a good quality mobile phone and 3G capable laptop. The phone should not be a smartphone, the aim should be for maximum battery life and talk time. The laptop should be more like a netbook than a desktop replacement, should come with a spare battery, and should dual boot to Windows and a useful Linux desktop distro. These days I’d shell out for a nice big SSD to stick in it too. Make sure there are no restrictions on international calling and no data caps for the phone and 3G card.
• Don’t give the direct number of the oncall phone. Instead, use an answering service as filter.  During office hours, the answering service should redirect enquiries to the regular helpdesk/support number.  Out of hours, the answering service should accept calls, take details and then pass these on to the oncall number. Additionally this service should text and email call details so there’s a record.
• Pay a fixed daily amount for being oncall. Pay 1.5 times this amount for being oncall on weekend days. Pay 2 times this amount for being oncall on a public holiday. Where someone is oncall on a public holiday, add one day to their holiday allowance.
• Pay a per-incident fee when oncall is used. Each oncall use should be tracked in your ticketing system. Make using oncall a business cost, thus giving the business a reason to make sure oncall is not used trivially, and a reason to make sure problems are fixed permanently and not just temporarily alleviated.
• When the oncall person has dealt with out of hours issues, don’t expect them in at the regular time the next day.  Expect them to use their judgement to make sure they are suitably rested.   You do not want an overly tired oncaller dealing with problems on production systems.
• The person oncall should never be taking on “out of hours” work. Want a database dumped and reloaded? Want a disk unmounted and fscked overnight when a server’s not busy? All those things can be done, but not by the oncall person. That person is there to respond to problems, not to perform routine or planned maintenance.
• Make it very clear that abuse of oncall is unacceptable. Oncall is there to fix customer or service affecting problems, not to help someone with Excel.
• Have a clear demarcation between production and testing/development/QA systems. The latter group are not oncall’s responsibility to fix.
• If you have offices around the world, have a “follow the sun” oncall system, get your offices to cover each other.
• Have realistic expectations of oncall response times. If you need to guarantee that problems are attended to within 20-30 minutes then you should be running an overnight shift, not oncall.
• Expect a daily report summarising the previous 24 hour oncall period, even if that report is “Nothing to report”. The weekend period could be lumped together on the Monday.
• Have a weekly oncall handover meeting between the outgoing and incoming oncall staff.
• During the day, have a junior or trainee sysadmin be oncall. It’s good practice for them.

Doing all of the above shows that you take oncall seriously, and you appreciate the impact being oncall has on someone’s life. Your oncall staff are the people who salvage the business’s reputation when the midden hits the windmill at some unworldly hour of the night. Keeping them happy and making them feel valued and respected can only be to the business’s benefit.

Update: I was going to post an HTML5 video here, but WordPress strips the <video> element. Here’s the direct link to the video file (Motion JPEG format, should play in most places):

video, now with shadows

Another one from the Blender book:

This is really an animation, but I’ve no idea how to stitch all 50 frames together into a movie yet.

I used to be a proper expert on Blender. I even went (in fact, I was invited and flown out to1) the Blender Users Conference in 2000. Since I’ve not used Blender in about 8 years, it’s quite hard to remember all the commands now, but I did make it through the “cup tutorial” from an old (paper) copy of the Blender Book (alternative free Blender manual).

Blender is like “vi”. And I mean that in both a good and a bad way … It’s incredibly powerful once you know how to use it, but damn hard to learn (or relearn in my case).

I’ll give potential users two tips: 1. The Fedora package is currently broken. If you hit “Space” and you don’t see a menu, it’s broken for you too. 2. Don’t even try using Blender without a full size keyboard with a keypad and a 3 button mouse. Really, don’t even begin. It’s just not possible. Laptop? Forget it or plug in an external keyboard and 3 button mouse.

1 That was back in the days when they had VC money …

Sniper: Ghost Warrior is a standard first-person shooter based on the Chrome 4 engine.  The marketing suggests this game is aimed at those who prefer stealth and strategy over outright shoot’em’up action.   After a brief, optional, training run you are dropped into the action, sniper rifle in hand.

Visually I find the game very odd.  Environments are very rich indeed and this game features some of the best foliage I have seen in a PC game. Certainly better than Just Cause 2, somewhat better than Crysis, and yet, at the same time, you look at scenery such as buildings and vehicles and are disappointed.  There’s none of the beautiful detail that JC2 set the standard for.

As you start the game, it’s all pretty obvious, move from point A to point B, perhaps shooting some baddies on the way.  You quickly realise just how tightly scripted the game is, and how shallow some of the segments are, and you very quickly feel hemmed in by the tiny (I’ve been playing JC2) sections of world you can explore.

It’s when you start to move around that the worst part of the game makes itself obvious.  You are a sniper, wearing a ghillie suit and moving slowly and quietly around in dense jungle foliage.   Bullets will start pocking around you, you won’t have a clue where from bar a very vague red arrow in the middle of your screen.  What’s happened is that you’ve tripped over some enemy AI who’s managed to pick you out from a distance of about 150 meters.

You won’t be able to see him, but he’ll cheerfully keep shooting at you, not often hitting, until you stand up so your head is out of the undergrowth and eventually work out where he is.  By this time you’ll have moved enough that more eagle-eyed AI join in.  You die, you get to do it again.  It’s very very dull.

At this point you’ll realise that the sniping system is confusing and not very good.  Other than the target’s movement there are no real visual clues as to why the bullet mark is where it is.   It feels a bit random and sterile.

You’re not always just sniping, there are some sections of standard let-rip-on-full-auto.  You’re a fully trained, fit, agile sniper clearing out an oil rig.  Yet you somehow cannot make it over a knee-high rail to go down some stairs.  Eventually you’ll realise there’s a small panel you have to shoot to make a section of the rail disappear.  You’ll check your calendar to make sure it’s not 5 years ago.

The bullet camera is a straight ripoff, accidental or intentional, I don’t know,  of that given to us by Sniper Elite back in 2005.  Given the choice between the two games, I’d suggest picking up Sniper Elite and playing that instead.

If you’re really intent on buying this game, I’d just wait a bit.  It’ll be reduced in price quickly enough as it really is fairly poor.