Planet ALUG

Syndicate content
Planet ALUG - http://planet.alug.org.uk/
Updated: 38 min 33 sec ago

Daniel Silverstone (Kinnison): STM32 and RTFM

Sun, 06/08/2017 - 17:23

I have been working with STM32 chips on-and-off for at least eight, possibly closer to nine years. About as long as ST have been touting them around. I love the STM32, and have done much with them in C. But, as my previous two posts may have hinted, I would like to start working with Rust instead of C. To that end, I have been looking with great joy at the work which Jorge Aparicio has been doing around Cortex-M3 and Rust. I've had many comments in person at Debconf, and also several people mention on Twitter, that they're glad more people are looking at this. But before I can get too much deeper into trying to write my USB stack, I need to sort a few things from what Jorge has done as demonstration work.

Okay, this is fast, but we need Ludicrous speed

All of Jorge's examples seem to leave the system clocks in a fairly default state, excepting turning on the clocks to the peripherals needed during the initialisation phase. Sadly, if we're going to be running the USB at all, we need the clocks to run a tad faster. Since my goal is to run something moderately CPU intensive on the end of the USB too, it makes sense to try and get our STM32 running at maximum clock speed. For the one I have, that's 72MHz rather than the 8MHz it starts out with. Nine times more cycles to do computing in makes a lot of sense.

As I said above, I've been doing STM32 in C a lot for many years; and fortunately I have built systems with the exact chip that's on the blue-pill before. As such, if I rummage, I can find some old C code which does what we need...

/* Enable HSE */ RCC_HSEConfig(RCC_HSE_ON); /* Wait till HSE is ready */ HSEStartUpStatus = RCC_WaitForHSEStartUp(); if (HSEStartUpStatus == SUCCESS) { /* Enable Prefetch Buffer */ FLASH_PrefetchBufferCmd(FLASH_PrefetchBuffer_Enable); /* Flash 2 wait state */ FLASH_SetLatency(FLASH_Latency_2); /* HCLK = SYSCLK */ RCC_HCLKConfig(RCC_SYSCLK_Div1); /* PCLK2 = HCLK */ RCC_PCLK2Config(RCC_HCLK_Div1); /* PCLK1 = HCLK/2 */ RCC_PCLK1Config(RCC_HCLK_Div2); /* ADCCLK = PCLK2/6 */ RCC_ADCCLKConfig(RCC_PCLK2_Div6); /* PLLCLK = 8MHz * 9 = 72 MHz */ RCC_PLLConfig(RCC_PLLSource_HSE_Div1, RCC_PLLMul_9); /* Enable PLL */ RCC_PLLCmd(ENABLE); /* Wait till PLL is ready */ while (RCC_GetFlagStatus(RCC_FLAG_PLLRDY) == RESET) {} /* Select PLL as system clock source */ RCC_SYSCLKConfig(RCC_SYSCLKSource_PLLCLK); /* Wait till PLL is used as system clock source */ while (RCC_GetSYSCLKSource() != 0x08) {} }

This code, rather conveniently, uses an 8MHz external crystal so we can almost direct-port it to the blue-pill Rust code and see how we go. If you're used to the CMSIS libraries for STM32, then you won't completely recognise the above since it uses the pre-CMSIS core libraries to do its thing. Library code from 2008 and it's still good on today's STM32s providing they're in the right family :-)

A direct conversion to Rust, using Jorge's beautifully easy to work with crates made from svd2rust results in:

fn make_go_faster(rcc: &RCC, flash: &FLASH) { rcc.cr.modify(|_, w| w.hseon().enabled()); while !rcc.cr.read().hserdy().is_ready() {} flash.acr.modify(|_, w| w.prftbe().enabled()); flash.acr.modify(|_, w| w.latency().two()); rcc.cfgr.modify(|_, w| w .hpre().div1() .ppre2().div1() .ppre1().div2() // .adcpre().bits(8) .pllsrc().external() .pllxtpre().div1() .pllmul().mul9() ); rcc.cr.modify(|_, w| w.pllon().enabled()); while rcc.cr.read().pllrdy().is_unlocked() {} rcc.cfgr.modify(|_,w| w.sw().pll()); while !rcc.cfgr.read().sws().is_pll() {} }

Now, I've not put the comments in which were in the C code, because I'm being very lazy right now, but if you follow the two together you should be able to work it through. I don't have timeouts for the waits, and you'll notice a single comment there (I cannot set up the ADC prescaler because for some reason the SVD is missing any useful information and so the generated crate only carries an unsafe function (bits()) and I'm trying to steer clear of unsafe for now. Still, I don't need the ADC immediately, so I'm okay with this.

By using this function in the beginning of the init() function of the blinky example, I can easily demonstrate the clock is going faster since the LED blinks more quickly.

This function demonstrates just how simple it is to take bit-manipulation from the C code and turn it into (admittedly bad looking) Rust with relative ease and without any of the actual bit-twiddling. I love it.

Mess with time, and you get unexpected consequences

Sadly, when you mess with the clock tree on a microcontroller, you throw a lot of things out of whack. Not least, by adjusting the clock frequency up we end up adjusting the AHB, APB1, and APB2 clock frequencies. This has direct consequences for peripherals floating around on those busses. Fortunately Jorge thought of this and while the blue-pill crate hard-wires those frequencies to 8MHz, they are, at least, configurable in code in some sense.

If we apply the make_go_faster() function to the serial loopback example, it simply fails to work because now the bus which the USART1 peripheral is connected to (APB2) is going at a different speed from the expected power-on default of 8MHz. If you remember from the function, we did .hpre().div1() which set HCLK to 72MHz, then .ppre1().div2() which sets the APB1 bus clock to be HCLK divided by 2, and .ppre2().div1() which sets APB2 bus clock to be HCLK. This means that we'd need to alter src/lib.rs to reflect these changes in the clock frequences and in theory loopback would start working once more.

It'd be awkward to try and demonstrate all that to you since I only have a phone camera to hand, but if you own a blue-pill then you can clone Jorge's repo and have a go yourself and see that I'm not bluffing you.

With all this done, it'll be time to see if we can bring the USB peripheral in the STM32 online, and that will be the topic of my next post in this discovery series.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM, part 2

Sat, 05/08/2017 - 17:08

Previously we talked about all the different kinds of descriptors which USB devices use to communicate their capability. This is important stuff because to write any useful USB device firmware we need to be able to determine how to populate our descriptors. However, having that data on the device is entirely worthless without an understanding of how it gets from the device to the host so that it can be acted upon. To understand that, let's look at the USB wire protocol.

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire

I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets.

There are a number of packet types which can appear on the USB link:

Packet type Purpose Token Packet When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction. Data(0/1) Packet Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets. Handshake Packet Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene. Start of Frame Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

  1. Setup Token - The host addresses the device and endpoint (OUT0)
  2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
  3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

  1. In Token - The host addresses the device and endpoint (IN0)
  2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
  3. Ack Packet - The host acknowledges the data packet
  4. In Token - The host addresses the device and endpoint (IN0)
  5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
  6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

  1. Out Token - The host addresses the device and endpoint (OUT0)
  2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
  3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor.

From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

  1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
  2. On the second ACK the low levels are still thinking.
  3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
  4. On the fourth ACK the control transaction is entirely complete.
Thinking at the low levels of the control interface

Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets.

Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format Field Name Byte start Byte length Encoding Meaning bmRequestType 0 1 Bitmap Describes the kind of request, and the target of it. See below. bRequest 1 1 Code The request code itself, meanings of the rest of the fields vary by bRequest wValue 2 2 Number A 16 bit value whose meaning varies by request type wIndex 4 2 Number A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint. wLength 6 2 Number A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet Field Name Value Meaning bmRequestType 0x08 Data direction is IN (from device to host), recipient is the device bRequest 0x06 GET_DESCRIPTOR (in this instance, the device descriptor is requested) wValue 0x0001 This means the device descriptor wIndex 0x0000 Irrelevant, there's only 1 device descriptor anyway wLength 12 This is the length of a device descriptor (12 bytes) SET_ADDRESS to set a device's USB address Field Name Value Meaning bmRequestType 0x00 Data direction is OUT (from host to device), recipient is the device bRequest 0x05 SET_ADDRESS (Set the device's USB address) wValue 0x00nn The address for the device to adopt (max 127) wIndex 0x0000 Irrelevant for address setting wLength 0 There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more.

One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace"

This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though...

When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended.

Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device.

The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports.

The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length.

The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero.

At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time.

Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete?

So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode.

If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today.

The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far.

The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely.

I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

  • How IN OUT and SETUP transfers work.
  • How the endpoint buffer memory is configured.
  • That all bus-powered devices MUST respond to suspend/resume properly
  • That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
  • There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
  • How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): USB Device Stacks, on RTFM

Fri, 04/08/2017 - 17:05

I have been spending time with Jorge Aparicio's RTFM for Cortex M3 framework for writing Rust to target Cortex-M3 devices from Arm (and particularly the STM32F103 from ST Microelectronics). Jorge's work in this area has been of interest to me ever since I discovered him working on this stuff a while ago. I am very tempted by the idea of being able to implement code for the STM32 with the guarantees of Rust and the language features which I have come to love such as the trait system.

I have been thinking to myself that, while I admire and appreciate the work done on the GNUK, I would like to, personally, have a go at implementing some kind of security token on an STM32 as a USB device. And with the advent of the RTFM for M3 work, and Jorge's magical tooling to make it easier to access and control the registers on an M3 microcontroller, I figured it'd be super-nice to do this in Rust, with all the advantages that entails in terms of isolating unsafe behaviour and generally having the potential to be more easily verified as not misbehaving.

To do this though, means that I need a USB device stack which will work in the RTFM framework. Sadly it seems that, thus-far, only Jorge has been working on drivers for any of the M3 devices his framework supports. And one person can only do so much. So, in my infinite madness, I decided I should investigate the complexity of writing a USB device stack in Rust for the RTFM/M3 framework. (Why I thought this was a good idea is lost to the mists of late night Googling, but hey, it might make a good talk at the next conference I go to). As such, this blog post, and further ones along these lines, will serve as a partial tour of what I'm up to, and a partial aide-memoir for me about learning USB. If I get something horribly wrong, please DO contact me to correct me, otherwise I'll just continue to be wrong. If I've simplified something but it's still strictly correct, just let me know if it's an oversimplification since in a lot of cases there's no point in me putting the full details into a blog posting. I will mostly be considering USB2.0 protocol details but only really for low and full speed devices. (The hardware I'm targetting does low-speed and full-speed, but not high-speed. Though some similar HW does high-speed too, I don't have any to hand right now)

A brief introduction to USB

In order to go much further, I needed a grounding in USB. It's a multi-layer protocol as you might expect, though we can probably ignore the actual electrical layer since any device we might hope to support will have to have a hardware block to deal with that. We will however need to consider the packet layer (since that will inform how the hardware block is implemented and thus its interface) and then the higher level protocols on top.

USB is a deliberately asymmetric protocol. Devices are meant to be significantly easier to implement, both in terms of hardware and software, as compared with hosts. As such, despite some STM32s having OTG ports, I have no intention of supporting host mode at this time.

USB is arranged into a set of busses which are, at least in the USB1.1 case, broadcast domains. As such, each device has an address assigned to it by the host during an early phase called 'configuration'. Once the address is assigned, the device is expected to only ever respond to messages addressed to it. Note that since everything is asymmetric in USB, the device can't send messages on its own, but has to be asked for them by the host, and as such the addressing is always from host toward device.

USB devices then expose a number of endpoints through which communication can flow IN to the host or OUT to the device. Endpoints are not bidirectional, but the in and out endpoints do overlap in numbering. There is a special pair of endpoints, IN0 and OUT0 which, between them, form what I will call the device control endpoints. The device control endpoints are important since every USB device MUST implement them, and there are a number of well defined messages which pass over them to control the USB device. In theory a bare minimum USB device would implement only the device control endpoints.

Configurations, and Classes, and Interfaces, Oh My!

In order for the host to understand what the USB device is, and what it is capable of, part of the device control endpoints' responsibility is to provide a set of descriptors which describe the device. These descriptors form a heirarchy and are then glommed together into a big lump of data which the host can download from the device in order to decide what it is and how to use it. Because of various historical reasons, where a multi-byte value is used, they are defined to be little-endian, though there are some BCD fields. Descriptors always start with a length byte and a type byte because that way the host can parse/skip as necessary, with ease.

The first descriptor is the device descriptor, is a big one, and looks like this:

Device Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (18) bDescriptorType 1 1 Constant Device Descriptor (0x01) bcdUSB 2 2 BCD USB spec version compiled with bDeviceClass 4 1 Class Code, assigned by USB org (0 means "Look at interface descriptors", common value is 2 for CDC) bDeviceSubClass 5 1 SubClass Code, assigned by USB org (usually 0) bDeviceProtocol 6 1 Protocol Code, assigned by USB org (usually 0) bMaxPacketSize 7 1 Number Max packet size for IN0/OUT0 (Valid are 8, 16, 32, 64) idVendor 8 2 ID 16bit Vendor ID (Assigned by USB org) idProduct 10 2 ID 16bit Product ID (Assigned by manufacturer) bcdDevice 12 2 BCD Device version number (same encoding as bcdUSB) iManufacturer 14 1 Index String index of manufacturer name (0 if unavailable) iProduct 15 1 Index String index of product name (0 if unavailable) iSerialNumber 16 1 Index String index of device serial number (0 if unavailable) bNumConfigurations 17 1 Number Count of configurations the device has.

This looks quite complex, but breaks down into a relatively simple two halves. The first eight bytes carries everything necessary for the host to be able to configure itself and the device control endpoints properly in order to communicate effectively. Since eight bytes is the bare minimum a device must be able to transmit in one go, the host can guarantee to get those, and they tell it what kind of device it is, what USB protocol it supports, and what the maximum transfer size is for its device control endpoints.

The encoding of the bcdUSB and bcdDevice fields is interesting too. It is of the form 0xMMmm where MM is the major number, mm the minor. So USB2.0 is encoded as 0x0200, USB1.1 as 0x0110 etc. If the device version is 17.36 then that'd be 0x1736.

Other fields of note are bDeviceClass which can be 0 meaning that interfaces will specify their classes, and idVendor/idProduct which between them form the primary way for the specific USB device to be identified. The Index fields are indices into a string table which we'll look at later. For now it's enough to know that wherever a string index is needed, 0 can be provided to mean "no string here".

The last field is bNumConfigurations and this indicates the number of ways in which this device might function. A USB device can provide any number of these configurations, though typically only one is provided. If the host wishes to switch between configurations then it will have to effectively entirely quiesce and reset the device.

The next kind of descriptor is the configuration descriptor. This one is much shorter, but starts with the same two fields:

Configuration Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Configuration Descriptor (0x02) wTotalLength 2 2 Number Size of the configuration in bytes, in total bNumInterfaces 4 1 Number The number of interfaces in this configuration bConfigurationValue 5 1 Number The value to use to select this configuration iConfiguration 6 1 Index The name of this configuration (0 for unavailable) bmAttributes 7 1 Bitmap Attributes field (see below) bMaxPower 8 1 Number Maximum bus power this configuration will draw (in 2mA increments)

An important field to consider here is the bmAttributes field which tells the host some useful information. Bit 7 must be set, bit 6 is set if the device would be self-powered in this configuration, bit 5 indicates that the device would like to be able to wake the host from sleep mode, and bits 4 to 0 must be unset.

The bMaxPower field is interesting because it encodes the power draw of the device (when set to this configuration). USB allows for up to 100mA of draw per device when it isn't yet configured, and up to 500mA when configured. The value may be used to decide if it's sensible to configure a device if the host is in a low power situation. Typically this field will be set to 50 to indicate the nominal 100mA is fine, or 250 to request the full 500mA.

Finally, the wTotalLength field is interesting because it tells the host the total length of this configuration, including all the interface and endpoint descriptors which make it up. With this field, the host can allocate enough RAM to fetch the entire configuration descriptor block at once, simplifying matters dramatically for it.

Each configuration has one or more interfaces. The interfaces group some endpoints together into a logical function. For example a configuration for a multifunction scanner/fax/printer might have an interface for the scanner function, one for the fax, and one for the printer. Endpoints are not shared among interfaces, so when building this table, be careful.

Next, logically, come the interface descriptors:

Interface Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (9) bDescriptorType 1 1 Constant Interface Descriptor (0x04) bInterfaceNumber 2 1 Number The number of the interface bAlternateSetting 3 1 Number The interface alternate index bNumEndpoints 4 1 Number The number of endpoints in this interface bInterfaceClass 5 1 Class The interface class (USB Org defined) bInterfaceSubClass 6 1 SubClass The interface subclass (USB Org defined) bInterfaceProtocol 7 1 Protocol The interface protocol (USB Org defined) iInterface 8 1 Index The name of the interface (or 0 if not provided)

The important values here are the class/subclass/protocol fields which provide a lot of information to the host about what the interface is. If the class is a USB Org defined one (e.g. 0x02 for Communications Device Class) then the host may already have drivers designed to work with the interface meaning that the device manufacturer doesn't have to provide host drivers.

The bInterfaceNumber is used by the host to indicate this interface when sending messages, and the bAlternateSetting is a way to vary interfaces. Two interfaces with the came bInterfaceNumber but different bAlternateSettings can be switched between (like configurations, but) without resetting the device.

Hopefully the rest of this descriptor is self-evident by now.

The next descriptor kind is endpoint descriptors:

Endpoint Descriptor Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (7) bDescriptorType 1 1 Constant Endpoint Descriptor (0x05) bEndpointAddress 2 1 Endpoint Endpoint address (see below) bmAttributes 3 1 Bitmap Endpoint attributes (see below) wMaxPacketSize 4 2 Number Maximum packet size this endpoint can send/receive bInterval 6 1 Number Interval for polling endpoint (in frames)

The bEndpointAddress is a 4 bit endpoint number (so there're 16 endpoint indices) and a bit to indicate IN vs. OUT. Bit 7 is the direction marker and bits 3 to 0 are the endpoint number. This means there are 32 endpoints in total, 16 in each direction, 2 of which are reserved (IN0 and OUT0) giving 30 endpoints available for interfaces to use in any given configuration. The bmAttributes bitmap covers the transfer type of the endpoint (more below), and the bInterval is an interval measured in frames (1ms for low or full speed, 125µs in high speed). bInterval is only valid for some endpoint types.

The final descriptor kind is for the strings which we've seen indices for throughout the above. String descriptors have two forms:

String Descriptor (index zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) wLangID[0] 2 2 Number Language code zero (e.g. 0x0409 for en_US) wLangID[n] 4.. 2 Number Language code n ...

This form (for descriptor 0) is that of a series of language IDs supported by the device. The device may support any number of languages. When the host requests a string descriptor, it will supply both the index of the string and also the language id it desires (from the list available in string descriptor zero). The host can tell how many language IDs are available simply by dividing bLength by 2 and subtracting 1 for the two header bytes.

And for string descriptors of an index greater than zero:

String Descriptor (index greater than zero) Field Name Byte start Byte length Encoding Meaning bLength 0 1 Number Size of the descriptor in bytes (variable) bDescriptorType 1 1 Constant String Descriptor (0x03) bString 2.. .. Unicode The string, in "unicode" format

This second form of the string descriptor is simply the the string is in what the USB spec calls 'Unicode' format which is, as of 2005, defined to be UTF16-LE without a BOM or terminator.

Since string descriptors are of a variable length, the host must request strings in two transactions. First a request for 2 bytes is sent, retrieving the bLength and bDescriptorType fields which can be checked and memory allocated. Then a request for bLength bytes can be sent to retrieve the entire string descriptor.

Putting that all together

Phew, this is getting to be quite a long posting, so I'm going to leave this here and in my next post I'll talk about how the host and device pass packets to get all that information to the host, and how it gets used.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Gitano 1.1

Thu, 03/08/2017 - 17:34

Today marks the release of Gitano 1.1. Richard(s) and I have spent quite a lot of time and effort on this release, and there's plenty of good stuff in it. We also released new versions of Lace, Supple, Luxio, and Gall to go alongside it, with bugfixes and improvements.

At this point, I intend to take a short break from Gitano to investigate some Rust-on-STM32 stuff, and then perhaps do some NetSurf work too.

Categories: LUG Community Blogs

Mick Morgan: a letter to our dear home secretary

Wed, 02/08/2017 - 15:56

Dear Amber

So,”real people” don’t care about privacy? All they really want is ease of use and a pretty GUI so that they can chat to all their friends on-line? Only “the enemy” (who is that exactly anyway?) needs encryption? Excuse me for asking, but what have you been smoking? Does the Home Office know about that?

I’m a real person. And I care deeply about privacy. I care enough to fund both my own Tor node and various openVPN servers dotted around the world just to get past your ludicrous attempts at gratuitous surveillance of my (and my family’s) routine use of the ‘net. I care about the security and privacy of my transactions with various commercial enterprises, including my bank (which is why I expect them to use TLS on their website). I care about privacy when I correspond with my Doctor and other professionals. I care about privacy when I use an on-line search engine (which, incidentally, is not Google). I care about privacy because privacy matters. I have the right to freedom of thought and expression. I have the right to discuss those thoughts with others of my choice – when I choose and how I choose. You may not like that, but it’s a fact of life. That doesn’t make me “the enemy”. Get over it.

Love and Kisses

Mick

(Note to readers: Aral Balkan has deconstructed Rudd’s ramblings. I commend the article to you.)

Categories: LUG Community Blogs

Jonathan McDowell: How to make a keyring

Mon, 31/07/2017 - 21:17

Every month or two keyring-maint gets a comment about how a key update we say we’ve performed hasn’t actually made it to the active keyring, or a query about why the keyring is so out of date, or told that although a key has been sent to the HKP interface and that is showing the update as received it isn’t working when trying to upload to the Debian archive. It’s frustrating to have to deal with these queries, but the confusion is understandable. There are multiple public interfaces to the Debian keyrings and they’re not all equal. This post attempts to explain the interactions between them, and how I go about working with them as part of the keyring-maint team.

First, a diagram to show the different interfaces to the keyring and how they connect to each other:

Public interfaces rsync: keyring.debian.org::keyrings

This is the most important public interface; it’s the one that the Debian infrastructure uses. It’s the canonical location of the active set of Debian keyrings and is what you should be using if you want the most up to date copy. The validity of the keyrings can be checked using the included sha512sums.txt file, which will be signed by whoever in keyring-maint did the last keyring update.

HKP interface: hkp://keyring.debian.org/

What you talk to with gpg --keyserver keyring.debian.org. Serves out the current keyrings, and accepts updates to any key it already knows about (allowing, for example, expiry updates, new subkeys + uids or new signatures without the need to file a ticket in RT or otherwise explicitly request it). Updates sent to this interface will be available via it within a few hours, but must be manually folded into the active keyring. This in general happens about once a month when preparing for a general update of the keyring; for example b490c1d5f075951e80b22641b2a133c725adaab8.

Why not do this automatically? Even though the site uses GnuPG to verify incoming updates there are still occasions we’ve seen bugs (such as #787046, where GnuPG would always import subkeys it didn’t understand, even when that subkey was already present). Also we don’t want to allow just any UID to be part of the keyring. It is thus useful to retain a final set of human based sanity checking for any update before it becomes part of the keyring proper.

Alioth/anonscm: https://anonscm.debian.org/git/keyring/keyring.git/

A public mirror of the git repository the keyring-maint team use to maintain the keyring. Every action is recorded here, and in general each commit should be a single action (such as adding a new key, doing a key replacement or moving a key between keyrings). Note that pulling in the updates sent via HKP count as a single action, rather than having a commit per key updated. This mirror is updated whenever a new keyring is made active (i.e. made available via the rsync interface). Until that point pending changes are kept private; we sometimes deal with information such as the fact someone has potentially had a key compromised that we don’t want to be public until we’ve actually disabled it. Every “keyring push” (as we refer to the process of making a new keyring active) is tagged with the date it was performed. Releases are also tagged with their codenames, to make it easy to do comparisons over time.

Debian archive

This is actually the least important public interface to the keyring, at least from the perspective of the keyring-maint team. No infrastructure makes use of it and while it’s mostly updated when a new keyring is made active we only make a concerted effort to do so when it is coming up to release. It’s provided as a convenience package rather than something which should be utilised for active verification of which keys are and aren’t currently part of the keyring.

Team interface Master repository: kaufmann.debian.org:/srv/keyring.debian.org/master-keyring.git

The master git repository for keyring maintenance is stored on kaufmann.debian.org AKA keyring.debian.org. This system is centrally managed by DSA, with only DSA and keyring-maint having login rights to it. None of the actual maintenance work takes place here; it is a bare repo providing a central point for the members of keyring-maint to collaborate around.

Private interface Private working clone

This is where all of the actual keyring work happens. I have a local clone of the repository from kaufmann on a personal machine. The key additions / changes I perform all happen here, and are then pushed to the master repository so that they’re visible to the rest of the team. When preparing to make a new keyring active the changes that have been sent to the HKP interface are copied from kaufmann via scp and folded in using the pull-updates script. The tree is assembled into keyrings with a simple make and some sanity tests performed using make test. If these are successful the sha512sums.txt file is signed using gpg --clearsign and the output copied over to kaufmann. update-keyrings is then called to update the active keyrings (both rsync + HKP). A git push public pushes the changes to the public repository on anonscm. Finally gbp buildpackage --git-builder='sbuild -d sid' tells git-buildpackage to use sbuild to build a package ready to be uploaded to the archive.

Hopefully that helps explain the different stages and outputs of keyring maintenance; I’m aware that it would be a good idea for this to exist somewhere on keyring.debian.org as well and will look at doing so.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): F/LOSS activity, July 2017

Mon, 31/07/2017 - 20:10

Once again, my focus was on Gitano, which we're working toward a 1.1 for. We had another one of our Gitano developer days which was attended by Richard maw and myself. You are invited to read the wiki page but a summary of what happened, which directly involved me, is:

  • Once again, we reviewed our current task state
  • We had a good discussion about our code of conduct including adopting a small change from upstream to improve matters
  • I worked on, and submitted a patch for, improving nested error message reports in Lace.
  • I reviewed and merged some work from Richard about pattern centralisation
  • I responded to comments on a number of in-flight series Richard had reviewed for me.
  • We discussed our plans for 1.1 and agreed that we'll be skipping a developer day in August because so much of it is consumed by DebConf and so on.

Other than that, related to Gitano during July I:

  • Submitted some code series before the developer day covering Gall cleanups and hook support in Gitano.
  • Reviewed and merged some more Makefile updates from Richard Ipsum
  • Reviewed and merged a Supple fix for environment cleardown from Richard Ipsum
  • Fixed an issue in one of the Makefiles which made it harder to build with dh-lua
  • I began work in earnest on Gitano CI, preparing a lot of scripts and support to sit around Jenkins (for now) for CIing packaging etc for Gitano and Debian
  • I began work on a system branch concept for Gitano CI which will let us handle the CI of branches in the repos, even if they cross repos.

I don't think I've done much non-Gitano F/LOSS work in July, but I am now in Montréal for debconf 2017 so hopefully more to say next month.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in July 2017

Mon, 31/07/2017 - 18:35

Here is my monthly update covering what I have been doing in the free software world during July 2017 (previous month):

  • Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
    • Moved the default mirror from ftp.de.debian.org to deb.debian.org. []
    • Create a sensible debian/changelog file if one does not exist. []
  • Updated django-slack, my library to easily post messages to the Slack group-messaging utility:
    • Merged a PR to clarify the error message when a channel could not be found. []
    • Reviewed and merged a suggestion to add a TestBackend. []
  • Added Pascal support to Louis Taylor's anyprint hack to add support for "print" statements from other languages into Python. []
  • Filed a PR against Julien Danjou's daiquiri Python logging helper to clarify an issue in the documentation. []
  • Merged a PR to Strava Enhancement Suite — my Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker — to remove Zwift activities with maps. []
  • Submitted a pull request for Redis key-value database store to fix a spelling mistake in a binary. []
  • Sent patches upstream to the authors of the OpenSVC cloud engine and the Argyll Color Management System to fix some "1204" typos.
  • Fixed a number of Python and deployment issues in my stravabot IRC bot. []
  • Correct a "1204" typo in Facebook's RocksDB key-value store. []
  • Corrected =+ typos in the Calibre e-book reader software. []
  • Filed a PR against the diaspy Python interface to the DIASPORA social network to correct the number of seconds in a day. []
  • Sent a pull request to remedy a =+ typo in sparqlwrapper, a SPARQL endpoint interface for Python. []
  • Filed a PR against Postfix Admin to fix some =+ typos. []
  • Fixed a "1042" typo in ImageJ, a Java image processing library. []
  • On a less-serious note, I filed an issue for Brad Abraham's bot for the Reddit sub-reddit to add some missing "hit the gym" advice. []

I also blogged about my recent lintian hacking and installation-birthday package.

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

(I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.)

This month I:

  • Assisted Mattia with a draft of an extensive status update to the debian-devel-announce mailing list. There were interesting follow-up discussions on Hacker News and Reddit.
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
  • I also submitted 5 patches to fix specific reproducibility issues in autopep8, castle-game-engine, grep, libcdio & tinymux.
  • Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
  • Worked on publishing our weekly reports. (#114 #115, #116 & #117)

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • comparators.xml:
    • Fix EPUB "missing file" tests; they ship a META-INF/container.xml file. []
    • Misc style fixups. []
  • APK files can also be identified as "DOS/MBR boot sector". (#868486)
  • comparators.sqlite: Simplify file detection by rewriting manual recognizes call with a Sqlite3Database.RE_FILE_TYPE definition. []
  • comparators.directory:
    • Revert the removal of a try-except. (#868534)
    • Tidy module. []

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Add missing File::Temp imports in the JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982. (#868077)

buildinfo.debian.net

buildinfo.debian.net is my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them.

  • Avoid a race condition between check-and-creation of Buildinfo instances. []


Debian

My activities as the current Debian Project Leader are covered in my "Bits from the DPL emails to the debian-devel-announce mailing list.

Patches contributed
  • obs-studio: Remove annoying "click wrapper" on first startup. (#867756)
  • vim: Syntax highlighting for debian/copyright files. (#869965)
  • moin: Incorrect timezone offset applied due to "84600" typo. (#868463)
  • ssss: Add a simple autopkgtest. (#869645)
  • dch: Please bump $latest_bpo_dist to current stable release. (#867662)
  • python-kaitaistruct: Remove Markdown and homepage references from package long descriptions. (#869265)
  • album-data: Correct invalid Vcs-Git URI. (#869822)
  • pytest-sourceorder: Update Homepage field. (#869125)

I also made a very large number of contributions to the Lintian static analysis tool. To avoid duplication here, I have outlined them in a separate post.

Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 1014-1 for libclamunrar, a library to add unrar support to the Clam anti-virus software to fix an arbitrary code execution vulnerability.
  • Issued DLA 1015-1 for the libgcrypt11 crypto library to fix a "sliding windows" information leak.
  • Issued DLA 1016-1 for radare2 (a reverse-engineering framework) to prevent a remote denial-of-service attack.
  • Issued DLA 1017-1 to fix a heap-based buffer over-read in the mpg123 audio library.
  • Issued DLA 1018-1 for the sqlite3 database engine to prevent a vulnerability that could be exploited via a specially-crafted database file.
  • Issued DLA 1019-1 to patch a cross-site scripting (XSS) exploit in phpldapadmin, a web-based interface for administering LDAP servers.
  • Issued DLA 1024-1 to prevent an information leak in nginx via a specially-crafted HTTP range.
  • Issued DLA 1028-1 for apache2 to prevent the leakage of potentially confidential information via providing Authorization Digest headers.
  • Issued DLA 1033-1 for the memcached in-memory object caching server to prevent a remote denial-of-service attack.
Uploads
  • redis:
    • 4:4.0.0-1 — Upload new major upstream release to unstable.
    • 4:4.0.0-2 — Make /usr/bin/redis-server in the primary package a symlink to /usr/bin/redis-check-rdb in the redis-tools package to prevent duplicate debug symbols that result in a package file collision. (#868551)
    • 4:4.0.0-3 — Add -latomic to LDFLAGS to avoid a FTBFS on the mips & mipsel architectures.
    • 4:4.0.1-1 — New upstream version. Install 00-RELEASENOTES as the upstream changelog.
    • 4:4.0.1-2 — Skip non-deterministic tests that rely on timing. (#857855)
  • python-django:
    • 1:1.11.3-1 — New upstream bugfix release. Check DEB_BUILD_PROFILES consistently, not DEB_BUILD_OPTIONS.
  • bfs:
    • 1.0.2-2 & 1.0.2-3 — Use help2man to generate a manpage.
    • 1.0.2-4 — Set hardening=+all for bindnow, etc.
    • 1.0.2-5 & 1.0.2-6 — Don't use upstream's release target as it overrides our CFLAGS & install RELEASES.md as the upstream changelog.
    • 1.1-1 — New upstream release.
  • libfiu:
    • 0.95-4 — Apply patch from Steve Langasek to fix autopkgtests. (#869709)
  • python-daiquiri:
    • 1.0.1-1 — Initial upload. (ITP)
    • 1.1.0-1 — New upstream release.
    • 1.1.0-2 — Tidy package long description.
    • 1.2.1-1 — New upstream release.

I also reviewed and sponsored the uploads of gtts-token 1.1.1-1 and nlopt 2.4.2+dfsg-3.

Debian bugs filed
  • ITP: python-daiquiri — Python library to easily setup basic logging functionality. (#867322)
  • twittering-mode: Correct incorrect time formatting due to "84600" typo. (#868479)
FTP Team

As a Debian FTP assistant I ACCEPTed 45 packages: 2ping, behave, cmake-extras, cockpit, cppunit1.13, curvedns, flask-mongoengine, fparser, gnome-shell-extension-dash-to-panel, graphene, gtts-token, hamlib, hashcat-meta, haskell-alsa-mixer, haskell-floatinghex, haskell-hashable-time, haskell-integer-logarithms, haskell-murmur-hash, haskell-quickcheck-text, haskell-th-abstraction, haskell-uri-bytestring, highlight.js, hoel, libdrm, libhtp, libpgplot-perl, linux, magithub, meson-mode, orcania, pg-dirtyread, prometheus-apache-exporter, pyee, pytest-pep8, python-coverage-test-runner, python-digitalocean, python-django-imagekit, python-rtmidi, python-transitions, qdirstat, redtick, ulfius, weresync, yder & zktop.

I additionally filed 5 RC bugs against packages that had incomplete debian/copyright files against: cockpit, cppunit, cppunit1.13, curvedns & highlight.js.

Categories: LUG Community Blogs

Chris Lamb: More Lintian hacking

Sat, 29/07/2017 - 09:31

Lintian is static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer.

I seem to have found myself hacking on it a bit more recently (see my previous installment). In particular, here's the code of mine — which made for a total of 20 bugs closed — that made it into the recent 2.5.52 release:

New tags
  • Check for the presence of an .asc signature in a .changes file if an upstream signing key is present. (#833585, tag)
  • Warn when dpkg-statoverride --add is called without a corresponding --list. (#652963, tag)
  • Check for years in debian/copyright that are later than the top entry in debian/changelog. (#807461, tag)
  • Trigger a warning when DEB_BUILD_OPTIONS is used instead of DEB_BUILD_MAINT_OPTIONS. (#833691, tag)
  • Look for "FIXME" and similar placeholders in various files in the debian directory. (#846009, tag)
  • Check for useless build-dependencies on dh-autoreconf or autotools-dev under Debhelper compatibility levels 10 or higher. (#844191, tag)
  • Emit a warning if GObject Introspection packages are missing dependencies on ${gir:Depends}. (#860801, tag)
  • Check packages do not contain upstart configuration under /etc/init. (#825348, tag)
  • Emit a classification tag if maintainer scripts such as debian/postinst is an ELF binary. (tag)
  • Check for overly-generic manual pages such as README.3pm.gz. (#792846, tag)
  • Ensure that (non-ELF) maintainer scripts begin with #!. (#843428, tag)
Regression fixes
  • Ensure r-data-without-readme-source checks the source package, not the binary; README.source files are not installed in the latter. (#866322, tag)
  • Don't emit source-contains-prebuilt-ms-help-file for files generated by Halibut. (#867673, tag)
  • Add .yml to the list of file extensions to avoid false positives when emitting extra-license-file. (#856137, tag)
  • Append a regression test for enumerated lists in the "a) b) c) …" style, which would previously trigger a "duplicate word" warning if the following paragraph began with an "a." (#844166, tag)
Documentation updates
  • Rename copyright-contains-dh-make-perl-boilerplate to copyright-contains-automatically-extracted-boilerplate as it can be generated by other tools such as dh-make-elpa. (#841832, tag)
  • Changes to new-package-should-not-package-python2-module (tag):
    • Upgrade from I: to W:. (#829744)
    • Clarify wording in description to make the justification clearer.
  • Clarify justification in debian-rules-parses-dpkg-parsechangelog. (#865882, tag)
  • Expand the rationale for the latest-debian-changelog-entry-without-new-date tag to mention possible implications for reproducible builds. (tag)
  • Update the source-contains-prebuilt-ms-help-file description; there exists free software to generate .chm files. (tag)
  • Append an example shell snippet to explain how to prevent init.d-script-sourcing-without-test. (tag)
  • Add a missing "contains" verb to the description of the debhelper-autoscript-in-maintainer-scripts tag. (tag)
  • Consistently use the same "Debian style" RFC822 date format for both "Mirror timestamp" and "Last updated" on the Lintian index page. (#828720)
Misc
  • Allow the use of suppress-tags=<tag>[,<tag>[,<tag>]] in ~/.lintianrc. (#764486)
  • Improve the support for "3.0 (git)" packages. However, they remain marked as unsupported-source-format as they are not accepted by the Debian archive. (#605999)
  • Apply patch from Dylan Aïssi to also check for .RData files (not just .Rdata) when checking for the copyright status of R Project data files. (#868178, tag)
  • Match more Lena Söderberg images. (#827941, tag)
  • Refactor a hard-coded list of possible upstream key locations to the common/signing-key-filenames Lintian::Data resource.
Categories: LUG Community Blogs

Jonathan McDowell: Learning to love Ansible

Mon, 24/07/2017 - 18:41

This post attempts to chart my journey towards getting usefully started with Ansible to manage my system configurations. It’s a high level discussion of how I went about doing so and what I got out of it, rather than including any actual config snippets - there are plenty of great resources out there that handle the actual practicalities of getting started much better than I could.

I’ve been convinced about the merits of configuration management for machines for a while now; I remember conversations about producing an appropriate set of recipes to reproduce our haphazard development environment reliably over 4 years ago. That never really got dealt with before I left, and as managing systems hasn’t been part of my day job since then I never got around to doing more than working my way through the Puppet Learning VM. I do, however, continue to run a number of different Linux machines - a few VMs, a hosted dedicated server and a few physical machines at home and my parents’. In particular I have a VM which handles my parents’ email, and I thought that was a good candidate for trying to properly manage. It’s backed up, but it would be nice to be able to redeploy that setup easily if I wanted to move provider, or do hosting for other domains in their own VMs.

I picked Ansible, largely because I wanted something lightweight and the agentless design appealed to me. All I really need to do is ensure Python is on the host I want to manage and everything else I can bootstrap using Ansible itself. Plus it meant I could use the version from Debian testing on my laptop and not require backports on the stable machines I wanted to manage.

My first attempt was to write a single Ansible YAML file which did all the appropriate things for the email VM; installed Exim/Apache/Roundcube, created users, made sure the appropriate SSH keys were in place, installed configuration files, etc, etc. This did the job, but I found myself thinking it was no better than writing a shell script to do the same things.

Things got a lot better when instead of concentrating on a single host I looked at what commonality was shared between hosts. I started with simple things; Debian is my default distro so I created an Ansible role debian-system which configured up APT and ensured package updates were installed. Then I added a task to setup my own account and install my SSH keys. I was then able to deploy those 2 basic steps across a dozen different machine instances. At one point I got an ARM64 VM from Scaleway to play with, and it was great to be able to just add it to my Ansible hosts file and run the playbook against it to get my basic system setup.

Adding email configuration got trickier. In addition to my parents’ email VM I have my own email hosted elsewhere (along with a whole bunch of other users) and the needs of both systems are different. Sitting down and trying to manage both configurations sensibly forced me to do some rationalisation of the systems, pulling out the commonality and then templating the differences. Additionally I ended up using the lineinfile module to edit the Debian supplied configurations, rather than rolling out my own config files. This helped ensure more common components between systems. There were also a bunch of differences that had grown out of the fact each system was maintained by hand - I had about 4 copies of each Let’s Encrypt certificate rather than just putting one copy in /etc/ssl and pointing everything at that. They weren’t even in the same places on different systems. I unified these sorts of things as I came across them.

Throughout the process of this rationalisation I was able to easily test using containers. I wrote an Ansible role to create systemd-nspawn based containers, doing all of the LVM + debootstrap work required to produce a system which could then be managed by Ansible. I then pointed the same configuration as I was using for the email VM at this container, and could verify at each step along the way that the results were what I expected. It was still a little nerve-racking when I switched over the live email config to be managed by Ansible, but it went without a hitch as hoped.

I still have a lot more configuration to switch to being managed by Ansible, especially on the machines which handle a greater number of services, but it’s already proved extremely useful. To prepare for a jessie to stretch upgrade I fired up a stretch container and pointed the Ansible config at it. Most things just worked and the minor issues I was able to fix up in that instance leaving me confident that the live system could be upgraded smoothly. Or when I want to roll out a new SSH key I can just add it to the Ansible setup, and then kick off an update. No need to worry about whether I’ve updated it everywhere, or correctly removed the old one.

So I’m a convert; things were a bit more difficult by starting with existing machines that I didn’t want too much disruption on, but going forward I’ll be using Ansible to roll out any new machines or services I need, and expect that I’ll find that new deployment to be much easier now I have a firm grasp on the tools available.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Yay, finished my degree at last

Tue, 18/07/2017 - 22:56

A little while back, in June, I sat my last exam for what I hoped would be the last module in my degree. For seven years, I've been working on a degree with the Open University and have been taking advantage of the opportunity to have a somewhat self-directed course load by taking the 'Open' degree track. When asked why I bothered to do this, I guess my answer has been a little varied. In principle it's because I felt like I'd already done a year's worth of degree and didn't want it wasted, but it's also because I have been, in the dim and distant past, overlooked for jobs simply because I had no degree and thus was an easy "bin the CV".

Fed up with this, I decided to commit to the Open University and thus began my journey toward 'qualification' in 2010. I started by transferring the level 1 credits from my stint at UCL back in 1998/1999 which were in a combination of basic programming in Java, some mathematics including things like RSA, and some psychology and AI courses which at the time were aiming at a degree called 'Computer Science with Cognitive Sciences'.

Then I took level 2 courses, M263 (Building blocks of software), TA212 (The technology of music) and MS221 (Exploring mathematics). I really enjoyed the mathematics course and so...

At level 3 I took MT365 (Graphs, networks and design), M362 (Developing concurrent distributed systems), TM351 (Data management and analysis - which I ended up hating), and finally finishing this June with TM355 (Communications technology).

I received an email this evening telling me the module result for TM355 had been posted, and I logged in to find I had done well enough to be offered my degree. I could have claimed my degree 18+ months ago, but I persevered through another two courses in order to qualify for an honours degree which I have now been awarded. Since I don't particularly fancy any ceremonial awarding, I just went through the clicky clicky and accepted my qualification of 'Batchelor of Science (Honours) Open, Upper Second-class Honours (2.1)' which grants me the letters 'BSc (Hons) Open (Open)' which, knowing me, will likely never even make it onto my CV because I'm too lazy.

It has been a significant effort, over the course of the past few years, to complete a degree without giving up too much of my personal commitments. In addition to earning the degree, I have worked, for six of the seven years it has taken, for Codethink doing interesting work in and around Linux systems and Trustable software. I have designed and built Git server software which is in use in some universities, and many companies, along with a good few of my F/LOSS colleagues. And I've still managed to find time to attend plays, watch films, read an average of 2 novel-length stories a week (some of which were even real books), and be a member of the Manchester Hackspace.

Right now, I'm looking forward to a stress free couple of weeks, followed by an immense amount of fun at Debconf17 in Montréal!

Categories: LUG Community Blogs

Jonathan McDowell: Just because you can, doesn't mean you should

Mon, 17/07/2017 - 19:41

There was a recent Cryptoparty Belfast event that was aimed at a wider audience than usual; rather than concentrating on how to protect ones self on the internet the 3 speakers concentrated more on why you might want to. As seems to be the way these days I was asked to say a few words about the intersection of technology and the law. I think people were most interested in all the gadgets on show at the end, but I hope they got something out of my talk. It was a very high level overview of some of the issues around the Investigatory Powers Act - if you’re familiar with it then I’m not adding anything new here, just trying to provide some sort of details about why it’s a bad thing from both a technological and a legal perspective.

Download

Categories: LUG Community Blogs

Chris Lamb: Installation birthday

Fri, 14/07/2017 - 11:08

Fancy receiving congratulations on the anniversary of when you installed your system?

Installing the installation-birthday package on your Debian machines will celebrate each birthday of your system by automatically sending a message to the local system administrator.

The installation date is based on the system installation time, not the package itself.

installation-birthday is available in Debian 9 ("stretch") via the stretch-backports repository, as well as in the testing and unstable distributions:

$ apt install installation-birthday

Enjoy, and patches welcome. :)

Categories: LUG Community Blogs

Steve Engledow (stilvoid): An evening of linux on the desktop

Mon, 10/07/2017 - 23:14

Last time, I wrote about trying a few desktop environments to see what's out there, keep things fresh, and keep me from complacency. Well, as with desktop environments, so with text editors. I decided briefly that I would try a few of the more recent code editors that are around these days. Lured in by their pleasing, modern visuals and their promises of a smooth, integrated experience, I've been meaning to give these a go for a while. Needless to say, as a long-time vim user, I just found myself frustrated that I wasn't able to get things done as efficiently in any of those editors as I could in vim ;) I tried installing vim keybindings in Atom but it just wasn't the same as a very limited set of functionality was there. As for the integrated environment, when you have tmux running by default, everything's integrated anyway.

And, as with editors, so once again with desktop environments. I've decided to retract my previous hasty promise and no longer to bother with trying any other environments; i3 is more than fine :)

However, I did spend some time this evening making things a bit prettier so here are some delicious configs for posterity:

Configs Xresources

I've switched back to xterm from urxvt because, er... dunno.

Anyway, I set some nice colours for terminals and some magic stuff that makes man pages all colourful :)

XTerm*faceName: xft:Hack:regular:size=12 *termName: xterm-256color ! Colourful man pages *VT100.colorBDMode: true *VT100.colorBD: cyan *VT100.colorULMode: true *VT100.colorUL: darkcyan *VT100.colorITMode: true *VT100.colorIT: yellow *VT100.veryBoldColors: 518 ! terminal colours *foreground:#CCCCCC *background:#2B2D2E !black darkgray *color0: #2B2D2E *color8: #808080 !darkred red *color1: #FF0044 *color9: #F92672 !darkgreen green *color2: #82B414 *color10: #A6E22E !darkyellow yellow *color3: #FD971F *color11: #E6DB74 !darkblue blue *color4: #266C98 *color12: #7070F0 !darkmagenta magenta *color5: #AC0CB1 *color13: #D63AE1 !darkcyan cyan *color6: #AE81FF *color14: #66D9EF !gray white *color7: #CCCCCC *color15: #F8F8F2 Vimrc

Nothing exciting here except for discovering a few options I hadn't previous known about:

" Show a marker at the 80th column to encourage nice code set colorcolumn=80 highlight ColorColumn ctermbg=darkblue " Scroll the text when we're 3 lines from the top or bottom set so=3 " Use browser-style incremental search set incsearch " Override the default background colour in xoria256 to match the terminal background highlight Normal ctermbg=black " I like this theme colorscheme xoria256 i3

I made a few colour tweaks to my i3 config so I get colours that match my new Xresources. One day, I might see if it's easy enough to have them both read colour definitions from the same place so I don't have to define things twice.

The result

Here's what it looks like:

Categories: LUG Community Blogs

Jonathan McDowell: Going to DebConf 17

Mon, 10/07/2017 - 18:54

Completely forgot to mention this earlier in the year, but delighted to say that in just under 4 weeks I’ll be attending DebConf 17 in Montréal. Looking forward to seeing a bunch of fine folk there!

Outbound:

2017-08-04 11:40 DUB -> 13:40 KEF WW853 2017-08-04 15:25 KEF -> 17:00 YUL WW251

Inbound:

2017-08-12 19:50 YUL -> 05:00 KEF WW252 2017-08-13 06:20 KEF -> 09:50 DUB WW852

(Image created using GIMP, fonts-dkg-handwriting and the DebConf17 Artwork.)

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): Gitano - Approaching Release - Access Control Changes

Sat, 08/07/2017 - 10:31

As mentioned previously I am working toward getting Gitano into Stretch. A colleague and friend of mine (Richard Maw) did a large pile of work on Lace to support what we are calling sub-defines. These let us simplify Gitano's ACL files, particularly for individual projects.

In this posting, I'd like to cover what has changed with the access control support in Gitano, so if you've never used it then some of this may make little sense. Later on, I'll be looking at some better user documentation in conjunction with another friend of mine (Lars Wirzenius) who has promised to help produce a basic administration manual before Stretch is totally frozen.

Sub-defines

With a more modern lace (version 1.3 or later) there is a mechanism we are calling 'sub-defines'. Previously if you wanted to write a ruleset which said something like "Allow Steve to read my repository" you needed:

define is_steve user exact steve allow "Steve can read my repo" is_steve op_read

And, as you'd expect, if you also wanted to grant read access to Jeff then you'd need yet set of defines:

define is_jeff user exact jeff define is_steve user exact steve define readers anyof is_jeff is_steve allow "Steve and Jeff can read my repo" readers op_read

This, while flexible (and still entirely acceptable) is wordy for small rulesets and so we added sub-defines to create this syntax:

allow "Steve and Jeff can read my repo" op_read [anyof [user exact jeff] [user exact steve]]

Of course, this is generally neater for simpler rules, if you wanted to add another user then it might make sense to go for:

define readers anyof [user exact jeff] [user exact steve] [user exact susan] allow "My friends can read my repo" op_read readers

The nice thing about this sub-define syntax is that it's basically usable anywhere you'd use the name of a previously defined thing, they're compiled in much the same way, and Richard worked hard to get good error messages out from them just in case.

No more auto_user_XXX and auto_group_YYY

As a result of the above being implemented, the support Gitano previously grew for automatically defining users and groups has been removed. The approach we took was pretty inflexible and risked compilation errors if a user was deleted or renamed, and so the sub-define approach is much much better.

If you currently use auto_user_XXX or auto_group_YYY in your rulesets then your upgrade path isn't bumpless but it should be fairly simple:

  1. Upgrade your version of lace to 1.3
  2. Replace any auto_user_FOO with [user exact FOO] and similarly for any auto_group_BAR to [group exact BAR].
  3. You can now upgrade Gitano safely.
No more 'basic' matches

Since Gitano first gained support for ACLs using Lace, we had a mechanism called 'simple match' for basic inputs such as groups, usernames, repo names, ref names, etc. Simple matches looked like user FOO or group !BAR. The match syntax grew more and more arcane as we added Lua pattern support refs ~^refs/heads/${user}/. When we wanted to add proper PCRE regex support we added a syntax of the form: user pcre ^/.+?... where pcre could be any of: exact, prefix, suffix, pattern, or pcre. We had a complex set of rules for exactly what the sigils at the start of the match string might mean in what order, and it was getting unwieldy.

To simplify matters, none of the "backward compatibility" remains in Gitano. You instead MUST use the what how with match form. To make this slightly more natural to use, we have added a bunch of aliases: is for exact, starts and startswith for prefix, and ends and endswith for suffix. In addition, kind of match can be prefixed with a ! to invert it, and for natural looking rules not is an alias for !is.

This means that your rulesets MUST be updated to support the more explicit syntax before you update Gitano, or else nothing will compile. Fortunately this form has been supported for a long time, so you can do this in three steps.

  1. Update your gitano-admin.git global ruleset. For example, the old form of the defines used to contain define is_gitano_ref ref ~^refs/gitano/ which can trivially be replaced with: define is_gitano_ref ref prefix refs/gitano/
  2. Update any non-zero rulesets your projects might have.
  3. You can now safely update Gitano

If you want a reference for making those changes, you can look at the Gitano skeleton ruleset which can be found at https://git.gitano.org.uk/gitano.git/tree/skel/gitano-admin/rules/ or in /usr/share/gitano if Gitano is installed on your local system.

Next time, I'll likely talk about the deprecated commands which are no longer in Gitano, and how you'll need to adjust your automation to use the new commands.

Categories: LUG Community Blogs

Daniel Silverstone (Kinnison): F/LOSS activity, June 2017

Sat, 01/07/2017 - 13:59

It seems to be becoming popular to send a mail each month detailing your free software work for that month. I have been slowly ramping my F/LOSS activity back up, after years away where I worked on completing my degree. My final exam for that was in June 2017 and as such I am now in a position to try and get on with more F/LOSS work.

My focus, as you might expect, has been on Gitano which reached 1.0 in time for Stretch's release and which is now heading gently toward a 1.1 release which we have timed for Debconf 2017. My friend a colleague Richard has been working hard on Gitano and related components during this time too, and I hope that Debconf will be an opportunity for him to meet many of my Debian friends too. But enough of that, back to the F/LOSS.

We've been running Gitano developer days roughly monthly since March of 2017, and the June developer day was attended by myself, Richard Maw, and Richard Ipsum. You are invited to read the wiki page for the developer day if you want to know exactly what we got up to, but a summary of my involvement that day is:

  • I chaired the review of our current task state for the project
  • I chaired the decision on the 1.1 timeline.
  • I completed a code branch which adds rudimentary hook support to Gitano and submitted it for code review.
  • I began to learn about git-multimail since we have a need to add support for it to Gitano

Other than that, related to Gitano during June I:

  • Reviewed Richard Ipsum's lua-scrypt patches for salt generation
  • Reviewed Richard Maw's work on centralising Gitano's patterns into a module.
  • Responded to reviews of my hook work, though I need to clean it up some before it'll be merged.

My non-Gitano F/LOSS related work in June has been entirely centred around the support I provide to the Lua community in the form of the Lua mailing list and website. The host on which it's run is ailing, and I've had to spend time trying to improve and replace that.

Hopefully I'll have more to say next month. Perhaps by doing this reporting I'll get more F/LOSS done. Of course, July's report will be sent out while I'm in Montréal for debconf 2017 (or at least for debcamp at that point) so hopefully more to say anyway.

Categories: LUG Community Blogs

Chris Lamb: Free software activities in June 2017

Sat, 01/07/2017 - 00:29

Here is my monthly update covering what I have been doing in the free software world (previous month):

  • Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
    • Support Debian "buster". (commit)
    • Set TRAVIS=true environment variable when running autopkgtests. (#45)
  • Updated the documentation in django-slack, my library to easily post messages to the Slack group-messaging utility to link to Slack's own message formatting documentation. (#66)
  • Added "buster" support to local-debian-mirror, my package to easily maintain and customise a local Debian mirror via the DebConf configuration tool. (commit)
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source. Multiple third-parties then can come to a consensus on whether a build was compromised or not.

I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.

This month I:

  • Chaired our monthly IRC meeting. (Summary, logs, etc.)
  • Presented at Hong Kong Open Source Conference 2017.
  • Presented at LinuxCon China.
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
    • cracklib2: Ensuring /var/cache/cracklib/src-dicts are reproducible. (#865623)
    • fontconfig: Ensuring the cache files are reproducible. (#864082)
    • nfstrace: Make the PDF footers reproducible. (#865751)
  • Submitted 6 patches to fix specific reproducibility issues in cd-hit, janus, qmidinet, singularity-container, tigervnc & xabacus.
  • Submitted a wishlist request to the TeX mailing list to ensure that PDF files are reproducible even if generated from a difficult path after identifying underlying cause. (Thread)
  • Categorised a large number of packages and issues in the Reproducible Builds notes.git repository.
  • Worked on publishing our weekly reports. (#110, #111, #112 & #113)
  • Updated our website with 13 missing talks (e291180), updated the metadata for some existing talks (650a201) and added OpenEmbedded to the projects page (12dfcf0).

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.


strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Add libarchive-cpio-perl with the !nocheck build profile. (01e408e)
  • Add dpkg-dev dependency build profile. (f998bbe)


Debian

My activities as the current Debian Project Leader are covered in my "Bits from the DPL" email to the debian-devel-announce mailing list. However, I:

Debian LTS

This month I have been paid to work 16 hours hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 974-1 fixing a command injection vulnerability in picocom, a dumb-terminal emulation program.
  • Issued DLA 972-1 which patches a double-free vulnerability in the openldap LDAP server.
  • Issued DLA 976-1 which corrects a buffer over-read vulnerability in the yodl ("Your Own Document Language") document processor.
  • Issued DLA 985-1 to address a vulnerability in libsndfile (a library for reading/writing audio files) where a specially-crafted AIFF file could result in an out-of-bounds memory read.
  • Issued DLA 990-1 to fix an infinite loop vulnerability in the expat, an XML parsing library.
  • Issued DLA 999-1 for the openvpn VPN server — if clients used a HTTP proxy with NTLM authentication, a man-in-the-middle attacker could cause the client to crash or disclose stack memory that was likely to contain the proxy password.
Uploads
  • bfs (1.0.2-1) — New upstream release, add basic/smoke autopkgtests.
  • installation-birthday (5) — Add some basic autopkgtest smoke tests and correct the Vcs-{Git,Browser} headers.
  • python-django:
    • 1:1.11.2-1 — New upstream minor release & backport an upstream patch to prevent a test failure if the source is not writable. (#816435)
    • 1:1.11.2-2 — Upload to unstable, use !nocheck profile for build dependencies that are only required for tests and various packaging updates.

I also made the following non-maintainer uploads (NMUs):

  • kluppe (0.6.20-1.1) — Fix segmentation fault caused by passing a truncated pointer instead of a GtkType. (#863421)
  • porg (2:0.10-1.1) — Fix broken LD_PRELOAD path for libporg-log.so. (#863495)
  • ganeti-instance-debootstrap (0.16-2.1) — Fix "illegal option for fgrep" error by using "--" to escape the search needle. (#864025)
  • pavuk (0.9.35-6.1) — Fix segmentation fault when opening the "Limitations" window due to pointer truncation in src/gtkmulticol.[ch]. (#863492)
  • timemachine (0.3.3-2.1) — Fix two segmentation faults in src/gtkmeter.c and gtkmeterscale.c caused by passing a truncated pointers using guint instead of a GtkType. (#863420)
  • jackeq (0.5.9-2.1) — Fix another segmentation fault caused by passing a truncated pointer instead of a GtkType. (#863416)
Debian bugs filed
  • debhelper: Don't run dh_installdocs if nodoc is specified in DEB_BUILD_PROFILES? (#865869)
  • python-blessed: Non-determistically FTBFS due to unreliable timing in tests. (#864337)
  • apt: Please print a better error message if zero certificates are loaded from the system CA store. (#866377)
FTP Team

As a Debian FTP assistant I ACCEPTed 16 packages: faceup, golang-github-andybalholm-cascadia, haskell-miniutter, libplack-builder-conditionals-perl, libprelude, lua-argparse, network-manager-l2tp, node-gulp-concat, node-readable-stream, node-stream-assert, node-xterm, pydocstyle, pytest-bdd, python-iso3166, python-zxcvbn & stressant.

Categories: LUG Community Blogs

Steve Engledow (stilvoid): The day of linux on the desktop

Thu, 15/06/2017 - 03:41

It's been a while since I last tried out a different desktop environment on my laptop and I've been using i3 for some time now so it's only fair to give other things a go ;)

To test these out, I ran another X display - keeping my original one running so I could switch back and forth to take notes - and started each environment with DISPLAY=:1 <the command to start the desktop>.

I'll start with just one today and perhaps review some others another time.

Deepin

In summary: bits of Gnome Shell, Chrome OS, and Mac OSX but not quite as polished as any of them.

The Deepin Desktop Environment (DDE - from the Deepin distribution) installed easily enough under Arch with a quick pacman -S deepin deepin-extra. It also started up easily with an unambiguous startdde.

Immediately on startup, DDE plays a slightly annoying chime presumably just to remind you of how far we've come since Windows 95. The initial view of the desktop looks similar to OSX or Chrome OS with file icons on the desktop and a launcher bar centred across the bottom of the screen.

The first thing I tried was clicking on a button labelled "Multitasking view" only to be presented with a prompt telling me "Kindly reminder: This application can not run without window effect" and an OK button. So far, so enigmatic. So then I tried a trusty right-click on the desktop which brought up the expected context menu. In the menu was a "Display settings" option so I plumped for that, thinking that perhaps that was where I could enable the mystic "window effect". Clicking the "Display settings" button opened a dark-themed panel from the right-hand side, similar to the information panel you get in OSX. I searched through that panel for a good couple of minutes but could find no allusion to any "window effect".

Unperturbed, I decided to press on and see what other features Deepin had to offer...

Moving the mouse around the desktop a bit, I discovered that Deepin has borrowed some ideas from Gnome shell as well as OSX and Chrome OS. Moving the mouse pointer into the top-left corner of the screen brings up an application list similar to Gnome's launcher. The bottom-right corner reveals the settings panel. The top-right does nothing and the bottom-left, wonder of wonders, brings up my old favourite, the "kindly reminder".

I poked around in the settings a bit more but didn't really see anything of interest so I fired up what looks to be the last part of Deepin left for me to explore: the file manager. It does the job and it's not very interesting although I did discover that Deepin also has it's own terminal emulator (unsurprisingly called deepin-terminal) which has a snazzy Matrix theme to it but is otherwise uninteresting.

That's it, I'm bored. Next!

I tried Budgie and LXQT for a few minutes each at this point but they weren't immediately interesting enough to make me want to write about them just now :)

Categories: LUG Community Blogs

Mick Morgan: it is now

Tue, 06/06/2017 - 15:35

Back in January 2011, I posted a brief note about a site hosted at the domain “ismycomputeroff.com“. I have just had occasion to look again at that site and found that the domain is now definitely off. It is parked at sedo and is up for sale at the ludicrous price of 599 euros.

Tell you what, you can have my “theinternetisoff.net” domain for the bargain price of half that – after all, it only cost me about a tenner.

Categories: LUG Community Blogs