menno.ioHighly irregular postings about technology and lifeZola2024-02-27T00:00:00+00:00/atom.xmlPython Security For Developers2024-02-27T00:00:00+00:002024-02-27T00:00:00+00:00Unknown/posts/python-security/<p>I gave a talk <a href="https://www.meetup.com/nzpug-christchurch/events/299307048/">tonight</a>
at the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python meetup</a> titled
"Security For Python Developers". It touched on general security issues from a
Python perspective such as typo-squatting, and very Python specific issues such
as the <a href="https://github.com/trailofbits/fickling">dark side</a> of the
<a href="https://docs.python.org/3/library/pickle.html">pickle</a>.</p>
<p>The slides are available <a href="/presentations/python-security/index.html">here</a>.</p>
<p>Thanks to everyone who came along and for the interesting digressions during
and after the talk.</p>
Reducing Fan Noise on Intel NUCs2024-02-25T00:00:00+00:002024-02-25T00:00:00+00:00Unknown/posts/intel-nuc-fan-noise/<p>I have an Intel NUC that I've been using for monitoring and backups on my home
network. It lives in the corner of a living room and the noise of the fan has
been surprisingly and annoyingly loud, even when the machine is idle.</p>
<p>On other machines, I've had success using
<a href="https://wiki.archlinux.org/title/Fan_speed_control#Fancontrol_(lm-sensors)">fancontrol</a>
to control fan speeds using hardware temperature sensors on the CPU and motherboard so I tried
that first. It turns out the NUC has no software controllable fans so that was a non-starter.</p>
<img src="/images/pwmconfig-no.png" />
<p>After some research I found that the BIOS in the NUC is actually quite sophisticated and offers a <a href="https://www.intel.com/content/www/us/en/support/articles/000005946/intel-nuc.html">number of options to control cooling and BIOS</a>.</p>
<img src="/images/nuc-bios-cooling.png" />
<p>Here's what I ended up tweaking:</p>
<ul>
<li>Set the Fan Control Mode to "Custom" so that I could change the precise cooling settings myself.</li>
<li>Set the Minimum Duty Cycle to 20%. This reduced the idle fan speed noise considerably.</li>
<li>Turned off Turbo Boost so that the CPU doesn't get so hot. This host doesn't need to do anything CPU intensive so the loss of performance isn't an issue.</li>
<li>Reduced the maximum CPU speed down to 1.2GHz (was 1.8Ghz), again to reduce CPU heat and power usage.</li>
<li>Set the Power Management State to "Low Power". I'm not sure exactly what this does but given how this host is used, low power seems sensible.</li>
</ul>
<p>After saving these settings and rebooting the NUC the fan noise was almost inaudible yet temperatures were well within the acceptable range. These changes should hopefully shave a little off the power bill too.</p>
IMAPClient 3.0.0 is out2023-10-29T00:00:00+00:002023-10-29T00:00:00+00:00Unknown/posts/imapclient-300/<p><a href="https://github.com/mjs/imapclient">IMAPClient</a> <a href="https://pypi.org/project/IMAPClient/3.0.0/">3.0.0</a> is out!</p>
<p>The big news here is that support for Python 2 - and older Python 3 versions -
has been dropped. IMAPClient now support Python 3.7 and later. This allows for
the removal of various compatibility shims, expands the set of Python language
features that can be used and reduces testing load. </p>
<p>Dropping support for older Python versions has also opened up the possibility
for type hints to be added throughout the code base. This is an ongoing effort
although much has already been done.</p>
<p>With the focus on modernizing the project, there aren't many bug fixes or new
features in this release. Upcoming releases will include fixes for various bugs
around IDLE amongst other things.</p>
<p>Many thanks to everyone who contributed to this release, especially <a href="https://github.com/JohnVillalovos">John
Villalovos</a> for his efforts around
modernizing the code base and project infrastructure.</p>
<p>Note that release notes are no longer maintained in the <a href="https://imapclient.readthedocs.io/">project
documentation</a> and are instead
automatically generated as part of each <a href="https://github.com/mjs/imapclient/releases">GitHub
release</a>. This takes advantage of GitHub's
<a href="https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes">automatic release notes generation</a>
feature and uses <a href="https://github.com/mjs/imapclient/blob/master/.github/release.yml">PR labels to categorise changes</a>. </p>
<p>As always, IMAPClient can be installed using pip (<code>pip install imapclient</code>). Full documentation is available at <a href="https://imapclient.readthedocs.io/en/3.0.0/">Read the Docs</a>. Any bugs or feature requests should be files as <a href="https://github.com/mjs/imapclient/issues">GitHub Issues</a> and any questions can be asked using <a href="https://github.com/mjs/imapclient/discussions">GitHub Discussions</a>.</p>
pytest is Awesome2023-08-29T00:00:00+00:002023-08-29T00:00:00+00:00Unknown/posts/pytest-talk/<p>I gave a presentation today for the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a> that talks about why pytest
is so such a fantastic testing framework and how to approach common testing
tasks and problems with it. Some potential areas of confusion around pytest's
"magic" were raised by the audience which led to some interesting unprompted
discussions.</p>
<p>The slides are <a href="/presentations/pytest.pdf">available here</a>.</p>
All About (Python) Iteration2023-05-07T00:00:00+00:002023-05-07T00:00:00+00:00Unknown/posts/python-iteration/<p>I gave a talk this week for the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a> that went into how
iteration works in Python in some detail. Iterators, iterables, generator
functions, list expressions and the various language protocols involved with
iteration in Python were all covered. The <code>while</code> loop even gets brief mention.</p>
<p>The slides are <a href="/presentations/python-iteration/index.html">available here</a>.</p>
Storing & Exchanging Data with Python2023-02-08T00:00:00+00:002023-02-08T00:00:00+00:00Unknown/posts/python-data-formats-talk/<p>Mike Kittridge and I ran a talk last week for the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a> that covered a large
assortment of ways that Python programs can store data and exchange data with
other programs. We covered the ins and outs of binary files, text files,
buffering, text encoding, newline ending, mmap, pickle, JSON and friends, MessagePack,
Protobuf, tabular data formats, multi-dimensional data formats and key-value
stores. It was quite the tour!</p>
<p>The slides are available here:</p>
<ul>
<li><a href="https://docs.google.com/presentation/d/e/2PACX-1vRAG-Vm_dG2W_nAp7md1bvqoWSM3gAo148VEkKrgo4MIRNejYCSoj9dxFzkG90jb6mzirNvIot7p2fV/pub?start=false&loop=false&delayms=10000">online</a></li>
<li><a href="/presentations/python-data-formats.pdf">PDF</a></li>
</ul>
Faster software through register based calling2021-11-23T00:00:00+00:002021-11-23T00:00:00+00:00Unknown/posts/golang-register-calling/<p>The <a href="https://golang.org/doc/go1.17#compiler">release notes for Go 1.17</a> mention
an interesting change in the Go compiler:
<q>function arguments and return values will now be passed using registers
instead of the stack</q>. The
<a href="https://go.googlesource.com/proposal/+/refs/changes/78/248178/1/design/40724-register-calling.md">proposal document</a>
for the feature mentions an expected <q>5-10% throughput improvement across a
range of applications</q> which is significant, especially for no effort on the
developers part aside from recompiling with a newer version of the Go compiler.
I was curious to see what this actually looks like and decided to take a deeper
look. This will get more than a little nerdy so buckle up!</p>
<p>Note that although the catalyst for this blog article was a change in Go, much
of this article should be of interest generally even if you don't use Go.</p>
<h3 id="refresher-registers">Refresher: Registers</h3>
<p>At this point it's helpful to remind ourselves of what CPU registers are. In a
nutshell, they are small amounts of high speed temporary memory built into the
processor. Each register has a name and stores one
<a href="https://en.wikipedia.org/wiki/Word_(computer_architecture)">word</a> of data
each - this is 64 bits on almost all modern computers.</p>
<p>Some registers are general purpose while others have specific functions. In this
article you'll come across the AX, BX and CX general purpose registers, as well
as the SP (stack pointer) register which is special purpose.</p>
<h3 id="refresher-the-stack">Refresher: the Stack</h3>
<p>It's also useful to remind ourselves what the stack is in a computer program.
It's a chunk of memory that's placed at the top of a program's memory.
The stack is typically used store local variables, function arguments, function
return values and function return addresses. The stack grows downwards as items
are added to it.</p>
<p>Eli Bendersky has an <a href="https://eli.thegreenplace.net/2011/02/04/where-the-top-of-the-stack-is-on-x86">excellent article</a> about how stacks work which contains this helpful diagram:</p>
<img src="https://eli.thegreenplace.net/images/2011/02/stack1.png" style="width:301px; height: 314px; margin-left: auto; margin-right: auto; display: block">
<p>When items are added to the stack we say that they are <em>pushed</em> onto the stack.
When items are removed from the stack we say that they are <em>popped</em> off the
stack. There are x86 CPU instructions for pushing and popping data onto and off
the stack.</p>
<p>The SP register mentioned earlier points to the item currently at the top of
the stack.</p>
<p>Note that I'm taking some liberties here. The stack works like this on x86
computers (and many other CPU architectures) but not on all of them.</p>
<h3 id="calling-conventions">Calling Conventions</h3>
<p>In compiled software, when some code wants to call a function, the arguments
for that function need to somehow be passed to the function (and the return
values need to be passed back somehow when the function completes). There are
different agreed-upon ways to do this and each style of passing arguments and
return values around is a "calling convention". </p>
<p>The part of the Go 1.17 release notes quoted above is really about a change in
Go's calling conventions.</p>
<p>This is all hidden from you unless you're programming in assembler or are
trying to make bits of code written in different programming languages work
together. Even so, it's still interesting to see how the machinery works under
the hood. </p>
<h3 id="a-small-program">A Small Program</h3>
<p>In order to compare the code the Go compiler generates in 1.16 vs 1.17 we
need a simple test program. It doesn't have to do much, just call a function
that takes a couple of arguments which then returns a value. Here's the trivial
program I came up with:</p>
<pre data-lang="go" style="background-color:#2b303b;color:#c0c5ce;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#b48ead;">package </span><span style="color:#bf616a;">main
</span><span>
</span><span style="color:#b48ead;">import </span><span>"</span><span style="color:#a3be8c;">fmt</span><span>"
</span><span>
</span><span style="color:#b48ead;">func </span><span style="color:#8fa1b3;">add</span><span>(</span><span style="color:#bf616a;">i</span><span>, </span><span style="color:#bf616a;">j </span><span style="color:#b48ead;">int</span><span>) </span><span style="color:#b48ead;">int </span><span>{
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#bf616a;">i </span><span>+ </span><span style="color:#bf616a;">j
</span><span>}
</span><span>
</span><span style="color:#b48ead;">func </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> </span><span style="color:#bf616a;">z </span><span>:= </span><span style="color:#bf616a;">add</span><span>(</span><span style="color:#d08770;">22</span><span>, </span><span style="color:#d08770;">33</span><span>)
</span><span> </span><span style="color:#bf616a;">fmt</span><span>.</span><span style="color:#bf616a;">Println</span><span>(</span><span style="color:#bf616a;">z</span><span>)
</span><span>}
</span></code></pre>
<h3 id="disassembling">Disassembling</h3>
<p>In order to see the CPU instructions being generated by the Go compiler we need
a disassembler. One tool that can do this is venerable
<a href="https://linux.die.net/man/1/objdump">objdump</a> which comes with the GNU
binutils suite and may already be installed if you're running
Linux. I'll be using <code>objdump</code> in this article.</p>
<div class="callout">
The Go build pipeline is a little unusual in that it generates
a kind of bespoke abstract assembly language before converting this to actual
machine specific instructions. This intermediate assembly language can be seen
using the <code>go tool objdump</code> command. </p>
<p>It's tempting to use this output for our exploration here but this intermediate
assembly language <a href="https://www.mit.edu/afs.new/sipb/project/golang/doc/asm.html">isn't necessarily a direct
representation</a>
of the machine code that will be generated for a given platform. For this reason I've chosen
to stick with objdump.
</div>
<h3 id="go-1-16-s-output">Go 1.16's Output</h3>
<p>Let's take a look at the output from Go 1.16 which we expect to be using
stack based calling. First let's build the binary using Go 1.16 and make sure
it works:</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#bf616a;">$</span><span> go1.16.10 build</span><span style="color:#bf616a;"> -o</span><span> prog-116 ./main.go
</span><span style="color:#bf616a;">$</span><span> ./prog-116
</span><span style="color:#bf616a;">55
</span></code></pre>
<div class="callout">
My Linux distro already had Go 1.17.3 installed and I used the approach described in the <a href="https://golang.org/doc/manage-install#installing-multiple">official docs</a> for installing Go 1.16.10.
</div>
<p>Great! Now lets disassemble it to see the generated instructions:</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#bf616a;">$</span><span> objdump</span><span style="color:#bf616a;"> -d</span><span> prog-116 > prog-116.asm
</span></code></pre>
<p>The first thing I noticed is that there's quite of lot of code:</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#bf616a;">$</span><span> wc</span><span style="color:#bf616a;"> -l</span><span> prog-116.asm
</span><span style="color:#bf616a;">164670</span><span> prog-116.asm
</span></code></pre>
<p>That's a lot of instructions for such a small program but this is because every
Go program includes the Go runtime which is a non-trivial amount of software for
scheduling goroutines and providing all the conveniences we expect as Go developers.
Fortunately for us, the instructions directly relating to the code in our test
program are right at the bottom:</p>
<p>(I'm omitting the offsets and raw bytes that objdump normally provides for
clarity; also some of Go's setup code)</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>0000000000497640 <main.main>:
</span><span> ...
</span><span> movq $0x37,(%rsp)
</span><span> call 40a3e0 <runtime.convT64>
</span><span> mov 0x8(%rsp),%rax
</span><span> xorps %xmm0,%xmm0
</span><span> movups %xmm0,0x40(%rsp)
</span><span> lea 0xaa7e(%rip),%rcx # 4a2100 <type.*+0xa100>
</span><span> mov %rcx,0x40(%rsp)
</span><span> mov %rax,0x48(%rsp)
</span><span> mov 0xb345d(%rip),%rax # 54aaf0 <os.Stdout>
</span><span> lea 0x4290e(%rip),%rcx # 4d9fa8 <go.itab.*os.File,io.Writer>
</span><span> mov %rcx,(%rsp)
</span><span> mov %rax,0x8(%rsp)
</span><span> lea 0x40(%rsp),%rax
</span><span> mov %rax,0x10(%rsp)
</span><span> movq $0x1,0x18(%rsp)
</span><span>
</span><span> movq $0x1,0x20(%rsp)
</span><span>
</span><span> nop
</span><span> call 491140 <fmt.Fprintln>
</span><span> ...
</span></code></pre>
<p>Weird! This doesn't look like our code at all. Where's the call to our <code>add</code>
function? In fact, <code>movq $0x37,(%rsp)</code> (move the value 0x37 to memory
location pointed to by the stack pointer register) looks super suspicious. 22 +
33 = 55 which is 0x37 in hex. It looks like the Go compiler has optimised the
code, working out the addition at compile time, eliminating most of our code in
the process!</p>
<p>In order to study this further we need to tell the Go compiler to not
<a href="https://dave.cheney.net/tag/inlining">inline</a> the add function which can be
done using a special comment to annotate the add function. <code>add</code> now looks like this:</p>
<pre data-lang="go" style="background-color:#2b303b;color:#c0c5ce;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#65737e;">//go:noinline
</span><span style="color:#b48ead;">func </span><span style="color:#8fa1b3;">add</span><span>(</span><span style="color:#bf616a;">i</span><span>, </span><span style="color:#bf616a;">j </span><span style="color:#b48ead;">int</span><span>) </span><span style="color:#b48ead;">int </span><span>{
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#bf616a;">i </span><span>+ </span><span style="color:#bf616a;">j
</span><span>}
</span></code></pre>
<p>Compiling the code and running <code>objdump</code> again, the disassembly looks more as
we might expect. Let's start with <code>main()</code> - I've broken up the disassembly
into pieces and added commentary.</p>
<p>The main func begins with base pointer and stack pointer initialisation:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>0000000000497660 <main.main>:
</span><span> mov %fs:0xfffffffffffffff8,%rcx
</span><span>
</span><span> cmp 0x10(%rcx),%rsp
</span><span> jbe 497705 <main.main+0xa5>
</span><span> sub $0x58,%rsp
</span><span> mov %rbp,0x50(%rsp)
</span><span> lea 0x50(%rsp),%rbp
</span></code></pre>
<p>followed by,</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> movq $0x16,(%rsp)
</span><span> movq $0x21,0x8(%rsp)
</span></code></pre>
<p>Here we see the arguments to <code>add</code> being pushed onto the stack in preparation
for the function call. 0x16 (22) is moved to where the stack pointer is
pointing. 0x21 (33) is copied to 8 bytes after where the stack pointer is
pointing (so earlier in the stack). The offset of 8 is important because we're
dealing with 64-bit (8 byte) integers. An 8 byte offset means the 33 is placed
on the stack directly after the 22. </p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> call 4976e0 <main.add>
</span><span> mov 0x10(%rsp),%rax
</span><span> mov %rax,0x30(%rsp)
</span></code></pre>
<p>Here's where the add function actually gets called. When the <code>call</code> instruction
is executed by the CPU, the current value of the instruction pointer is pushed
to the stack and execution jumps to the <code>add</code> function. Once <code>add</code> returns,
execution continues here where <code>z</code> (stack pointer + 0x30 as it turns out) is
assigned to the returned value (stack pointer + 0x10). The AX register is used
as temporary storage when moving the return value.</p>
<p>There's more code that follows in main to handle the call the fmt.Println but
that's outside the scope of this article.</p>
<p>One thing I found interesting looking at this code is that the classic <code>push</code>
instructions aren't being used to add values onto the stack. Values are placed
onto the stack using <code>mov</code>. It turns out that this is for performance reasons.
A <code>mov</code> <a href="https://agner.org/optimize/instruction_tables.pdf">generally requires fewer CPU cycles</a> than a <code>push</code>.</p>
<p>We should also have a look at <code>add</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>0000000000497640 <main.add>:
</span><span> mov 0x10(%rsp),%rax
</span><span> mov 0x8(%rsp),%rcx
</span><span> add %rcx,%rax
</span><span> mov %rax,0x18(%rsp)
</span><span> ret
</span></code></pre>
<p>The second argument (at SP + 0x10) is copied to the AX register, and the first
argument (at SP + 0x08) is copied to the CX register. But hang on, weren't the
arguments at SP and SP+0x10? They were but when the <code>call</code> instruction was
executed, the instruction pointer was pushed to the stack which means the stack
pointer had to be decremented to make room for it - this means the offsets to
the arguments have to be adjusted to account for this.</p>
<p>The <code>add</code> instruction is easy enough to understand. Here CX and AX are added
(with the result left in AX). The result is then pushed to the return location
(SP + 0x18).</p>
<p>The <code>ret</code> (return) instruction grabs the return address off the stack and
starts execution just after the <code>call</code> in <code>main</code>.</p>
<p>Phew! That's a lot of code for a simple program. Although it's useful to
understand what's going on, be thankful that we don't have to write assembly
language very often these days!</p>
<h3 id="examining-go-1-17-s-output">Examining Go 1.17's Output</h3>
<p>Now let's take a look at the same program compiled with Go 1.17. The
compilation and disassembly steps are similar to Go 1.16:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ go build -o prog-117 ./main.go
</span><span>$ objdump -d prog-117 > prog-117.asm
</span></code></pre>
<p>The main disassembly starts the same as under Go 1.16 but - as expected -
differs in the call to <code>add</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> mov $0x16,%eax
</span><span> mov $0x21,%ebx
</span><span> xchg %ax,%ax
</span><span> call 47e260 <main.add>
</span></code></pre>
<p>Instead of copying the function arguments onto the stack, they're copied into
the AX and BX registers. This is the essence of register based calling. </p>
<p><s>The <code>xchg %ax,%ax</code> instruction is a bit more mysterious and I only have
theories regarding what it's for. Email me if you know and I'll add the detail
here.</s></p>
<p><a name="xchg-update"></a>
<strong>Update</strong>: The <code>xchg %ax,%ax</code> instruction is almost certainly there to work
around a <a href="https://www.intel.com/content/www/us/en/support/articles/000055650/processors.html">bug</a>
in a number of Intel processors. The instruction ("exchange AX
with AX") does nothing but introduces two bytes of padding before the <code>call</code>
instruction that follows - this serves to work around the processor bug. There's a
Go <a href="https://github.com/golang/go/issues/35881">GitHub issue</a> which has much more detail.
Thank you to the many people who wrote about this.</p>
<p>As we've already seen earlier, the <code>call</code> instruction moves execution to the
<code>add</code> function.</p>
<p>Now let's take a look at <code>add</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>000000000047e260 <main.add>:
</span><span> add %rbx,%rax
</span><span> ret
</span></code></pre>
<p>Well that's simple! Unlike the Go 1.16 version, there's no need to move
arguments from the stack into registers in order to add them, and there's no
need to move the result back to the stack. The function arguments are expected to
be in the AX and BX registers, and the return value is expected to come back
via AX. The <code>ret</code> instruction moves execution back to where <code>call</code> was executed,
using the return address that <code>call</code> left on the stack.</p>
<p>With so much less work being done when handling function arguments and return
values, it's starting to become clearer why register based calling might be
faster.</p>
<h3 id="performance-comparison">Performance Comparison</h3>
<p>So how much faster is register based calling? I created a simple Go benchmark program to check:</p>
<pre data-lang="go" style="background-color:#2b303b;color:#c0c5ce;" class="language-go "><code class="language-go" data-lang="go"><span style="color:#b48ead;">package </span><span style="color:#bf616a;">main
</span><span>
</span><span style="color:#b48ead;">import </span><span>"</span><span style="color:#a3be8c;">testing</span><span>"
</span><span>
</span><span style="color:#65737e;">//go:noinline
</span><span style="color:#b48ead;">func </span><span style="color:#8fa1b3;">add</span><span>(</span><span style="color:#bf616a;">i</span><span>, </span><span style="color:#bf616a;">j </span><span style="color:#b48ead;">int</span><span>) </span><span style="color:#b48ead;">int </span><span>{
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#bf616a;">i </span><span>+ </span><span style="color:#bf616a;">j
</span><span>}
</span><span>
</span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">result </span><span style="color:#b48ead;">int
</span><span>
</span><span style="color:#b48ead;">func </span><span style="color:#8fa1b3;">BenchmarkIt</span><span>(</span><span style="color:#bf616a;">b </span><span>*</span><span style="color:#bf616a;">testing</span><span>.</span><span style="color:#b48ead;">B</span><span>) {
</span><span> </span><span style="color:#bf616a;">x </span><span>:= </span><span style="color:#d08770;">22
</span><span> </span><span style="color:#bf616a;">y </span><span>:= </span><span style="color:#d08770;">33
</span><span> </span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">z </span><span style="color:#b48ead;">int
</span><span> </span><span style="color:#b48ead;">for </span><span style="color:#bf616a;">i </span><span>:= </span><span style="color:#d08770;">0</span><span>; </span><span style="color:#bf616a;">i </span><span>< </span><span style="color:#bf616a;">b</span><span>.</span><span style="color:#bf616a;">N</span><span>; </span><span style="color:#bf616a;">i</span><span>++ {
</span><span> </span><span style="color:#bf616a;">z </span><span>= </span><span style="color:#bf616a;">add</span><span>(</span><span style="color:#bf616a;">x</span><span>, </span><span style="color:#bf616a;">y</span><span>)
</span><span> }
</span><span> </span><span style="color:#bf616a;">result </span><span>= </span><span style="color:#bf616a;">z
</span><span>}
</span></code></pre>
<p>Note the use of a variable outside of the benchmark function to ensure that the
<a href="https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go">compiler won't optimise</a>
the assignment to <code>z</code> away.</p>
<p>The benchmark can be run like this:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>go test bench_test.go -bench=.
</span></code></pre>
<p>On my somewhat long in the tooth laptop, the best result I could get under Go 1.16 was:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>BenchmarkIt-4 512385438 2.292 ns/op
</span></code></pre>
<p>With Go 1.17:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>BenchmarkIt-4 613585278 1.915 ns/op
</span></code></pre>
<p>That's a noticeable improvement - a 16% reduction in execution time for our
example. Not bad, especially as the improvement comes for free for all Go
programs just by using a newer version of the compiler.</p>
<h3 id="conclusion">Conclusion</h3>
<p>I hope you found it interesting to explore some lower level details of software
that we don't think about much these days and that you learned something along
the way. </p>
<p>Many thanks to <a href="https://benhoyt.com/writings/">Ben Hoyt</a> and <a href="https://hardbyte.nz/">Brian
Thorne</a> for their detailed reviews and input into this
article.</p>
<p><strong>Update</strong>: This article ended up generating quite a bit of discussion elsewhere, in particular:</p>
<ul>
<li><a href="https://www.reddit.com/r/golang/comments/r0apqf/a_deep_dive_into_the_register_based_calling/">Reddit</a></li>
<li><a href="https://news.ycombinator.com/item?id=29316415">Hacker News</a></li>
</ul>
New Features in Python 3.8 and 3.92021-09-07T00:00:00+00:002021-09-07T00:00:00+00:00Unknown/posts/new-python-38-39/<p>I gave an online (COVID lockdown) presentation to the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
Meetup</a> tonight that covered a host
of <a href="https://docs.python.org/3/whatsnew/">new features</a> that were added to
Python in versions 3.8 and 3.9. While preparing for the presentation I was
blown away by how quickly Python is evolving (and I didn't even get to the new
<a href="https://www.python.org/dev/peps/pep-0636/">pattern matching features</a> coming in 3.10).</p>
<p>The slides are <a href="/presentations/new-python-features-3.8-3.9/index.html">available here</a>.</p>
IMAPClient 2.2.02021-01-16T00:00:00+00:002021-01-16T00:00:00+00:00Unknown/posts/imapclient-220/<p><a href="https://imapclient.freshfoo.com/">IMAPClient</a> <a href="https://pypi.org/project/IMAPClient/2.2.0/">2.2.0</a> is out!</p>
<p>The most critical change in this release is a fix to avoid an exception when
creating an IMAPClient instance under Python 3.9. imaplib (used by IMAPClient
internally) now supports connection timeouts and this conflicted with
IMAPClient's own timeout handling.</p>
<p>Other highlights for this release:</p>
<ul>
<li>Python 3.8 and 3.9 are now officially supported</li>
<li>Many performance improvments around string handling (thanks Carson!)</li>
<li>MULTIAPPEND and LITERAL+ support (thanks Devin Bayer)</li>
<li>SASL authentication improvements (thanks Mantas!)</li>
<li>Connection timeouts actually apply under Python 2.7 now</li>
<li>A fix around IMAP protocol parsing of numeric strings (thanks Jasper!) </li>
<li><a href="https://github.com/psf/black">Black</a> has been run over the entire codebase.
Black's way of formatting code is now the official way for the project.</li>
</ul>
<p>There are many more changes in this release. See the <a href="https://imapclient.readthedocs.io/en/2.2.0/releases.html#version-2-2-0">release notes</a> for more details. Thanks you so much to the <a href="https://imapclient.readthedocs.io/en/2.2.0/#authors">many contributors</a> to the project.</p>
<p>As always, IMAPClient can be installed using pip (<code>pip install imapclient</code>). Full documentation is available at <a href="https://imapclient.readthedocs.io/en/2.2.0/">Read the Docs</a>.</p>
Better Emacs Config: use-package2020-05-10T00:00:00+00:002020-05-10T00:00:00+00:00Unknown/posts/use-package/<p><a href="https://github.com/jwiegley/use-package">use-package</a> is an
<a href="https://en.wikipedia.org/wiki/Emacs">Emacs</a> package which allows
packages to be loaded declaratively. It's been around for ages and
I've seen it used in other people's configurations, but I've only
recently paid some real attention to it. I wish I'd learned how to use
it sooner - it's really improved my <a href="https://github.com/mjs/emacs.d">Emacs config</a>.</p>
<p>Let's look at an example:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(</span><span style="color:#96b5b4;">use-package</span><span> magit
</span><span> :bind (("</span><span style="color:#a3be8c;">C-x g</span><span>" . magit-status)
</span><span> ("</span><span style="color:#a3be8c;">C-x C-g</span><span>" . magit-status)))
</span></code></pre>
<p>When this is run, the use of the <a href="https://magit.vc/">Magit</a> package is declared and
two key-bindings for its main function are defined. What's really nice
here is that autoloads are installed for those key-bindings - the
package isn't actually loaded until I actually type either of those
shortcuts for the first time. This means Emacs can start faster and
resources aren't used for features I don't use often (that said, Magit is
something I do use <em>all</em> the time!).</p>
<p>Note use-package's clear, compact syntax. There's less ceremony for
common startup tasks such as setting up key-bindings and I love how
use-package encourages all setup related to a given package to be
neatly grouped together.</p>
<h4 id="configuring-packages">Configuring Packages</h4>
<p>It's common to need to run some code before or after a package is
loaded in order to set it up. With use-package this is done using
<code>:init</code> (before loading) and <code>:config</code> (after loading). Here's the
above example with some "helpful" messages inserted printed before and
after the magit package is loaded:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(</span><span style="color:#96b5b4;">use-package</span><span> magit
</span><span> :init
</span><span> (message "</span><span style="color:#a3be8c;">Loading Magit!</span><span>")
</span><span> :config
</span><span> (message "</span><span style="color:#a3be8c;">Loaded Magit!</span><span>")
</span><span> :bind (("</span><span style="color:#a3be8c;">C-x g</span><span>" . magit-status)
</span><span> ("</span><span style="color:#a3be8c;">C-x C-g</span><span>" . magit-status)))
</span></code></pre>
<p>Again, because loading of the package is deferred until one of the
key-bindings is used, the messages won't appear until I actually hit
one of those keys.</p>
<h4 id="deferred-loading">Deferred Loading</h4>
<p>Key-bindings are just one way that a deferred package might be loaded
by use-package. Other mechanisms include assigning a file extension to
a mode defined in the package (via <code>:mode</code>) or by adding a function
from the package into a hook (via <code>:hook</code>). use-package provides
a variety of syntactic sugar to make this painless and concise.</p>
<h4 id="immediate-loading">Immediate Loading</h4>
<p>Of course there are some packages which you always want to be loaded
immediately. use-package can handle this too through the <code>:demand</code>
keyword. Here's an example:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(</span><span style="color:#96b5b4;">use-package</span><span> evil
</span><span> :demand </span><span style="color:#d08770;">t
</span><span>
</span><span> :custom
</span><span> (evil-esc-delay </span><span style="color:#d08770;">0.001 </span><span>"</span><span style="color:#a3be8c;">avoid ESC/meta mixups</span><span>")
</span><span> (evil-shift-width </span><span style="color:#d08770;">4</span><span>)
</span><span> (evil-</span><span style="color:#96b5b4;">search</span><span>-module 'evil-</span><span style="color:#96b5b4;">search</span><span>)
</span><span>
</span><span> :bind (:</span><span style="color:#96b5b4;">map</span><span> evil-normal-state-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">S</span><span>" . </span><span style="color:#96b5b4;">replace</span><span>-symbol-at-point))
</span><span>
</span><span> :config
</span><span> </span><span style="color:#65737e;">;; Enable evil-mode in all buffers.
</span><span> (evil-mode </span><span style="color:#d08770;">1</span><span>))
</span></code></pre>
<p>Here we have the <a href="https://github.com/emacs-evil/evil">Evil</a> package
being loaded immediately (due to the <code>:demand t</code>) with some configuration
set before it's loaded (some customisations need to be set before
loading), a key-binding added, and evil-mode being enabled globally.</p>
<p>Note the use of the <code>:custom</code> keyword here. This is a clean way of
setting customisations that you could also set with Emacs' customize
functionality. It can be nice to keep customizations with the
use-package declaration, although I'm not that consistent about this
myself.</p>
<h4 id="key-bindings-in-keymaps">Key-bindings in Keymaps</h4>
<p>In the previous example, the key-binding is slightly different to
earlier examples because the binding is being set inside a specific
keymap instead of globally. use-package provides clean syntax for
this. Here's an example with multiple key-bindings being set up across
multiple keymaps:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(</span><span style="color:#96b5b4;">use-package</span><span> evil-args
</span><span> :bind (:</span><span style="color:#96b5b4;">map</span><span> evil-inner-text-objects-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">a</span><span>" . evil-inner-arg)
</span><span> :</span><span style="color:#96b5b4;">map</span><span> evil-outer-text-objects-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">a</span><span>" . evil-outer-arg)
</span><span>
</span><span> :</span><span style="color:#96b5b4;">map</span><span> evil-normal-state-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">L</span><span>" . evil-forward-arg)
</span><span> ("</span><span style="color:#a3be8c;">K</span><span>" . evil-jump-out-args)
</span><span>
</span><span> :</span><span style="color:#96b5b4;">map</span><span> evil-normal-state-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">H</span><span>" . evil-backward-arg)
</span><span> ("</span><span style="color:#a3be8c;">L</span><span>" . evil-forward-arg)
</span><span>
</span><span> :</span><span style="color:#96b5b4;">map</span><span> evil-motion-state-</span><span style="color:#96b5b4;">map
</span><span> ("</span><span style="color:#a3be8c;">H</span><span>" . evil-backward-arg)
</span><span> ("</span><span style="color:#a3be8c;">L</span><span>" . evil-forward-arg)))
</span></code></pre>
<p>Not bad! Much better than:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(define-key evil-inner-text-objects-</span><span style="color:#96b5b4;">map </span><span>(kbd "</span><span style="color:#a3be8c;">a</span><span>") 'evil-inner-arg)
</span><span>(define-key evil-outer-text-objects-</span><span style="color:#96b5b4;">map </span><span>(kbd "</span><span style="color:#a3be8c;">a</span><span>") 'evil-outer-arg)
</span><span>(define-key evil-normal-state-</span><span style="color:#96b5b4;">map </span><span>(kbd "</span><span style="color:#a3be8c;">L</span><span>") 'evil-forward-arg)
</span><span style="color:#65737e;">; you get the idea ...
</span></code></pre>
<h4 id="package-manager-integration">Package Manager Integration</h4>
<p>By default, use-package only loads packages that have already
installed somehow, but it can integrate with a package manager too.</p>
<p>If you're already using the built-in Emacs package manager
(<a href="https://www.emacswiki.org/emacs/InstallingPackages#installing-packages">package.el</a>)
then simply adding <code>:ensure t</code> to a <code>use-package</code> block will cause
use-package to download and install the package if it's not already
there. Extending our first example slightly:</p>
<pre data-lang="el" style="background-color:#2b303b;color:#c0c5ce;" class="language-el "><code class="language-el" data-lang="el"><span>(</span><span style="color:#96b5b4;">use-package</span><span> magit
</span><span> :ensure </span><span style="color:#d08770;">t
</span><span> :bind (("</span><span style="color:#a3be8c;">C-x g</span><span>" . magit-status)
</span><span> ("</span><span style="color:#a3be8c;">C-x C-g</span><span>" . magit-status)))
</span></code></pre>
<p>This avoids the need to separately call package.el's <code>install-package</code>
function or use the <code>list-packages</code> interface to install a package.</p>
<p>use-package can also work with other package managers. The powerful
<a href="https://github.com/raxod502/straight.el">straight.el</a> package
manager has tight integration with use-package (it's what I use now).</p>
<h4 id="what-next">What Next?</h4>
<p>If you want to learn more about use-package, the <a href="https://github.com/jwiegley/use-package/blob/master/README.md">official
README</a>
is approachable and comprehensive. There's plenty more to it than what
I've covered here, although you don't need to know much to start see
its benefits.</p>
<p>Other articles that you might also find helpful:</p>
<ul>
<li><a href="https://medium.com/@suvratapte/configuring-emacs-from-scratch-use-package-c30382297877">Configuring Emacs from Scratch — use-package</a> by Suvrat Apte.</li>
<li><a href="http://cachestocaches.com/2015/8/getting-started-use-package/">Getting Started With use-package</a> by Gregory J Stein.</li>
</ul>
<p><strong>Edit 2020-05-15</strong>: Use <code>:custom</code> instead of <code>setq</code> in <code>:init</code>. Thanks Canatella.</p>
My Python Toolbox2019-10-30T00:00:00+00:002019-10-30T00:00:00+00:00Unknown/posts/my-python-toolbox/<img src="/images/toolbox.jpg" style="max-width:250px; float:right">
<p>After a recent <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a>, I was asked to
create a list of Python libraries and tools that I tend to gravitate
towards when working on Python projects. The request was aimed at
helping newer Pythonistas find a way through the massive Python
ecosystem that exists today.</p>
<p>This is my attempt at such a list. It's highly subjective, being
coloured by my personal journey. I've done a lot of Python in my
career but that doesn't mean that I've necessarily picked the
best tool for each task. Still, I hope it's useful as a starting
point (I don't think there's any big surprises here).</p>
<h2 id="black">Black</h2>
<p>Many programming language communities are waking up to the fact that
having a tool which takes care of code formatting for you is a productivity booster
and avoids pointless arguments within teams. <a href="https://golang.org">Go</a> can take much of the credit for the
recent interest in code formatters with the <code>gofmt</code> tool that ships
with the standard Go toolchain. Rust has
<a href="https://github.com/rust-lang/rustfmt">rustfmt</a>, most JavaScript
projects seem to prefer <a href="https://prettier.io/">prettier</a> and an
automatic formatter seems to now be de rigueur for any new language.</p>
<p>A few formatters exist for Python but
<a href="https://github.com/psf/black#line-length">Black</a> seems to be fast
becoming the default choice, and rightly so. It makes sensible
formatting decisions (the way <a href="https://github.com/psf/black#line-length">it handles line
length</a> is particularly
smart) and has few configuration options so everyone's code ends up
looking the same across projects.</p>
<p>All Python projects should all be using Black!</p>
<h2 id="argparse">argparse</h2>
<p>Python has a number of options for processing command line
arguments but I prefer good old
<a href="https://docs.python.org/3/library/argparse.html">argparse</a> which has
been in the standard library since Python 3.2 (and 2.7). It has a
logical API and is capable enough for the needs of most programs.</p>
<h2 id="pytest">pytest</h2>
<p>The standard library has a perfectly fine <a href="https://en.wikipedia.org/wiki/XUnit">xUnit
style</a> testing package in the
form of <a href="https://docs.python.org/3/library/unittest.html">unittest</a>
but <a href="https://pytest.org/">pytest</a> requires less ceremony and is
just more fun. I really like the detailed failure output when tests
fail and the fixtures mechanism which provides a more powerful and
clearer way of reusing test common functionality than the classic
setup and teardown approach. It also encourages composition in tests
over inheritance.</p>
<p><code>pytest</code>'s extension mechanisms are great too. We have a handy <a href="https://github.com/TheCacophonyProject/cacophony-api/blob/177e49bac4311f19b19c5ee0a7b4b7e39187c500/test/conftest.py#L46">custom test report hook</a> for
for the API server tests at <a href="https://cacophony.org.nz/">The Cacophony Project</a> which includes
recent API server logs in the output when tests fail.</p>
<h2 id="csv">csv</h2>
<p>So much data ends up being available in
<a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> or
similarly formatted files and I've done my fair share of extracting
data out of them or producing CSV files for consumption by other
software. The <a href="https://docs.python.org/3/library/csv.html">csv</a>
package in the standard library is well designed and flexible
workhorse that deserves more praise.</p>
<h2 id="dates-and-times">Dates and times</h2>
<p>The standard
<a href="https://docs.python.org/3/library/datetime.html">datetime</a> package
from the standard library is excellent and ends up getting used in
almost every Python program I work on. It provides convenient ways to
represent and manipulate timestamps, time intervals and time zones. I
frequently pop open a Python shell just to do some quick ad hoc date
calculations.</p>
<p><code>datetime</code> intentionally doesn't try to get too involved with the
vagaries of time zones. If you need to represent timestamps in
specific timezones or convert between them, the
<a href="https://pypi.org/project/pytz/">pytz</a> package is your friend.</p>
<p>There are times where you need to do more complicated things with
timestamps and that's where
<a href="https://dateutil.readthedocs.io/">dateutil</a> comes in. It supports
generic date parsing, complex recurrence rules and relative delta
calculations (e.g. "what is next Monday?"). It also has a complete
timezone database built in so you don't need <code>pytz</code> if you're using
<code>dateutil</code>.</p>
<h2 id="plumbum">plumbum</h2>
<p>Shell scripts are great for what they are but there are also real
benefits to using a more rigorous programming language for the tasks
that shell scripts are typically used for, especially once a script
get beyond a certain size or complexity. <a href="/posts/never-write-shell-scripts-again/">One way
forward</a> is to use Python for
its expressiveness and cleanliness and the
<a href="https://plumbum.readthedocs.io/">plumbum</a> package to provide the
shell-like ease of running and chaining commands together that Python
lacks on it's own.</p>
<p>Here's a somewhat contrived example showing <code>plumbum</code>'s command chaining
capabilities combined with some Python to extract the first 5 lines:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">from </span><span>plumbum.cmd </span><span style="color:#b48ead;">import </span><span>find, grep, sort
</span><span>
</span><span>output = (find['</span><span style="color:#a3be8c;">-name</span><span>', '</span><span style="color:#a3be8c;">*.py</span><span>'] | grep['</span><span style="color:#a3be8c;">-v</span><span>', '</span><span style="color:#a3be8c;">python2.7</span><span>'] | sort)()
</span><span style="color:#b48ead;">for </span><span>line </span><span style="color:#b48ead;">in </span><span>output.</span><span style="color:#bf616a;">splitlines</span><span>()[:</span><span style="color:#d08770;">5</span><span>]
</span><span> </span><span style="color:#96b5b4;">print</span><span>(line)
</span></code></pre>
<p>In case you're wondering, the name is Latin for <em>lead</em>, which is what
pipes used to be made from (and also why we also have <em>plumbers</em>).</p>
<h2 id="attrs">attrs</h2>
<p>Python class creation with a lot less
boilerplate. <a href="https://attrs.readthedocs.io/">attrs</a> turns up all over
the place and with good reason - you end up with classes that require
fewer lines to define and behave correctly in terms of Python's
comparison operators.</p>
<p>Here's a quick example of some of the things that attrs gives you:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>>>> </span><span style="color:#b48ead;">import </span><span>attr
</span><span>>>> @attr.s
</span><span style="color:#d08770;">... </span><span style="background-color:#bf616a;color:#2b303b;">class</span><span> Point:
</span><span style="color:#d08770;">... </span><span>x = attr.</span><span style="color:#bf616a;">ib</span><span>(</span><span style="color:#bf616a;">default</span><span>=</span><span style="color:#d08770;">0</span><span>)
</span><span style="color:#d08770;">... </span><span>y = attr.</span><span style="color:#bf616a;">ib</span><span>(</span><span style="color:#bf616a;">default</span><span>=</span><span style="color:#d08770;">0</span><span>)
</span><span style="color:#d08770;">...
</span><span>
</span><span>>>> p0 = </span><span style="color:#bf616a;">Point</span><span>() </span><span style="color:#65737e;"># using default values
</span><span>>>> p1 = </span><span style="color:#bf616a;">Point</span><span>(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>) </span><span style="color:#65737e;"># specifying attribute values
</span><span>
</span><span style="color:#65737e;"># equality implemented by comparing attributes
</span><span>>>> p0 == p1
</span><span style="color:#d08770;">True
</span><span>>>> p2 = </span><span style="color:#bf616a;">Point</span><span>(</span><span style="color:#d08770;">3</span><span>, </span><span style="color:#d08770;">4</span><span>)
</span><span>>>> p0 == p2
</span><span style="color:#d08770;">False
</span><span>
</span><span>>>> </span><span style="color:#96b5b4;">repr</span><span>(p2) </span><span style="color:#65737e;"># nice repr values
</span><span>'</span><span style="color:#a3be8c;">Point(x=3, y=4)</span><span>'
</span></code></pre>
<p>There's a lot more to attrs than this example covers, and most default
behaviour is customisable.</p>
<p>It's worth nothing that <a href="https://docs.python.org/3.7/library/dataclasses.html">data classes</a> in
Python 3.7 and later offer some of the features of attrs, so you could
use those if you want to stick to the standard library. <code>attrs</code> offers a
richer feature set though.</p>
<h2 id="requests">requests</h2>
<p>If you're making HTTP 1.0/1.1 requests with Python then you should almost certainly be
using <a href="https://requests.kennethreitz.org/">requests</a>. It can do
everything you need and then some, and has a lovely API.</p>
<p>As far as <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP 2.0</a> goes, it seems that <a href="https://3.python-requests.org/">requests 3</a>
will have that covered, but it's a work in progress at time of writing.</p>
<h2 id="pew">pew</h2>
<p>Effective use of virtual environments is crucial for a happy Python
development experience. After <a href="https://menno.io/posts/virtualenvs-talk/">trying out a few
approaches</a> for managing
virtualenvs, I've settled on <a href="https://github.com/berdario/pew">pew</a> as
my preferred tool of choice. It feels clean and fits the way I work.</p>
<hr />
<p>That's what is in my Python toolbox. What's in yours? I'd love to hear
your thoughts in the comments.</p>
Python virtualenvs talk2019-08-08T00:00:00+00:002019-08-08T00:00:00+00:00Unknown/posts/virtualenvs-talk/<p>I had the pleasure of giving a talk about Python virtual environments
at this week's <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a>. It described the
problem that virtualenvs solve, some gotchas and the tools people use
to create and manage them. We also spent some time on some of the
newer entrants in this space including
<a href="https://github.com/berdario/pew">pew</a>,
<a href="https://docs.pipenv.org/">pipenv</a> and
<a href="https://poetry.eustace.io/">poetry</a>. <a href="/presentations/virtualenvs/">The slides</a> are available.</p>
<p>Giving presentations is a great way of solidifying your knowledge of a
particular subject - you want to make sure you're describing things
accurately so end up doing extra research and thinking more
deeply. I'm sure I get as much out of preparing for a talk as the
people who attend.</p>
<p>pew is rapidly becoming my preferred virtualenv tool (over
<a href="https://virtualenvwrapper.readthedocs.io/en/latest/">virtualenvwrapper</a>). It
feels a little cleaner to use and some nice touches.</p>
Introduction to attrs2018-11-25T00:00:00+00:002018-11-25T00:00:00+00:00Unknown/posts/python-attrs/<p>I've just published a <a href="https://jupyter.org/">Jupyter Notebook</a> I used to present an introduction
to the excellent Python <a href="https://attrs.readthedocs.io/">attrs</a> package at the November 2018
<a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python meetup</a>.</p>
<p>You can find it on <a href="https://github.com/chch-python-meetup/2018-11-attrs/blob/master/attrs.ipynb">Github</a>.</p>
influx-spout 2.12018-09-11T00:00:00+00:002018-09-11T00:00:00+00:00Unknown/posts/influx-spout-v210/<p><a href="https://github.com/jumptrading/influx-spout/">influx-spout</a> 2.1 has just been released and it includes a bunch of
exciting new features. Here's the highlights...</p>
<h4 id="downsampler">Downsampler</h4>
<p>The biggest addition is a new "<a href="https://github.com/jumptrading/influx-spout#downsampler">downsampler</a>" component which is
useful for creating rolled up versions of metrics streams for long
term archive. Generating rolled up versions of measurements in
real-time is more straightforward than trying to create them from
already stored measurements later, especially at large volumes. This
approach also eliminates the extra load on the short-term database
caused by batch rollup operations.</p>
<p>The downsampler reads measurements once they have been processed by
influx-spout's <a href="https://github.com/jumptrading/influx-spout#filter">filter</a> and averages them over a configured period. The
averaged measurements are emitted at the end of each sampling period
at clean time boundaries. For example if the sampling period is 10
minutes then measurements will be emitted at 0 minutes, 10 minutes, 20
minutes and so on past the hour.</p>
<p>Downsampled lines are emitted for consumption by an influx-spout
writer which will deliver them to an InfluxDB instance.</p>
<h4 id="new-tags-filter-rule-type">New "tags" filter rule type</h4>
<p>influx-spout's filter component now supports a new filter rule type
which allows matching against one or more tags in measurement
lines. Using this rule type is much faster than using regex rules to
achieve the same kinds of matches - 80-1000% faster depending on
number tags to match and line size. This approach is also much safer
as matching is independent of tag order and matches tag keys and
values precisely.</p>
<h4 id="tag-sorting">Tag sorting</h4>
<p>The filter component now orders tag keys in all measurement lines
passing through it. This ensures the best performance when
measurements are inserted into InfluxDB. Predictable tag ordering is
also required by the downsampler.</p>
<h4 id="filter-performance">Filter performance</h4>
<p>Various optimisations have been made in the filter to avoid
unnecessary processing and allocations around line unescaping. </p>
<p>The hashes of measurement names are now only computed once which
speeds up matching when multiple "basic" rules are in use.</p>
<h4 id="get-it">Get It!</h4>
<p>If you or your organisation is a power user of InfluxDB you should
really take a look at influx-spout. Version 2.1 is available via the
project's <a href="https://github.com/jumptrading/influx-spout/releases">Releases</a> page on Github.</p>
<p>Also, if you're into Go development and performance sensitive
software, influx-spout's <a href="https://github.com/jumptrading/influx-spout/">code</a> is worth studying.</p>
IMAPClient 2.1.02018-09-05T00:00:00+00:002018-09-05T00:00:00+00:00Unknown/posts/imapclient-210/<p><a href="http://imapclient.freshfoo.com">IMAPClient</a> 2.1.0 has just been
released! Here's the main highlights:</p>
<ul>
<li>Python 3.7 is now officially suppported</li>
<li>Testing against PyPy (version 2 and 3)</li>
<li>Added support for the <a href="https://tools.ietf.org/html/rfc2087">QUOTA extension</a> </li>
<li>Helper for <a href="https://imapclient.readthedocs.io/en/2.1.0/api.html#imapclient.IMAPClient.find_special_folder">locating special folders</a></li>
<li>Document usage for using self-signed TLS certificates</li>
<li>Document how to use the <code>email</code> package from the standard library to parse fetched emails</li>
<li>Handling NIL values for INTERNALDATE</li>
</ul>
<p>As always, IMAPClient can be installed using pip (<code>pip install imapclient</code>). Full documentation is available at <a href="https://imapclient.readthedocs.io/en/2.1.0/">Read the Docs</a>.</p>
<p>Enjoy!</p>
Introducing influx-spout2018-07-03T00:00:00+00:002018-07-03T00:00:00+00:00Unknown/posts/introducing-influx-spout/<p>As well as my <a href="https://cacophony.org.nz/">main gig</a>, I do some work with the excellent folks at
<a href="http://jumptrading.com/">Jump Trading</a>. My main focus there so far has been finalising
and open sourcing a project - implemented in <a href="https://golang.org/">Go</a> - called
<a href="https://github.com/jumptrading/influx-spout/">influx-spout</a>.</p>
<p>influx-spout will primarily be of interest to you if:</p>
<ul>
<li>you use the <a href="https://www.influxdata.com/time-series-platform/influxdb/">InfluxDB</a> time series database</li>
<li>you have a lot of data going into InfluxDB</li>
<li>you want flexibility in how incoming InfluxDB data is handled</li>
</ul>
<p>influx-spout sits in between an incoming fire-hose of InfluxDB
measurements (potentially from thousands of hosts) and one or more
InfluxDB instances (or other services which accept the <a href="https://docs.influxdata.com/influxdb/v1.5/write_protocols/line_protocol_reference/">InfluxDB Line
Protocol</a> such as <a href="https://www.influxdata.com/time-series-platform/kapacitor/">Kapacitor</a>). It accepts InfluxDB measurements over
either UDP or HTTP, applies sanity checks, batching, filtering and
routing, and then writes out the measurements to their final
destinations.</p>
<p>influx-spout provides flexibility: measurements can be sharded
across backends, some classes of measurements can duplicated to multiple
backends and measurements which are no longer important can be dropped
before they get near a disk.</p>
<p>influx-spout also allows for easy changes in the way measurements are
handled without requiring changes to the systems producing the
measurements. External systems are configured to send measurements to
single, static endpoint (i.e. influx-spout) with any changes to the
way measurements are handled taken care of by changes to
influx-spout's configuration.</p>
<p>As a Go developer, influx-spout is interesting because it of the high
volumes of data that it needs to support. Here's a few things we've
done to ensure that influx-spout can handle large data volumes:</p>
<ul>
<li>The data path is entirely in RAM: no disk is involved.</li>
<li>Scale out: the various functions of influx-spout are implemented as
separate processes which can run on a single machine or scaled out
multiple machines.</li>
<li>Custom algorithms for parsing and data conversion where required.</li>
<li>Minimal memory allocation: wherever possible buffers are allocated
once and reused.</li>
<li>Minimal copying: wherever possible data is read into a buffer once
and used from there.</li>
<li>Performance regression testing: there are automated benchmarks which
compare the performance of key areas of the code against an earlier
reference revision. The checks are run for every pull request to
detect regressions.</li>
</ul>
<p>If you're into Go development and performance sensitive software, the
<a href="https://github.com/jumptrading/influx-spout/">influx-spout code</a> is worth studying.</p>
<p>influx-spout v2.0.0 has just been released can downloaded from the
project's <a href="https://github.com/jumptrading/influx-spout/releases">Releases</a> page. There's lots more information about the
project in the <a href="https://jumptrading.github.io/influxdb/metrics/monitoring/devops/production/engineering/2018/03/09/introducing-influx-spout.html">project launch post</a> and in the <a href="https://github.com/jumptrading/influx-spout/blob/master/README.md">README</a>.</p>
Listing S3 objects with NodeJS2018-06-26T00:00:00+00:002018-06-26T00:00:00+00:00Unknown/posts/listing-s3-objects-with-nodejs/<p>I recently had to write some NodeJS code which uses the <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/">AWS SDK</a> to
list all the objects in a <a href="https://en.wikipedia.org/wiki/Amazon_S3">S3</a> bucket which potentially contains many
objects (currently over 80,000 in production). The S3 <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property">listObjects</a>
API will only return up to 1,000 keys at a time so you have to make
multiple calls, setting the <code>Marker</code> field to page through all the
keys.</p>
<p>It turns out there's a lot of sub-optimal examples out there for how
to do this which often involve global state and complicated recursive
callbacks. I'm also a fan of the clarity of JavaScript's newer
async/await feature for handling asynchronous code so I was keen on a
solution which uses that style.</p>
<p>Here's what I came up with:</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span>
</span><span style="color:#b48ead;">async function </span><span style="color:#8fa1b3;">allBucketKeys</span><span>(</span><span style="color:#bf616a;">s3</span><span>, </span><span style="color:#bf616a;">bucket</span><span>) {
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#bf616a;">params </span><span>= {
</span><span> Bucket: </span><span style="color:#bf616a;">bucket</span><span>,
</span><span> };
</span><span>
</span><span> </span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">keys </span><span>= [];
</span><span> </span><span style="color:#b48ead;">for </span><span>(;;) {
</span><span> </span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">data </span><span>= </span><span style="color:#b48ead;">await </span><span style="color:#bf616a;">s3</span><span>.</span><span style="color:#8fa1b3;">listObjects</span><span>(</span><span style="color:#bf616a;">params</span><span>).</span><span style="color:#8fa1b3;">promise</span><span>();
</span><span>
</span><span> </span><span style="color:#bf616a;">data</span><span>.</span><span style="color:#bf616a;">Contents</span><span>.</span><span style="color:#96b5b4;">forEach</span><span>((</span><span style="color:#bf616a;">elem</span><span>) </span><span style="color:#b48ead;">=> </span><span>{
</span><span> </span><span style="color:#bf616a;">keys </span><span>= </span><span style="color:#bf616a;">keys</span><span>.</span><span style="color:#96b5b4;">concat</span><span>(</span><span style="color:#bf616a;">elem</span><span>.</span><span style="color:#bf616a;">Key</span><span>);
</span><span> });
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>(!</span><span style="color:#bf616a;">data</span><span>.</span><span style="color:#bf616a;">IsTruncated</span><span>) {
</span><span> </span><span style="color:#b48ead;">break</span><span>;
</span><span> }
</span><span> </span><span style="color:#bf616a;">params</span><span>.</span><span style="color:#bf616a;">Marker </span><span>= </span><span style="color:#bf616a;">data</span><span>.</span><span style="color:#bf616a;">NextMarker</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#bf616a;">keys</span><span>;
</span><span>}
</span></code></pre>
<p>It's called like this:</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span>
</span><span style="color:#65737e;">// Remember to catch exceptions somewhere...
</span><span style="color:#b48ead;">const </span><span style="color:#bf616a;">s3 </span><span>= </span><span style="color:#8fa1b3;">connectToS3Somehow</span><span>();
</span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">keys </span><span>= </span><span style="color:#b48ead;">await </span><span style="color:#8fa1b3;">allBucketKeys</span><span>(</span><span style="color:#bf616a;">s3</span><span>, "</span><span style="color:#a3be8c;">my_bucket</span><span>");
</span></code></pre>
<p>This solution is clean, concise and hopefully straightforward.</p>
<p>An important aspect that supports this solution is that the AWS API
can return a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promise</a> for a call (via <code>.promise()</code>) which can then be
used with <code>await</code>. Given the need to conditionally call <code>listObjects</code>
multiple times, an arguably clearer code structure can be achieved using
await instead of callbacks.</p>
Never Write Shell Scripts Again2018-05-14T00:00:00+00:002018-05-14T00:00:00+00:00Unknown/posts/never-write-shell-scripts-again/<p>I recently gave a talk at the <a href="https://www.meetup.com/NZPUG-Christchurch/">Christchurch Python
meetup</a> which discussed the
strengths and weaknesses of shell scripts, and why you might want to
consider using Python instead of shell scripts. I went into some areas
where Python is arguably worse than shell scripts, and then we dived
into the excellent <a href="https://pypi.org/project/plumbum/">Plumbum</a> package
which nicely addresses most of those weaknesses.</p>
<p>The slides for the talk are <a href="http://menno.io/presentations/never-write-shell-scripts-again/">available
here</a>.</p>
IMAPClient 2.0.02018-01-13T00:00:00+00:002018-01-13T00:00:00+00:00Unknown/posts/imapclient-200/<p>I'm very happy to announce that IMAPClient
<a href="https://pypi.python.org/pypi/IMAPClient/2.0.0">2.0.0</a> is out. Many
thanks to all the contributors who helped to shape what is one of the
biggest IMAPClent releases to date.</p>
<p>A major focus of this release was removing the dependency on
backports.ssl and pyOpenSSL for
<a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> support.
These dependencies were introduced in IMAPClient 1.0 to provide
consistent TLS support, even for Python versions which didn't have good
support in the standard library. Unfortunately they caused installation
headaches for many people and had various rough edges.</p>
<p>Now that Python has solid built-in TLS support in readily available
Python versions (2.7.9+ and 3.4+), it was time for IMAPClient to go back
to using the standard library <code>ssl</code> package. This should make most
IMAPClient installation problems are thing of the past, and also
significantly reduces the dependences that get pulled in when IMAPClient
is installed.</p>
<p>Other highlights in this release:</p>
<ul>
<li>Python 2.6 and 3.3 are no longer supported. This version supports
Python 2.7, 3.4, 3.5 and 3.6.</li>
<li>Support for the MOVE extension (<a href="https://tools.ietf.org/html/rfc6851">RFC
6851</a>)</li>
<li>More precise execeptions are used (see
<a href="https://github.com/mjs/imapclient/blob/2.x/imapclient/exceptions.py">imapclient.exceptions</a>)</li>
<li>Logs are now handled by the Python logging module. The <code>debug</code> and
<code>log_file</code> attributes are gone.</li>
<li>GMail labels using international characters are now handled
properly.</li>
<li>All non-library code moved out of the <code>imapclient</code> package.</li>
<li>Connection and read/write operations timeouts can now be distinct.</li>
<li>New <code>welcome</code> property to allow access to IMAP server greeting.</li>
<li>Significant
<a href="https://github.com/mjs/imapclient/blob/2.x/README.rst">README</a> and
<a href="http://imapclient.readthedocs.io/en/2.x/">documentation</a>
improvements.</li>
<li>Many bug fixes.</li>
</ul>
<p>See the <a href="https://imapclient.readthedocs.io/en/2.x/releases.html">release
notes</a> for more
details.</p>
IMAPClient 1.1.02017-11-23T00:00:00+00:002017-11-23T00:00:00+00:00Unknown/posts/imapclient-110/<p>IMAPClient <a href="https://pypi.python.org/pypi/IMAPClient/1.1.0">1.1.0</a> has
just been released! Many thanks to the recent contributors and the
project's other maintainers, Nicolas Le Manchet and Maxime Lorant. This
release is full of great stuff because of them. Here's some highlights:</p>
<ul>
<li><a href="https://imapclient.readthedocs.io/en/1.x/#imapclient.IMAPClient.search">search</a>
now supports nested criteria so that more complex criteria can be
expressed. IMAPClient will add parentheses in the right place. See</li>
<li><a href="https://imapclient.readthedocs.io/en/1.x/#imapclient.IMAPClient.plain_login">PLAIN
authentication</a>
support</li>
<li><a href="https://imapclient.readthedocs.io/en/1.x/#imapclient.IMAPClient.unselect_folder">UNSELECT</a>
support</li>
<li><a href="https://imapclient.readthedocs.io/en/1.x/#imapclient.IMAPClient.enable">ENABLE</a>
support</li>
<li><a href="https://imapclient.readthedocs.io/en/1.x/#imapclient.IMAPClient.expunge">UID
EXPUNGE</a>
support</li>
<li>IMAP modified UTF-7 encoding/decoding now works correctly in all
cases</li>
<li>the mock package is no longer installed by default (just as a test
dependency)</li>
<li>many, many bug fixes</li>
</ul>
<p>See the <a href="https://imapclient.readthedocs.io/en/1.x/releases.html">release
notes</a> for more
details.</p>
<p>Much work has already gone into the 2.0 release and it won't be too far
behind 1.1.0. The headline change there is reworking IMAPClient's
handling of TLS.</p>
UC Data Science talk2017-10-27T00:00:00+00:002017-10-27T00:00:00+00:00Unknown/posts/uc-data-science-talk/<p>This week I had the pleasure of giving a talk to Data Science masters
students at the University of Canterbury (NZ). It was my attempt at
giving some real-world advice from many years working as a technology
geek. I think it went well and I hope the audience got something useful
out of it.</p>
<p>The <a href="/presentations/uc-data-science/index.html">slides</a> are now available.</p>
<p>I also managed to promote my <a href="https://cacophony.org.nz/">current gig</a> as
well. Expect an article about what I've been up to soon.</p>
Showing all Go compiler errors2017-04-07T00:00:00+00:002017-04-07T00:00:00+00:00Unknown/posts/showing-all-go-compiler-errors/<p>The Go compiler normally stops after it finds 10 errors, aborting with a
<code>too many errors</code> message. For example:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ go build
</span><span># sandbox/manyerrs
</span><span>./errs.go:4: undefined: w
</span><span>./errs.go:5: undefined: w
</span><span>./errs.go:6: undefined: w
</span><span>./errs.go:7: undefined: w
</span><span>./errs.go:8: undefined: w
</span><span>./errs.go:9: undefined: w
</span><span>./errs.go:10: undefined: w
</span><span>./errs.go:11: undefined: w
</span><span>./errs.go:12: undefined: w
</span><span>./errs.go:13: undefined: w
</span><span>./errs.go:13: too many errors
</span></code></pre>
<p>This is useful default behaviour - if there's lots problems you usually
don't care about seeing all of the issues. You just fix what you can see
and try again.</p>
<p>Sometimes though you really want to see all the errors.</p>
<p>The trick is to use the <code>-gcflags</code> option to the <code>go</code> tool to pass <code>-e</code>
(show all errors) to the compiler.</p>
<p>Here's how you do it:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ go build -gcflags="-e"
</span><span>./errs.go:4: undefined: w
</span><span>./errs.go:5: undefined: w
</span><span>./errs.go:6: undefined: w
</span><span>./errs.go:7: undefined: w
</span><span>./errs.go:8: undefined: w
</span><span>./errs.go:9: undefined: w
</span><span>./errs.go:10: undefined: w
</span><span>./errs.go:11: undefined: w
</span><span>./errs.go:12: undefined: w
</span><span>./errs.go:13: undefined: w
</span><span>./errs.go:14: undefined: w
</span><span>./errs.go:15: undefined: w
</span><span>./errs.go:16: undefined: w
</span><span>./errs.go:17: undefined: w
</span></code></pre>
<p>Given that this was surprisingly difficult to find this I thought I'd
write it down here. Hope this was useful.</p>
Python async talk2017-04-05T00:00:00+00:002017-04-05T00:00:00+00:00Unknown/posts/python-async-talk/<p>I gave a talk at last night's <a href="https://www.meetup.com/NZPUG-Christchurch/events/238573201/">Christchurch Python
meetup</a>
about Python's relatively new asynchronous programming features. To be
honest, I didn't know all that much about the topic and signed myself up
for the talk to force myself to learn :)</p>
<p>I used <a href="http://jupyter.org/">Jupyter Notebook</a> for the presentation. Its
ability to mix together text and interactive Python snippets works
really well for this kind of talk. I've published <a href="https://github.com/mjs/python-async-talk/blob/master/async.ipynb">the
slides</a>
on Github as it has native support for rendering Jupyter notebooks
(thanks Github!).</p>
IMAPClient has moved to Github2016-08-29T00:00:00+00:002016-08-29T00:00:00+00:00Unknown/posts/imapclient-has-moved-to-github/<p>The code has been imported, tickets have migrated and the Bitbucket
repository is now a redirect: the IMAPClient project is now using Git
and Github.</p>
<p>The Github link is: <a href="https://github.com/mjs/imapclient">https://github.com/mjs/imapclient</a></p>
<p>(<a href="http://imapclient.freshfoo.com">http://imapclient.freshfoo.com</a> redirects to the new location)</p>
<p>This change has been planned for a long time. I've noticed a that many
contributors struggle with Mercurial and Bitbucket, and I greatly prefer
Git myself nowadays.</p>
<p>Please <a href="https://groups.io/g/imapclient">email the mailing list</a> or <a href="mailto:menno@freshfoo.com">me
personally</a> if you see any problems with
migration or have any questions.</p>
IMAPClient 1.0.22016-08-22T00:00:00+00:002016-08-22T00:00:00+00:00Unknown/posts/imapclient-102/<p>IMAPClient 1.0.2 is out! This is release comes with a few small fixes
and tweaks, as well as a some documentation improvements.</p>
<p>Specifically:</p>
<ul>
<li>There's now an explicit check that the pyOpenSSL version that
IMAPClient is seeing is sufficient. This is to help with situations
(typically on OS X) where the (old) system pyOpenSSL takes
precedence over the version that IMAPClient needs. Use of
virtualenvs is highly recommended.</li>
<li>Python 3.5 is now officially supported and tested against.</li>
<li>setup.py can now be used even if it's not in the current directory.</li>
<li>Handling of RFC2822 group address syntax has been
<a href="http://imapclient.readthedocs.io/en/stable/#imapclient.response_types.Envelope">documented</a>.</li>
<li>The <a href="http://imapclient.readthedocs.io/en/stable/#configuration-file-format">INI file
format</a>
used by the live tests and interactive shell has finally been
documented.</li>
<li>Links to ReadTheDocs now go to readthedocs.io</li>
<li>The project README has been arranged so that all the essentials are
right at the top.</li>
</ul>
<p>I announced that the project would be moving to Git and Github some time
ago and this is finally happening. This release will be the last release
where the project is on Bitbucket.</p>
Nikola2016-07-10T00:00:00+00:002016-07-10T00:00:00+00:00Unknown/posts/nikola/<p>Those paying attention may have noticed that <a href="http://freshfoo.com/">this
site</a> has changed. Up until recently, freshfoo.com
was generated using a combination of
<a href="https://pyblosxom.github.io/">PyBlosxom</a> and
<a href="http://www.voidspace.org.uk/python/rest2web/">rest2web</a> but it's now
using the wonderful <a href="https://getnikola.com/">Nikola</a> static site
generator. The change was prompted by a desire to have more flexibility
about freshfoo.com's hosting situation - moving a static site around is
far easier than a dynamically generated one. A static site also provides
a snappier experience for the end user and there really wasn't a need
for the site to be dynamically generated anyway.</p>
<p>Nikola also provides some nice tooling for the site author to help with
the writing process. The <a href="https://getnikola.com/features/auto.html">nikola
auto</a> command which serves up
the site locally, automatically rebuilding as you work on it and
automatically refreshing browser tabs which are viewing the site is
especially useful.</p>
<p>The pre-existing articles for the site were written using a combinartion
of raw HTML (back from the <a href="http://blogger.com/">Blogger</a> days!) and
<a href="https://en.wikipedia.org/wiki/ReStructuredText">reStructuredText</a> which
are both supported by Nikola (as well as
<a href="https://daringfireball.net/projects/markdown/">Markdown</a> and more), so
converting them for use by Nikola required only minor massaging. The URL
scheme for posts and articles has changed a little but there are
permanent redirects in place to map the old URLs to their new locations.
I think the new URL scheme is somewhat cleaner anyway.</p>
<p>The <a href="http://disqus.com/">Disqus</a> comments for historic articles have
been preserved despite the URL changes for articles. Disqus provides a
handy <a href="https://help.disqus.com/customer/portal/articles/912757-url-mapper">URL
Mapper</a>
tool to support these situations.</p>
<p>I'm really happy with the result but it's possible that I might have
missed something. Please leave a comment below if you see something that
isn't right.</p>
IMAPClient mailing list moved to Groups.io2016-06-07T00:00:00+00:002016-06-07T00:00:00+00:00Unknown/posts/imapclient-groupsio/<p>The IMAPClient mailing list has been hosted at
<a href="http://librelist.com/">Librelist</a> for a while. It worked OK initially
but the site doesn't seem to be getting much love these days. People
also tend to find the "send an email to subscribe" model confusing.</p>
<p>After looking at a number of options, I've decided to shift the mailing
list to <a href="http://groups.io">Groups.io</a>. It's a well run mailing list
service which allows interaction via both the web and email. There's
lots of nice bells and whistles but the main benefit I see is that it
seems to be well thought out and efficient.</p>
<p>To join the new list or to find out more, take a look at
<a href="https://groups.io/g/imapclient">https://groups.io/g/imapclient</a>.</p>
<p>The change to the new mailing list is effective immediately. Please use
the new one from now on. I'm going to send out invites to people who
were active on the old list. IMAPClient's documentation has already been
updated to reflect the change.</p>
<p>To ensure they aren't lost, the historical messages from the old list on
librelist.com will be imported into new one.</p>
<p>Please <a href="mailto:menno@freshfoo.com">email me</a> or comment here if you have
any feedback about the change.</p>
IMAPClient 1.0.02015-11-24T00:00:00+00:002015-11-24T00:00:00+00:00Unknown/posts/imapclient-1.0.0/<p>IMAPClient 1.0.0 is finally here! This is a monster release, bursting
with new features and fixes.</p>
<p>Here's the highlights:</p>
<p><strong>Enhanced TLS support</strong>: The way that IMAPClient establishes
<a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a>
connections has been completely reworked. By default, IMAPClient will
attempt certificate verification and certificate hostname checking, and
will not use known-insecure TLS settings and protocols. In addition, TLS
parameters are now highly configurable.</p>
<p>This change necessitates that backwards compatibility has been broken,
and also means that IMAPClient has a bunch of new dependencies. Please
see the earlier blog article about the <a href="http://freshfoo.com/blog/imapclient-1.0-tls">TLS
changes</a> as well as the
<a href="http://imapclient.readthedocs.org/en/stable/releases.html#version-1-0-0">release
notes</a>
for more information.</p>
<p><strong>STARTTLS support</strong>: When the server supports it, IMAPClient can now
establish an encrypted connection after initially starting with an
unencrypted connection using the STARTTLS command. The starttls method
takes an SSL context object for controlling the parameters of the TLS
negotiation.</p>
<p>Many thanks to Chris Arndt for his extensive initial work on this.</p>
<p><strong>Robust search criteria handling</strong>: IMAPClient's methods that accept
search criteria have been changed to provide take criteria in a more
straightforward and robust way. In addition, the way the <em>charset</em>
argument interacts with search criteria has been improved. These changes
make it easier to pass search criteria and have them handled correctly.</p>
<p>Unfortunately these changes also mean that small changes may be required
to existing code that uses IMAPClient. Please see the earlier blog
article about the <a href="http://freshfoo.com/blog/imapclient-1.0-search">search
changes</a> as well as the
<a href="http://imapclient.readthedocs.org/en/stable/releases.html#version-1-0-0">release
notes</a>
for more information.</p>
<p><strong>Socket timeout support</strong>: IMAPClient now accepts a timeout at creation
time. The timeout applies while establishing the connection and for all
operations on the socket connected to the IMAP server.</p>
<p><strong>Semantic versioning</strong>: In order to better indicate version
compatibility to IMAPClient's users, the project will now strictly
adhere to the <a href="http://semver.org">Semantic Versioning</a> scheme.</p>
<p><strong>Performance optimisation for parsing message id lists</strong>: A short
circuit is now used when parsing a list of message ids which greatly
speeds up parsing time.</p>
<p><strong>Installation via wheels</strong>: In addition to .zip and .tar.gz files,
IMAPClient releases are now also available as universal
<a href="http://pythonwheels.com/">wheels</a>.</p>
<p>There have also been many other <a href="http://imapclient.readthedocs.org/en/stable/releases.html#other">smaller fixes and
improvements</a>.
See the <a href="http://imapclient.readthedocs.org/en/stable/releases.html#version-1-0-0">release
notes</a>
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded via <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
<p>This release couldn't have been possible with the amazing support of
<a href="https://nylas.com/">Nylas</a>. If you're developing software that needs to
deal with email, save yourself a whole lot of pain and check out their
email platform. If you're after a modern, extensible, cross-platform
email client check out <a href="https://nylas.com/n1">N1</a>.</p>
Search Changes in IMAPClient 1.02015-09-15T00:00:00+00:002015-09-15T00:00:00+00:00Unknown/posts/imapclient-1.0-search/<p>Following on from <a href="http://freshfoo.com/blog/imapclient-1.0-tls">last week's
post</a> about the upcoming
IMAPClient 1.0 release, I'd like to draw attention to some significant,
compatibility breaking changes related to the handling of search
criteria.</p>
<p>IMAPClient's methods that accept search criteria
(<a href="http://imapclient.readthedocs.org/en/latest/#imapclient.IMAPClient.search">search</a>,
<a href="http://imapclient.readthedocs.org/en/latest/#imapclient.IMAPClient.sort">sort</a>,
<a href="http://imapclient.readthedocs.org/en/latest/#imapclient.IMAPClient.thread">thread</a>,
<a href="http://imapclient.readthedocs.org/en/latest/#imapclient.IMAPClient.gmail_search">gmail_search</a>)
have been changed to provide take criteria in a more straightforward and
robust way. In addition, the way the <em>charset</em> argument interacts with
search criteria has been improved. These changes make it easier to pass
search criteria and have them handled correctly but unfortunately also
mean that small changes may be required to existing code that uses
IMAPClient.</p>
<h1 id="search-criteria">Search criteria</h1>
<p>The preferred way to specify criteria now is as a list of strings, ints
and dates (where relevant). The list should be flat with all the
criteria parts run together. Where a criteria takes an argument, just
provide it as the next element in the list.</p>
<p>Some valid examples:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">DELETED</span><span>'])
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">NOT</span><span>', '</span><span style="color:#a3be8c;">DELETED</span><span>'])
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">FLAGGED</span><span>', '</span><span style="color:#a3be8c;">SUBJECT</span><span>', '</span><span style="color:#a3be8c;">foo</span><span>', '</span><span style="color:#a3be8c;">BODY</span><span>', '</span><span style="color:#a3be8c;">hello world</span><span>'])
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">NOT</span><span>', '</span><span style="color:#a3be8c;">DELETED</span><span>', '</span><span style="color:#a3be8c;">SMALLER</span><span>', </span><span style="color:#d08770;">1000</span><span>])
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">SINCE</span><span>', </span><span style="color:#bf616a;">date</span><span>(</span><span style="color:#d08770;">2006</span><span>, </span><span style="color:#d08770;">5</span><span>, </span><span style="color:#d08770;">3</span><span>)])
</span></code></pre>
<p>IMAPClient will perform all required conversion, quoting and encoding.
Callers do not need to and should not attempt to do this themselves.
IMAPClient will automatically send criteria parts as IMAP literals when
required (i.e. when the encoded part is 8-bit).</p>
<p>Some previously accepted ways of passing search criteria will not work
as they did in previous versions of IMAPClient. Small changes will be
required in these cases. Here are some examples of how to update code
written against older versions of IMAPClient:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">NOT DELETED</span><span>']) </span><span style="color:#65737e;"># Before
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">NOT</span><span>', '</span><span style="color:#a3be8c;">DELETED</span><span>']) </span><span style="color:#65737e;"># After
</span><span>
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">TEXT "foo"</span><span>']) </span><span style="color:#65737e;"># Before
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">TEXT</span><span>', '</span><span style="color:#a3be8c;">foo</span><span>']) </span><span style="color:#65737e;"># After (IMAPClient will add the quotes)
</span><span>
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">DELETED</span><span>', '</span><span style="color:#a3be8c;">TEXT "foo"</span><span>']) </span><span style="color:#65737e;"># Before
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">DELETED</span><span>', '</span><span style="color:#a3be8c;">TEXT</span><span>', '</span><span style="color:#a3be8c;">foo</span><span>']) </span><span style="color:#65737e;"># After
</span><span>
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">SMALLER 1000</span><span>']) </span><span style="color:#65737e;"># Before
</span><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">SMALLER</span><span>', </span><span style="color:#d08770;">1000</span><span>]) </span><span style="color:#65737e;"># After
</span></code></pre>
<p>It is also possible to pass a single string as the search criteria.
IMAPClient will not attempt quoting in this case, allowing the caller to
specify search criteria at a lower level. Specifying criteria using a
sequence of strings is preferable however. The following examples
(equivalent to those further above) are valid:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>c.</span><span style="color:#bf616a;">search</span><span>('</span><span style="color:#a3be8c;">DELETED</span><span>')
</span><span>c.</span><span style="color:#bf616a;">search</span><span>('</span><span style="color:#a3be8c;">NOT DELETED</span><span>')
</span><span>c.</span><span style="color:#bf616a;">search</span><span>('</span><span style="color:#a3be8c;">FLAGGED SUBJECT "foo" BODY "hello world"</span><span>')
</span><span>c.</span><span style="color:#bf616a;">search</span><span>('</span><span style="color:#a3be8c;">NOT DELETED SMALLER 1000</span><span>')
</span><span>c.</span><span style="color:#bf616a;">search</span><span>('</span><span style="color:#a3be8c;">SINCE 03-May-2006</span><span>')
</span></code></pre>
<h1 id="search-charset">Search charset</h1>
<p>The way that the search <em>charset</em> argument is handled has also changed.</p>
<p>Any unicode criteria arguments will now be encoded by IMAPClient using
the supplied charset. The charset must refer to an encoding that is
capable of handling the criteria's characters or an error will occur.
The charset must obviously also be one that the server supports! (UTF-8
is common)</p>
<p>Any criteria given as bytes will not be changed by IMAPClient, but the
provided charset will still be passed to the IMAP server. This allows
already encoding criteria to be passed through as-is. The encoding
referred to by <em>charset</em> should match the actual encoding used for the
criteria.</p>
<p>The following are valid examples:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>c.</span><span style="color:#bf616a;">search</span><span>(['</span><span style="color:#a3be8c;">TEXT</span><span>', </span><span style="color:#b48ead;">u</span><span>'</span><span style="color:#96b5b4;">\u263a</span><span>'], '</span><span style="color:#a3be8c;">utf-8</span><span>') </span><span style="color:#65737e;"># IMAPClient will apply UTF-8 encoding
</span><span>c.</span><span style="color:#bf616a;">search</span><span>([</span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">TEXT</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#96b5b4;">\xe2\x98\xba</span><span>'], '</span><span style="color:#a3be8c;">utf-8</span><span>') </span><span style="color:#65737e;"># Caller has already applied UTF-8 encoding
</span></code></pre>
<p>The documentation and tests for search, gmail_search, sort and thread
has updated to account for these changes and have also been generally
improved.</p>
<p>For those willing to try out the changes now please install from
IMAPClient's tip. Any feedback on the changes and/or documentation would
be hugely appreciated.</p>
TLS Changes in IMAPClient 1.02015-09-08T00:00:00+00:002015-09-08T00:00:00+00:00Unknown/posts/imapclient-1.0-tls/<p>IMAPClient 1.0 is <strong>really</strong> close to being done now and it's going to
be one of the biggest releases in its history (thanks largely to the
support of the good people at <a href="https://nylas.com/">Nylas</a>).</p>
<p>The headline feature of this release is the completely revamped
<a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> support.
With 1.0, IMAPClient will perform certificate verification, certificate
hostname checking, and will not use known-insecure TLS settings and
protocols - <em>by default</em>. In order to work around Python historically
patchy TLS support<sup class="footnote-reference"><a href="#1">1</a></sup>, IMAPClient uses
<a href="https://pypi.python.org/pypi/backports.ssl/">backports.ssl</a> and
<a href="https://pypi.python.org/pypi/pyOpenSSL">pyOpenSSL</a> to provide
consistent TLS functionality across all supported Python versions (2.6,
2.7, 3.3 and 3.4).</p>
<p>All this goodness doesn't come for free however. There were some
compromises and compatibility breaks required to make it work:</p>
<ol>
<li>Due to lack of support in some of the new dependencies, IMAPClient
no longer supports Python 3.2.</li>
<li>The keyword arguments that were accepted by the IMAPClient
constructor to pass through to imaplib's IMAP4 constructor are no
longer supported. This was in place to provide access to imaplib's
SSL arguments which are no longer relevant. Please pass a SSL
context object instead.</li>
<li>When using the default SSL context that IMAPClient creates
(recommended), certificate verification is enabled and various bad
TLS settings are disabled. This means that IMAPClient may fail to
establish TLS connections to servers that used to work with previous
versions of IMAPClient (especially if a self-signed certificate is
used by the server). Fortunately it's not difficult to supply your
own CA certificates or disable verification if required.</li>
</ol>
<p>There's a <a href="http://imapclient.readthedocs.org/en/latest/#tls-ssl">new section in the
manual</a> which has
more details and includes examples of how to tweak the SSL context for
some common scenarios.</p>
<p>For those willing to try out the changes now please install from
IMAPClient's tip. Any feedback would be hugely appreciated.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Note that due to the hard work of various folks, TLS support is
<em>much</em> better in Python 3.4 and 2.7.9.</p>
</div>
IMAPClient 0.132015-07-01T00:00:00+00:002015-07-01T00:00:00+00:00Unknown/posts/imapclient-0.13/<p>I'm chuffed to announce that IMAPClient 0.13 is out!</p>
<p>Here's what's new:</p>
<ul>
<li>Added support for the ID command (as per RFC2971). Many thanks to
Eben Freeman from Nylas.</li>
<li>Fixed exception with NIL address in envelope address list. Thomas
Steinacher gets a big thank you for this one.</li>
<li>Fixed a regression in the handling of NIL/None SEARCH responses.
Thanks again to Thomas Steinacher.</li>
<li>Don't traceback when an unparseable date is seen in ENVELOPE or
INTERNALDATE responses. None is now returned instead.</li>
<li>Extended timestamp parsing support to allow for quirky timestamp
strings which use dots for the time separator.</li>
<li>Replaced the horrible INTERNALDATE parsing code</li>
<li>The datetime_to_imap top-level function has been moved to the
datetime_util module and is now called datetime_to_INTERNALDATE.
This will only affect you in the unlikely case that you were
importing this function out of the IMAPClient package.</li>
<li>The docs for various IMAPClient methods, and the HACKING.rst file
have been improved.</li>
<li>CONDSTORE live test is now more reliable (especially when running
against Gmail)</li>
</ul>
<p>See the
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS.rst">NEWS.rst</a> file
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
<p>I'm also excited to announce that <a href="https://nylas.com/">Nylas</a> (formerly
Inbox) has now employed me to work on IMAPClient part time. There should
be a significant uptick in the development of IMAPClient.</p>
<p>The next major version of IMAPClient will be 1.0.0, and will be
primarily focussed on enhancing TLS/SSL support.</p>
IMAPClient 0.122015-06-30T00:00:00+00:002015-06-30T00:00:00+00:00Unknown/posts/imapclient-0.12/<p>I'm very happy to announce that IMAPClient 0.12 is out!</p>
<p>This is a big release. Some highlights:</p>
<ul>
<li>Unicode handling has been fixed. Some bad decisions were made during
the Python 3 port (v0.10) and this release fixes that. Bytes are now
returned in most places (instead of unicode strings).</li>
<li>MODSEQ parts in SEARCH responses are now handled correctly. A crash
has been fixed when MODSEQ queries (part of the CONDSTORE extension)
are made with the search method. The returned MODSEQ value is now
available via the "modseq" attribute on the returned list of ids.</li>
<li>Extra __init__ keyword args are passed through. This allows
access to SSL options that the underlying imaplib library might
support (Python version dependent).</li>
<li>Python 3.4 is now officially supported.</li>
<li>More control over OAUTH2 parameters.</li>
<li>The deprecated get_folder_delimiter() method has been removed.</li>
</ul>
<p>See the
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS.rst">NEWS.rst</a> file
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>Many thanks go to <a href="https://www.inboxapp.com/">Inbox</a> for sponsoring the
significant unicode changes in this release.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
<p>The next major version of IMAPClient will be 1.0.0, and will be
primarily focussed on enhancing TLS/SSL support.</p>
IMAPClient introduction2015-03-08T00:00:00+00:002015-03-08T00:00:00+00:00Unknown/posts/imapclient-intro/<p>I gave a presentation introducting
<a href="http://imapclient.freshfoo.com/">IMAPClient</a> at the <a href="http://www.meetup.com/NZPUG-Christchurch/">monthly
Christchurch Python meetup</a>
on Thursday. It included a brief introduction to the IMAP protocol, the
motivation for creating the IMAPClient package, some examples of how it
compares to using imaplib from the standard library, and some discussion
of future plans. There also may have been some silly pictures. The talk
seemed to be well received. Thanks to everyone who attended.</p>
<p>I'm not sure how useful they'll be without the vocal part but <a href="/presentations/imapclient-intro/">the
slides</a> are now available online.</p>
<p>Apart from my talk, a wide range of topics came up including installing
and managing multiple versions of Python side-by-side, TLS (SSL) and
Python and the suckiness (or not) of JavaScript. As always, lots of fun.
There's always a great bunch of people who make it along.</p>
IMAPClient now all at Bitbucket2015-03-01T00:00:00+00:002015-03-01T00:00:00+00:00Unknown/posts/imapclient-moved-to-bb/<p>I've been wanting to do this for a while: IMAPClient is now completely
hosted on Bitbucket. The Trac instance is no more and all tickets have
been <a href="https://bitbucket.org/unayok/trac2bb">migrated</a> to the Bitbucket
issue tracker. <a href="http://imapclient.freshfoo.com">http://imapclient.freshfoo.com</a> now redirects to the
Bitbucket site.</p>
<p>The primary motivation for this change is that it makes it possible for
anyone to easily interact with IMAPClient's tickets using the Bitbucket
account they probably already have. Due to spam issues, I had to turn
off public write access to the Trac bug tracker a long time ago meaning
I was the only one creating and updating tickets. This change also means
less maintenance overhead for me, so more time to spend on IMAPClient.</p>
<p>Please let me know if you see anything related to the migration that
doesn't look right.</p>
Go for Pythonistas2014-08-08T00:00:00+00:002014-08-08T00:00:00+00:00Unknown/posts/go-for-pythonistas/<p>I gave a talk last night at the monthly <a href="http://www.meetup.com/NZPUG-Christchurch/events/193772932/">Christchurch Python
meetup</a>
titled "Go For Pythonistas". It was an introduction to the <a href="http://golang.org/">Go
programming language</a> from the perspective of a
Python developer. The talk went well, with plenty of questions and
comments throughout. Thanks to all who attended for your interest!</p>
<p>I'm not sure how useful they'll be on their own but the slides for the
talk are <a href="/presentations/go-for-pythonistas/">available here</a>.</p>
<p>I mentioned <a href="http://www.ubuntu.com/cloud/juju">Juju</a> in the talk a few
times which piqued a few people's interest so I also ended up doing a
short impromptu talk about that.
<a href="http://www.ubuntu.com/cloud/juju">Juju</a> is the amazing cloud
orchestration software that I hack on at
<a href="http://www.canonical.com/">Canonical</a> these days (written in Go). It's
worth a few blog articles in itself.</p>
IMAPClient 0.112014-07-16T00:00:00+00:002014-07-16T00:00:00+00:00Unknown/posts/imapclient-0.11/<p>This is a somewhat belated announcement that IMAPClient 0.11 is out (and
has been for a little over a week now). Notable changes:</p>
<ul>
<li>Support for raw Gmail search API when connected to Gmail's IMAP</li>
<li>ENVELOPE FETCH responses are now returned as Envelope instances.
These objects are namedtuples providing convenient attribute and
positional based access to envelope fields and the Date field is
converted to a datetime instance.</li>
<li>BODY and BODYSTRUCTURE responses are now processed recusively so
multipart sections within other multipart sections are returned
correctly. NOTE: code that expects the old (broken) behaviour will
need to be updated.</li>
<li>SELECT response bug fix with respect to handling of square brackets
in flags</li>
</ul>
<p>Big thanks go out to John Louis del Rosario, Naveen Nathan, Brandon
Rhodes and Benjamin Morrise for their contributions towards this
release.</p>
<p>See the
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS.rst">NEWS.rst</a> file
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
Inbox is sponsoring IMAPClient development2014-07-16T00:00:00+00:002014-07-16T00:00:00+00:00Unknown/posts/inboxapp/<p>Now that they have <a href="http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launch-inbox-a-next-generation-email-platform/">officially
launched</a>
I can happily announce that the good folks at
<a href="https://www.inboxapp.com/">Inbox</a> are sponsoring the development of
certain features and fixes for
<a href="http://imapclient.freshfoo.com/">IMAPClient</a>. Inbox have just released
the initial version of their open source email sync engine which
provides a clean REST API for dealing with email - hiding all the
archaic intricacies of protocols such as IMAP and MIME. IMAPClient is
used by the Inbox engine to interact with IMAP servers.</p>
<p>The sponsorship of IMAPClient by Inbox will help to increase the speed
of IMAPClient development and all improvements will be open source,
feeding directly in to trunk so that all IMAPClient users benefit.
Thanks Inbox!</p>
<p>The first request from Inbox is to fix some unicode/bytes handling
issues that crept in as part of the addition of Python 3 support. It's a
non-trivial amount of work but things are going well. Watch this
space...</p>
IMAPClient 0.102013-06-06T00:00:00+00:002013-06-06T00:00:00+00:00Unknown/posts/imapclient-0.10/<p>IMAPClient 0.10 has just been released. This is an important release
because it's the first to support Python 3!</p>
<p>Here's the highlights:</p>
<ul>
<li>Python 3.2 and 3.3 are now officially supported. This release also
means that Python versions older than 2.6 are no longer supported.
Be sure to see the
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS.rst">NEWS.rst</a>
file for more information on this change.</li>
<li>The <code>HIGHESTMODSEQ</code> item in <code>SELECT</code> responses is now parsed
correctly</li>
<li>Fixed daylight saving handling in FixedOffset class</li>
<li>Fixed <code>--port</code> command line bug in imapclient.interact when SSL
connections are made.</li>
<li>Michael Foord's excellent Mock library is now longer included with
the IMAPClient package (it is listed as external test dependency)</li>
<li>Live tests that aren't UID related are now only run once per run</li>
<li>Live tests now perform far less logins to the server under test</li>
<li>Unit tests can now be run for all supported Python versions using
<code>tox</code>.</li>
<li><code>python setup.py test</code> now runs the unit tests</li>
<li>Many documentation fixes and improvements.</li>
</ul>
<p>A massive thank you to Mathieu Agopian for his massive contribution to
getting the Python 3 support finished. His changes and ideas feature
heavily in this release.</p>
<p>See the
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS.rst">NEWS.rst</a> file
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
Monitoring Audio Levels with PulseAudio2013-05-08T00:00:00+00:002013-05-08T00:00:00+00:00Unknown/posts/pulseaudio_monitoring/<p>I'm working on driving an analog <a href="http://en.wikipedia.org/wiki/VU_meter">VU
meter</a> from my <a href="http://www.raspberrypi.org/">Raspberry
Pi</a> using whatever audio is going out the
Pi's sound outputs. The de facto Linux sound system,
<a href="http://www.freedesktop.org/wiki/Software/PulseAudio">PulseAudio</a>,
allows any sound output (or "sink" in PulseAudio's nonclementure) to be
monitored. In PulseAudio land, each sink has a corresponding "source"
called the monitor source which can be read just like any other other
PulseAudio input such as a microphone. In fact, to help with volume
meter style applications, PulseAudio even allows you to ask for peak
level measurements, which means you can sample the monitor sink at a low
frequency, with low CPU utilisation, but still produce a useful volume
display. When this feature is used, each sample read indicates the peak
level since the last sample.</p>
<p>The main PulseAudio API is asynchronous and callback based, and the
<a href="http://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/Developer">documentation</a>
is primarly just an API reference. This makes it a little difficult to
figure out how to get everthing to hang together. Using the code from
various open source projects (primarily
<a href="https://code.google.com/p/veromix-plasmoid/">veromix-plasmoid</a> and
<a href="http://0pointer.de/lennart/projects/pavumeter/">pavumeter</a>), along with
the API reference, I was able to develop a fairly minimal code example
that will hopefully be useful to others trying to do something similar.
Although this example is written in Python, it is using the PulseAudio C
API directly (via ctypes) so it should hopefully still be relevant if
your application is written in C or another language.</p>
<p>Here's the demo code. The latest version is also available <a href="https://gitlab.com/menn0/peak-detect">on
GitLab</a>. Note that in order to run
this example you need to install Vincent Breitmoser's ctypes PulseAudio
wrapper available at <a href="https://github.com/Valodim/python-pulseaudio">https://github.com/Valodim/python-pulseaudio</a>.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>sys
</span><span style="color:#b48ead;">from </span><span>Queue </span><span style="color:#b48ead;">import </span><span>Queue
</span><span style="color:#b48ead;">from </span><span>ctypes </span><span style="color:#b48ead;">import </span><span style="color:#bf616a;">POINTER</span><span>, c_ubyte, c_void_p, c_ulong, cast
</span><span>
</span><span style="color:#65737e;"># From https://github.com/Valodim/python-pulseaudio
</span><span style="color:#b48ead;">from </span><span>pulseaudio.lib_pulseaudio </span><span style="color:#b48ead;">import </span><span style="color:#d08770;">*
</span><span>
</span><span style="color:#65737e;"># edit to match your sink
</span><span style="color:#bf616a;">SINK_NAME </span><span>= '</span><span style="color:#a3be8c;">alsa_output.pci-0000_00_1b.0.analog-stereo</span><span>'
</span><span style="color:#bf616a;">METER_RATE </span><span>= </span><span style="color:#d08770;">344
</span><span style="color:#bf616a;">MAX_SAMPLE_VALUE </span><span>= </span><span style="color:#d08770;">127
</span><span style="color:#bf616a;">DISPLAY_SCALE </span><span>= </span><span style="color:#d08770;">2
</span><span style="color:#bf616a;">MAX_SPACES </span><span>= </span><span style="color:#bf616a;">MAX_SAMPLE_VALUE </span><span>>> </span><span style="color:#bf616a;">DISPLAY_SCALE
</span><span>
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">PeakMonitor</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">sink_name</span><span>, </span><span style="color:#bf616a;">rate</span><span>):
</span><span> </span><span style="color:#bf616a;">self</span><span>.sink_name = sink_name
</span><span> </span><span style="color:#bf616a;">self</span><span>.rate = rate
</span><span>
</span><span> </span><span style="color:#65737e;"># Wrap callback methods in appropriate ctypefunc instances so
</span><span> </span><span style="color:#65737e;"># that the Pulseaudio C API can call them
</span><span> </span><span style="color:#bf616a;">self</span><span>._context_notify_cb = </span><span style="color:#bf616a;">pa_context_notify_cb_t</span><span>(</span><span style="color:#bf616a;">self</span><span>.context_notify_cb)
</span><span> </span><span style="color:#bf616a;">self</span><span>._sink_info_cb = </span><span style="color:#bf616a;">pa_sink_info_cb_t</span><span>(</span><span style="color:#bf616a;">self</span><span>.sink_info_cb)
</span><span> </span><span style="color:#bf616a;">self</span><span>._stream_read_cb = </span><span style="color:#bf616a;">pa_stream_request_cb_t</span><span>(</span><span style="color:#bf616a;">self</span><span>.stream_read_cb)
</span><span>
</span><span> </span><span style="color:#65737e;"># stream_read_cb() puts peak samples into this Queue instance
</span><span> </span><span style="color:#bf616a;">self</span><span>._samples = </span><span style="color:#bf616a;">Queue</span><span>()
</span><span>
</span><span> </span><span style="color:#65737e;"># Create the mainloop thread and set our context_notify_cb
</span><span> </span><span style="color:#65737e;"># method to be called when there's updates relating to the
</span><span> </span><span style="color:#65737e;"># connection to Pulseaudio
</span><span> _mainloop = </span><span style="color:#bf616a;">pa_threaded_mainloop_new</span><span>()
</span><span> _mainloop_api = </span><span style="color:#bf616a;">pa_threaded_mainloop_get_api</span><span>(_mainloop)
</span><span> context = </span><span style="color:#bf616a;">pa_context_new</span><span>(_mainloop_api, '</span><span style="color:#a3be8c;">peak_demo</span><span>')
</span><span> </span><span style="color:#bf616a;">pa_context_set_state_callback</span><span>(context, </span><span style="color:#bf616a;">self</span><span>._context_notify_cb, </span><span style="color:#d08770;">None</span><span>)
</span><span> </span><span style="color:#bf616a;">pa_context_connect</span><span>(context, </span><span style="color:#d08770;">None</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">None</span><span>)
</span><span> </span><span style="color:#bf616a;">pa_threaded_mainloop_start</span><span>(_mainloop)
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__iter__</span><span>(</span><span style="color:#bf616a;">self</span><span>):
</span><span> </span><span style="color:#b48ead;">while </span><span style="color:#d08770;">True</span><span>:
</span><span> </span><span style="color:#b48ead;">yield </span><span style="color:#bf616a;">self</span><span>._samples.</span><span style="color:#bf616a;">get</span><span>()
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">context_notify_cb</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">context</span><span>, </span><span style="color:#bf616a;">_</span><span>):
</span><span> state = </span><span style="color:#bf616a;">pa_context_get_state</span><span>(context)
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>state == </span><span style="color:#bf616a;">PA_CONTEXT_READY</span><span>:
</span><span> </span><span style="color:#b48ead;">print </span><span>"</span><span style="color:#a3be8c;">Pulseaudio connection ready...</span><span>"
</span><span> </span><span style="color:#65737e;"># Connected to Pulseaudio. Now request that sink_info_cb
</span><span> </span><span style="color:#65737e;"># be called with information about the available sinks.
</span><span> o = </span><span style="color:#bf616a;">pa_context_get_sink_info_list</span><span>(context, </span><span style="color:#bf616a;">self</span><span>._sink_info_cb, </span><span style="color:#d08770;">None</span><span>)
</span><span> </span><span style="color:#bf616a;">pa_operation_unref</span><span>(o)
</span><span>
</span><span> </span><span style="color:#b48ead;">elif </span><span>state == </span><span style="color:#bf616a;">PA_CONTEXT_FAILED </span><span>:
</span><span> </span><span style="color:#b48ead;">print </span><span>"</span><span style="color:#a3be8c;">Connection failed</span><span>"
</span><span>
</span><span> </span><span style="color:#b48ead;">elif </span><span>state == </span><span style="color:#bf616a;">PA_CONTEXT_TERMINATED</span><span>:
</span><span> </span><span style="color:#b48ead;">print </span><span>"</span><span style="color:#a3be8c;">Connection terminated</span><span>"
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">sink_info_cb</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">context</span><span>, </span><span style="color:#bf616a;">sink_info_p</span><span>, </span><span style="color:#bf616a;">_</span><span>, </span><span style="color:#bf616a;">__</span><span>):
</span><span> </span><span style="color:#b48ead;">if </span><span>not sink_info_p:
</span><span> </span><span style="color:#b48ead;">return
</span><span>
</span><span> sink_info = sink_info_p.contents
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">-</span><span>'* </span><span style="color:#d08770;">60
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">index:</span><span>', sink_info.index
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">name:</span><span>', sink_info.name
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">description:</span><span>', sink_info.description
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>sink_info.name == </span><span style="color:#bf616a;">self</span><span>.sink_name:
</span><span> </span><span style="color:#65737e;"># Found the sink we want to monitor for peak levels.
</span><span> </span><span style="color:#65737e;"># Tell PA to call stream_read_cb with peak samples.
</span><span> </span><span style="color:#b48ead;">print
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">setting up peak recording using</span><span>', sink_info.monitor_source_name
</span><span> </span><span style="color:#b48ead;">print
</span><span> samplespec = </span><span style="color:#bf616a;">pa_sample_spec</span><span>()
</span><span> samplespec.channels = </span><span style="color:#d08770;">1
</span><span> samplespec.format = </span><span style="color:#bf616a;">PA_SAMPLE_U8
</span><span> samplespec.rate = </span><span style="color:#bf616a;">self</span><span>.rate
</span><span>
</span><span> pa_stream = </span><span style="color:#bf616a;">pa_stream_new</span><span>(context, "</span><span style="color:#a3be8c;">peak detect demo</span><span>", samplespec, </span><span style="color:#d08770;">None</span><span>)
</span><span> </span><span style="color:#bf616a;">pa_stream_set_read_callback</span><span>(pa_stream,
</span><span> </span><span style="color:#bf616a;">self</span><span>._stream_read_cb,
</span><span> sink_info.index)
</span><span> </span><span style="color:#bf616a;">pa_stream_connect_record</span><span>(pa_stream,
</span><span> sink_info.monitor_source_name,
</span><span> </span><span style="color:#d08770;">None</span><span>,
</span><span> </span><span style="color:#bf616a;">PA_STREAM_PEAK_DETECT</span><span>)
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">stream_read_cb</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">stream</span><span>, </span><span style="color:#bf616a;">length</span><span>, </span><span style="color:#bf616a;">index_incr</span><span>):
</span><span> data = </span><span style="color:#bf616a;">c_void_p</span><span>()
</span><span> </span><span style="color:#bf616a;">pa_stream_peek</span><span>(stream, data, </span><span style="color:#bf616a;">c_ulong</span><span>(length))
</span><span> data = </span><span style="color:#bf616a;">cast</span><span>(data, </span><span style="color:#bf616a;">POINTER</span><span>(c_ubyte))
</span><span> </span><span style="color:#b48ead;">for </span><span>i </span><span style="color:#b48ead;">in </span><span style="color:#bf616a;">xrange</span><span>(length):
</span><span> </span><span style="color:#65737e;"># When PA_SAMPLE_U8 is used, samples values range from 128
</span><span> </span><span style="color:#65737e;"># to 255 because the underlying audio data is signed but
</span><span> </span><span style="color:#65737e;"># it doesn't make sense to return signed peaks.
</span><span> </span><span style="color:#bf616a;">self</span><span>._samples.</span><span style="color:#bf616a;">put</span><span>(data[i] - </span><span style="color:#d08770;">128</span><span>)
</span><span> </span><span style="color:#bf616a;">pa_stream_drop</span><span>(stream)
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">main</span><span>():
</span><span> monitor = </span><span style="color:#bf616a;">PeakMonitor</span><span>(</span><span style="color:#bf616a;">SINK_NAME</span><span>, </span><span style="color:#bf616a;">METER_RATE</span><span>)
</span><span> </span><span style="color:#b48ead;">for </span><span>sample </span><span style="color:#b48ead;">in </span><span>monitor:
</span><span> sample = sample >> </span><span style="color:#bf616a;">DISPLAY_SCALE
</span><span> bar = '</span><span style="color:#a3be8c;">></span><span>' * sample
</span><span> spaces = ' ' * (</span><span style="color:#bf616a;">MAX_SPACES </span><span>- sample)
</span><span> </span><span style="color:#b48ead;">print </span><span>' </span><span style="color:#d08770;">%3d %s%s</span><span style="color:#96b5b4;">\r</span><span>' % (sample, bar, spaces),
</span><span> sys.stdout.</span><span style="color:#bf616a;">flush</span><span>()
</span><span>
</span><span style="color:#b48ead;">if </span><span>__name__ == '</span><span style="color:#a3be8c;">__main__</span><span>':
</span><span> </span><span style="color:#bf616a;">main</span><span>()
</span></code></pre>
<p>When running this demo, you'll need to modify SINK_NAME to match the
name of the sink you want to monitor. If you're not sure of the sink,
just run it - the program prints the details of all available sinks to
stdout. If all goes to plan you should see a basic volume display in the
console (when sound is actually playing!).</p>
<p>In this demo, the PeakMonitor class does all the interaction with the
PulseAudio API. It needs the name of a sink to monitor and the sampling
rate. Iterating over a PeakMonitor instance will give 8-bit peak level
samples (actually they're from 128 to 255 - see comment in code and
comment from Tanu on this article).</p>
<p>The <code>main</code> function implements a simple volume display. Some of the
constants at the top of the code control how the volume level is
displayed.</p>
<p>As mentioned earlier, the PulseAudio API is asynchronous, making heavy
use of callbacks. This goes for API calls that return information about
the sound configuration of the system as well as API calls that return
or accept actual sound data. The reason for this is that the available
sound devices may change at any point as a program is running (e.g. when
USB audio devices are connected) and PulseAudio clients may need to be
able to handle such changes.</p>
<p>Here's how the demo program interacts with PulseAudio:</p>
<ul>
<li><code>__init__</code> sets up the PulseAudio processing loop in a separate
thread, establishes a connection to the PulseAudio daemon and asks
PulseAudio to call the <code>context_notify_cb</code> method with updates about
the status of the connection to PulseAudio.</li>
<li><code>context_notify_cb</code> will be called several times by the PulseAudio
API as the connection is established. All things going to plan, the
connection will reach the <code>PA_CONTEXT_READY</code> state. At this point,
we request that the <code>sink_info_cb</code> method be called with information
about each available sink.</li>
<li><code>sink_info_cb</code> is called once for each available sink. If a sink
with the name matching the one passed to <code>__init__</code> is seen, a
stream is created on it's monitor source and <code>stream_read_cb</code> is set
to be called with 8-bit peak level samples.</li>
<li><code>stream_read_cb</code> is then called at the rate requested when the
PeakMonitor class was instantiated. <code>pa_stream_peek</code> reads the
available sample(s) from the stream and <code>pa_stream_drop</code> tells
PulseAudio that we're done with the stream data (presumably so the
buffer can be re-used or deallocated). The callback functions are
all called on the mainloop thread created by the PulseAudio API so
the <code>_samples</code> Queue instance is used to safely return samples back
to the main thread (<code>__iter</code> reads the samples from this queue).</li>
</ul>
<p>When running this demo on my fairly high-spec laptop it has no
noticeable impact on CPU utilisation, but it's another story when it
runs on the Raspberry Pi: around 25% of the CPU is required when the
monitor sampling rate is 344 Hz. The issue is that we've got PulseAudio
calling our Python function (stream_read_cb) at a siginficant
frequency and Python just isn't that fast on the Raspberry Pi's 700MHz
CPU. The pointer manipulation being done in <code>stream_read_cb</code>, which
would be incredibily fast in C, is being done using a significant amount
of Python bytecode and function calls (partially because ctypes is being
used to do them).</p>
<p>I'm not too comfortable with constantly pegging the Raspberry Pi CPU at
25% as I'd like to eventually to have a always-running running daemon
process driving the VU meter on the Raspberry Pi. It doesn't seem like a
good idea to constantly have the CPU doing that much work. I plan on
trying out Cython to convert the PulseAudio interation parts to C or
rewriting the entire program in pure C since it's not really taking much
advantage of Python's features. More on this to come.</p>
<p>I hope this article is useful to developers who are interested in using
the PulseAudio API. Please note that I'm by no means a PulseAudio
expert. Please let me know if there's anything I could be doing better
and I'll update the article.</p>
<p>(This article is actually quite far behind where I'm actually at with
the VU meter project. I actually already have a DAC connected to the
GPIO ports, driving an analog VU meter using the sound going out the
Raspberry Pi's audio outputs. I'm hoping to publish some more articles
this week.)</p>
<p><strong>Update (2013-02-14):</strong> I've updated the code and article to explain
about why peak samples only range from 128 to 255. See Tanu's comment
below too.</p>
<p><strong>Update (2013-05-06):</strong> There's now a <a href="http://freshfoo.com/blog/raspberry_pi_vu_meter">new
article</a> that describes
how this code is used to actually drives a VU meter from the Raspberry
Pi.</p>
Raspberry Pi: driving a VU meter using a digital-to-analog converter2013-05-08T00:00:00+00:002013-05-08T00:00:00+00:00Unknown/posts/raspberry_pi_vu_meter/<p>As I've mentioned in previous
<a href="http://freshfoo.com/blog/pulseaudio_monitoring">blog</a>
<a href="http://freshfoo.com/blog/overengineered_blinking_leds">articles</a>, my
wife and I have been working on driving an analog VU meter based on the
sound going out the Raspberry Pi's audio outputs. This now works!</p>
<p>Here's a video demonstrating the result:</p>
<div class="youtube-container">
<iframe
src="https://www.youtube-nocookie.com/embed/K0NM2YxhWtI"
allow="encrypted-media"
frameborder="0"
webkitallowfullscreen
mozallowfullscreen
allowfullscreen>
</iframe>
</div>
<p>The music<sup class="footnote-reference"><a href="#1">1</a></sup> is playing from a Raspberry Pi, with software running on
the Pi digitally sampling the peak output audio level and writing that
out to an 8-bit <a href="http://en.wikipedia.org/wiki/Digital-to-analog_converter">digital-to-analog
converter</a>
(DAC). The DAC output is then used to drive the analog meter. If you're
interesting in knowing how all this hangs together, keep reading.</p>
<p>The Raspberry Pi is very capable of generating digital signals using its
GPIO pins but an analog meter requires an analog signal. To convert a
digital value emitted from the Raspberry Pi to an analog voltage we need
a DAC. It's possible, and rewarding, to <a href="http://hackaday.com/2011/02/17/your-first-digital-to-analog-converter-build/">build a
DAC</a>
using discrete components, but it's more convenient, accurate and
reliable if you use an off-the-shelf DAC packaged as an integrated
circuit (IC).</p>
<p>We've chosen to use the Analog Devices
<a href="http://www.analog.com/static/imported-files/data_sheets/AD557.pdf">AD557</a>.
It has the following features:</p>
<ul>
<li>8-bit parallel input: It takes 8-bit values as input (0-255) read
via 8 individual (parallel) pins which are sampled at the same time.</li>
<li>5V single supply operation: The AD557 requires just a single 5V
supply voltage. This is ideal for the Raspberry Pi where we have 5V
power readily available. Many DAC ICs require a supply voltage
somewhat higher than 5V making them trickier to use with the Pi.</li>
<li>0 to 2.55V output: A value of 0 at the input gives 0V at the output.
A value of 255 at the input (the maximum) gives 2.55V at the output.
This is easy to work with and think about.</li>
<li>Flexible input voltage: The digital inputs accept values of 0 to
0.8V as "0", and 2 to 5V+ as "1", making them compatible with the
3.3V digital outputs on the Raspberry Pi.</li>
<li>Low cost: Less concern about blowing it up!</li>
</ul>
<p>Here's how the Raspberry Pi, DAC and VU meter were connected together:</p>
<img alt="Circuit diagram" src="/images/vu_meter.png" width="600" height="399" />
<p>Various GPIO pins on the Raspberry Pi are used to provide a number
between 0 and 255 to the data input pins (1 through 7) on the DAC. The
power supply and ground for the DAC are provided by the 5V power and
ground pins available on the Raspberry Pi GPIO header.</p>
<p>The slightly odd seeming output configuration involving three Vout pins
is actually just what is required for normal operation. These three pins
allow for alternate configurations where a non-standard output voltage
range is required. See the AD577 documentation for more details.</p>
<p>The CS (chip select) and CE (chip enable) pins on the AD557 warrant some
explanation. The DAC will only change its output voltage when both of
these pins are low (at ground). This means that you can ensure you have
the right values set on the data pins before telling the AD557 to update
the output to the next value, ensuring unintended changes in output
voltage are avoided while a new value is being set on the inputs.</p>
<p>There are use cases that require both the CS and CE pins to be used
which I won't go in to here. Since we only need one of these pins here,
CS is connected permanently to ground. CE is connected to a GPIO pin so
we can control it from software running on the Raspberry Pi. The value
on CE pin needs to be dropped from 1 to ground 0 for at least 225ns in
order for the AD557 to update it's output voltage to a new value.</p>
<p>The 15K resistor is there to limit current. The <a href="http://www.sifam.com/Downloads/Content/Vintage_AL%20Meters-A4_web.pdf">Sifam audio level
meter</a>
we're using is current driven, and needs very little current at that. It
will show 0 (where the meter goes in to the red) at around 300uA. At the
maximum output voltage from the DAC (5V), a 15K resistor will ensure the
meter gets around 333uA (5 / 15000) which puts the needle nicely just in
to the red zone.</p>
<p>Note that although this meter is marked as a VU meter, we are in no way
indicating a true VU reading. The VU term has a <a href="http://en.wikipedia.org/wiki/VU_meter">strict
definition</a> which we're not
trying to emulate in any real way here.</p>
<img alt="Photo of prototype circuit" src="/images/vu_meter_circuit.jpg" width="336" height="450" />
<p>The code that drives this circuit is available <a href="https://gitlab.com/menn0/raspberry-pi-vu-meter">on
GitLab</a>. The files are:</p>
<ul>
<li>dac.py: Some simple functions to initialise the GPIO ports and do
writes of 8-bit values to the DAC pins.</li>
<li>peak_monitor.py: Functionality to sample peak output audio levels
using PulseAudio. See the <a href="/posts/pulseaudio_monitoring">previous
article</a> for more on how this works.</li>
<li>vu_meter.py: Top level functionality which reads peak audio samples
and writes them to the GPIO pins.</li>
</ul>
<p>Have a look around. There really isn't that much code involved.</p>
<p>I hope you've found this post interesting and/or educational. My wife
and I are about to solder up the circuit and get it and the Pi in to
some kind of box. I'll certainly post some pictures of the result once
that's done.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Irving Berlin's Russian Lullaby, performed by John Coltrane and band.</p>
</div>
IMAPClient 0.9.22013-03-29T00:00:00+00:002013-03-29T00:00:00+00:00Unknown/posts/IMAPClient-0.9.2/<p>IMAPClient 0.9.2 was released yesterday. In this release:</p>
<ul>
<li>The IMAP THREAD command is now supported. Thanks to Lukasz Mierzwa
for the patches.</li>
<li>Enhanced CAPABILITY querying</li>
<li>Better documentation for contributors (see
<a href="https://bitbucket.org/mjs0/imapclient/src/tip/HACKING">HACKING</a>
file)</li>
<li>Copyright date update for 2013.</li>
</ul>
<p>See the <a href="https://bitbucket.org/mjs0/imapclient/src/tip/NEWS">NEWS</a> file
and <a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient site</a>.</p>
<p>Note that the official project source repository is now <a href="https://bitbucket.org/mjs0/imapclient/">on
Bitbucket</a>.
<a href="http://imapclient.freshfoo.com/">http://imapclient.freshfoo.com/</a> is still the offical home page and is
still used for project tracking. It is only the source respository that
has moved.</p>
IMAPClient 0.9.12012-11-13T00:00:00+00:002012-11-13T00:00:00+00:00Unknown/posts/IMAPClient-0.9.1/<p>IMAPClient 0.9.1 is out! In this release:</p>
<ul>
<li>OAUTH2 support: OAUTH v1 is still supported by Gmail but is now
deprecated. Thanks to Zac Witte for the patch for version 2 support.</li>
<li>Stream support: It is now possible to have IMAPClient run an
external command to establish a connection to an IMAP server. This
is useful for exotic connection or authentication setups. Thanks to
Dave Eckhardt for the original patch.</li>
<li>livetest updates to deal with the way Gmail now handles with new
messages.</li>
</ul>
<p>See the <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a> file and
<a href="http://imapclient.readthedocs.org/">manual</a> for more details.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
itertools.groupby2012-11-05T00:00:00+00:002012-11-05T00:00:00+00:00Unknown/posts/itertools_groupby/<p>A relatively unknown part of the Python standard library that I find
myself using fairly regularly at work these days is the
<a href="http://docs.python.org/library/itertools.html#itertools.groupby">groupby</a>
function in the
<a href="http://docs.python.org/library/itertools.html">itertools</a> module. In a
nutshell, groupby takes an iterator and breaks it up into sub-iterators
based on changes in the "key" of the main iterator. This is of course
done without reading the entire source iterator into memory.</p>
<p>The "key" is almost always based on some part of the items returned by
the iterator. It is defined by a "key function", much like the
<a href="http://docs.python.org/library/functions.html#sorted">sorted</a> builtin
function. groupby probably works best when the data is grouped by the
key but this isn't strictly necessary. It depends on the use case.</p>
<p>I've successfully used groupby for splitting up the results of large
database queries or the contents of large data files. The resulting code
ends up being clean and small.</p>
<p>Here's an example:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">from </span><span>itertools </span><span style="color:#b48ead;">import </span><span>groupby
</span><span style="color:#b48ead;">from </span><span>operator </span><span style="color:#b48ead;">import </span><span>itemgetter
</span><span>
</span><span>things = [('</span><span style="color:#a3be8c;">2009-09-02</span><span>', </span><span style="color:#d08770;">11</span><span>),
</span><span> ('</span><span style="color:#a3be8c;">2009-09-02</span><span>', </span><span style="color:#d08770;">3</span><span>),
</span><span> ('</span><span style="color:#a3be8c;">2009-09-03</span><span>', </span><span style="color:#d08770;">10</span><span>),
</span><span> ('</span><span style="color:#a3be8c;">2009-09-03</span><span>', </span><span style="color:#d08770;">4</span><span>),
</span><span> ('</span><span style="color:#a3be8c;">2009-09-03</span><span>', </span><span style="color:#d08770;">22</span><span>),
</span><span> ('</span><span style="color:#a3be8c;">2009-09-06</span><span>', </span><span style="color:#d08770;">33</span><span>)]
</span><span>
</span><span style="color:#b48ead;">for </span><span>key, items </span><span style="color:#b48ead;">in </span><span style="color:#bf616a;">groupby</span><span>(things, </span><span style="color:#bf616a;">itemgetter</span><span>(</span><span style="color:#d08770;">0</span><span>)):
</span><span> </span><span style="color:#b48ead;">print </span><span>key
</span><span> </span><span style="color:#b48ead;">for </span><span>subitem </span><span style="color:#b48ead;">in </span><span>items:
</span><span> </span><span style="color:#b48ead;">print </span><span>subitem
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">-</span><span>' * </span><span style="color:#d08770;">20
</span><span>
</span></code></pre>
<p>Here the dummy data in the "things" list is grouped by the first item of
each element (that is, the key is the first element). For each key, the
key is printed followed by the items returned by each sub-iterator.</p>
<p>The output looks like:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>2009-09-02
</span><span>('2009-09-02', 11)
</span><span>('2009-09-02', 3)
</span><span>--------------------
</span><span>2009-09-03
</span><span>('2009-09-03', 10)
</span><span>('2009-09-03', 4)
</span><span>('2009-09-03', 22)
</span><span>--------------------
</span><span>2009-09-06
</span><span>('2009-09-06', 33)
</span><span>-------------------
</span></code></pre>
<p>The "things" list is a contrived example. In a real world situation this
could be a database cursor object or a CSV reader object. Any iterable
object can be used.</p>
<p>Here's a closer look at what groupby is doing using the Python
interactive shell:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>>>> iterator = groupby(things, itemgetter(0))
</span><span>>>> iterator
</span><span><itertools.groupby object at 0x95d3acc>
</span><span>>>> iterator.next()
</span><span>('2009-09-02', <itertools._grouper object at 0x95e0d0c>)
</span><span>>>> iterator.next()
</span><span>('2009-09-03', <itertools._grouper object at 0x95e0aec>)
</span></code></pre>
<p>You can see how a key and sub-iterator are returned for each pass
through the groupby iterator.</p>
<p>groupby is a handy tool to have under your belt. Think of it whenever
you need to split up a dataset by some criteria.</p>
Tiling addiction2012-10-02T00:00:00+00:002012-10-02T00:00:00+00:00Unknown/posts/tiling_addiction/<p>So I have this problem ... Well it's not really a problem - I can stop
whenever I want, really I can. My problem is that I have a thing for
<a href="http://en.wikipedia.org/wiki/Tiling_window_manager">tiling window
managers</a> (WMs)<sup class="footnote-reference"><a href="#1">1</a></sup>.
I love the efficient window management, keyboard focussed operation,
extensive customisability and lightweight feel that most tiling WMs
offer. For the X Window System <sup class="footnote-reference"><a href="#2">2</a></sup> there's an <a href="http://en.wikipedia.org/wiki/Tiling_window_manager#List_of_tiling_window_managers_for_X">awful lot to choose
from</a>,
and I've been obsessed for some time now with finding a great tiling WM
that works for me and then configuring it to perfection.</p>
<p>I've spent an embarrassing amount of time installing, learning about,
configuring and tinkering with tiling WMs. Over the last 6 years I've
tried out, with at least some seriousness, all of the following:</p>
<ul>
<li>various versions of
<a href="http://en.wikipedia.org/wiki/Ion_(window_manager)">Ion</a>: my first
love. The project is pretty well dead now due to the author having
various issues with the free software community.</li>
<li><a href="http://dwm.suckless.org/">dwm</a> (and <a href="http://0mark.unserver.de/projects/dwm-sprinkles/">various
forks</a>): Written
in C, configured by editing the C source. Super fast. An inspiration
for many other tiling WMs.</li>
<li><a href="http://aerosuidae.net/musca.html">Musca</a>: Fairly minimal. The
project hasn't been worked on for a long time.</li>
<li><a href="http://www.nongnu.org/ratpoison/">Ratpoison</a>: The closest thing to
GNU Screen for X Windows. I used it for ages.</li>
<li><a href="http://www.nongnu.org/stumpwm/">Stumpwm</a>: Ratpoison but re-written
by the same author in Common Lisp. Also very good.</li>
<li><a href="http://code.google.com/p/euclid-wm/">euclid-wm</a></li>
<li><a href="https://code.google.com/p/wmii/">wmii</a>: Very modular. Takes a lot
of ideas from the Plan 9 OS.</li>
<li><a href="http://xmonad.org/">xmonad</a>: Popular and mature. Written and
configured in Haskell. Requires a good grasp of Haskell in order to
do anything beyond the basics.</li>
<li><a href="http://qtile.org/">qtile</a>: A promising tiling WM written in Python.
I would have looked at it more seriously if I could get to it run on
the ancient Linux distro we use at work.</li>
<li><a href="http://awesome.naquadah.org/">awesome</a> (versions 2 and 3): I
happily used the stable and simple awesome 2 for ages but version 3
is definitely a big step up.</li>
</ul>
<p>These tiling WMs all have their pros and cons but I've ended up
settling<sup class="footnote-reference"><a href="#3">3</a></sup> on awesome (version 3) as it is stable (i.e. not someone's
project that was never finished and hasn't been touched in years),
highly customisable (almost everything can be changed using Lua), sane
to configure with sensible defaults, handles multiple monitors well and
lacks irritating bugs.</p>
<p>I use awesome both at home and at work, and until recently I'd been
maintaining two separate configurations. After being frustrated by the
process of repeatedly synchronising changes between the two
configurations, I decided to merge the two and clean up a bunch of stuff
at the same time.</p>
<p>The result can be seen on GitLab at:
<a href="https://gitlab.com/menn0/awesome-config">https://gitlab.com/menn0/awesome-config</a></p>
<p>My goals for the new configuration were:</p>
<ul>
<li><strong>A single configuration for multiple hosts</strong>: My work and home
awesome configurations are mostly the same, however at home I use a
laptop and at work I use a workstation with four monitors. The
configuration needs to be able to dynamically handle such
differences.</li>
<li><strong>Allow for per-host customisation where needed</strong>: At work, I need a
widget to monitor production environments as well as clocks for
other time zones. My laptop needs a battery widget. Obviously, the
way the panels at the top of each screen are put together is host
specific. My laptop and work machine also have quite different
keyboards so some bindings need to be different (although most
bindings are identical).</li>
<li><strong>Modularisation and abstraction</strong>: Most people's awesome
configurations involve a single lengthy, poorly organised Lua file
(rc.lua). After some organic evolution such a configuration tends
toward difficult-to-understand spaghetti code. I've also seen
configurations where the same patterns are copy-and-pasted over and
over - this screams out for abstraction. I wanted to apply some
basic software development principles because when you work on your
awesome configuration you're actually doing software development
<sup class="footnote-reference"><a href="#4">4</a></sup>.</li>
</ul>
<p>Some highlights from my configuration:</p>
<ul>
<li><strong>A very short rc.lua and many small modules</strong>: Instead of being a
huge, monolithic Lua file,
<a href="https://gitlab.com/menn0/awesome-config/-/blob/master/rc.lua">rc.lua</a>
simply imports a handful of modules which in turn end up using other
modules. This makes it easier to find things and to see the overall
structure.</li>
<li><strong>Per host configuration</strong>: <a href="https://gitlab.com/menn0/awesome-config/-/blob/master/config-bazhou.lua">Configuration
parameters</a>,
<a href="https://gitlab.com/menn0/awesome-config/-/blob/master/keys.lua">extra keyboard
bindings</a>
and the <a href="https://gitlab.com/menn0/awesome-config/-/blob/master/panel-bazhou.lua">panel
setup</a>
are loaded from different modules based on the local hostname. I've
written a simple helper function called
<a href="https://gitlab.com/menn0/awesome-config/-/blob/master/utils.lua">require_by_host</a>
to do the hostname-based dynamic loading of modules.</li>
<li><strong>Minimal global state</strong>: In Lua, variables and functions are global
by default so it's possible to introduce subtle bugs through
unintended shadowing or sharing of state. I've tried to minimise
this through the (hopefully correct) use of modules. I'm sure this
could be improved in some areas.</li>
</ul>
<p>Note that I'm by no means a Lua expert. I'd love to hear feedback on
what I've done from more experienced Lua developers and awesome users.</p>
<p>If you're new to tiling window managers, I hope this post has piqued
your interest (or at least hasn't put you off them!). If you're already
using awesome, I hope I may have encouraged you to re-think your
configuration. I know I'm obsessive about these things, but I think
there's much to gain from optimising the way you interact with the
software you use the most, and your window manager certainly meets that
criteria.</p>
<p><strong>Update</strong>: As requested, a screenshot of how my config looks (on my
laptop).</p>
<img src="/images/awesome.png" width="400" height="250" />
<p>The panel at the top has from left to right: a button for the
application launcher menu, the tag (workspace) list, a window list for
the current tag, CPU and memory usage graphs, a battery widget, volume
widget, system tray (containing the Dropbox and Network Manager icons),
a clock and a awesome layout indicator applet. Most/all of the UI
elements are implemented in Lua.</p>
<p>Three application windows are shown in the tag, organised using the
"tile" layout. Chrome in the "master" position with Emacs and a shell
off to the right.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>If you're unsure what tiling window managers are about, or why
you'd considering using one, there are a
<a href="http://superuser.com/questions/52082/why-use-a-tiling-window-manager">few</a>
<a href="http://tychoish.com/rhizome/why-tiling-window-managers-matter/">useful</a>
<a href="http://tuomov.iki.fi/software/">pages</a> around.</p>
</div>
<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
<p>I use Linux almost exclusively.</p>
</div>
<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
<p>By "settled" I mean, "have used exclusively for about a year now".</p>
</div>
<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
<p>The same goes for your <a href="http://ola-bini.blogspot.co.uk/2008/05/how-large-is-your-emacs.html">editor
configuration</a>,
but that's another story.</p>
</div>
Somewhat over-engineered blinking LEDs2012-07-21T00:00:00+00:002012-07-21T00:00:00+00:00Unknown/posts/overengineered_blinking_leds/<p>My <a href="http://www.raspberrypi.org/">Raspberry Pi</a> arrived a couple of weeks
ago and I've been working on turning it into a mini-audio server to
connect to the home stereo in the living room.</p>
<p>As part of the project I'd like to drive an analog <a href="http://en.wikipedia.org/wiki/VU_meter">VU
meter</a> from the sound signal.</p>
<p>This week my (enthusiastic!) wife and I played around with attaching
some basic electronics to the GPIO ports so that we could get more
comfortable the Raspberry Pi. Our electronics knowledge is more than a
little rusty but surprisingly everything we tried worked first go.</p>
<p>On the first night we attached a LED in line with a resistor directly to
a GPIO port and were able to programmatically turn it on and off. Easily
done.</p>
<p>Then we took that a little further by using a transistor to switch the
LED from the Pi's 5V power supply. This is a better option because the
circuit can be arranged so that a minimal current is pulled from the
GPIO pin but a higher current can be put through the LED to give a nice
bright light. The amount of current you can put through the GPIO pins
without damaging the Pi is limited so this is a safer option (although
not strictly needed for a single LED). There's an <a href="http://elinux.org/RPi_Tutorial_EGHS:LED_output#Circuit_2_-_LED_Driving_Circuit_.28using_Transistor_Switching_Circuit.29">excellent
page</a>
on elinux.org which explains this arrangment.</p>
<p>Here's the result (sorry about the dodgy video quality):</p>
<div class="youtube-container">
<iframe
src="https://www.youtube-nocookie.com/embed/4Ba8csSsaVA"
allow="encrypted-media"
frameborder="0"
webkitallowfullscreen
mozallowfullscreen
allowfullscreen>
</iframe>
</div>
<p>The Python code driving the LED looks like:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>itertools
</span><span style="color:#b48ead;">import </span><span>time
</span><span style="color:#b48ead;">import </span><span>RPi.</span><span style="color:#bf616a;">GPIO </span><span style="color:#b48ead;">as </span><span style="color:#bf616a;">GPIO
</span><span>
</span><span style="color:#65737e;"># to use Raspberry Pi board pin numbers
</span><span>GPIO.</span><span style="color:#bf616a;">setmode</span><span>(</span><span style="color:#bf616a;">GPIO</span><span>.</span><span style="color:#bf616a;">BCM</span><span>)
</span><span>
</span><span style="color:#65737e;"># set up GPIO output channel
</span><span>GPIO.</span><span style="color:#bf616a;">setup</span><span>(</span><span style="color:#d08770;">17</span><span>, </span><span style="color:#bf616a;">GPIO</span><span>.</span><span style="color:#bf616a;">OUT</span><span>)
</span><span>
</span><span>pin_values = itertools.</span><span style="color:#bf616a;">cycle</span><span>([</span><span style="color:#bf616a;">GPIO</span><span>.</span><span style="color:#bf616a;">HIGH</span><span>, </span><span style="color:#bf616a;">GPIO</span><span>.</span><span style="color:#bf616a;">LOW</span><span>])
</span><span>
</span><span style="color:#b48ead;">for </span><span>pin_value </span><span style="color:#b48ead;">in </span><span>pin_values:
</span><span> GPIO.</span><span style="color:#bf616a;">output</span><span>(</span><span style="color:#d08770;">17</span><span>, pin_value)
</span><span> time.</span><span style="color:#bf616a;">sleep</span><span>(</span><span style="color:#d08770;">1</span><span>)
</span></code></pre>
<p>On the second night we duplicated this circuit 5 times to drive 5 LEDs:</p>
<div class="youtube-container">
<iframe
src="https://www.youtube-nocookie.com/embed/ADx86lg6DwI"
allow="encrypted-media"
frameborder="0"
webkitallowfullscreen
mozallowfullscreen
allowfullscreen>
</iframe>
</div>
<p>We were quite chuffed that we managed to pack everything neatly into one
end of the breadboard so that all the LEDs were in a line.</p>
<p>The plan now is to get
<a href="http://en.wikipedia.org/wiki/PulseAudio">PulseAudio</a> working. It
provides a nice way to intercept the sound going to the audio output. It
should be possible to use that to drive these LEDs like the lights on a
retro 80's hi-fi system.</p>
<p>And after comes driving an analog meter which will require
digital-to-analog conversion either by using the PWM channels or an
external DAC chip. More on that to come.</p>
IMAPClient 0.9 released2012-05-17T00:00:00+00:002012-05-17T00:00:00+00:00Unknown/posts/IMAPClient-0.9/<p>I'm pleased to announce version 0.9 of IMAPClient, the easy-to-use and
Pythonic IMAP client library.</p>
<p>Highlights for this release:</p>
<ul>
<li>Support for Gmail's label API. Thanks to Brian Neal for the patches
for this.</li>
<li>Significant cleanup and refactoring in preparation for Python 3
compatibility.</li>
<li>The "livetest" module can now be safely used against IMAP accounts
with real data. Previously it could only be used with dummy accounts
due to the destructive nature of the tests.</li>
<li>Fixed handling of IMAP servers that return all-digit folder name
without quotes. Thanks to Rhett Garber for the bug report.</li>
<li>Much improved test coverage (again, in preparation for Python 3
support)</li>
<li>Fixed rename live test so that it uses folder namespaces</li>
<li>Parse STATUS responses robustly - fixes folder_status() with MS
Exchange.</li>
<li>Numerous livetest fixes to work around oddities with the MS Exchange
IMAP implementation.</li>
</ul>
<p>The <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a> file and
<a href="http://imapclient.readthedocs.org/">manual</a> have more details on all of
the above.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
<p>The main focus of the next release (0.10) will be Python 3 support as
this is easily the most requested feature. Watch this space for more
news on this.</p>
Introducing elemental2012-04-08T00:00:00+00:002012-04-08T00:00:00+00:00Unknown/posts/introducting_elemental/<p>The first release of my first Elisp project is out. The project is
called "elemental" and the intro from the README goes:</p>
<blockquote>
<p>elemental is a set of Emacs Lisp functions for intelligently jumping
between and transposing list/tuple/dictionary/function-parameter
elements. These functions are primarily useful when editing software
source code.</p>
</blockquote>
<p>It's probably easier to get an understanding what it does by
demonstration so I've uploaded a <a href="http://www.youtube.com/watch?v=V9MRMm884mo">quick
screencast</a> to Youtube.</p>
<p>The project is hosted on GitHub: <a href="https://github.com/mjs/elemental">https://github.com/mjs/elemental</a></p>
<p>Feedback and patches welcome!</p>
IMAPClient 0.8.1 released2012-01-12T00:00:00+00:002012-01-12T00:00:00+00:00Unknown/posts/IMAPClient-0.8.1/<p>Version 0.8.1 of IMAPClient has just been released. This version works
around a subtle bug in distutils which was preventing installation on
Windows from working. Thanks to Bob Yexley for the bug report.</p>
<p>This release also contains a few small documentation updates and
packaging fixes. The <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a>
file has more details.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
IMAPClient 0.8 released2011-11-10T00:00:00+00:002011-11-10T00:00:00+00:00Unknown/posts/IMAPClient-0.8/<p>Version 0.8 of IMAPClient is out! Although I didn't get everything into
this release that I had hoped to, there's still plenty there. Thanks to
Johannes Heckel and Andrew Scheller for their contributions to this
release.</p>
<p>Highlights for 0.8:</p>
<ul>
<li>OAUTH authentication support</li>
<li>IDLE support
(<a href="http://imapclient.freshfoo.com/browser/imapclient/examples/idle_example.py">example</a>)</li>
<li>Full NOOP support</li>
<li>Comprehensive Sphinx based docs. A HTML version of the docs is
included in the source distribution. They are also hosted online at
<a href="http://imapclient.readthedocs.org/">http://imapclient.readthedocs.org/</a></li>
<li>Folder rename support</li>
<li>New "debug" property to simplify protocol debugging</li>
<li>Live test and interactive shell features are now part of the
imapclient Python package and can be called from the command-line.
(For example: <code>python -m imapclient.interact ...</code>)</li>
<li>New normalise_times attribute allows caller to select whether
datetimes returned by fetch() are native or not</li>
<li>interactive shell now works with both IPython 0.10 and 0.11 (and
later)</li>
<li>BODY/BODYSTRUCTURE parsing fixes</li>
<li>Programmer friendly version information now available
(imapclient.version_info)</li>
</ul>
<p>The <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a> file and <a href="http://imapclient.readthedocs.org/">main
documentation</a> has more details on
all of the above.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
Fibonacci Quilt2011-03-13T00:00:00+00:002011-03-13T00:00:00+00:00Unknown/posts/fib-quilt/<p>A quilt incorporating the starting numbers of the <a href="http://en.wikipedia.org/wiki/Fibonacci_number">Fibonacci
sequence</a>. Designed by
me, expertly crafted by Susanna and adorably used by Amelia.</p>
<a href="/images/quilt1.jpg">
<img src="/images/quilt1-thumb.jpg" width="250" />
</a>
<a href="/images/quilt2.jpg">
<img src="/images/quilt2-thumb.jpg" width="250" />
</a>
Domain renewal scam2011-02-18T00:00:00+00:002011-02-18T00:00:00+00:00Unknown/posts/domain-renewal-group/<div>
<a href="/images/domain-renewal-group-scaled.png">
<img alt="Domain Renewal Group letter" src="/images/domain-renewal-group-thumb.png" width="124" height="200" style="float: left; padding-right:0.8em" />
</a>
<p>My domain is up for renewal soon and I recently received a <a href="/images/domain-renewal-group-scaled.png">very
official looking letter</a> from a
company called Domain Renewal Group. On the surface it looks like a
renewal notice from my registrar but if you read more closely mention
that the letter "isn't an invoice" and that if you return the form and
pay them you'd be transferring the domain from your current registrar to
them. Never mind that the price they're asking is almost 5 times higher
than what my actual register wants for an annual renewal.</p>
<p>I bet this catches plenty of people out - that's what their business
model depends on. Bottom feeding scum.</p>
<p><a href="http://www.google.com/search?q=domain+renewal+group+scam">Plenty of
others</a> have
blogged about these guys as well, but I thought I'd add my voice. As
more people learn about these kinds of scams they become less effective.</p>
</div>
IMAPClient 0.7 released2011-02-10T00:00:00+00:002011-02-10T00:00:00+00:00Unknown/posts/IMAPClient-0.7/<p>The next version of IMAPClient was quietly released on the weekend. I've
been pretty busy so I'm just getting to telling the world about it now.</p>
<p>This release is earlier then planned because IMAPClient is featured in
the IMAP chapter of <a href="http://rhodesmill.org/brandon/">Brandon Rhodes</a>'
new edition of the book <a href="http://www.amazon.co.uk/Foundations-Python-Network-Programming-Comprehensive/dp/1430230037">Foundations of Python Network
Programming</a>.
Brandon had made several bug reports and feature requests while working
on the book and some of the examples in the book relied on unreleased
changes to IMAPClient. IMAPClient 0.7 works with book's examples.</p>
<p>What this does mean is that IMAPClient 0.7 doesn't have Python 3
support. I have been making headway with this however and with a little
luck and a little more time, 0.8 should be Python 3 compatible.</p>
<p>Highlights for 0.7:</p>
<ul>
<li>BODY and BODYSTRUCTURE FETCH response parsing was fixed. Previously,
the object returned for multipart messages was difficult to deal
with reliably (and not really correct). Existing code that makes
BODY/BODYSTRUCTURE FETCH requests with IMAPClient may need to be
updated.</li>
<li>the live test scripts and the unit tests have been converted to use
the excellent <a href="http://pypi.python.org/pypi/unittest2">unittest2
framework</a>. This makes
writing tests easier, improves test failure output and, for the live
tests, provides significant improvement over the previous crude test
script.</li>
<li>livetest.py now takes an INI file for the IMAP account parameters to
test against instead of requiring that they are passed on command
line.</li>
<li>support for the NAMESPACE command IMAP added</li>
<li>fetch() now supports optional modifiers. These are required for
extensions such as RFC 4551 (conditional store). Thanks to Thomas
Jost for the patch.</li>
<li>the live tests now work with the FastMail.fm IMAP implementation.
Some special casing was required for some tests.</li>
</ul>
<p>The <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a> document has
more details on all the above.</p>
<p>Proper documentation is also on its way. I've been slowly making headway
with Sphinx based documentation.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
IMAPClient 0.6.2 is out2010-09-18T00:00:00+00:002010-09-18T00:00:00+00:00Unknown/posts/IMAPClient-0.6.2/<p>This release fixes some parsing issues when dealing with FETCH items
that include square brackets (eg. "BODY[HEADER.FIELDS (From
Subject)]") and includes the example when the package is installed
using PyPI.</p>
<p>Also, IMAPClient now uses the excellent
<a href="http://packages.python.org/distribute/">Distribute</a> instead of
setuptools.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com/">IMAPClient
site</a>.</p>
<p>In the pipeline now is better (Sphinx based) documentation, cleaning up
the tests, OAUTH support and Python 3 support.</p>
IMAPClient 0.6.1 released2010-09-03T00:00:00+00:002010-09-03T00:00:00+00:00Unknown/posts/IMAPClient-0.6.1/<p>I've just released IMAPClient 0.6.1.</p>
<p>The only functional change in the release is that it now automatically
patches imaplib's IMAP4_SSL class to fix <a href="http://bugs.python.org/issue5949">Python Issue
5949</a>. This is a bug that's been fixed
in later Python 2.6 versions and 2.7 but still exists in Python versions
that are in common use. Without fix this you may experience hangs when
using SSL.</p>
<p>The patch is only applied if the running Python version is known to be
one of the affected versions. It is applied when IMAPClient is imported.</p>
<p>The only other change in this release is that I've now marked IMAPClient
as "production ready" on PyPI and have updated the README to match. This
was prompted by a request to clarify the current status of the project
and seeing that all current functionality is solid and, I don't plan to
change the existing APIs in backwards-incompatible ways, I've decided to
indicate the project as suitable for production use.</p>
<p>As always, IMAPClient can be installed from PyPI
(<code>pip install imapclient</code>) or downloaded from the <a href="http://imapclient.freshfoo.com//">IMAPClient
site</a>. Feedback, bug reports and
patches are most welcome.</p>
IMAPClient 0.6 released2010-05-12T00:00:00+00:002010-05-12T00:00:00+00:00Unknown/posts/IMAPClient-0.6/<p>IMAPClient 0.6 is finally out! Highlights:</p>
<ul>
<li>Completely new response parsing code. Complex response items like
BODYSTRUCTURE and ENVELOPE are now handled properly and interaction
with the Gmail and MS Exchange IMAP implementations works correctly.</li>
<li>Support for the COPY command</li>
<li>Support for the XLIST extension (used by Gmail)</li>
<li>select_folder() now returns a parsed response containing all
reported details about the selected folder.</li>
<li>the return value from list_folders(), list_sub_folders() and
xlist_folders() now include the IMAP folder flags and folder
delimiter.</li>
<li>Handling of internationalised folder names has been cleaned up.
Folder names now are always returned as unicode strings.</li>
<li>Many bug fixes.</li>
</ul>
<p>Be aware that there have been several API changes with this release. See
the <a href="http://imapclient.freshfoo.com/browser/NEWS">NEWS</a> file for further
details.</p>
<p>IMAPClient can be installed from PyPI (<code>pip install imapclient</code>) or
downloaded from the <a href="http://imapclient.freshfoo.com//">IMAPClient site</a>.
As always, feedback, bug reports and patches are most welcome.</p>
<p>Special thanks goes to <a href="http://starship.python.net/~skippy/">Mark
Hammond</a>. He has contributed a
significant amount of code and effort to this release. Incidentally,
Mark is using IMAPClient as part of the backend for the
<a href="https://mozillalabs.com/raindrop">Raindrop</a> Mozilla Labs project.</p>
TypeError: object.__init__() takes no parameters2010-01-31T00:00:00+00:002010-01-31T00:00:00+00:00Unknown/posts/object__init__takes_no_parameters/<p>At my employer we are in the process of migrating from Python 2.4 to
2.6. When running some existing code under Python 2.6 we started getting
DeprecationWarnings about "object.__new__() takes no parameters" and
"object.__init__() takes no parameters".</p>
<p>A simple example that triggers the warning:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">MyClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__new__</span><span>(</span><span style="color:#bf616a;">cls</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MyClass.__new__</span><span>', a, b
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#96b5b4;">super</span><span>(MyClass, </span><span style="color:#bf616a;">cls</span><span>).</span><span style="color:#96b5b4;">__new__</span><span>(</span><span style="color:#bf616a;">cls</span><span>, a, b)
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MyClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(MyClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>(a, b)
</span><span>
</span><span>obj = </span><span style="color:#bf616a;">MyClass</span><span>(</span><span style="color:#d08770;">6</span><span>, </span><span style="color:#d08770;">7</span><span>)
</span></code></pre>
<p>This gives:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ python2.4 simple-warning.py
</span><span>MyClass.__new__ 6 7
</span><span>MyClass.__init__ 6 7
</span><span>
</span><span>$ python2.6 simple-warning.py
</span><span>MyClass.__new__ 6 7
</span><span>simple-warning.py:5: DeprecationWarning: object.__new__() takes no parameters
</span><span> return super(MyClass, cls).__new__(cls, a, b)
</span><span>MyClass.__init__ 6 7
</span><span>simple-warning.py:9: DeprecationWarning: object.__init__() takes no parameters
</span><span> super(MyClass, self).__init__(a, b)
</span></code></pre>
<p>It turns out that a <a href="http://svn.python.org/view?view=rev&revision=54539">change to
Python</a> for 2.6 (and
3) means that object.__new__ and object.__init__ no longer take
arguments - a TypeError is raised when arguments are passed. To avoid
breaking too much pre-existing code, there is a special case that will
cause a DeprecationWarning instead of TypeError if both __init__ and
__new__ are overridden. This is the case we were running into with
our code at work.</p>
<p>The reason for this change seems to make enough sense: object doesn't do
anything with arguments to __init__ and __new__ so it shouldn't
accept them. Raising an error when arguments are passed highlights code
where the code might be doing the wrong thing.</p>
<p>Unfortunately this change also breaks Python's multiple inheritance in a
fairly serious way when cooperative
<a href="http://docs.python.org/library/functions.html#super">super</a> calls are
used. Looking at the <a href="http://bugs.python.org/issue1683368">ticket</a> for
this change, this issue was thought about but perhaps the implications
were not fully understood. Given that using super with multiple
inheritance is common and "correct" practice, it would seem that this
change to Python is a step backwards.</p>
<p>Let's look at an even simpler example, one that triggers a TypeError
under Python 2.6.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">MyClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MyClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(MyClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>(a, b)
</span><span>
</span><span>obj = </span><span style="color:#bf616a;">MyClass</span><span>(</span><span style="color:#d08770;">6</span><span>, </span><span style="color:#d08770;">7</span><span>)
</span></code></pre>
<p>The output looks like:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ python2.4 simple.py
</span><span>MyClass.__init__ 6 7
</span><span>
</span><span>$ python2.6 simple.py
</span><span>MyClass.__init__ 6 7
</span><span>Traceback (most recent call last):
</span><span> File "simple.py", line 7, in <module>
</span><span> obj = MyClass(6, 7)
</span><span> File "simple.py", line 5, in __init__
</span><span> super(MyClass, self).__init__(a, b)
</span><span>TypeError: object.__init__() takes no parameters
</span></code></pre>
<p>The fix would seem to be to not pass the arguments to
object.__init__:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">MyClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MyClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(MyClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>()
</span><span>
</span><span>obj = </span><span style="color:#bf616a;">MyClass</span><span>(</span><span style="color:#d08770;">6</span><span>, </span><span style="color:#d08770;">7</span><span>)
</span></code></pre>
<p>On the surface, this deals with the issue:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ python2.6 simple-fixed.py
</span><span>MyClass.__init__ 6 7
</span></code></pre>
<p>But what about when multiple inheritance is brought into the picture?</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">MyClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MyClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(MyClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>()
</span><span>
</span><span>
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">AnotherClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">object</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">AnotherClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(AnotherClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>()
</span><span>
</span><span>
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">MultiClass</span><span style="color:#eff1f5;">(</span><span style="color:#a3be8c;">MyClass</span><span style="color:#eff1f5;">, </span><span style="color:#a3be8c;">AnotherClass</span><span style="color:#eff1f5;">):
</span><span>
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#96b5b4;">__init__</span><span>(</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">a</span><span>, </span><span style="color:#bf616a;">b</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">MultiClass.__init__</span><span>', a, b
</span><span> </span><span style="color:#96b5b4;">super</span><span>(MultiClass, </span><span style="color:#bf616a;">self</span><span>).</span><span style="color:#96b5b4;">__init__</span><span>(a, b)
</span><span>
</span><span>obj = </span><span style="color:#bf616a;">MultiClass</span><span>(</span><span style="color:#d08770;">6</span><span>, </span><span style="color:#d08770;">7</span><span>)
</span></code></pre>
<p>Things don't go so well:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ python2.6 problem.py
</span><span>MultiClass.__init__ 6 7
</span><span>MyClass.__init__ 6 7
</span><span>Traceback (most recent call last):
</span><span> File "problem.py", line 21, in <module>
</span><span> obj = MultiClass(6, 7)
</span><span> File "problem.py", line 19, in __init__
</span><span> super(MultiClass, self).__init__(a, b)
</span><span> File "problem.py", line 5, in __init__
</span><span> super(MyClass, self).__init__()
</span><span>TypeError: __init__() takes exactly 3 arguments (1 given)
</span></code></pre>
<p>Not passing the arguments to the super call in MyClass.__init__
means that AnotherClass.__init__ is called with the wrong number of
arguments. This can be "fixed" by putting the arguments to the
__init__ calls back in but then these classes can no longer be used
by themselves.</p>
<p>Another "fix" could be to remove the super call in MyClass.__init__
-after all, object.__init__ doesn't do anything. This avoids the
TypeError but also means that AnotherClass.__init__ isn't called at
all! This isn't right either.</p>
<p>What is the correct way to deal with this situation in Python 2.6+? Wise
developers avoid multiple inheritance where possible but it is sometimes
the best solution to a problem. If Python claims to support multiple
inheritance then it should do so in a way that the language feature that
accompanies it (super) works. This change to the behaviour of
object.__init__ and object.__new__ means that super can't really
be used with class hierarchies that involves multiple inheritance.</p>
<p>Should this get fixed or am I missing something?</p>
<p>(Thanks to Christian Muirhead for helping with investigations into this
issue)</p>
rst_break plugin for PyBlosxom2010-01-31T00:00:00+00:002010-01-31T00:00:00+00:00Unknown/posts/rst_break-plugin/<p>I just scratched an itch by writing a small plugin for PyBlosxom that
allows the
<a href="http://pyblosxom.sourceforge.net/registry/text/rst.html">rst</a>
(<a href="http://docutils.sourceforge.net/rst.html">reStructured Text</a>) and
<a href="http://pyblosxom.sourceforge.net/registry/display/readmore.html">readmore</a>
plugins to work together<sup class="footnote-reference"><a href="#1">1</a></sup>. It defines a reST "break" directive which
gets transformed into the breakpoint string the readmore plugin looks
out for. This allows for "Read more..." breaks to be inserted in for
reST based articles.</p>
<p>For further information see the <a href="/code/">Code</a> page here and at the top
of the
<a href="/hg/pyblosxom-plugins/trunk/raw-file/9afdcd2bf738/rst_break/rst_break.py">plugin</a>
itself.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Yes, the audience for this plugin is probably tiny!</p>
</div>
IMAPClient 0.5.22010-01-25T00:00:00+00:002010-01-25T00:00:00+00:00Unknown/posts/IMAPClient-0.5.2/<p>IMAPClient 0.5.2 has just been released. This release fixes 2 bugs
(<a href="http://imapclient.freshfoo.com/ticket/28">#28</a> and
<a href="http://imapclient.freshfoo.com/ticket/33">#33</a>). Much thanks to Fergal
Daly and Mark Eichin for reporting these bugs.</p>
<p>Install from the <a href="/projects/IMAPClient/IMAPClient-0.5.2.tar.gz">tarball</a>
or <a href="/projects/IMAPClient/IMAPClient-0.5.2.zip">zip</a> or upgrade using
easy_install or pip.</p>
A little thing about cron2010-01-13T00:00:00+00:002010-01-13T00:00:00+00:00Unknown/posts/crontab-updates/<p>Here's something I just learned the hard way.</p>
<p>If you edit a crontab with "crontab -e", cron won't reload the updated
crontab immediately. Changes will be read at 1 second past the next
minute boundary. For example, if you change the crontab at 10:54:32,
cron will reload it at 10:55:01. This means if you're trying to test how
something runs under cron and you're impatient so you set that thing to
run at the next minute, you won't see it run!</p>
<p>I spent a good half hour chasing my tail on this one. Set the test entry
to run 2 minutes ahead instead.</p>
IMAPClient Trac instance now allows for user registrations2009-12-27T00:00:00+00:002009-12-27T00:00:00+00:00Unknown/posts/IMAPClient-Trac_AccountManagerPlugin/<p>I've had several requests over the last few weeks to open up the
IMAPClient Trac instance so that anyone can submit tickets. Initially I
changed access levels so that anoymous users could create tickets. This
turned out to be fairly inflexible: it doesn't allow people to add
attachments or modify tickets later. This approach also resulted in one
strange ticket being created where all fields were filled with random
characters - a bot looking for buffer overruns?</p>
<p>Since then, I've disabled anonymous ticket creation and have set up the
fantastic
<a href="http://trac-hacks.org/wiki/AccountManagerPlugin">AccountManagerPlugin</a>
which allows people to register accounts for themselves. Once someone
has created an account and logged in they can create and modify tickets.
I have a feeling I'm going to have to turn on the optional CAPTCHA
support, but I'm willing to see how it goes for a while first.</p>
IMAPClient 0.5.12009-12-16T00:00:00+00:002009-12-16T00:00:00+00:00Unknown/posts/IMAPClient-0.5.1/<p>I've just made a quick point release of IMAPClient. <a href="http://python.net/crew/skippy/">Mark
Hammond</a> is interested in using it for a
<a href="https://mozillalabs.com/raindrop">project</a> he's working on but the
licenses (GPL and MPL) were incompatible. I was thinking about <a href="http://imapclient.freshfoo.com/ticket/8">relaxing
the license</a> of IMAPClient
anyway so this presented a good opportunity to make the switch.</p>
<p>Work on the 0.6 release is coming along. This version will fix a number
issues with parsing of FETCH responses - the FETCH parsing code is being
completely rewritten. This is the first time that IMAPClient will bypass
most of <a href="http://docs.python.org/library/imaplib.html">imaplib</a> for some
functionality. It's looking like that at some point IMAPClient may not
use imaplib at all.</p>
<p>IMAPClient can be installed from PyPI using <code>pip install IMAPClient</code> or
<code>easy_install IMAPClient</code>. It can also be downloaded from the
<a href="http://imapclient.freshfoo.com">IMAPClient project page</a>. As always,
feedback and patches are most welcome.</p>
PyBlosxom to Disqus import script2009-11-26T00:00:00+00:002009-11-26T00:00:00+00:00Unknown/posts/disqus-import-script/<p>I've had a few requests for the little hack I created to import comments
from PyBlosxom into Disqus. A cleaned up version of disqus-import.py is
now on the <a href="/code/">Code</a> page. There's some docs at the top of the
file.</p>
Setting PYTHON_EGG_CACHE when deploying Python apps using FastCGI2009-11-20T00:00:00+00:002009-11-20T00:00:00+00:00Unknown/posts/fastcgi-python-egg-cache/<p>I recently sorted out an issue with the
<a href="http://imapclient.freshfoo.com/">IMAPClient</a> Trac instance that's been
bugging me for a while.</p>
<p>The problem was that whenever the web server logs were rotated logrotate
would restart <a href="http://www.lighttpd.net/">Lighttpd</a>. The web server
restart would in turn restart the Trac (FastCGI) processes.
Unfortunately, the Trac processes would fail to start with the following
error.</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>pkg_resources.ExtractionError: Can't extract file(s) to egg cache
</span><span>
</span><span>The following error occurred while trying to extract file(s) to the Python egg
</span><span>cache:
</span><span>
</span><span> [Errno 13] Permission denied: '/root/.python-eggs'
</span><span>
</span><span>The Python egg cache directory is currently set to:
</span><span>
</span><span> /root/.python-eggs
</span></code></pre>
<p>Bang, no <a href="http://imapclient.freshfoo.com/">IMAPClient</a> web site (the
rest of the site was ok). To band-aid the problem when it happened (and
I noticed!) I issue a <code>sudo /etc/init.d/lighttpd restart</code> and everything
would be fine again.</p>
<p>After some investigation I found that running
<code>/etc/init.d/lighttpd restart</code> as root always triggered the problem
where-as restarting using sudo always worked. My guess is that
restarting when logged in as root was leaving $HOME at /root even after
Lighttpd had dropped to its unprivileged user account. The unprivileged
user isn't allowed to write to /root so Trac blows up. setuptools seems
to use $HOME instead of looking up the actual home directory of the
current user.</p>
<p>The fix for me was to set the PYTHON_EGG_CACHE environment variable
for the FastCGI processes to somewhere they are allowed to write to.
This is done with the bin-environment option if you're using Lighttpd
like me.</p>
<p>I imagine similar problems can happen with any Python app deployed using
FastCGI.</p>
Why I chose Disqus Comments2009-11-06T00:00:00+00:002009-11-06T00:00:00+00:00Unknown/posts/disqus/<p>I recently moved my blog comments away the
<a href="http://pyblosxom.sourceforge.net/">PyBlosxom</a>
<a href="http://pyblosxom.sourceforge.net/registry/input/comments/comments.html">comments</a>
plugin to a hosted system. The main driver was the ability for people to
subscribe to comments for an article using email or RSS. It's a pain for
people to have to check back to the site to see if someone has replied
to their comments. I was also keen on user-experience-enhancing features
such as integration with external systems like OpenID, Twitter and
Yahoo.</p>
<p>My criteria were:</p>
<ul>
<li>email subscription, RSS a bonus</li>
<li>support for pre-formatted text sections in comments (essential for
code samples)</li>
<li>an import mechanism for existing comments</li>
<li>threading of comments to allow commenters to respond to each other
sensibly</li>
<li>clean look with some ability to customise</li>
<li>support for a variety of authentication/profile systems</li>
</ul>
<p>There are a number of hosted comment systems out there. The most popular
options seem to be <a href="http://disqus.com/">Disqus</a>, <a href="http://js-kit.com/">JS-Kit
Echo</a> and
<a href="http://www.intensedebate.com/">IntenseDebate</a>.</p>
<p>IntenseDebate was eliminated first because it doesn't seem to provide an
import mechanism for custom websites. Import only seems to be supported
for well known blog platforms such as Wordpress. There's no comment API
either. The approach seems to be to leave your old comment system in
place and just have new comments go into IntenseDebate. Not good enough,
I wanted to completely replace the existing comments system.</p>
<p>After some deliberation I decided on JS-Kit Echo for one tiny reason: it
supports the <pre> tag. The closest Disqus supported was the
<code> tag which doesn't preserve white-space (useless for Python
code samples).</p>
<p>So I paid my US$12 (it's the only service that doesn't have a free
option) and started looking at how to import my existing comments using
their API ... and quickly found that it sucks. Comments can be submitted
but you can't specify a timestamp so they are dated with the import
date. Far from ideal. Then there's the API for retrieving comments: it
returns the data as JavaScript code (no not JSON)! It's pretty clear
that the API is what they use with the JavaScript for Echo itself and
geared for that use only. They've just thrown it out there and
documented it, warts and all.</p>
<p>Back to the drawing board.</p>
<p>The only showstopper for Disqus was the lack of <pre>. Everything
else about it was great: it met all my requirements and the API was
clean and comprehensive. If only there was a way to have properly
formatted source code in the comments.</p>
<p>Light bulb moment: use a CSS hack to make <code> in comments
behave like <pre>. The trick is to turn code into a block element
and change how white-space is handled. The CSS snippet looks like:</p>
<pre data-lang="css" style="background-color:#2b303b;color:#c0c5ce;" class="language-css "><code class="language-css" data-lang="css"><span style="color:#8fa1b3;">.</span><span style="color:#d08770;">dsq-comment-message </span><span style="color:#bf616a;">code </span><span>{
</span><span> display:block;
</span><span> white-space:pre;
</span><span>}
</span></code></pre>
<p>Works great.</p>
<p>With the only blocker gone, I wrote a Python script with the help of Ian
Lewis' excellent
<a href="http://code.google.com/p/disqus-python-client/">disqus-python-client</a>
package to pull in the existing comments from the old system. Within an
hour or so it was ready to go.</p>
<p>Hopefully this article saves someone else some time if they decide to
use one of these systems. Getting things running chewed up a lot more
time then I had expected.</p>
IMAPClient has a new home2009-10-19T00:00:00+00:002009-10-19T00:00:00+00:00Unknown/posts/new-imapclient-site/<p>I've just (finally) finished setting up a proper website for IMAPClient.
The new home for the project is <a href="http://imapclient.freshfoo.com/">http://imapclient.freshfoo.com/</a>.</p>
<p>It's a Trac instance with Mercurial support that monitors the main trunk
repository. All items from the TODO file in the source have been
converted to tickets in the bug tracker. I've even created a hokey
little logo.</p>
<p>Let me know me know if anything looks broken.</p>
<p>Time to work on some long-standing bugs...</p>
Problem installing Trac under Lighttpd+FastCGI2009-08-28T00:00:00+00:002009-08-28T00:00:00+00:00Unknown/posts/trac_fastcgi_bug/<p>I just finished configuring a <a href="http://imapclient.freshfoo.com">Trac
instance</a> for IMAPClient. To keep page
response times tolerable I'm using FastCGI.</p>
<p>It turns out there's a little gotcha when you serve an app at the root
(ie. "/") using FastCGI with Lighttpd. It passes the wrong SCRIPT_NAME
and PATH_INFO to the app causing unexpected behaviour. The problem
isn't there if the app is served from a prefix (eg. "/trac").</p>
<p>The problem is apparently fixed somewhere in Lighttpd 1.5 but I'm on
1.4.13 as shipped with Debian.</p>
<p>With Trac there is a workaround. If you modify Trac's FastCGI runner
(fcgi_frontend.py) to correct the SCRIPT_NAME and PATH_INFO
environment variables before the rest of the app sees them, the problem
is solved. Here's what my fcgi_frontend.py now looks like (header
comments removed):</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>pkg_resources
</span><span>
</span><span style="color:#b48ead;">from </span><span>trac </span><span style="color:#b48ead;">import </span><span>__version__ </span><span style="color:#b48ead;">as </span><span style="color:#bf616a;">VERSION
</span><span style="color:#b48ead;">from </span><span>trac.web.main </span><span style="color:#b48ead;">import </span><span>dispatch_request
</span><span>
</span><span style="color:#b48ead;">from </span><span>os </span><span style="color:#b48ead;">import </span><span>environ
</span><span style="color:#b48ead;">import </span><span>_fcgi
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">application</span><span>(</span><span style="color:#bf616a;">environ</span><span>, </span><span style="color:#bf616a;">start_request</span><span>):
</span><span> environ['</span><span style="color:#a3be8c;">PATH_INFO</span><span>'] = environ['</span><span style="color:#a3be8c;">SCRIPT_NAME</span><span>'] + environ['</span><span style="color:#a3be8c;">PATH_INFO</span><span>']
</span><span> environ['</span><span style="color:#a3be8c;">SCRIPT_NAME</span><span>'] = ''
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#bf616a;">dispatch_request</span><span>(environ, start_request)
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">run</span><span>():
</span><span> _fcgi.</span><span style="color:#bf616a;">WSGIServer</span><span>(application).</span><span style="color:#bf616a;">run</span><span>()
</span><span>
</span><span style="color:#b48ead;">if </span><span>__name__ == '</span><span style="color:#a3be8c;">__main__</span><span>':
</span><span> pkg_resources.</span><span style="color:#bf616a;">require</span><span>('</span><span style="color:#a3be8c;">Trac==</span><span style="color:#d08770;">%s</span><span>' % </span><span style="color:#bf616a;">VERSION</span><span>)
</span><span> </span><span style="color:#bf616a;">run</span><span>()
</span></code></pre>
<p>Figuring this out cost me lots of time. Hopefully this information helps
someone else out.</p>
<p>Relevant links:</p>
<ul>
<li><a href="http://redmine.lighttpd.net/issues/show/729">http://redmine.lighttpd.net/issues/show/729</a></li>
<li><a href="http://trac.edgewall.org/ticket/2418">http://trac.edgewall.org/ticket/2418</a></li>
</ul>
Dear Bazaar...2009-07-29T00:00:00+00:002009-07-29T00:00:00+00:00Unknown/posts/dear_bzr/<p>Our relationship has only <a href="http://freshfoo.com/blog/repos_accessible">just
started</a> but it's already
time for me to move on. Although I think you're great,
<a href="http://mercurial.selenic.com/wiki/">Mecurial</a> has become my new
<a href="/hg/">best</a> <a href="/repos/">friend</a>. Things with her are just a little
easier. We <a href="http://mercurial.selenic.com/wiki/ConvertExtension">just
clicked</a>.</p>
<p>I get along so well with
<a href="http://mercurial.selenic.com/wiki/HgWebDirStepByStep">her</a>
<a href="http://trac.edgewall.org/wiki/TracMercurial">friends</a>; I never did
quite fit in <a href="https://launchpad.net/loggerhead">with</a>
<a href="http://bazaar-vcs.org/TracBzr">yours</a>. They always seemed somewhat
immature.</p>
<p>It's also hard to ignore <a href="http://www.python.org/dev/peps/pep-0374/#decision">recent
developments</a>. It
seems like Mercurial is going to become quite important in the future.</p>
<p>No hard feelings I hope. I truly wish you all the best for the future.</p>
<p>Menno</p>
Banishing trailing whitespace with Emacs2009-07-03T00:00:00+00:002009-07-03T00:00:00+00:00Unknown/posts/trailing_whitespace_removal_emacs/<p>One of my pet peeves, especially with Python code, is trailing
whitespace. It serves no purpose, introduces noise in diffs and wastes
valuable bytes dammit (yes I'm being pedantic).</p>
<p>In order to see make trailing whitespace visible in Emacs you can use
the show-trailing-whitespace variable. Emacs of course has a command to
remove trailing whitespace: delete-trailing-whitespace.</p>
<p>Better yet, to get rid of trailing whitespace automatically on save you
can add a function to the write-contents hook. The following snippet
will cause trailing whitespace to be removed on save, but just for
Python files.</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>;; Automatically remove trailing whitespace when saving Python file
</span><span>(add-hook 'python-mode-hook
</span><span> (lambda () (add-hook 'write-contents-hooks 'delete-trailing-whitespace t)))
</span></code></pre>
<p>The one problem with doing this is that when changing an existing
codebase your commits could end up with many whitespace deltas, making
it difficult to see the meat of your changes. Use with care.</p>
draftsdir plugin2009-06-26T00:00:00+00:002009-06-26T00:00:00+00:00Unknown/posts/draftsdir_plugin/<p>I've just published a simple little
<a href="http://pyblosxom.sourceforge.net">PyBlosxom</a> plugin called draftsdir.
It solves a personal itch for me: I'd like to review draft blog entries
on the site before they go live. The plugin lets you define a directory
for draft entries which aren't shown unless you add a query arg to the
URL (the default arg name is "drafts" but there's an option to change
it). When you're happy with an entry you move it from the drafts
directory to where your other published entries are. Simple.</p>
<p>For more details see the <a href="/code/">Code section</a> of this site, the <a href="/repos/pyblosxom-plugins/trunk">bzr
repo</a> or download it
<a href="/downloads/draftsdir.py">directly</a>. There install and configuration
instructions at the top of the file.</p>
Emacs with Viper mode: editor nirvana?2009-06-23T00:00:00+00:002009-06-23T00:00:00+00:00Unknown/posts/editor_nirvana/<p>I've been a long time <a href="http://vim.org">Vim</a> user and fan. Once you're
used proficient with the Vi way of editing it's difficult go back to
anything else. Other editors just feel inefficient and clunky.</p>
<p>That said, I've been jealous of certain Emacs features that I've seen
while looking over the shoulders of my colleagues. Multiple frames (GUI
windows), the clean extension system (via elisp), the tight process
integration (shell, SQL, debuggers etc) and all sorts of unexpected
bells and whistles; these goodies slowly ate away at my curiosity.</p>
<p>So 2 weeks ago I caved and decided to give Emacs another go. I hadn't
really used it since university. Starting with my colleague's
configuration and making some minor tweaks, I began using Emacs for
serious work.</p>
<p>A few days in and I was enjoying exploring my new toy but it didn't feel
quite right. Although I had a reasonable grasp of Emacs' editing keys
and commands, most tasks took way too long, requiring convoluted hand
gymnastics. My left pinky was permanently sore from constantly reaching
for the Ctrl and Alt keys. I was missing those surgical, efficient Vi
commands.</p>
<p>At the suggestion of one Emacs-using colleague I gave Viper (viper-mode)
a try. It's an emulation that attempts to provide a fairly accurate Vi
experience while still allowing full access to the rest of Emacs. To be
honest I was expecting it to be a half-assed kludge. I was wrong. Viper
is a mature Emacs extension and it does a great job of mitigating
conflicts between the Emacs and Vi ways of doing things.</p>
<p>Viper mode proved to be the tipping point; because of it I'm sticking
with Emacs. As far as I'm concerned it's the best of both worlds.</p>
<p>For anyone who's interested, my Emacs config is available in the
<a href="/viewrepo/bzr/config/trunk/files">repository browser</a> here or via
<a href="/repos/config/trunk/">bzr</a>. This is my personal configuration branch so
it will update as I make changes to the setup. Note that I'm using the
latest development (but rock solid) Emacs so there might be references
to features in the config which only exist in this version.</p>
<p>Some items of note in my config:</p>
<p><strong>vimpulse</strong>: Viper mode only emulates classic Vi. vimpulse provides a bunch of
extra features which a Vim user will probably miss such as various "g" commands
and visual select mode.</p>
<p><strong>ido-mode</strong>: This is a standard feature of Emacs which isn't bound to keys by
default. It gives amazing power by replacing the standard find-file and
switch-buffer keystrokes with beefed up alternatives. The key features are
quick directory switching and fuzzy, recursive name matching (but that's not
all).</p>
<p><strong>ibuffer</strong>: I've replaced the standard buffer list binding (C-x C-b) with ibuffer.
This gives a more powerful and easier to use version of the standard buffer
list and allows for crazy batch manipulation of buffers.</p>
<p><strong>yasnippet</strong>: Mode specific template expansion. Powerful and super useful for
cranking out commonly used sections of text (programming is full of them).</p>
<p><strong>flymake - pyflakes integration</strong>: Flymake runs arbitrary programs over
buffers on-the-fly. For Python files flymake has been configured to run
<a href="http://www.divmod.org/trac/wiki/DivmodPyflakes">pyflakes</a> and highlight errors
in code as I type. I might change this to use
<a href="http://www.logilab.org/project/pylint">pylint</a> at some stage because pylint
finds a wider range of problems.</p>
<p>Some useful Emacs config hacking links:</p>
<ul>
<li><a href="http://steve.yegge.googlepages.com/effective-emacs">http://steve.yegge.googlepages.com/effective-emacs</a></li>
<li><a href="http://www.emacswiki.org/">http://www.emacswiki.org/</a></li>
<li><a href="http://xahlee.org/emacs/elisp_idioms.html">http://xahlee.org/emacs/elisp_idioms.html</a></li>
</ul>
bzr repositories now available2009-06-22T00:00:00+00:002009-06-22T00:00:00+00:00Unknown/posts/repos_accessible/<p>I've just made my personal bzr repositories publically available so that
anyone can easily get to them (including me!) and so I can refer to
things from blog articles. The repos are available for branching using
bzr under <a href="/repos/"><a href="http://freshfoo.com/repos/">http://freshfoo.com/repos/</a></a> and in <a href="/viewrepo/bzr/">human
browseable form</a>. See also the links in the left sidebar
and in the <a href="/code/">code section</a> of the site.</p>
<p>I'm using <a href="https://launchpad.net/loggerhead">Loggerhead</a> to provide the
web viewable form (proxied via the main lighttpd server). It was very
easy to setup (using serve-branches). I just wrote a simple init.d
script to ensure it stays running.</p>
Announcing IMAPClient 0.52009-04-29T00:00:00+00:002009-04-29T00:00:00+00:00Unknown/posts/IMAPClient-0.5/<p>It's been a long time between releases but I've just released IMAPClient
0.5.</p>
<p>From the NEWS file:</p>
<blockquote>
<p>Folder names are now encoded and decoded transparently if required
(using modified UTF-7). This means that any methods that return folder
names may return unicode objects as well as normal strings [API
CHANGE]. Additionally, any method that takes a folder name now
accepts unicode object too. Use the folder_encode attribute to
control whether encode/decoding is performed.</p>
<p>Unquoted folder names in server responses are now handled correctly.
Thanks to Neil Martinsen-Burrell for reporting this bug.</p>
<p>Fixed a bug with handling of unusual characters in folder names.</p>
<p>Timezones are now handled correctly for datetimes passed as input and
for server responses. This fixes a number of bugs with timezones.
Returned datetimes are always in the client's local timezone [API
CHANGE].</p>
<p>Many more unit tests, some using Michael Foord's excellent mock.py.
(<a href="http://www.voidspace.org.uk/python/mock/">http://www.voidspace.org.uk/python/mock/</a>)</p>
</blockquote>
<p>IMAPClient can be installed from PyPI using <code>easy_install IMAPClient</code> or
downloaded from my <a href="http://freshfoo.com/code/">Code</a> page. As always,
feedback and patches are most welcome.</p>
Initial thoughts for PyCon 20092009-03-28T00:00:00+00:002009-03-28T00:00:00+00:00Unknown/posts/initial_pycon_thoughts/<p>I'm at <a href="http://us.pycon.org">PyCon</a> 2009 in Chicago. So much awesome
stuff has already been discussed and we're only a few hours in. Much of
it very relevant for my employer (which is nice since they're paying).</p>
<p>Some random thoughts so far:</p>
<ul>
<li><a href="http://getwindmill.com">Windmill</a> (the web testing framework) looks
incredibly promising. If you're doing web apps and you're a Python
shop you should be using this. The Python shell integration rocks.</li>
<li><a href="http://code.google.com/p/the-cassandra-project/">Cassandra</a> is
worth exploring. It's a large-scale, distributed DB that's used (and
hacked on) by Facebook with good Python bindings.</li>
<li>Twitter has added a facinating new dimension to the conference.
Everyone is tweeting using the #pycon tag. There's all sorts of
interesting discussion going on in real-time as talks happen.
<a href="http://twitterfall.com">Twitterfall</a> is a great way to watch the
action.</li>
</ul>
<p>The conference seems really well run and the space at the hotel is
great. The only problem I (and many others) have noticed so far is that
the wifi can be a tad unreliable. Not suprising given the huge number of
wireless devices around.</p>
Upgrading iPhone firmware using VMWare on Linux2009-01-01T00:00:00+00:002009-01-01T00:00:00+00:00Unknown/posts/iphone_upgrade_with_vmware/<p>Well that was a bit scary. I've just updated the firmware on my iPhone
to 2.2 (the horrors!). Actually, this normally shouldn't be too much of
a trial. The difference here is that I did it from Windows running
inside VMWare on a Linux host.</p>
<p>I initially tried the naive approach of applying the update normally
using iTunes (but running inside VMWware). <strong>Don't do this!</strong> The host
(Linux) USB subsystem gets in the way leaving you with a somewhat
useless iPhone in "recovery mode". It seems that the iPhone firmware
upgrade procedure does something tricky with USB that doesn't play
nicely with VMWare running on Linux.</p>
<p>To workaround the issue, some USB modules have to not be loaded on the
Linux host during the upgrade. Extending the tips on <a href="http://stevereads.com/weblog/2008/09/12/vmware-xp-iphone-firmware-version-21-update-no/">Stephen Laniel's
blog</a>
I created a blacklist file for modprobe that prevents certain USB
modules from loading. Just unloading the modules manually isn't enough.
The iPhone reconnects to the USB bus at least once during the upgrade
process causing udev to reload the modules you've manually unloaded.</p>
<p>So before attempting the upgrade, create a file named something like
/etc/modprobe.d/blacklist-usb with the following contents:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>blacklist snd_usb_audio
</span><span>blacklist usbhid
</span><span>blacklist ehci_hcd
</span></code></pre>
<p>I'm not sure if ehci_hcd needs to be disabled, but I did this in my
case.</p>
<p>Reload udev so that it knows about the new modprobe config file:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sudo /etc/init.d/udev reload
</span></code></pre>
<p>Now make sure these modules aren't already loaded:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sudo /sbin/modprobe -r snd_usb_audio
</span><span>sudo /sbin/modprobe -r usbhid
</span><span>sudo /sbin/modprobe -r usbhid
</span></code></pre>
<p>Now go back to VMWare, cross your fingers and attempt the firmware
upgrade using iTunes. Things should progress along slowly.</p>
<p>You might find that at one point during the upgrade that iTunes sits
there with a dialog saying "Preparing iPhone for upgrade". If this goes
on for several minutes it could be that the iPhone isn't connected to
VMWare anymore. iTunes is just waiting for the device to appear. If this
happens, reattach the iPhone to VMWare using the "Removable Devices"
option on VMWare's VM menu. It's a good idea to occasionally check that
the iPhone is connected to VMWare during the upgrade.</p>
<p>Once the upgrade is complete you can remove the modprobe config file and
reload VMWare: :</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sudo rm /etc/modprobe.d/blacklist-usb
</span><span>sudo /etc/init.d/udev reload
</span></code></pre>
<p>For the record, this was done using iTunes 8 running on Vista running
inside VMware Workstation 6.5.0 build-118166 on Ubuntu Intrepid (8.10).</p>
All contacts in sync (+1 for the iPhone)2008-10-26T00:00:00+00:002008-10-26T00:00:00+00:00Unknown/posts/contacts_sync/<p>So <a href="http://batstrading.co.uk">new employer</a> has given me an iPhone and
despite my dislike of vendor lock-in I really like it.</p>
<p>One thing I've now managed to achieve is a single, central set of
contact data. Like most geeks I had people's email addresses, phone
numbers and postal addresses scattered across various devices and email
profiles. In my case this included a Nokia 6130, my laptop's Thunderbird
profile, various work Thunderbird profiles, my Gmail account and my
trusty old Palm Tungsten E2. None of these stores of contact data were
being synced. Contacts were added haphazardly as required.</p>
<p>No longer. The mess has been solved through the use of a variety of
tools.</p>
<p>Getting an iPhone was the tipping point. Suddenly it was possible to
sync contact (and calendar) data over the air, without needing to
connect to a PC. This is very attractive to me. I like being able to
sync even when away from the computer where iTunes is installed. I'm
using the excellent (and free) <a href="http://www.nuevasync.com/">NeuvaSync</a>
service for this. It hijacks the iPhone's Microsoft Exchange support to
push changes to contact data to and from Google Contacts.</p>
<p>I've chosen Google Contacts as the central storage point for this data
because it's reliable, I already have a populated gmail account and
(mainly) because of the large number of tools that can interface with
it.</p>
<p>Syncing to the various Thunderbird instances I use is done using the
<a href="http://www.zindus.com/">Zindus</a> extension. This syncs Thunderbird's
address book with Google Contacts. It works fairly well as long as you
remember to stick to Google's rules about contact entries (eg. email
addresses must be unique across all contacts). It will warn you if
there's a problem.</p>
<p>So, that covers syncing contact data to everywhere I need it: iPhone,
Gmail and Thunderbird.</p>
<p>The final challenge was to pull together all the contact entries I had
around the place and merge them into Google Contacts. This was a long
and messy process that was helped by some adhoc Python hacking. To get
the data out of my Palm I used the pilot-address tool that ships with
the pilot-link package. The excellent <a href="http://www.gnokii.org/">Gnokii</a>
tool pulled the contacts out of my Nokia (over Bluetooth, no less!). I
wrote some Python scripts to help merge the highly imperfect sets of
data, trying to make intelligent guesses where possible. Finally,
OpenOffice Calc was used to manually fix the CSV before importing into
Google Contacts.</p>
<p>It took a long time and many editing iterations but I now have a clean,
centralised set of contacts. Say what you like about me needing to get
out more, but I think this is awesome (and it was a worthwhile learning
process).</p>
freshfoo.com now using rest2web2008-09-25T00:00:00+00:002008-09-25T00:00:00+00:00Unknown/posts/freshfoo-using-rest2web/<p>I've now converted 2 popular <a href="/articles/">articles</a><sup class="footnote-reference"><a href="#1">1</a></sup> on freshfoo.com
so that they are generated via <a href="http://www.voidspace.org.uk/">Michael
Foord's</a> excellent
<a href="http://www.voidspace.org.uk/python/rest2web/">rest2web</a>. Previously one
of these articles was in plain HTML and the other was on the site wiki.
I've got a few other articles in the works so I decided it was time to
migrate the articles to a clean, consistent format. <a href="http://docutils.sourceforge.net/rst.html">reStructured
Text</a> fits the bill perfectly.</p>
<p>I recently also started using the <a href="http://pyblosxom.sourceforge.net/registry/text/rst.html">rst
entryparser</a>
for PyBlosxom so now most of the site is generated using reStructured
Text.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>according to the HTTP server logs</p>
</div>
Linux on the Sony Vaio VGN-BZ112008-09-25T00:00:00+00:002008-09-25T00:00:00+00:00Unknown/posts/linux-on-vaio-vgn-bz11/<p>I've just posted an <a href="/articles/linux_on_vaio_bz11.html">article</a> on my
experiences with Linux on my shiny new Sony Vaio
<a href="http://vaio.sony.co.uk/view/ShowProduct.action?product=VGN-BZ11XN">VGN-BZ11XN</a>
notebook. Hopefully it's useful to other Linux users who own or are
considering purchasing this laptop.</p>
<p>Questions, updates and feedback most definitely welcome.</p>
markup.py2008-09-18T00:00:00+00:002008-09-18T00:00:00+00:00Unknown/posts/markup.py/<p>Glenn and I discovered <a href="http://markup.sourceforge.net/">markup.py</a>
today. It's a lightweight, simple Python module for generating HTML and
XML. Tags are generated using Python objects, with calls to methods on
those objects dynamically creating child tags (using __getattr__
trickery). A useful feature is that repeated tags of the same type can
be blatted out by passing sequences of values.</p>
<p>The module provides a natural way to create correct HTML/XML from Python
and suited our needs perfectly. My only complaint is that the output
isn't indented so it can be harder to read than it needs to be. It would
probably be straightforward to add such an option. Also, it would be
nice if the output could be streamed to a file-like object. Currently it
builds the output in memory which could be a potential problem for large
pages.</p>
<p>markup.py is worth adding to your toolbox if you ever have to generate
bits of XML from Python. For simple jobs, it beats using a complex XML
library and is less error prone than plain old string interpolation.</p>
Slides online for Python on the Openmoko Neo Freerunner2008-09-16T00:00:00+00:002008-09-16T00:00:00+00:00Unknown/posts/Python-on-Openmoko-slides/<p>PyCon UK was fantastic. Many thanks to the organisers for such a great
event.</p>
<p>As promised at the end of the talk, I've just put the <a href="/presentations/PyCon_UK-2008/">slides and code
online</a> for my Python on Openmoko
presentation. I'm hoping this stuff is useful to anyone looking to
develop Python apps for Openmoko phone.</p>
PyCon UK 20082008-08-29T00:00:00+00:002008-08-29T00:00:00+00:00Unknown/posts/PyCon-UK-2008/<p><a href="http://www.pyconuk.org">PyCon UK 2008</a> is approaching fast. If you're a
Python programmer in the UK (or are just Py-curious) then you really
should be going. The talk schedule looks hugely exciting and the
in-the-hallways and at-the-pub action will undoubtably be fun and
engaging. I guarantee you'll learn something about Python and come away
feeling inspired.</p>
<p>Disclaimer: I'm presenting :) I'll be doing a talk on <a href="http://www.pyconuk.org/talk_abstracts.html#11">Python on the
Openmoko Freerunner</a> as
well as helping out
<a href="http://www.voidspace.org.uk/python/weblog/">Michael</a> and
<a href="http://babbageclunk.com/">Christian</a> with a 4 hour tutorial:
<a href="http://www.pyconuk.org/community/IronPythonTutorial">Developing with
IronPython</a></p>
Neo Freerunner: first impressions2008-07-30T00:00:00+00:002008-07-30T00:00:00+00:00Unknown/posts/Neo_Freerunner-first_impressions/<p>I finally got my hands on my <a href="http://openmoko.com/">Neo Freerunner</a> two
weeks ago and have been playing with it when time allows (so much so
that I haven't given myself time to blog about it).</p>
<p>Overall, the hardware is great. The first thing you notice is that the
unit feels very solid and the quality of the display is excellent;
bright and high resolution. I've had success with wifi, GPS, the SD card
slot and basic GSM usage. I haven't had a change to try out the
accelerometers yet, mainly due to a (surprising) lack of software that
uses them.</p>
<p>The Freerunner is a fantastic device but the software is still very much
a work in progress. I wouldn't advise anyone to get one unless they're a
developer or a Linux geek. This will change though as the software
(rapidly) matures. Things get better each day as updates are released.</p>
<p>Perhaps best of all, Python is readily installed and there's Python
API's to access the hardware and GUI toolkits. Python also forms a
crucial part of the in-development <a href="http://wiki.openmoko.org/wiki/OpenmokoFramework">FreeSmartphone.Org
(FSO)</a> software stack
which will one day form the standard base system for Openmoko phones.
I'll be presenting a
<a href="http://www.pyconuk.org/talk_abstracts.html#1037">talk</a> on Python and
Openmoko at <a href="http://www.pyconuk.org">PyCon UK</a> in September.</p>
<p>The Neo Freerunner is uber exciting and reeks of potential. I'm really
looking forward to seeing where the community takes it and contributing
where I can. Watch this space.</p>
<p>ps. SSHing to your phone is neat.<br />
pps. Setting up your phone to dual boot between different images <em>while
waiting in the doctor's waiting room</em> is really really neat :)</p>
OpenMoko release2008-06-27T00:00:00+00:002008-06-27T00:00:00+00:00Unknown/posts/OpenMoko_release/<p>The <a href="http://www.openmoko.com/product.html">OpenMoko Freerunner</a> has been
released! This is big news for people who'd like an open and free phone
(running Linux) with some interesting hardware: GSM, GPRS data, WiFi,
GPS, accelerometers, USB host support, accelerated graphics, SD card
slot and much more. The software is still a work in progress so the
phone is primarily for <a href="http://openmoko.org">developers</a> at this stage</p>
<p>The <a href="https://www.truebox.co.uk/trueboxportal/index.php?wk=OpenMoko">UK
distributor</a>
has been swamped by the amount of people interested in buying one. This
is certainly an encouraging for the potential success of the project.
I'm on the list to get one in the next batch, fingers crossed.</p>
Catching Unhandled Exceptions in .NET2008-05-23T00:00:00+00:002008-05-23T00:00:00+00:00Unknown/posts/dotnet_exceptions/<p>At Resolver, we've been looking at better ways of dealing with unhandled
exceptions that occur during test runs. Apart from the need to log that
a problem occurred it is important that the dialog boxes that Windows
generates don't block the test run (ideally they wouldn't appear at
all). We had a hack in place to deal with these dialogs that I won't go
into here. Let's just say we've been finding our hack inadequate.</p>
<p>In the .NET world there's 2 APIs that your program can use to be
notified about unhandled exceptions. Each covers exceptions that happen
in different parts of your code. In order to be comprehensive about
catching unhandled exceptions you really need to use both APIs.</p>
<p>The first is the Application.ThreadException event. Despite what the
name seems to indicate, this catches unhandled errors that occur during
normal execution of Windows Forms applications. That is, in event
handlers once Application.Run has been called.</p>
<p>Here's a quick example of how to use it from IronPython.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>clr
</span><span>clr.</span><span style="color:#bf616a;">AddReference</span><span>('</span><span style="color:#a3be8c;">System</span><span>')
</span><span>clr.</span><span style="color:#bf616a;">AddReference</span><span>('</span><span style="color:#a3be8c;">System.Windows.Forms</span><span>')
</span><span style="color:#b48ead;">from </span><span>System.Threading </span><span style="color:#b48ead;">import </span><span>ThreadExceptionEventHandler
</span><span style="color:#b48ead;">from </span><span>System.Windows.Forms </span><span style="color:#b48ead;">import </span><span>Application, UnhandledExceptionMode, Form, Button
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">handler</span><span>(</span><span style="color:#bf616a;">sender</span><span>, </span><span style="color:#bf616a;">event</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">Oh no!</span><span>'
</span><span> </span><span style="color:#b48ead;">print </span><span>event.Exception
</span><span> Application.</span><span style="color:#bf616a;">Exit</span><span>()
</span><span>
</span><span>Application.ThreadException += </span><span style="color:#bf616a;">ThreadExceptionEventHandler</span><span>(handler)
</span><span>Application.</span><span style="color:#bf616a;">SetUnhandledExceptionMode</span><span>(UnhandledExceptionMode.CatchException)
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">die</span><span>(</span><span style="color:#bf616a;">_</span><span>, </span><span style="color:#bf616a;">__</span><span>):
</span><span> </span><span style="color:#b48ead;">raise </span><span style="color:#bf616a;">ValueError</span><span>('</span><span style="color:#a3be8c;">foo</span><span>')
</span><span>form = </span><span style="color:#bf616a;">Form</span><span>()
</span><span>button = </span><span style="color:#bf616a;">Button</span><span>(</span><span style="color:#bf616a;">Text</span><span>='</span><span style="color:#a3be8c;">Click me</span><span>')
</span><span>button.Click += die
</span><span>form.Controls.</span><span style="color:#bf616a;">Add</span><span>(button)
</span><span>Application.</span><span style="color:#bf616a;">Run</span><span>(form)
</span></code></pre>
<p>The second API is AppDomain.CurrentDomain.UnhandledException. This event
is triggered when unhandled exceptions occur in threads other than the
main thread. There is one caveat with using this event: Windows will
still pop up its own dialog box even if you've installed your own
handler! This is somewhat frustrating when you want to run unattended
test runs as we do at Resolver. Our builds would block until someone
comes along and closes the dialog.</p>
<p>It seems that many others face the same problem and there's no good
solutions reported online. The only option I could find was to write
some C++/C# that bypasses the .NET handler by using the
SetUnhandledExceptionFilter Win32 call. The CLR uses the same underlying
Win32 API to do its unhandled exception handling so by installing your
own handler here you can prevent the .NET handler from firing and
prevent the dialog box from appearing. The problem with this approach is
that you probably don't have access to a useful CLR traceback.</p>
<p>This morning it occurred to me that perhaps if the process terminates
during the unhandled exception handler then the CLR won't have the
opportunity to show its own dialog. We had tried Application.Exit()
without success (the dialog still appears) but terminating the current
process with a Kill() did the trick! Here's how the code looks...</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>clr
</span><span>clr.</span><span style="color:#bf616a;">AddReference</span><span>('</span><span style="color:#a3be8c;">System</span><span>')
</span><span>clr.</span><span style="color:#bf616a;">AddReference</span><span>('</span><span style="color:#a3be8c;">System.Drawing</span><span>')
</span><span>
</span><span style="color:#b48ead;">from </span><span>System </span><span style="color:#b48ead;">import </span><span>AppDomain, UnhandledExceptionEventHandler
</span><span style="color:#b48ead;">from </span><span>System.Diagnostics </span><span style="color:#b48ead;">import </span><span>Process
</span><span style="color:#b48ead;">from </span><span>System.Threading </span><span style="color:#b48ead;">import </span><span>Thread, ThreadStart
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">handler</span><span>(</span><span style="color:#bf616a;">sender</span><span>, </span><span style="color:#bf616a;">event</span><span>):
</span><span> </span><span style="color:#b48ead;">print </span><span>'</span><span style="color:#a3be8c;">AppDomain error!</span><span>'
</span><span> </span><span style="color:#b48ead;">print </span><span>event.ExceptionObject
</span><span> Process.</span><span style="color:#bf616a;">GetCurrentProcess</span><span>().</span><span style="color:#bf616a;">Kill</span><span>()
</span><span>
</span><span>AppDomain.CurrentDomain.UnhandledException += </span><span style="color:#bf616a;">UnhandledExceptionEventHandler</span><span>(handler)
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">die</span><span>():
</span><span> </span><span style="color:#b48ead;">raise </span><span style="color:#bf616a;">ValueError</span><span>('</span><span style="color:#a3be8c;">foo</span><span>')
</span><span>t = </span><span style="color:#bf616a;">Thread</span><span>(</span><span style="color:#bf616a;">ThreadStart</span><span>(die))
</span><span>t.</span><span style="color:#bf616a;">Start</span><span>()
</span><span>Thread.</span><span style="color:#bf616a;">Sleep</span><span>(</span><span style="color:#d08770;">5000</span><span>)
</span></code></pre>
<p>It's ugly but it works. I'd love to know if there's a better way of
dealing with this.</p>
<p>Hopefully this helps some people out there who have struggled with the
same issue. Although these examples are in IronPython, the principles
should easily translate to C# and other .NET languages.</p>
PyCon Italia presentation materials are online2008-05-16T00:00:00+00:002008-05-16T00:00:00+00:00Unknown/posts/PyCon-Italia-Due-presentation-is-online/<p>As promised at the end of my talk, I just uploaded my <a href="/presentations/PyCon-Italia-Due/">slides and sample
code</a> from <a href="http://www.pycon.it/">PyCon
Italia</a>. Included are the S5 slides, a simple
Resolver One sample, the IronPython shell example and code for the demo
program, mp3stats.</p>
<p>Thanks to <a href="http://www.voidspace.org.uk/ironpython/">Michael Foord</a> for
the basis of much of the slide content.</p>
PyCon Italia Due2008-05-11T00:00:00+00:002008-05-11T00:00:00+00:00Unknown/posts/PyCon-Italia-Due/<p>I'm enjoying the wonderful weather in Florence this weekend while
attending <a href="http://www.pycon.it/">PyCon Italy 2</a>. Yesterday's highlight
was Richard Stallman's thought-provoking keynote titled <em>Free Software
in ethics and in practice</em> held in the jaw-dropping <a href="http://en.wikipedia.org/wiki/Palazzo_Vecchio">Palazzo
Vecchio</a>. Stallman's
alter-ego Saint IGNUcious (of the church of Emacs) even made an
appearance.</p>
<p>My presentation on Sunday covers Application Development in IronPython.
It's mainly an introduction to IronPython for Python programmers. Being
very much an an Italian language conference, there's real-time
translation of English presentations to Italian (mine included).
Conversely there's translation from Italian to English in one stream.</p>
<p>The conferences organisers and attendees are being patient with my lack
of Italian language skills. I feel very lucky that many Italians can
speak English. It's easy to be complacent about learning other languages
when you already know English.</p>
<p>I've been fortunate to have met some great people including Raymond
Hettinger (core Python team), Arkadiusz Wahlig (Skype4Py) and some of
the organisers Simone, Giovanni and Lawrence. It's always great to be
able talk shop, exchange perspectives and be inspired. (Resolver One has
been getting plugged too!) I'm looking forward to more conversations as
the conference continues. The best stuff at conferences always happens
outside of the lecture theatres.</p>
<p>ps. The food rocks! The conference lunch today was amazing and last
night's Florentine style steak was super-tasty.</p>
Resolver One 1.0 Released2008-01-19T00:00:00+00:002008-01-19T00:00:00+00:00Unknown/posts/Resolver-1.0/<p>As
<a href="http://www.voidspace.org.uk/python/weblog/arch_d7_2008_01_12.shtml#e921">mentioned</a>
by <a href="http://tartley.com/?p=231">several</a>
<a href="http://holdenweb.blogspot.com/2008/01/resolver-released.html">others</a>,
Resolver One 1.0 was
<a href="http://www.resolversystems.com/news/?p=36">released</a> on Wednesday!</p>
<p>For those of you who don't know, Resolver One is a unique spreadsheet
application written in
<a href="http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython">IronPython</a>,
that allows you to easily add functionality using the Python language.
The entire spreadsheet is represented as a IronPython program which can
be eyeballed and extended. Changes to the spreadsheet grid are reflected
in the Python code and changes to the code are reflected on the
spreadsheet grid. It's really neat. Resolver is great for people who
want to develop complex spreadsheets in a clean, powerful and testable
way. It's also useful (and fun!) for programmers when prototyping.</p>
<p>One of the Resolver developers Andrzej, has written a nice article
describing <a href="http://andrzejonsoftware.blogspot.com/2008/01/5-reasons-to-try-resolver-one.html">5 Reasons To Try Resolver
One</a>.
Of course there are more than just 5 reasons :) If you do try out
Resolver One, make sure to check out the <a href="http://www.resolverhacks.net/">Resolver
Hacks</a> site too.</p>
<p>The 1.0 release of Resolver One is the result of over 2 years of work.
Having joined the team very recently I've only been part of the very
last bit of that. One thing I've found really interesting is the time
leading up to the release date. Compared to other commercial software
projects I've worked on the atmosphere felt under-control and (almost)
relaxed. It seems that the XP development practices used by the
development team are really paying off. No horrible integration issues
right before the release, no unexpected bugs on release day, no huge
schedule blow-outs. Such a nice difference.</p>
Announcing IMAPClient 0.42008-01-12T00:00:00+00:002008-01-12T00:00:00+00:00Unknown/posts/IMAPClient-0.4/<p>I've just released IMAPClient 0.4. Changes are:</p>
<ul>
<li>Support for folder subscribing and unsubscribing.</li>
<li>Support for server capabilities checks.</li>
<li>Support for folder status checks.</li>
<li>Tests for the above.</li>
<li>More robust date handling.</li>
</ul>
<p>Many thanks to Helder Guerreiro for contributing patches for some of the
features in this release.</p>
<p>IMAPClient can be installed from PyPI using <code>easy_install IMAPClient</code> or
downloaded from my <a href="http://freshfoo.com/wiki/CodeIndex">Code</a> page. As
always, feedback and patches are most welcome</p>
NYC Encounters2008-01-12T00:00:00+00:002008-01-12T00:00:00+00:00Unknown/posts/NYC_Encounters/<p>Susanna and I spent the Christmas and New Year period in New York City
staying with the the wonderful Libby and Phillip. Libby is a long time
friend of Susanna's from New Zealand who's been studying in NYC for many
years now.</p>
<p>On one of our first mornings in NYC we were walking out the door of
<a href="http://goodenoughtoeat.com/">Good Enough to Eat</a> after a delicious
brunch when I hear "Menno?". By freak coincidence <a href="http://skvidal.wordpress.com">Seth
Vidal</a> and his partner in crime Eunice
were walking in the door at the same time as we were leaving. Seth and I
used to work together a lot on the
<a href="http://linux.duke.edu/projects/yum/">Yum</a> project. I don't live in NYC
and neither does he, but some how we end up at the same place in a huge
city at the same time. A super weird but pleasant surprise. Seth
describes the incident <a href="http://skvidal.wordpress.com/2007/12/26/superduper-big-apple-weirdness/">on his
blog.</a></p>
<p>The remainer of our visit didn't offer any more strange co-incidences
but we had excellent fun. I caught up with Rohan, Susan and
<a href="http://tartley.com/">Jon</a> one afternoon which was awesome. Thanks Rohan
for showing us some sights
(<a href="http://www.mcsorleysnewyork.com/">McSorely's</a> is a must-do
experience).</p>
<p>As always, photos to come...</p>
spamquestion 0.22007-11-21T00:00:00+00:002007-11-21T00:00:00+00:00Unknown/posts/spamquestion-0.2/<p>I released version 0.2 of the spamquestion plugin last night. It now
conforms to the requirements for a PyBlosxom plugin. The code is much
cleaner too. Available from the <a href="/code/">usual place</a>.</p>
Blog spam update2007-11-06T00:00:00+00:002007-11-06T00:00:00+00:00Unknown/posts/spamquestion_update/<p>Well, two comment spams have made it past the spamquestion plugin. This
makes me wonder if either the submissions were done manually or whether
the software the spammers use is at least human assisted. I guess it's
also possible that the spam software is so good that it can
automatically work out my simpler arithmetic questions.</p>
<p>The web server logs give some clues. There's literally hundreds of
obviously automated POST attempts to various pages on my blog. The
requested related to the two comments that made it through however seem
far more human however. Here's one example:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>68.187.226.250 freshfoo.com - [03/Nov/2007:01:41:57 +0000] "GET /blog/Holland_photos_online.1024px HTTP/1.1" 200 11367 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
</span><span>68.187.226.250 freshfoo.com - [03/Nov/2007:01:42:06 +0000] "POST /blog/Holland_photos_online#comment_anchor HTTP/1.1" 200 14928 "http://freshfoo.com/blog/Holland_photos_online.1024px" "Mozilla/4.0 (compatible; MSIE 6.0; Windows
</span><span>NT 5.1; SV1)"
</span></code></pre>
<p>These are the only two HTTP requests made for the first spam that made
it through; no dumb, repeated automatic requests like some of the other
attempts in the logs. Notice how the parent page was visited first and
then 9 seconds later the POST was made. That's pretty quick for someone
to fill out the form manually but it's possible, especially if the spam
body was ready in the clipboard. If their system is partially automated
then the short delay is even more plausible.</p>
<p>To test whether some spambots are actually capable of doing simple
arithmetic by themselves, I've removed all the addition and subtraction
questions from my spamquestion configuration and have added more
questions that are harder to answer programmatically. If the spam
continues, then I'm going to conclude that there's definitely some human
assistance going on. If it stops, then it's more likely that the spambot
software was actually able to solve some of my arithmetic questions
itself.</p>
<p>I also need to look at is short-term blocking of spamming IPs. When
examining my logs I found there had been almost 500 comment spam
attempts for just today! I'd rather not be dealing with that bandwidth
on my server. Dropping all packets from a spammer's IP for a few hours
would slow them right down.</p>
<p>Fun fun fun...</p>
Comments renabled, announcing spamquestion2007-11-01T00:00:00+00:002007-11-01T00:00:00+00:00Unknown/posts/spamquestion/<p>Due to my <a href="/blog/spamity_spam_spam">recent comment spam issues</a> I've
created a new PyBlosxom plugin called spamquestion. It is similar to the
existing <a href="http://pyblosxom.sourceforge.net/blog/registry/input/comments/comment_magicword">Magic Word
plugin</a>
but instead of using just the one question for any comment entry on the
blog, it randomly selects a question from a larger set of configured
questions. This makes it much harder for spammers to get past the
comment form using automated software. Unlike CAPTCHA systems this
scheme doesn't disadvantage visually impaired people or those on text
based browsers.</p>
<p>The spamquestion plugin can be downloaded from my <a href="/code/">Code</a> page.</p>
<p>Comments are now re-enabled on my site, with spamquestion enabled. It'll
be interesting to see how the scheme holds up. I also plan to install
the <a href="http://pyblosxom.sourceforge.net/blog/registry/input/comments/comment_akismet">Akismet
plugin</a>
as a second line of defense.</p>
Spam: good news and bad news2007-10-25T00:00:00+00:002007-10-25T00:00:00+00:00Unknown/posts/spamity_spam_spam/<h4 id="the-bad-news">The bad news...</h4>
<p>My blog was hit by a comment spammer last week. Hundreds of entries were
made, interestingly focussing only a few articles (perhaps with a higher
Google ranking?). Running without a CAPTCHA system or similar was good
while it lasted. Comments are now disabled until I get around to
installing a CAPTCHA style plugin.</p>
<p>Lazy web: what anti comment-spam technologies do you find work well for
you? Is CAPTCHA the best option we have?</p>
<h4 id="the-good-news">The good news...</h4>
<p>I started using SpamAssassin for my personal email over a month ago.
Having seen the complete ineffectiveness of some anti-spam systems I was
fairly pessimistic about how effective it would be. Boy was I wrong.
Without any tweaks to the default filtering config (except for ensuring
that the latest rules are being used) it stops virtually spam hitting my
mailbox with zero false-positives so far. I get 20-40 spams a day and 1
or 2 a month get through to my inbox.</p>
<p>My mail volume is comparatively low so I just set <a href="http://wiki.apache.org/spamassassin/UsedViaProcmail">Procmail to invoke
SpamAssassin</a> for
each inbound message. For higher volume situations something like SA's
spamd should probably be used. Using Procmail has the nice benefit of
being able to direct spam to a separate folder for later persual and
deletion.</p>
<p>A cron job is set to run <code>sa-update</code> ever night to ensure the latest
default checks are being used. This is important; spammers develop new
tricks to bypass anti-spam systems all the time.</p>
<p>Currently I have all suspected spam going to a spam folder. However SA
has been so successful that I'm thinking of getting Procmail to
automatically delete higher scoring spam and send only the lower scoring
spams to the spam folder. Depending on attitudes towards false-positives
some might just delete all emails that SA thinks is spam. Personally,
I'd rather be a bit cautious. Losing real email scares me.</p>
<p>It's so nice when something works beyond expectation.</p>
Announcing IMAPClient 0.32007-10-17T00:00:00+00:002007-10-17T00:00:00+00:00Unknown/posts/IMAPClient-0.3/<p>I've just made a new release of IMAPClient (0.3). Changes are:</p>
<ul>
<li>The distribution has been renamed from imapclient to IMAPClient so
as follow the conventions used by modern Python packages. The Python
package name remains imapclient however.</li>
<li>Fixed a bug reported by Brian Jackson which meant more complex fetch
part selections weren't being handled correctly. (eg.
<code>"BODY[HEADER.FIELDS (FROM)]"</code>). Thanks Brian!</li>
<li>IMAPClient is now distributed using setuptools.</li>
</ul>
<p>IMAPClient can be installed from PyPI using <code>easy_install IMAPClient</code> or
downloaded from my <a href="http://freshfoo.com/wiki/CodeIndex">Code</a> page. As
always, feedback and patches are most welcome</p>
FuzzyFinder2007-10-12T00:00:00+00:002007-10-12T00:00:00+00:00Unknown/posts/fuzzyfinder/<p><a href="http://www.vim.org/scripts/script.php?script_id=1984">FuzzyFinder</a> is a
useful Vim extension that I've discovered recently (nothing to do with
<a href="http://www.voidspace.org.uk/python/weblog/">Fuzzyman</a>). It has proven
to be a great productivity enhancer, especially when dealing with large
codebases with many files.</p>
<p>FuzzyFinder provides a mechanism to search through files on disk or Vim
buffers using fuzzy filename matching. When activated it interactively
searches the current directory for files matching the name you entered.
Matching is very loose, so if for example you enter "abc", you'll get a
list of all files matching <code>*a*b*c*</code>. It sounds strange at first but is
very effective in practice.</p>
<p>Here's a screen shot of FuzzyFinder when first activated. A list of all
files in the current directory is displayed.</p>
<div class="screenshot">
<p><a href="/images/fuzzyfinder1.png"><img src="/images/fuzzyfinder1.png" alt="" /></a></p>
</div>
<p>The arrow keys can be used to make a selection from the list (useful if
you can see what you want). If the list is long, start filtering!</p>
<p>This screenshot shows what happens after a few characters have been
entered. The list of available choices is filtered to match. Very
powerful.</p>
<div class="screenshot">
<p><a href="/images/fuzzyfinder2.png"><img src="/images/fuzzyfinder2.png" alt="" /></a></p>
</div>
<p>FuzzyFinder can also do recursive matching using the <code>**</code> wildcard. This
is great for large source code trees.</p>
<div class="screenshot">
<p><a href="/images/fuzzyfinder3.png"><img src="/images/fuzzyfinder3.png" alt="" /></a></p>
</div>
Converted to PyBlosxom2007-09-16T00:00:00+00:002007-09-16T00:00:00+00:00Unknown/posts/moved_to_pyblosxom/<p>I've converted the blog part of this site to use
<a href="http://pyblosxom.sourceforge.net">PyBlosxom</a>. It used to be powered by
Blogger using their FTP-upload-to-your-own-host feature but I found
various parts of the Blogger system to be inflexible and buggy.</p>
<p>PyBlosxom is an old-school CGI requires a certain amount of expertise to
get set up but it's solid and has a very powerful plugin system. Most of
the features on this page from the archive list to the tag list are
provided via plugins. Because PyBlosxom is written in Python it's easy
to extend, tweak or write new plugins.</p>
<p>I'm not completely happy with the styling of the pages but it will do
for now. I also need to import the comments from the old blog. Please
let me know if you see any problems.</p>
Introducing imapclient2007-05-03T00:00:00+00:002007-05-03T00:00:00+00:00Unknown/posts/Introducing_imapclient/<p>Today I released the first versions of imapclient, an IMAP4 client
library I've been working on. From the README:</p>
<blockquote>
<p>imapclient aims to be a easy-to-use, Pythonic and complete IMAP client
library with no dependencies outside the Python standard library.
Features:</p>
<ul>
<li>Arguments and return values are natural Python types.</li>
<li>IMAP server responses are fully parsed and readily useable.</li>
<li>IMAP unique message IDs (UIDs) are handled transparently. There is
no need to call different methods to use UIDs.</li>
<li>Convenience methods are provided for commonly used functionality.</li>
<li>Exceptions are raised when errors occur.</li>
</ul>
</blockquote>
<p>imapclient makes IMAP useable from Python with little effort. If you've
used the imaplib module from the standard library, you'll know that a
lot of supporting code is needed just to get simple things done. I'd
wager that everyone who has used imaplib has their own fragile regexes
and management code to go with it. I hope that imapclient will put an
end to this.</p>
<p>imapclient can be downloaded from my
<a href="http://freshfoo.com/wiki/CodeIndex">Code</a> page. Feedback and patches
most welcome!</p>
Data on the rocks2007-04-05T00:00:00+00:002007-04-05T00:00:00+00:00Unknown/posts/Data_on_the_rocks/<p>I was asked to help a friend recently whose hard disk was dying. The
system wouldn't boot anymore with the BIOS reporting disk or read
errors. Ironically this starting happening the moment after my friend
mentioned to his girlfriend that they should "really start doing
backups".</p>
<p>Said friend was instructed by me to buy a new hard disk and an external
hard disk enclosure and I went around armed with various Linux based
<a href="http://trinityhome.org/">rescue</a> <a href="http://www.sysresccd.org">disks</a> and
a Windows XP install disk.</p>
<p>Things didn't start out well. I booted using one of the rescue disks and
tried mounting the failing hard drive. The <code>mount</code> process hung and
<code>dmesg</code> showed the kernel spewing out IDE related error messages at a
great number of Hz. I ended up having to forcibly kill the <code>mount</code>
process. Several more attempts failed in a similar way.</p>
<p>I had heard that failing hard disks can sometimes be made to work if
cooled down. The theories about why this works seem flimsy (something
about contracting the metal inside the drive so that components go back
into alignment), but since options were limited at this stage I figured
it was worth a try.</p>
<p>The new hard disk went into the computer and the faulty one was
installed into the enclosure. I then wrapped the enclosure in several
plastic bags and put it into the freezer with the cables hanging out of
the door. With fingers crossed I connected the enclosure to the computer
and turned everything on. This time I was able to mount the disk from
the rescue disk without even a single kernel error. I couldn't believe
it! I hurriedly partitioned the new drive and began copying before
something went wrong.</p>
<p>There was a lot of data to copy (56GB). This gave plenty of time for
worrying thoughts like "<em>what if condensation occurs inside the cold
drive and the electronics short out?</em>". Fortunately all went well and
every byte was recovered from the failing drive. After a bit more
fiddling with <code>boot.ini</code>, NTFS conversion and incorrect file attributes
the system was working normally again. Great!</p>
<p>After all this, I did some research to see if other people use this
trick with success (one might question why I didn't do this research
before I tried it...). It turns out that in most cases
<a href="http://geeksaresexy.blogspot.com/2006/01/freeze-your-hard-drive-to-recover-data.html">it</a>
<a href="http://www.trap17.com/index.php/hard-drive-freezer-trick_t26193.html">does</a>
<a href="http://www.macosxhints.com/article.php?story=2006110111270170">work</a>.
I'd love to hear a solid explanation of why...</p>
cert2rss.py retired2007-03-22T00:00:00+00:002007-03-22T00:00:00+00:00Unknown/posts/cert2rss.py_retired/<p>A long time ago, I wrote a little script called cert2rss.py that takes
the URLs from the <a href="http://www.us-cert.gov/">US-CERT</a> <a href="http://www.us-cert.gov/channels/bulletins.rdf">Summary Bulletin
Feed</a>, parsed the HTML
summary pages and generated a feed of containing each item individually.
I needed something like this for work as I need to keep track security
vulnerabilities for all things Linux. The script has been broken for
some time because the HTML of summary pages has changed.</p>
<p>Today I finally got around to looking at the problem and found that the
HTML of the summary pages is now so bad that it is very difficult to
extract useful data from. The HTML looks fine when rendered but it is
full of incorrectly escaped text, missing tags and bizarre formatting.
Whoever is responsible for generating that HTML shouldn't have their
jobs.</p>
<p>So, I'm retiring cert2rss.py. I've updated my <a href="/code/">Code</a> page to
reflect this.</p>
<p>As an alternative, there's some reasonable <a href="http://nvd.nist.gov/download.cfm">RSS
feeds</a> from the National Vulnerability
Database that provide similar functionality.</p>
ssh-agent-applet 0.12007-02-14T00:00:00+00:002007-02-14T00:00:00+00:00Unknown/posts/ssh-agent-applet_0.1/<p>I've finally managed to do the first release of ssh-agent-applet. I've
actually been using it for some time on my own PC but it needed a bit of
love and documentation to make it easier to install.</p>
<p>From the README:</p>
<blockquote>
<p>ssh-agent-applet is a Gnome applet that allows you to conveniently
keep your ssh key(s) on external media. This means that if your
computer is cracked or stolen, the attacker will not have a copy of
private ssh key(s).</p>
</blockquote>
<blockquote>
<p>Using ssh-agent-applet, your keys are loaded into ssh-agent as soon as
you insert your "key drive" into a USB port. The drive is
automatically unmounted once the key loaded so you can remove it from
the USB port immediately.</p>
</blockquote>
<p>For developers, ssh-agent-applet is a reasonable example of how to write
Gnome applets in Python and how to interface with HAL from Python using
DBUS. The HAL/DBUS interactions are non-trivial and go beyond what most
documentation covers. It took a fair amount of experimentation to get
some of this stuff working. Hopefully the applet or at least its' code
is of use to other people.</p>
<p>ssh-agent-applet can be found on
<a href="https://gitlab.com/menn0/ssh-agent-applet">GitLab</a>.</p>
Found photos2007-01-09T00:00:00+00:002007-01-09T00:00:00+00:00Unknown/posts/Found_photos/<p>I just stumbled across <a href="http://www.superfluous.com/">superfluous.com</a>
while looking for something completely different. It's a small
collection of anonymous photos found discarded on the street. The site
invites you to think about the possible histories and stories behind the
pictures. Somewhat haunting. See the
<a href="http://www.superfluous.com/about.html">About</a> page for more details.</p>
Starting with Ruby2007-01-06T00:00:00+00:002007-01-06T00:00:00+00:00Unknown/posts/Starting_with_Ruby/<p>This <a href="http://bendiken.net/2007/01/01/the-road-to-lisp">excellent read</a>
from Arto Bendiken has inspired me to do 2 things:</p>
<ol>
<li>start learning Ruby</li>
<li>keep going with learning Scheme which I haven't touched much since I
<a href="http://freshfoo.com/blog/2006/10/scheming-on-way-to-oz.html">started teaching
myself</a>
on the plane to Australia</li>
</ol>
<p>I hunted around for some Python to Ruby guides and comparisons and found
the following useful links:</p>
<ul>
<li><a href="http://blog.ianbicking.org/ruby-python-power.html">Ian Bicking's awesome comparison between Python and
Ruby</a>: Ian tries
hard to stay impartial and keep the discussion on the really
important issues.</li>
<li>A <a href="http://www.dthg.net/docs/ruby.html">Ruby introduction for
programmers</a>: useful quick notes
for those who already program in other dynamic languages.</li>
<li>The <a href="http://www.rubycentral.com/book/intro.html">intro</a> from the
Programming Ruby book isn't bad.</li>
</ul>
<p>Now I just have to make some time for all this. I still want to keep
contributing to Yum and starting doing a Dutch language course as well
(my Dutch isn't too bad but I want to get a let more proficient). Yoga
is on the cards as well. I'm sure it can be done with a bit of planning
and enthusiasm.</p>
USB Sniffing / Optus 3G and Linux2006-11-13T00:00:00+00:002006-11-13T00:00:00+00:00Unknown/posts/USB_Sniffing___Optus_3G_and_Linux/<p>I'm living at my brother's new place while I'm in Australia. He doesn't
have an internet connection (no home computer) and doesn't have a home
phone (just uses mobile). To work around this obvious lack of
connectivity I've borrowed an <a href="http://www.optus.com.au/">Optus</a> branded
<a href="http://www.option.com/products/3gquad.shtml">GlobeTrotter 3G Quad</a>
PCMCIA card. The card is supplied as part of the Optus Wireless Connect
package.</p>
<p>The Windows drivers and software worked immediately on my laptop but I
rarely use Windows so was keen to get it working under my <a href="http://fedoraproject.org/">Linux
distro</a>.</p>
<p>I spent many hours trying the configurations described at the
<a href="http://pharscape.org/">PHARscape</a> site (and others). PPP negotiation
would start but an IP address would never be allocated. I figured there
was something different about the Optus 3G network but was using the
Optus Access Point Name (APN) documented on the net ("internet").</p>
<p>I ended up sniffing the USB traffic to and from the card under Windows
by using the excellent
<a href="http://sourceforge.net/projects/usbsnoop/">SnoopyPro</a> USB sniffer. I
captured the URBs the Windows drivers sent to first serial port endpoint
on the card and then used <code>strings</code> on the capture file under Linux. The
AT commands used by the Windows drivers magically appeared.</p>
<p>It turns out that I was using the wrong APN; I should have been using
"connect" instead of "internet". Once the pppd chat script was updated
things started working. Fantastic!</p>
<p>More details on the specifics of the pppd configuration I'm using can be
found on the freshfoo.com
<a href="http://freshfoo.com/wiki/Using_Optus_3G_with_Linux">wiki</a>.</p>
HTTP Auth and Selenium2006-11-07T00:00:00+00:002006-11-07T00:00:00+00:00Unknown/posts/HTTP_Auth_and_Selenium/<p>I'm starting to look at
<a href="http://www.openqa.org/selenium-rc/">Selenium-RC</a> at work today. We're
moving a lot of web based functional tests away from
<a href="http://pbp.berlios.de/">PBP</a>.</p>
<p>I ran into a major problem straight away. Our web interface uses Basic
HTTP Auth and Firefox insists on presenting a dialog to the
(non-existent) user when you open a http://user:pass@site.com/ style
URL. The tests would hang waiting for the Ok button click.</p>
<p>After a lot of searching I finally found the
<a href="http://kb.mozillazine.org/Network.http.phishy-userpass-length">network.http.phishy-userpass-length</a>.
option. When set to 255 the authentication warning dialog box isn't
shown.</p>
<p>Selenium creates a fresh Firefox profile every time it starts Firefox so
modifying the option in <code>about:config</code> doesn't help. You need to edit
firefox.js and add a line like
<code>pref("network.http.phishy-userpass-length", 255);</code>. This ensures that
the option is set in any new profile. On Windows firefox.js can probably
be found at <code>Program Files\Mozilla Firefox\defaults\prefs\firefox.js</code>.
It'll probably be at <code>/usr/lib/firefox-*/defaults/pref/firefox.js</code> on a
Linux machine.</p>
<p>I wonder if Selenium-RC could set this option in profiles it creates
itself? It already seems to set a bunch of other options.</p>
GStreamer and Python experimentation2006-10-28T00:00:00+00:002006-10-28T00:00:00+00:00Unknown/posts/GStreamer_and_Python_experimentation/<p>I recently played with <a href="http://gstreamer.org/">GStreamer</a> for the first
time. I'd heard plenty of good stuff about the framework but had never
spent any time developing with it.</p>
<p>In a nutshell, it's very powerful and very cool. Complex multimedia apps
can be created with very little code. You link together small blocks to
form a useful chain of functionality. The Python API is quite nice
because the Python bindings are well supported by the developers.</p>
<p>The big problem is lack of documentation. The best starting point I
found was <a href="http://jonobacon.org/">Jono Bacon</a>'s excellent <a href="http://www.jonobacon.org/?p=750">Getting
Started with GStreamer with Python</a>.
After that, the C based docs and the <code>gst-inspect</code> tool are probably
your best friends. They make sense once you get your head around the
general architecture.</p>
<p>In the interest of expanding the amount of reference information
available for GStreamer I've put the results of a few hours of messing
around online. It's a simple program that illustrates some interesting
stuff using GStreamer. Despite the low line-count, what it does is
somewhat impressive; it takes 2 arbitrary audio files and plays them
over the top of each mixing in real-time. More details and the download
link can be found on my <a href="/wiki/CodeIndex">code</a> page.</p>
Martini Design Patterns?2006-10-21T00:00:00+00:002006-10-21T00:00:00+00:00Unknown/posts/Martini_Design_Patterns/<p>Sometimes the random text to put off filters in spam emails is
hilarious. This is an excerpt of one I received today (odd line breaks
as per source):</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>In a way that lets you put principles will help the "Trading Spaces" show. support in your own code. so that you can spend You want to learn about the patterns that
</span><span>to know how they you get to take same problems. the patterns that in between sips of a martini.
</span><span>
</span><span>the patterns that or on the real relationship "secret language" Head First book, you know challenging. Something But you don't just
</span><span> in between sips of a martini.
</span><span>the next time you're Design Patterns, you'll avoid alone. At any given moment, used in the Java API deep understanding of why
</span></code></pre>
<p>Looks like they randomised the words of some design patterns book,
although I don't know where the martini references come from.</p>
Scheming on the way to Oz2006-10-20T00:00:00+00:002006-10-20T00:00:00+00:00Unknown/posts/Scheming_on_the_way_to_Oz/<p>I arrived in Australia 2 days ago to sort out various passport related
issues and to get some time in the <a href="http://netboxblue.com/">office</a>(it's
been 6 months). Since the flight gave me a spare 24 hours to fill I
decided it would be an excellent time to start learning
<a href="http://schemers.org/">Scheme</a>. This is something I've been meaning to
do for ages, just to stretch my programming mind in new ways, but also
because I'm just curious about Scheme/Lisp.</p>
<p>Before I left I installed the Fred Bayer's excellent
<a href="http://www.lispme.de/index.html#lispme">LispMe</a> Scheme interpreter on
my Palm. This is a relatively complete, almost standard, Scheme
implementation which includes APIs for Palm graphics, databases and UI
elements. Not that I got that far ... I'm definitely closer to the
"Hello world" end of the spectrum when it comes to Scheme.</p>
<p>For documentation I downloaded Dorai Sitaram's excellent free tutorial
<a href="http://www.ccs.neu.edu/home/dorai/t-y-scheme/t-y-scheme.html">Teach Yourself Scheme in Fixnum
Days</a> on
to my Palm. It's freely available in HTML, a page per chapter. A bit of
fiddling with <a href="http://www.pyrite.org/publisher/index.html">Pyrite
Publisher</a> gave me a
suprisingly readable Palm version, one document per chapter. The output
wasn't perfect but completely useable. Given a little more preprocessing
I reckon I could get the whole tutorial into one document.</p>
<p>Although I would have loved to, I didn't get the classic Scheme/Lisp
text <a href="http://mitpress.mit.edu/sicp/full-text/book/book.html">Structure and Interpretation of Computer
Programs</a> onto my
Palm. It's damn big and I couldn't figure out a way to convert it nicely
in the time I had to do it. I think it is doable though.</p>
<p>By setting up shortcuts in the Palm launcher I was able to quickly
switch between the book and LispMe and happily spend the much of the
flight(s) teaching myself the basics of Scheme. Some thoughts:</p>
<ul>
<li>LispMe is cool! There wasn't anything that I covered in the tutorial
that it couldn't do. Some of the demos programs are really
impressive too, for example a 3D rotating cube!</li>
<li>Teach Yourself Scheme in Fixnum Days is a nice tutorial. It doesn't
mess around much and gets into more difficult ground fairly quickly.
It's probably a good introduction for programmers experienced in
other languages.</li>
<li>I'd still like to go through
<a href="http://mitpress.mit.edu/sicp/full-text/book/book.html">SICP</a> at
some stage to get a more complete grounding in Scheme.</li>
<li>Entering Lisp-like code via the Palm graffiti input system requires
a certain amount of dedication. There is a Palm hack that comes with
LispMe to help with parentheses matching but unfortunately it
doesn't work on more recent Palm's like my Tungsten.</li>
<li>I don't think I could see myself using Scheme for any major
projects, not when I already find Python to be such a good fit for
most things. That isn't really the point of this exercise though
(and I should never say never).</li>
</ul>
<p>I have a lot more to learn. I certainly didn't get through the whole
tutorial but skipped ahead a bit to see what other stuff is covered.
Continuations, the non-deterministic operator (amb) and macros look
really interesting. The brain stretching really begins when you start
getting on to these more advanced topics. Maybe for the flight back to
the UK...</p>
Planet London Python is online2006-10-10T00:00:00+00:002006-10-10T00:00:00+00:00Unknown/posts/Planet_London_Python_is_online/<p>I've just installed the <a href="http://londonpython.webfactional.com/">Planet London
Python</a> site.
<a href="http://planetplanet.org/">Planet</a> took a small amount of fiddling but
it's all quite logical. A little more customisation is probably
required. Some more feeds wouldn't go astray either!</p>
<p>Thanks to Remi from <a href="http://webfaction.com">WebFaction</a> for the hosting
space.</p>
Selenium at London 2.02006-09-06T00:00:00+00:002006-09-06T00:00:00+00:00Unknown/posts/Selenium_at_London_2.0/<p>I went to the regular <a href="http://www.magpiebrain.com/blog/2006/08/03/london-20-rc-7-tuesday-the-5th-of-september/">London
2.0</a>
meetup last night. Great people, interesting conversations and a little
too much beer.</p>
<p>One of the <a href="http://www.openqa.org/selenium/">Selenium</a> developers,
<a href="http://www.jrandolph.com/blog/">Jason Huggins</a> demoed Selenium and
<a href="/posts/Selenium_at_London_2.0/%20http://www.openqa.org/selenium-ide/">Selenium-IDE</a> on a Macbook
notebook running the
<a href="http://www.apple.com/downloads/macosx/system_disk_utilities/parallelsdesktopformac.html">Parallels</a>
virtual machine software. He had several virtual machines running
different platforms including Ubuntu Linux, Windows XP and Mac OS X with
different browsers on each. Testing was triggered by checking in code to
the SVN repository of Buildix virtual machine also running on the
laptop.</p>
<p>When a check-in occurs the code is automagically tested on the browsers
running on each of the browsers running on the different platforms in
each virtual machine. The beauty of selenium is that the tests run
inside the browser so your interface is tested just as it is used. All
your JavaScript runs and works as for the end user. I loved the fact
that all the browser sessions are
<a href="http://www.unixuser.org/~euske/vnc2swf/">captured</a> to a Flash movie so
that if a test fails you can watch the footage of the steps leading up
to the failure.</p>
<p>The crux Jason's demo was that you could have a complete multiplatform
web application test system running on a bottom level Macbook laptop
(with extra RAM). The machine appeared to run very quickly despite the 5
or 6 VMs that were running. Jason mentioned that if you don't care about
Mac browser testing (Safari) then you could set up a similar system
using VMWare or <a href="http://www.xensource.com/">Xen</a> on Linux or Windows.</p>
<p>Selenium-IDE also looks seriously cool. It's a Firefox plugin that
provides a complete environment for developing test cases. It can record
your steps as you use a web interface and output to a variety of test
script formats including a HTML stype sequential output, Python and
Ruby. If using Python or Ruby scripts you can use all the features of
the language such as conditions and loops. Selenium-IDE is going to make
developing effective complete web tests so much easier (and fun!).</p>
<p>We need to start looking at this <a href="http://netboxblue.com/">where I work</a>.
Our current web interface tests use <a href="http://twill.idyll.org/">Twill</a>
which works ok but has many shortcomings. Creating test cases is
tedious, the JavaScript parts of the UI aren't tested at all and there's
no testing of cross-browser support. Our UI is using more and more
JavaScript so something like Selenium is going to become vital. I'm
looking forward to using it.</p>
Rockbox rocks2006-07-21T00:00:00+00:002006-07-21T00:00:00+00:00Unknown/posts/Rockbox_rocks/<p>I discovered <a href="http://rockbox.org/">Rockbox</a> today. It's a complete open
source replacment for the firmware of several portable audio players
include the iPod. It supports many audio formats including MP3, Ogg
Vorbis and WMA and has cool stuff like gapless playback (great for mix
CDs), themes, bookmarks and much more. I also love the fact that no
special directory layout is required. Use your normal shell commands
(eg. <code>cp</code> or <code>rsync</code>) or file management interface (eg. Nautilus) to
copy tracks to your iPod.</p>
<p>I installed it on my iPod this afternoon; wow. It's not perfect (the tag
cache system is fiddily) but it's stable and works as advertised. My
main criticism is that there's too many options and features to fiddle
with; great for geeks but not for the general consumer. But hey, it's
open source so someone could create a friendlier version at some point.
I love the idea that I can play with the code if I want to. It also has
a plugin system for add-on apps which I'd love to check out at some
point.</p>
<p>Rockbox is definitely staying on my iPod (yes, the process is
reversible).</p>
Met Office English2006-06-15T00:00:00+00:002006-06-15T00:00:00+00:00Unknown/posts/Met_Office_English/<p>From today's weather forecast courtesy of the <a href="http://www.met-office.gov.uk">UK Met
Office</a>...</p>
<blockquote>
<p>There will be rather large cloud amounts, but some brightness.</p>
</blockquote>
<p>Hmmm ...</p>
Kernel hacking, Software Suspend (Suspend2) and other coolness2006-04-06T00:00:00+00:002006-04-06T00:00:00+00:00Unknown/posts/Kernel_hacking,_Software_Suspend_(Suspend2)_and_other_coolness/<p>Lots of interesting stuff in yesterday's sessions. Did the full day
kernel hacking tutorial with Rusty Russell and Robert Love which despite
the disorganisation and the fact that we ran out of time was a great
intro and a lot of fun. Although they might not realise it, Rusty and
Robert have a good dynamic that stops things from getting boring.</p>
<p>The open source quiz was great to watch. Some fairly crazy madness.</p>
<p>I hung around until late for some of the Birds of a Feather (BOF)
sessions. The most interesting for me was the <a href="http://www.suspend2.net">Software
Suspend</a> "installfest". This stuff is seriously
cool. Fully featured suspend to hard disk with no special hardware/power
management support required. You really don't ever need to shutdown your
Linux system again. I was lucky enough to have all the Suspend2
developers helping me get it working on my notebook. It was suprisingly
easy. I'm planning on contributing a cleaner kernel Fedora RPM with the
Suspend2 patches. The guys are pushing to get this into the mainline
kernel soon which should make this a lot more accessible.</p>
<p>At around 10pm we were kicked out by the organisers and some of us
headed down to the Wig and Pen (again!). Drank too much (again!) and
chatted to lots of cool people. Finally managed to track down <a href="http://www.bytebot.net">Colin
Charles</a> who Seth Vidal told me to look out for
at the conference. I should really easy up on the drinking one of these
days...</p>
Much Yum progress2006-04-06T00:00:00+00:002006-04-06T00:00:00+00:00Unknown/posts/Much_Yum_progress/<p>Stayed up late last night working on the yum sqlite integration. Hacked
out a new iterator style parser so yum doesn't load all the metadata
into memory (and dump to a pickle) when importing into sqlite. I'm quite
happy with it.</p>
<p>Also cleaned up a few other parts of Gijs' TODOs (and then added more of
my own) :). All in all a productive and fun night although I'm more than
a little tired this morning. In an effort to regain some fitness I've
been walking every morning but I'm thinking that might not happen today
:)</p>
<p>Seth emailed Gijs and me this morning in order to get CVS commit access
organised. Sweet.</p>
Yum plugins?2006-04-06T00:00:00+00:002006-04-06T00:00:00+00:00Unknown/posts/Yum_plugins/<p><a href="https://lists.dulug.duke.edu/pipermail/yum-devel/2005-March/000730.html">Reasonable</a>
<a href="https://lists.dulug.duke.edu/pipermail/yum-devel/2005-March/000743.html">amount</a>
of
<a href="https://lists.dulug.duke.edu/pipermail/yum-devel/2005-March/000773.html">activity</a>
on yum-devel recently. Some arguing about how/if new features should be
done and lots of bits and pieces being worked on. The arguing is mainly
people proposing new features and Seth <a href="https://lists.dulug.duke.edu/pipermail/yum-devel/2005-March/000762.html">fighting against
them</a>.
I see where Seth's coming from: he's got to make sure that not any old
thing gets into yum creating a maintenance and usability nightmare.
Still, he seems to be fighting awfully hard... Sometimes I think its
better just to write some proof of concept code to show how the proposal
might work. This gets everyone on the same page and makes the arguments
more productive.</p>
<p>To help resolve some of the discussions I <a href="https://lists.dulug.duke.edu/pipermail/yum-devel/2005-March/000773.html">brought
up</a>
the idea of maybe doing plugins for yum so that people can do external
hacks for the more wierd shit without them making it into the main yum
codebase. Funnily enough Panu and Gijs piped up straight away; they'd
been thinking the same thing. Seth initially wasn't too keen on the idea
but then gave a little ground. Seth now wants us to solidify the idea
somewhat. I've got some rough ideas but haven't had the energy to put
them to the list. Maybe tomorrow.</p>