IMAPClient 1.1.0

IMAPClient 1.1.0 has just been released! Many thanks to the recent contributors and the project's other maintainers, Nicolas Le Manchet and Maxime Lorant. This release is full of great stuff because of them. Here's some highlights:

  • search now supports nested criteria so that more complex criteria can be expressed. IMAPClient will add parentheses in the right place. See
  • PLAIN authentication support
  • UNSELECT support
  • ENABLE support
  • UID EXPUNGE support
  • IMAP modified UTF-7 encoding/decoding now works correctly in all cases
  • the mock package is no longer installed by default (just as a test dependency)
  • many, many bug fixes

See the release notes for more details.

Much work has already gone into the 2.0 release and it won't be too far behind 1.1.0. The headline change there is reworking IMAPClient's handling of TLS.

UC Data Science talk

This week I had the pleasure of giving a talk to Data Science masters students at the University of Canterbury (NZ). It was my attempt at giving some real-world advice from many years working as a technology geek. I think it went well and I hope the audience got something useful out of it.

The slides are now available.

I also managed to promote my current gig as well. Expect an article about what I've been up to soon.

Showing all Go compiler errors

The Go compiler normally stops after it finds 10 errors, aborting with a too many errors message. For example:

$ go build
# sandbox/manyerrs
./errs.go:4: undefined: w
./errs.go:5: undefined: w
./errs.go:6: undefined: w
./errs.go:7: undefined: w
./errs.go:8: undefined: w
./errs.go:9: undefined: w
./errs.go:10: undefined: w
./errs.go:11: undefined: w
./errs.go:12: undefined: w
./errs.go:13: undefined: w
./errs.go:13: too many errors

This is useful default behaviour - if there's lots problems you usually don't care about seeing all of the issues. You just fix what you can see and try again.

Sometimes though you really want to see all the errors.

Read more…

Python async talk

I gave a talk at last night's Christchurch Python meetup about Python's relatively new asynchronous programming features. To be honest, I didn't know all that much about the topic and signed myself up for the talk to force myself to learn :)

I used Jupyter Notebook for the presentation. Its ability to mix together text and interactive Python snippets works really well for this kind of talk. I've published the slides on Github as it has native support for rendering Jupyter notebooks (thanks Github!).

IMAPClient has moved to Github

The code has been imported, tickets have migrated and the Bitbucket repository is now a redirect: the IMAPClient project is now using Git and Github.

The Github link is: https://github.com/mjs/imapclient

(http://imapclient.freshfoo.com redirects to the new location)

This change has been planned for a long time. I've noticed a that many contributors struggle with Mercurial and Bitbucket, and I greatly prefer Git myself nowadays.

Please email the mailing list or me personally if you see any problems with migration or have any questions.

IMAPClient 1.0.2

IMAPClient 1.0.2 is out! This is release comes with a few small fixes and tweaks, as well as a some documentation improvements.

Specifically:

  • There's now an explicit check that the pyOpenSSL version that IMAPClient is seeing is sufficient. This is to help with situations (typically on OS X) where the (old) system pyOpenSSL takes precedence over the version that IMAPClient needs. Use of virtualenvs is highly recommended.
  • Python 3.5 is now officially supported and tested against.
  • setup.py can now be used even if it's not in the current directory.
  • Handling of RFC2822 group address syntax has been documented.
  • The INI file format used by the live tests and interactive shell has finally been documented.
  • Links to ReadTheDocs now go to readthedocs.io
  • The project README has been arranged so that all the essentials are right at the top.

I announced that the project would be moving to Git and Github some time ago and this is finally happening. This release will be the last release where the project is on Bitbucket.

Nikola

Those paying attention may have noticed that this site has changed. Up until recently, freshfoo.com was generated using a combination of PyBlosxom and rest2web but it's now using the wonderful Nikola static site generator. The change was prompted by a desire to have more flexibility about freshfoo.com's hosting situation - moving a static site around is far easier than a dynamically generated one. A static site also provides a snappier experience for the end user and there really wasn't a need for the site to be dynamically generated anyway.

Nikola also provides some nice tooling for the site author to help with the writing process. The nikola auto command which serves up the site locally, automatically rebuilding as you work on it and automatically refreshing browser tabs which are viewing the site is especially useful.

Read more…

IMAPClient mailing list moved to Groups.io

The IMAPClient mailing list has been hosted at Librelist for a while. It worked OK initially but the site doesn't seem to be getting much love these days. People also tend to find the "send an email to subscribe" model confusing.

After looking at a number of options, I've decided to shift the mailing list to Groups.io. It's a well run mailing list service which allows interaction via both the web and email. There's lots of nice bells and whistles but the main benefit I see is that it seems to be well thought out and efficient.

To join the new list or to find out more, take a look at https://groups.io/g/imapclient.

The change to the new mailing list is effective immediately. Please use the new one from now on. I'm going to send out invites to people who were active on the old list. IMAPClient's documentation has already been updated to reflect the change.

To ensure they aren't lost, the historical messages from the old list on librelist.com will be imported into new one.

Please email me or comment here if you have any feedback about the change.

IMAPClient 1.0.0

IMAPClient 1.0.0 is finally here! This is a monster release, bursting with new features and fixes.

Here's the highlights:

Enhanced TLS support: The way that IMAPClient establishes TLS connections has been completely reworked. By default, IMAPClient will attempt certificate verification and certificate hostname checking, and will not use known-insecure TLS settings and protocols. In addition, TLS parameters are now highly configurable.

This change necessitates that backwards compatibility has been broken, and also means that IMAPClient has a bunch of new dependencies. Please see the earlier blog article about the TLS changes as well as the release notes for more information.

STARTTLS support: When the server supports it, IMAPClient can now establish an encrypted connection after initially starting with an unencrypted connection using the STARTTLS command. The starttls method takes an SSL context object for controlling the parameters of the TLS negotiation.

Many thanks to Chris Arndt for his extensive initial work on this.

Robust search criteria handling: IMAPClient's methods that accept search criteria have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly.

Unfortunately these changes also mean that small changes may be required to existing code that uses IMAPClient. Please see the earlier blog article about the search changes as well as the release notes for more information.

Socket timeout support: IMAPClient now accepts a timeout at creation time. The timeout applies while establishing the connection and for all operations on the socket connected to the IMAP server.

Semantic versioning: In order to better indicate version compatibility to IMAPClient's users, the project will now strictly adhere to the Semantic Versioning scheme.

Performance optimisation for parsing message id lists: A short circuit is now used when parsing a list of message ids which greatly speeds up parsing time.

Installation via wheels: In addition to .zip and .tar.gz files, IMAPClient releases are now also available as universal wheels.

There have also been many other smaller fixes and improvements. See the release notes and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded via IMAPClient site.

This release couldn't have been possible with the amazing support of Nylas. If you're developing software that needs to deal with email, save yourself a whole lot of pain and check out their email platform. If you're after a modern, extensible, cross-platform email client check out N1.

Search Changes in IMAPClient 1.0

Following on from last week's post about the upcoming IMAPClient 1.0 release, I'd like to draw attention to some significant, compatibility breaking changes related to the handling of search criteria.

IMAPClient's methods that accept search criteria (search, sort, thread, gmail_search) have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly but unfortunately also mean that small changes may be required to existing code that uses IMAPClient.

Search criteria

The preferred way to specify criteria now is as a list of strings, ints and dates (where relevant). The list should be flat with all the criteria parts run together. Where a criteria takes an argument, just provide it as the next element in the list.

Some valid examples:

c.search(['DELETED'])
c.search(['NOT', 'DELETED'])
c.search(['FLAGGED', 'SUBJECT', 'foo', 'BODY', 'hello world'])
c.search(['NOT', 'DELETED', 'SMALLER', 1000])
c.search(['SINCE', date(2006, 5, 3)])

IMAPClient will perform all required conversion, quoting and encoding. Callers do not need to and should not attempt to do this themselves. IMAPClient will automatically send criteria parts as IMAP literals when required (i.e. when the encoded part is 8-bit).

Some previously accepted ways of passing search criteria will not work as they did in previous versions of IMAPClient. Small changes will be required in these cases. Here are some examples of how to update code written against older versions of IMAPClient:

c.search(['NOT DELETED'])    # Before
c.search(['NOT', 'DELETED']) # After

c.search(['TEXT "foo"'])     # Before
c.search(['TEXT', 'foo'])    # After (IMAPClient will add the quotes)

c.search(['DELETED', 'TEXT "foo"'])    # Before
c.search(['DELETED', 'TEXT', 'foo'])   # After

c.search(['SMALLER 1000'])    # Before
c.search(['SMALLER', 1000])   # After

It is also possible to pass a single string as the search criteria. IMAPClient will not attempt quoting in this case, allowing the caller to specify search criteria at a lower level. Specifying criteria using a sequence of strings is preferable however. The following examples (equivalent to those further above) are valid:

c.search('DELETED')
c.search('NOT DELETED')
c.search('FLAGGED SUBJECT "foo" BODY "hello world"')
c.search('NOT DELETED SMALLER 1000')
c.search('SINCE 03-May-2006')

Search charset

The way that the search charset argument is handled has also changed.

Any unicode criteria arguments will now be encoded by IMAPClient using the supplied charset. The charset must refer to an encoding that is capable of handling the criteria's characters or an error will occur. The charset must obviously also be one that the server supports! (UTF-8 is common)

Any criteria given as bytes will not be changed by IMAPClient, but the provided charset will still be passed to the IMAP server. This allows already encoding criteria to be passed through as-is. The encoding referred to by charset should match the actual encoding used for the criteria.

The following are valid examples:

c.search(['TEXT', u'\u263a'], 'utf-8')         # IMAPClient will apply UTF-8 encoding
c.search([b'TEXT', b'\xe2\x98\xba'], 'utf-8')  # Caller has already applied UTF-8 encoding

The documentation and tests for search, gmail_search, sort and thread has updated to account for these changes and have also been generally improved.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback on the changes and/or documentation would be hugely appreciated.