Search Changes in IMAPClient 1.0

Following on from last week's post about the upcoming IMAPClient 1.0 release, I'd like to draw attention to some significant, compatibility breaking changes related to the handling of search criteria.

IMAPClient's methods that accept search criteria (search, sort, thread, gmail_search) have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly but unfortunately also mean that small changes may be required to existing code that uses IMAPClient.

Search criteria

The preferred way to specify criteria now is as a list of strings, ints and dates (where relevant). The list should be flat with all the criteria parts run together. Where a criteria takes an argument, just provide it as the next element in the list.

Some valid examples:['DELETED'])['NOT', 'DELETED'])['FLAGGED', 'SUBJECT', 'foo', 'BODY', 'hello world'])['NOT', 'DELETED', 'SMALLER', 1000])['SINCE', date(2006, 5, 3)])

IMAPClient will perform all required conversion, quoting and encoding. Callers do not need to and should not attempt to do this themselves. IMAPClient will automatically send criteria parts as IMAP literals when required (i.e. when the encoded part is 8-bit).

Some previously accepted ways of passing search criteria will not work as they did in previous versions of IMAPClient. Small changes will be required in these cases. Here are some examples of how to update code written against older versions of IMAPClient:['NOT DELETED'])    # Before['NOT', 'DELETED']) # After['TEXT "foo"'])     # Before['TEXT', 'foo'])    # After (IMAPClient will add the quotes)['DELETED', 'TEXT "foo"'])    # Before['DELETED', 'TEXT', 'foo'])   # After['SMALLER 1000'])    # Before['SMALLER', 1000])   # After

It is also possible to pass a single string as the search criteria. IMAPClient will not attempt quoting in this case, allowing the caller to specify search criteria at a lower level. Specifying criteria using a sequence of strings is preferable however. The following examples (equivalent to those further above) are valid:'DELETED')'NOT DELETED')'FLAGGED SUBJECT "foo" BODY "hello world"')'NOT DELETED SMALLER 1000')'SINCE 03-May-2006')

Search charset

The way that the search charset argument is handled has also changed.

Any unicode criteria arguments will now be encoded by IMAPClient using the supplied charset. The charset must refer to an encoding that is capable of handling the criteria's characters or an error will occur. The charset must obviously also be one that the server supports! (UTF-8 is common)

Any criteria given as bytes will not be changed by IMAPClient, but the provided charset will still be passed to the IMAP server. This allows already encoding criteria to be passed through as-is. The encoding referred to by charset should match the actual encoding used for the criteria.

The following are valid examples:['TEXT', u'\u263a'], 'utf-8')         # IMAPClient will apply UTF-8 encoding[b'TEXT', b'\xe2\x98\xba'], 'utf-8')  # Caller has already applied UTF-8 encoding

The documentation and tests for search, gmail_search, sort and thread has updated to account for these changes and have also been generally improved.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback on the changes and/or documentation would be hugely appreciated.

TLS Changes in IMAPClient 1.0

IMAPClient 1.0 is really close to being done now and it's going to be one of the biggest releases in its history (thanks largely to the support of the good people at Nylas).

The headline feature of this release is the completely revamped TLS support. With 1.0, IMAPClient will perform certificate verification, certificate hostname checking, and will not use known-insecure TLS settings and protocols - by default. In order to work around Python historically patchy TLS support [*], IMAPClient uses backports.ssl and pyOpenSSL to provide consistent TLS functionality across all supported Python versions (2.6, 2.7, 3.3 and 3.4).

All this goodness doesn't come for free however. There were some compromises and compatibility breaks required to make it work:

  1. Due to lack of support in some of the new dependencies, IMAPClient no longer supports Python 3.2.
  2. The keyword arguments that were accepted by the IMAPClient constructor to pass through to imaplib's IMAP4 constructor are no longer supported. This was in place to provide access to imaplib's SSL arguments which are no longer relevant. Please pass a SSL context object instead.
  3. When using the default SSL context that IMAPClient creates (recommended), certificate verification is enabled and various bad TLS settings are disabled. This means that IMAPClient may fail to establish TLS connections to servers that used to work with previous versions of IMAPClient (especially if a self-signed certificate is used by the server). Fortunately it's not difficult to supply your own CA certificates or disable verification if required.

There's a new section in the manual which has more details and includes examples of how to tweak the SSL context for some common scenarios.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback would be hugely appreciated.

[*] Note that due to the hard work of various folks, TLS support is much better in Python 3.4 and 2.7.9.

IMAPClient 0.13

I'm chuffed to announce that IMAPClient 0.13 is out!

Here's what's new:

  • Added support for the ID command (as per RFC2971). Many thanks to Eben Freeman from Nylas.
  • Fixed exception with NIL address in envelope address list. Thomas Steinacher gets a big thank you for this one.
  • Fixed a regression in the handling of NIL/None SEARCH responses. Thanks again to Thomas Steinacher.
  • Don't traceback when an unparseable date is seen in ENVELOPE or INTERNALDATE responses. None is now returned instead.
  • Extended timestamp parsing support to allow for quirky timestamp strings which use dots for the time separator.
  • Replaced the horrible INTERNALDATE parsing code
  • The datetime_to_imap top-level function has been moved to the datetime_util module and is now called datetime_to_INTERNALDATE. This will only affect you in the unlikely case that you were importing this function out of the IMAPClient package.
  • The docs for various IMAPClient methods, and the HACKING.rst file have been improved.
  • CONDSTORE live test is now more reliable (especially when running against Gmail)

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

I'm also excited to announce that Nylas (formerly Inbox) has now employed me to work on IMAPClient part time. There should be a significant uptick in the development of IMAPClient.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

IMAPClient 0.12

I'm very happy to announce that IMAPClient 0.12 is out!

This is a big release. Some highlights:

  • Unicode handling has been fixed. Some bad decisions were made during the Python 3 port (v0.10) and this release fixes that. Bytes are now returned in most places (instead of unicode strings).
  • MODSEQ parts in SEARCH responses are now handled correctly. A crash has been fixed when MODSEQ queries (part of the CONDSTORE extension) are made with the search method. The returned MODSEQ value is now available via the "modseq" attribute on the returned list of ids.
  • Extra __init__ keyword args are passed through. This allows access to SSL options that the underlying imaplib library might support (Python version dependent).
  • Python 3.4 is now officially supported.
  • More control over OAUTH2 parameters.
  • The deprecated get_folder_delimiter() method has been removed.

See the NEWS.rst file and manual for more details.

Many thanks go to Inbox for sponsoring the significant unicode changes in this release.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

IMAPClient introduction

I gave a presentation introducting IMAPClient at the monthly Christchurch Python meetup on Thursday. It included a brief introduction to the IMAP protocol, the motivation for creating the IMAPClient package, some examples of how it compares to using imaplib from the standard library, and some discussion of future plans. There also may have been some silly pictures. The talk seemed to be well received. Thanks to everyone who attended.

I'm not sure how useful they'll be without the vocal part but the slides are now available online.

Apart from my talk, a wide range of topics came up including installing and managing multiple versions of Python side-by-side, TLS (SSL) and Python and the suckiness (or not) of JavaScript. As always, lots of fun. There's always a great bunch of people who make it along.