Listing S3 objects with NodeJS

I recently had to write some NodeJS code which uses the AWS SDK to list all the objects in a S3 bucket which potentially contains many objects (currently over 80,000 in production). The S3 listObjects API will only return up to 1,000 keys at a time so you have to make multiple calls, setting the Marker field to page through all the keys.

It turns out there's a lot of sub-optimal examples out there for how to do this which often involve global state and complicated recursive callbacks. I'm also a fan of the clarity of JavaScript's newer async/await feature for handling asynchronous code so I was keen on a solution which uses that style.

Read more…

IMAPClient 1.1.0

IMAPClient 1.1.0 has just been released! Many thanks to the recent contributors and the project's other maintainers, Nicolas Le Manchet and Maxime Lorant. This release is full of great stuff because of them. Here's some highlights:

  • search now supports nested criteria so that more complex criteria can be expressed. IMAPClient will add parentheses in the right place. See
  • PLAIN authentication support
  • UNSELECT support
  • ENABLE support
  • UID EXPUNGE support
  • IMAP modified UTF-7 encoding/decoding now works correctly in all cases
  • the mock package is no longer installed by default (just as a test dependency)
  • many, many bug fixes

See the release notes for more details.

Much work has already gone into the 2.0 release and it won't be too far behind 1.1.0. The headline change there is reworking IMAPClient's handling of TLS.

UC Data Science talk

This week I had the pleasure of giving a talk to Data Science masters students at the University of Canterbury (NZ). It was my attempt at giving some real-world advice from many years working as a technology geek. I think it went well and I hope the audience got something useful out of it.

The slides are now available.

I also managed to promote my current gig as well. Expect an article about what I've been up to soon.

Showing all Go compiler errors

The Go compiler normally stops after it finds 10 errors, aborting with a too many errors message. For example:

$ go build
# sandbox/manyerrs
./errs.go:4: undefined: w
./errs.go:5: undefined: w
./errs.go:6: undefined: w
./errs.go:7: undefined: w
./errs.go:8: undefined: w
./errs.go:9: undefined: w
./errs.go:10: undefined: w
./errs.go:11: undefined: w
./errs.go:12: undefined: w
./errs.go:13: undefined: w
./errs.go:13: too many errors

This is useful default behaviour - if there's lots problems you usually don't care about seeing all of the issues. You just fix what you can see and try again.

Sometimes though you really want to see all the errors.

Read more…

Python async talk

I gave a talk at last night's Christchurch Python meetup about Python's relatively new asynchronous programming features. To be honest, I didn't know all that much about the topic and signed myself up for the talk to force myself to learn :)

I used Jupyter Notebook for the presentation. Its ability to mix together text and interactive Python snippets works really well for this kind of talk. I've published the slides on Github as it has native support for rendering Jupyter notebooks (thanks Github!).

IMAPClient has moved to Github

The code has been imported, tickets have migrated and the Bitbucket repository is now a redirect: the IMAPClient project is now using Git and Github.

The Github link is: https://github.com/mjs/imapclient

(http://imapclient.freshfoo.com redirects to the new location)

This change has been planned for a long time. I've noticed a that many contributors struggle with Mercurial and Bitbucket, and I greatly prefer Git myself nowadays.

Please email the mailing list or me personally if you see any problems with migration or have any questions.

IMAPClient 1.0.2

IMAPClient 1.0.2 is out! This is release comes with a few small fixes and tweaks, as well as a some documentation improvements.

Specifically:

  • There's now an explicit check that the pyOpenSSL version that IMAPClient is seeing is sufficient. This is to help with situations (typically on OS X) where the (old) system pyOpenSSL takes precedence over the version that IMAPClient needs. Use of virtualenvs is highly recommended.
  • Python 3.5 is now officially supported and tested against.
  • setup.py can now be used even if it's not in the current directory.
  • Handling of RFC2822 group address syntax has been documented.
  • The INI file format used by the live tests and interactive shell has finally been documented.
  • Links to ReadTheDocs now go to readthedocs.io
  • The project README has been arranged so that all the essentials are right at the top.

I announced that the project would be moving to Git and Github some time ago and this is finally happening. This release will be the last release where the project is on Bitbucket.