Jake McCrary

Which Clojure testing library is most used?

I’ve always assumed that the built-in clojure.test is the most widely used testing library in the Clojure community. Earlier this month I decided to test this assumption using the Google’s BigQuery GitHub dataset.

The BigQuery GitHub dataset contains over three terabytes of source code from more than 2.8 million open source GitHub repositories. BigQuery lets us quickly query this data using SQL.

Below is a table with the results (done in early March 2017) of my investigation. Surprising no one, clojure.test comes out as the winner and it is a winner by a lot.

1
2
3
4
5
6
7
8
| Library      | # Repos Using |
|--------------+---------------|
| clojure.test |         14304 |
| midje        |          1348 |
| expectations |           429 |
| speclj       |           207 |
| smidjen      |             1 |
| fudje        |             1 |

23,243 repositories were identified as containing Clojure (or ClojureScript) code. This means there were about 6,953 repositories that didn’t use any testing library1. This puts the “no tests or an obscure other way of testing” in a pretty solid second place.

You should take these numbers as ballpark figures and not exact answers. I know from using GitHub’s search interface that there are three public projects using fudje2.

So, why don’t all three of those projects show up? The dataset only includes projects where Google could identify the project as open source and the GitHub licenses API is used to do that3. Two of those three projects were probably unable to be identified as something with an appropriate license.

Another small problem is that since expectations is an actual word, it shows up outside of ns declarations. I ended up using a fairly simple query to generate this data and it only knows that expectations shows up somewhere in a file. I experimented with some more restrictive queries but they didn’t drastically change the result and I wasn’t sure they weren’t wrong in other ways. If you subtract a number between 100 and 150 you’ll probably have a more accurate expectations usage count.

Keep reading if you want to hear more about the steps to come up with the above numbers.

If you have other Clojure questions you think could be answered by querying this dataset, let me know in the comments or on twitter. I have some more ideas, so I wouldn’t be surprised if at least one more article gets written.

The Details

The process was pretty straightforward. Most of my time was spent exploring the tables, figuring out what the columns represented, figuring out what queries worked well, and manually confirming some of the results. BigQuery is very fast. Very little of my time was spent waiting for results.

1. Setup the data

You get 1 TB of free BigQuery usage a month. You can blow through this in a single query. Google provides sample tables that contain less data but I wanted to operate on the full set of Clojure(Script) files, so my first step was to execute some queries to create tables that only contained Clojure data.

First, I queried the github_repos.files table for all the Clojure(Script) files and saved that to a clojure.files table.

1
2
3
4
5
6
7
8
SELECT
  *
FROM
  [bigquery-public-data:github_repos.files]
WHERE
  (RIGHT(path, 4) = '.clj'
    OR RIGHT(path, 5) = '.cljc'
    OR RIGHT(path, 5) = '.cljs')

The above query took only 9.2 seconds to run and processed 328 GB of data.

Using the clojure.files table, we can select the source for all the Clojure code from the github_repos.contents. I saved this to a clojure.contents table.

1
2
3
SELECT *
FROM [bigquery-public-data:github_repos.contents]
WHERE id IN (SELECT id FROM clojure.files)

This query processed 1.84 TB of data in 21.5 seconds. So fast. In just under 30 seconds, I’ve blown through the free limit.

2. Identify what testing library (or libraries) a repo uses

We can guess that a file uses a testing library if it contains certain string. The strings we’ll search for are the namespaces we’d expect to see required or used in a ns declaration. The below query does this for each file and then rolls up the results by repository. It took 3 seconds to run and processed 611 MB of data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
SELECT
  files.repo_name,
  MAX(uses_clojure_test) uses_clojure_test,
  MAX(uses_expectations) uses_expectations,
  MAX(uses_midje) uses_midje,
  MAX(uses_speclj) uses_speclj,
  MAX(uses_fudje) uses_fudje,
  MAX(uses_smidjen) uses_smidjen,
FROM (
  SELECT
    id,
    contents.content LIKE '%clojure.test%' uses_clojure_test,
    contents.content LIKE '%expectations%' uses_expectations,
    contents.content LIKE '%midje%' uses_midje,
    contents.content LIKE '%speclj%' uses_speclj,
    contents.content LIKE '%fudje%' uses_fudje,
    contents.content LIKE '%smidjen%' uses_smidjen,
  FROM
    clojure.contents AS contents) x
JOIN
  clojure.files files ON files.id = x.id
GROUP BY
  files.repo_name

Below is a screenshot of the first few rows in the result.

BigQuery results for test library usage by repo

3. Export the data

At this point, we could continue doing the analysis using SQL and the BigQuery UI but I opted to explore the data using Clojure and the repl. There were too many rows to directly download the query results as a csv file, so I ended up having to save the results as a table and then export it to Google’s cloud storage and download from there.

The first few rows of the file look like this:

1
2
3
files_repo_name,uses_clojure_test,uses_expectations,uses_midje,uses_speclj,uses_fudje,uses_smidjen
wangchunyang/clojure-liberator-examples,true,false,false,false,false,false
yantonov/rex,false,false,false,false,false,false

4. Calculate some numbers

The code takes the csv file and does some transformations. You could do this in Excel or using any language of your choice. I’m not going to include code here, as it isn’t that interesting.

BigQuery thoughts

This was my first time using Google’s BigQuery. This wasn’t the most difficult analysis to do but I was impressed at the speed and ease of use. The web UI, which I used entirely for this, is neither really great or extremely terrible. It mostly just worked and I rarely had to look up documentation.

I don’t really feel comfortable making a judgment call on if the cost is expensive or not but this article cost a bit less than seven dollars to write. This doesn’t seem too outrageous to me.

Based on my limited usage of BigQuery, it is something I’d look into further if I needed its capabilities.


  1. Probably higher, as projects can and use more than one testing library.

  2. And those projects are jumarko/clojure-random, dpassen1/great-sort, and jimpil/fudje.

  3. Source is a Google Developer Advocate’s response on old HN post

Using lein-test-refresh with expectations

The 2.2.0 release1 of expectations adds a clojure.test compatible syntax. The release adds the defexpect macro which forces you to name your test but then generates code that is compatible with clojure.test.

Why would you want this? Because clojure.test is the built-in testing library for Clojure, an entire ecosystem has been built around it. Tool support for clojure.test is always going to be ahead of support for the original expectations. By using the new clojure.test compatible syntax, expectations can take advantage of all the tools built for clojure.test.

Using lein-test-refresh with expectations

If you move to the new clojure.test compatible syntax, you can start using lein-test-refresh to automatically rerun your tests when your code changes. lein-test-refresh is a fork of the original expectations autorunner, lein-autoexpect, but it has grown to have more features than its original inspiration. Now you can use it with expectations2.

Below is a sample project.clj that uses lein-test-refresh with the latest expectations.

1
2
3
4
5
(defproject expectations-project "0.1.0-SNAPSHOT"
  :description "Sample project using expectations"
  :dependencies [[org.clojure/clojure "1.8.0"]]
  :plugins [[com.jakemccrary/lein-test-refresh  "0.18.1"]]
  :profiles {:dev {:dependencies [[expectations "2.2.0-beta1"]]}})

Here is an example test file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(ns expectations-project.core-test
  (:require [expectations :refer :all]
            [expectations.clojure.test :refer [defexpect]]))

(defexpect two
  2 (+ 1 1))

(defexpect three
  3 (+ 1 1))

(defexpect group
  (expect [1 2] (conj [] 1 5))
  (expect #{1 2} (conj #{} 1 2))
  (expect {1 2} (assoc {} 1 3)))

And here is the result of running lein test-refresh.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ lein test-refresh
*********************************************
*************** Running tests ***************
:reloading (expectations-project.core-test)

FAIL in (group) (expectations_project/core_test.clj:11)
expected: [1 2]
  actual: [1 5] from (conj [] 1 5)

FAIL in (group) (expectations_project/core_test.clj:11)
expected: {1 2}
  actual: {1 3} from (assoc {} 1 3)

FAIL in (three) (expectations_project/core_test.clj:8)
expected: 3
  actual: 2 from (+ 1 1)

Ran 3 tests containing 5 assertions.n
3 failures, 0 errors.

Failed 3 of 5 assertions
Finished at 11:53:06.281 (run time: 0.270s)

After some quick edits to fix the test errors and saving the file, here is the output from the tests re-running.

1
2
3
4
5
6
7
8
9
10
11
12
13
*********************************************
*************** Running tests ***************
:reloading (expectations-project.core-test)

Ran 3 tests containing 5 assertions.
0 failures, 0 errors.
:reloading ()

Ran 3 tests containing 5 assertions.
0 failures, 0 errors.

Passed all tests
Finished at 11:53:59.045 (run time: 0.013s)

If you’re using expectations and switch to the new clojure.test compatible syntax, I’d encourage you to start using lein-test-refresh.


  1. As of 2016-02-27 2.2.0 isn’t out yet, but 2.2.0-beta1 has been released and has the changes.

  2. In fact, you have to use it if you use Leiningen and the new syntax and want your tests to run automatically.

Reading in 2016

The time has come for another end-of-year summary of my reading from the previous year. Here are links to my previous end-of-year reflections: 2013, 2014, 2015.

I’ve continued to keep track of my reading using Goodreads. My profile contains the full list and reviews of books I’ve read since 2010. Here is my full 2016 list.

2016 Goal

My 2016 goal was to read one or two biographies. It is a genre that I typically don’t read and I felt like branching out. I’m going to consider this goal achieved. I read Open (my review) by Andre Agassi (well, ghostwritten for him) and Medium Raw by Anthony Bourdain. Both have been tagged as memoirs so in the strictest sense maybe I shouldn’t count them towards my goal but I’m counting them. Open tells Andre Agassi’s life story and claims to have been fact checked so it is pretty close to a biography.

Of the two, I drastically preferred Open. I would not recommend Medium Raw unless you know you like Bourdain’s writing and the book description sounds interesting to you. Open has a broader appeal and tells an interesting story of a man’s conflict, struggle and success.

2016 Numbers

I read 59 books in 2016 for a total of 22,397 pages. I also read every issue of Amazon’s Day One weekly periodical (as I have every year since Day One started being published). Overall my rating distribution is pretty similar to 2015, with my three, four, and five star categories all containing two or three more books than the previous year.

Recommendations

I awarded twelve books a five star rating. I’ve listed them below in no particular order. The title links are affiliate links to Amazon and the review links to go my review on Goodreads.

While I enjoyed all of the above books, a few stand out.

Chasing the Scream by Johann Hari

This book is about the war on drugs. It brings together studies, history, and anecdotes to present a compelling read on addiction and drug policy. I think it would be hard to finish this book and not be persuaded to vote for policies of drug decriminalization or legalization and more humane addiction treatments. The goodreads reviews of this book are generally extremely positive, with 61% of them being five stars. Go read the book’s summary and a few reviews and you’ll want to read this book.

Deep Work by Cal Newport

This book is game changer for those of us that have hobbies or work in fields where distraction-free concentration is beneficial or required. Cal Newport makes the argument that the ability to perform deep work is critical for mastering complicated information and producing great results. The book is split in to two parts. The first defines and makes the argument for deep work. The second prescribes rules for enabling yourself to perform more deep work. I highly recommend reading this book and implementing the recommendations2.

A Little Life by Hanya Yanagihara

This is a depressing and challenging book. As this review says, this book is about a terribly broken character, Jude, and his struggles. The writing is excellent. If you feel up to reading a long, difficult book that exposes you to some terrible experiences then read this book. I’m having a hard time saying this was my favorite fiction book from last year but it is the only five star fiction book I’m specifically calling out.

Moral Tribes by Joshua Greene

I didn’t give this book five stars but I’m still going to recommend Moral Tribes by Joshua Greene. This book explains how conflict arises when two moral groups meet and interact (an “Us” vs “Them” situation) and proposes Utilitarianism as a solution to this problem. While some parts of the book were a chore to finish, I’m glad I’ve read this book. My review highlights more of what I found interesting in this book.

There is a section of the book that presents both sides of the abortion debate which I’ve shown to friends on both side of the debate. All of them seemed to enjoy reading this small section of the book.

Series read this year

I finish almost every book I start reading. I read quick enough where I find it worthwhile to finish marginal books just in case they turn out good. With a book series the end of each book provides a checkpoint for me to reevaluate if I want to continue the series. While none of the below series earned five stars, I’m highlighting them because each series represents me making the choice to continue staying in author’s world.

I continued The Expanse series this year and read books 4-6. This is just great space opera and I’ll continue reading it until it stops being written.

I started and finished Don Winslow’s The Power of the Dog series. It tells the fictionalized tale of drug cartels in Mexico. Parts are brutal and violent and unfortunately often based on real events.

I read the three main books of Ann Leckie’s Imperial Radch. It is a neat science fiction story that deals with interesting topics. As a warning, I found the first book in the series to be the weakest. Keep pushing and read the second before giving up on this story.

Scott Meyer’s Magic 2.0 was a fun read in which the main character realizes he can edit a file and change reality. It is a silly premise and it delivers good, funny, lighthearted reading.

I also read Neil Gaiman’s American Gods and Anasazi Boys. I devoured both of these books.

Technical books read

I didn’t read many books on software in 2016. I read the lowest number of software related books since I’ve started keeping track. I finished Ben Rady’s Serverless Single Page Apps and Ron Jeffries' The Nature of Software Development. I also read most of Google’s Site Reliability Engineering but did not finish it before the end of 2016. I read more non-fiction books than in 2015 but I would have liked to see more software books in the mix.

More stats

There were definitely a couple months where my reading took a definite dip. I don’t remember what I was doing during April or September but apparently I wasn’t reading.

Chart of reading per month

Unsurprisingly, ebooks continued to be my preferred format. This isn’t surprising and I expect that this trend continues. At this point the only reason I’ll even potentially look at this stat next year is because it forces me to go through and confirm each book had its format recorded correctly.

1
2
3
4
5
|           | 2014 | 2015 | 2016 |
|-----------+------+------+------|
| ebook     |   64 |   47 |   56 |
| hardcover |    1 |    1 |    0 |
| paperback |    4 |    3 |    3 |

My average rating was about the same as 2015.

1
2
3
4
5
6
7
8
| Year | Average Rating |
|------+----------------|
| 2011 |           3.84 |
| 2012 |           3.66 |
| 2013 |           3.67 |
| 2014 |           3.48 |
| 2015 |           3.86 |
| 2016 |           3.83 |

I read multiple books by quite a few authors. Below is a table summarizing my repeat

1
2
3
4
5
6
7
8
9
| Author           | Average Rating | Number of Books | Number of Pages |
|------------------+----------------+-----------------+-----------------|
| Ann Leckie       |          3.667 |               3 |            1205 |
| James S.A. Corey |          3.667 |               3 |            1673 |
| Scott Meyer      |              4 |               3 |            1249 |
| Don Winslow      |              4 |               2 |            1182 |
| Junot Díaz       |              4 |               2 |             565 |
| Michael Crichton |            3.5 |               2 |             814 |
| Neil Gaiman      |              4 |               2 |             981 |

2017 goals

This year I’m planning on revisiting some of my favorite books. I’m not going to set a concrete number for this goal and will just have to trust myself to honestly judge if I accomplish this goal.


  1. This review is currently blank on Goodreads because this was a book read for a book club I’m part of and we haven’t met to discuss the book yet. We typically don’t post reviews online of books we’ll be discussing. To any of my book club members that are reading this post, I apologize for spoiling the surprise of what star rating I’m planning on giving this book.

  2. I’m considering writing up more about this book and some changes I’ve made to help myself do more deep work.

Making code fast: Measure what you intend to measure

I’ve spent a significant portion of my career figuring out how to make software run faster. It is a problem I enjoy solving. One of the most important steps in an optimization task is to identify what you are trying to optimize and how you will measure it. Answer these questions wrong and you’ll waste your time solving the wrong problem.

Recently I joined a teammate on a task that involved identifying a bottleneck in a Clojure code base. We knew the code path we needed to optimize and turned to the Tufte library to take timing measurements. This was my first time using Tufte and, with my tiny amount of usage, I like what I see.

At some point in the process, we had code1 that looked similar to the translate function below (lines 20-24).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
(ns bench.core
  (:require [clojure.string :as string]
            [taoensso.tufte :as tufte]))

(defn raw->maps [lines]
  (map (fn [line]
         (zipmap [:a :b :c]
                 (map (fn [s] (Long/parseLong s))
                      (string/split line #"\|"))))
       lines))

(defn summarize [maps]
  (reduce (fn [r m]
            (-> r
                (update :a (fnil + 0) (:a m))
                (update :b (fnil + 0) (:b m))
                (update :c (fnil + 0) (:c m))))
          maps))

(defn translate [lines]
  (tufte/profile {}
                 (let [maps (tufte/p ::raw->maps (raw->maps lines))
                       summary (tufte/p ::summarize (summarize maps))]
                   summary)))

Here is some Tufte output from running some data through translate.

1
2
3
4
5
              pId      nCalls       Min        Max       MAD      Mean   Time% Time
:bench.core/summarize           1   346.0ms    346.0ms       0ns   346.0ms     100 346.0ms
:bench.core/raw->maps           1    2.46µs     2.46µs       0ns    2.46µs       0 2.46µs
       Clock Time                                                          100 346.05ms
   Accounted Time                                                          100 346.0ms

Notice anything surprising with the output?2

It surprised me that raw->maps took such a tiny amount of time compared to the summarize function. Then I realized that we had forgotten about Clojure’s lazy sequences. summarize is taking so much of the time because raw->maps is just creating a lazy sequence; all the work of realizing that sequence happens in summarize. By wrapping the call to raw->maps with a doall we were able to get the time measurements we intended.

This example demonstrates an important lesson. When you are profiling code, make sure you are measuring what you think you are measuring. This can be challenging in languages, such as Clojure, that have a concept of laziness. Reflect on your measurement results and perform a gut check that the results make sense with what you intended to measure. If anything feels off, confirm that you’re measuring what you meant to measure.


  1. Example built using clojure 1.8.0 and tufte 1.1.1. Also, sorry for the terrible names of functions. I was drawing a blank when coming up with this example.

  2. Imagine this output having 10 more lines in it. Now imagine it having 20. It starts getting quite a bit more difficult to notice oddities as more and more lines get added to this output. Try not to overwhelm yourself by having too much output.

Unify your project interfaces

Jeff Ramnani wrote an article about unifying your command line interactions across programming projects. I recommend that you read it. The basic gist is that we often find ourselves working on multiple projects at a time. Frequently these projects are in different languages and use different build tools. Remembering the necessary incantations to interact with the various projects is difficult and we’re lazy. We can do better by standardizing an interface to our projects.

This interface can take many forms. One option is to have a bin or scripts directory in each project and then consistently name the scripts you put in there (examples: run, test, and build). Another option is to use Makefiles with consistently named targets. Either way, your projects now have a standard way of interacting with them. This frees you from having to remember all the various commands and makes onboarding new developers easier.

I’ve been using a similar approach to Jeff Ramnani for years and highly recommend it. I’m a fan of the Makefile approach but either approach works. The unified targets I use across projects are the following:

  • up - Brings the system up
  • status - Is the system up and running?
  • logs - Show me the logs
  • local-db - Connect to my local database
  • build - Build the project
  • test - Run the tests

If you haven’t created a common interface for your projects I recommend that you do it. It definitely makes moving between projects easier.

HTML markup for better sharing on social media

For a bit more than a year I worked on a project that crawled the web and indexed articles. Two of our sources of data were articles shared on Facebook and Twitter. After seeing hundreds of article previews on these two social networks, I decided to improve how my own articles were previewed.

I thought figuring out the markup I needed to add would be a painless experience. Unfortunately, when you search for this information you end up at various SEO optimization and other similar sites where you get the pleasure of experiencing full screen pop-overs trying to get you to sign up for mailing lists and other annoying features of the modern web. Probably unsurprisingly, the least annoying source for this information turned out to be the social networks themselves.

Below is what you will want to add to the <head> section of your articles' markup. Items in all caps should be values that make sense for your articles. Most fields are pretty self-evident, but check Twitter’s and Facebook’s documentation for more details. The Open Graph documentation has more details as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!-- Twitter Card data -->
<meta name="twitter:card" content="SUMMARY" />
<meta name="twitter:site" content="TWITTER HANDLE OF SITE (@jakemcc for this site)" />
<meta name="twitter:creator" content="YOUR TWITTER HANDLE" />
<meta name="twitter:title" content="ARTICLE TITLE" />
<meta name="twitter:description" content="SHORT DESCRIPTION OF CONTENT" />
<meta name="twitter:image" content="IMAGE THAT SHOWS UP WITH PREVIEW" />

<!-- Open Graph data -->
<meta property="og:site_name" content="SITE TITLE" />
<meta property="og:url" content="CANONICAL URL" />
<meta property="og:title" content="ARTICLE TITLE" />
<meta property="og:description" content="SHORT DESCRIPTION OF CONTENT" />
<meta property="og:image" content="IMAGE THAT SHOWS UP WITH PREVIEW" />
<meta property="og:type" content="article" />
<meta property="article:published_time" content="PUBLISHED DATETIME" />

If you have control of your site’s markup and want better previews of your articles on the various social networks then you should add this markup to your web site 1. Hopefully this article has saved you from having a full screen pop-over prompt you to join yet another mailing list.


  1. You can actually remove the twitter:title, twitter:description, and twitter:image lines since Twitter will fallback to the equivalent Open Graph markup if they missing.

Better command history in your shell

My ideal command history would let me search the history of every shell but when I hit the up arrow it would only cycle through my current shell’s history. In February, I was able to achieve this setup in large part because of a utility called hstr.

What is hstr?

hstr is a neat Bash and Zsh utility that lets you easily search, view, and manage your command history. hstr provides a tool named hh that provides a text interface for manipulating your command history. To see what it looks like check out the README and this video tutorial. If you are running OS X and use Homebrew you can install it by running brew install hh.

Making global history searchable but arrows cycle through local history

hstr is a neat tool but my favorite part of my setup is how the global command history is searchable but only a shell’s local history is cycled through with the arrow keys. This is achieved by manipulating where history is written and tweaking some environment variables.

The first step is to change your $PROMPT_COMMAND to append your shell’s history to a global history file. Below is the snippet that does this from my .bashrc file.

1
2
# Whenever a command is executed, write it to a global history
PROMPT_COMMAND="history -a ~/.bash_history.global; $PROMPT_COMMAND"

The next step is to bind a keystroke to run hh, which is what hstr provides, with $HISTFILE pointing to ~/.bash_history.global. I wanted to fully replace the default command history searching (and I use Emacs style keyboard shortcuts) so I’ve bound these actions to ctrl-r.

1
2
# On C-r set HISTFILE and run hh
bind -x '"\C-r": "HISTFILE=~/.bash_history.global hh"'

With those two additions to my .bashrc I’ve achieved my ideal command history searching. When I hit ctrl-r I’m searching all of my history and yet I only cycle through a shell’s local history with the arrow keys. This small addition1 made my command line productivity higher.


  1. My setup was inspired by this StackExchange post.

Better code reloading in a Clojure web server

A couple weeks ago I released com.jakemccrary/reload. This tiny library provides a ring middleware that uses org.clojure/tools.namespace to reload changed Clojure code on incoming http requests.

This middleware was created because my team was running into problems using ring’s wrap-reload middleware. Unfortunately these problems happened about nine months ago and, since I didn’t write this post back then, I’ve since forgotten these problems. Regardless, this project has been used since the beginning of this year and has helped make my team’s development workflow smoother. If you are running into problems it might help you too.

Usage

If you’d like to give it a shot, then add the latest version (at the time of writing [com.jakemccrary/reload "0.1.0"]) to your project.clj.

Require com.jakemccrary.middleware.reload and wrap your handler with wrap-reload.

1
2
3
4
5
6
7
(ns example
  (:require
   ;; more deps
   [com.jakemccrary.middleware.reload :as reload]))

;; wherever you are setting up your middleware stack
(reload/wrap-reload routes)

reload/wrap-reload optionally takes a list of directories to monitor as a second parameter. By default it reloads the src directory.

AWS Elastic Beanstalk: Send a SQS message to a specific route in your worker environment

Lumanu uses AWS Elastic Beanstalk. Elastic Beanstalk (from now on abbreviated as EB) helps you provision and tie together Amazon services to fairly easily get web applications and services running with push button (or command line) deploys. We’ve been using EB with a multi-container docker deploy for nearly a year now and it pretty much just works.

EB has a concept of environment tiers and there are two different types; a web tier and a worker tier. Web tier environments provide the configuration and components necessary for serving HTTP requests in a scalable fashion. Worker environments are designed to run operations that you wouldn’t want performed by your front-end serving web application.

A major difference between the two environment tiers is that a worker environment provisions a SQS queue and provides a daemon that reads from this queue and POSTs messages to an instance of your worker service. This daemon prevents your worker service from having to connect to and manage a SQS queue. By default, the daemon POSTs messages to http://localhost/. You can optionally configure it to POST to a different route.

It is possible to have different messages POST to different routes. You can do this by setting the beanstalk.sqsd.path attribute on your SQS message. For example, if you want your worker service to receive a message at /trigger-email you would set the beanstalk.sqsd.path attribute to /trigger-email.

7 tips for a successful remote meeting

As mentioned in my tips for working from home article, I’ve been working remotely for nearly three years. In those three years I’ve been in countless meetings, both productive and unproductive. Meetings, both in-person and remote, are hard. Remote meetings pose some additional challenges but they also offer some unique benefits.

Below are seven tips that will help you have successful remote meetings. I wrote this article focusing on remote meetings but many of these tips will improve your co-located meetings as well.

Have an agenda

If you are organizing a meeting, you should make sure that the meeting has an agenda. The agenda doesn’t have to be a long, complicated document; it can be as simple as a list of topics and goals.

Why should you have an agenda? An agenda helps focus discussion by providing an outline of what the meeting is designed to cover.

Send out your agenda with the meeting invite. This gives invitees time to think about the topics and helps prevent people from showing up clueless. It also provides an opportunity for an invitee to excuse themself or suggest an alternative person if they don’t believe they will contribute to the meeting.

Start and end on time

You should start your meetings on time. You should end your meetings on time or early. If you are not starting and ending on time then you are not being respectful of your attendees time. The lack of punctuality contributes towards people dreading meetings.

If you are running out of time, wind down the meeting. If more discussions need to happen, reflect and see if more time needs to be scheduled with the entire group or if only a subset of the group is required.

Use video chat

Even if you don’t work remotely, you’ve probably had to dial in to a group audio chat. This is almost always a terrible experience. Without body language, it is near impossible to tell when someone is about to start speaking and, as a result, there are awkward pauses while everyone waits for someone else to speak and everyone speaks over each other. It is terrible.

This is why I recommend using video chat. Video chats let you see the other people on the call and this allows you to pick up on physical cues. These cues vastly improve communication in the meeting.

Co-located attendees should use their own device

Sometimes you’ll have a mixed meeting, some attendees are remote and others are together in an office. The co-located attendees should each use their own device to connect to the meeting.

Co-located attendees sharing a single device is non-optimal for many reasons. It is often hard for all the co-located attendees to be captured by the camera in a way that enables the remote attendees to reliably view them. Sharing a single microphone also makes it so some co-located attendees are easy to hear and others are barely audible.

Using a single device also makes it harder for all the co-located attendees to view the remote attendees. Without a clear view of the remote attendees, the co-located attendees often accidentally exclude the remote people by focusing on discussions between the co-located group.

Ignore distractions

Hopefully you have invited just the right people to the meeting and everyone is engaged in the discussion and paying attention. Realistically this doesn’t happen. Computers are incredibly good at so many things and one of those things is distracting the user.

When you are attending a remote meeting, minimize what can distract you. Close your email and hide the chat program. Put your phone out of arms reach. Try to focus intently on the meeting. If you find yourself not paying attention and not contributing, take this as a signal that you shouldn’t be in this meeting. If nothing on the rest of the agenda seems like it requires you, then leave the meeting and be more selective about what you join in the future.

If you notice other attendees not paying attention, gently call them out on it. This can be done by soliciting discussion from them or by being direct.

Have a shared artifact

This is one of the more important tips in this list and it is one of the areas where remote meetings have an advantage over in-person meetings.

When the meeting starts give everyone a link to a shared document that everyone can edit (for example a Google Doc). It can be useful to seed this document with the agenda. This shared document can be used to capture whatever you want. I’ve found it useful to capture options discussed, pros/cons lists, and follow-up actions. Writing in the shared document helps solidify ideas and gives the group a reference both during and after the meeting.

With in-person meetings, this shared artifact is often a whiteboard. Whiteboards are immensely useful but are barely editable by more than one person at once and are harder to reference after a meeting. I know I’m not the only person who dislikes trying to decipher terrible whiteboard handwriting captured by someone’s phone.

Except for when drawing diagrams, I’ve found the Google Docs style shared document used during a remote meeting more effective than using a whiteboard in an in-person meeting. You can always use a shared document in an in-person meeting as well but then you are requiring attendees to have a laptop open and that is an invitation for distracted attendees.

Assign responsibilities

Hopefully you are having a meeting to influence an outcome and not just hear everyone talk. As a result, you should be assigning follow-up responsibilities as the meeting progresses. Make the follow-up actions explicit and assigned to an individual. You can capture these responsibilities in your shared artifact.


Meetings can be difficult. You should do what you can to make them more successful. If you are being invited to a meeting without an agenda, ask for an agenda. If you’re in a meeting and you can tell someone is constantly distracted, try nicely calling them out on it (either privately or in the group). If there isn’t a shared artifact, make one and suggest it to the group. Meetings don’t have to be terrible. We can make them better.