Today I released lein-test-refresh 0.24.1¹. I don’t always announce new lein-test-refresh versions with an article but this release breaks some existing behavior so I thought it was worth it.

Each of these changes is the direct result of interacting with four different lein-test-refresh users. Some of this took place on GitHub and others through email. Thanks to all of you for taking the time to think about improvements and notice oddities and bring them to my attention.

Breaking change: Monitoring keystrokes to perform actions

Prior to this release, if you hit Ctrl-D then STDIN reads an EOF and test-refresh would quit. With version 0.24.1, test-refresh no longer does that. Instead, it stops monitoring for input and just keeps running tests. Since it stops monitoring for input it no longer notices when you hit Enter to cause your tests to rerun. You can still stop lein test-refresh by sending a SIGINT with Ctrl-C.

This change was made because there is some combination of environments where if test-refresh execs /bin/bash then it receives an EOF on STDIN. Before this change, that means test-refresh would quit unexpectedly. Now it will keep going.

Thanks Alan Thompson for bringing this to my attention and taking the time to help diagnose the problem.

You can supply your own narrowing test selector

Being able to tell test-refresh to narrow its focus by adding :test-refresh/focus as metadata on a test or namespace has quickly become a favorite feature of many users. Now you can configure a shorter keyword by specifying configuration in your profile. See the sample project.clj for how to set this up.

Thanks Yuri Govorushchenko for the suggestion.

Experimental: Run in a repl

I’ve turned down this feature in the past but a narrower request came up and I thought it seemed useful. test-refresh now exposes a function you can call in a repl to run test-refresh in that repl. This makes the repl useless for any other task. To do this, first add lein-test-refresh as a dependency instead of a plugin to your project.clj. Then, require the namespace and call the function passing in one or more paths to your test directories. Example below.

1
2
3
4
5

user=> (require 'com.jakemccrary.test-refresh)
nil
user=> (com.jakemccrary.test-refresh/run-in-repl "test")
*********************************************
*************** Running tests ***************

This request was done so that you can run it in Cursive’s repl and gain the ability to click on stacktraces. Thanks Klaus Wuestefeld for bringing this up again with a really solid and focused use case.

Better output on exceptions while reloading

This was a pull request from Minh Tuan Nguyen. Now figuring out where to look for the error will be a little easier.

Thank you

Thanks to all the users of lein-test-refresh. I’ve found it to be very valuable to the way I work and I’m very happy that others do as well.

I was recently adding a feature to an internal web UI that caught all unhandled JavaScript errors and reported them to the backend service. The implementation went smoothly with most of the effort spent figuring out how to test the code that was reporting the errors.

If the error reporting failed, I didn’t want to trigger reporting another error or completely lose that error. I decided to log a reporting error to the console. I wanted to write a test showing that errors reporting errors were handled so that a future me, or another developer, didn’t accidentally remove this special error handling and enable a never ending cycle of of reporting failed reporting attempts.

It took me a while to figure out how to do this. I searched the web and found various articles about using Jasmine to do async tests. They were helpful but I also wanted to mock out a function, console.error, and assert that it was called. None of the examples I found were explicit about doing something like this. I forget how many different approaches I tried, but it took a while to figure out the below solution.

Here is the code I wanted to test.

1
2
3
4
5
6
7

function reportEvent(event) {
  return fetch('/report-event', {
    method: 'POST',
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify({name: 'ui name', ...event})
  }).catch(function(e) { console.error('Problem reporting event:', e)});
}

It takes an incoming event object and merges it with a default value and posts that to the backing service. fetch returns a Promise and the code handles errors by calling catch on it and logging.

Below is what I eventually came up with for testing the error handling feature of reportEvent.

1
2
3
4
5
6
7
8
9
10

describe('reporting events', function() {
  it('logs errors', (done) => {
    spyOn(console, 'error').and.callFake(() => {
      expect(console.error).toHaveBeenCalled();
      done();
    });
    spyOn(window, 'fetch').and.returnValue(Promise.reject('error!'));
    reportEvent({level: 'WARN', msg: 'ERROR!'});
  });
});

This uses spyOn to mock out fetch and console.error. The fetch call is told to return a rejected Promise. The console.error spy is a bit more interesting.

The console.error spy is told to call a fake function. That function asserts that the console.error spy has been called. More importantly, it also calls a done function. That done function is a callback passed to your test by Jasmine. Calling done signals that your async work is completed.

If done is never called then Jasmine will fail the test after some amount of time. By calling done in our console.error fake, we’re able to signal to Jasmine that we’ve handled the rejected promise.

You don’t actually need the expect(console.error).toHaveBeenCalled(); as done won’t be called unless console.error has been called. If you don’t have it though then Jasmine will complain there are no assertions in the test.

So there we have it, an example of using some of Jasmine’s asynchronous test features with spies. I wish I had found an article like this when I started this task. Hopefully it saves you, and future me, some time.

Leiningen test selectors are great. They allow you to filter what tests run by applying a function to the test’s metadata. If that function returns a truthy value then that test will run. lein-test-refresh supports them and even includes a built in one for its focus feature.

I was recently asked if test-refresh could support filtering tests using a regular expression against the name of a namespace or test. Lucky for me, test-refresh already supports this because of its support of test selectors.

Most of the examples of Leiningen test selectors show very simple functions that look for the existence of a keyword in the metadata. We can do more than that. We can write a predicate that does whatever we want with the metadata.

To take a look at a test’s metadata, I generated a new project and looked at the generated default test file.

1
2
3
4
5
6
7

(ns selector.core-test
  (:require [clojure.test :refer :all]
            [selector.core :refer :all]))

(deftest a-test
  (testing "FIXME, I fail."
    (is (= 0 1))))

I then used my repl and to see what metadata was on the test.

1
2
3
4
5
6
7

selector.core-test> (meta #'a-test)
{:test #function[selector.core-test/fn--17267],
 :line 5,
 :column 1,
 :file "/Users/jake/src/jakemcc/blog-examples/selector/test/selector/core_test.clj",
 :name a-test,
 :ns #namespace[selector.core-test]}

Given the metadata above, I wrote the selector below which lets us select only integration tests.

1
2
3
4
5

:test-selectors {:integration (fn [m]
                                (or (clojure.string/includes? (str (:ns m))
                                                              "integration")
                                    (clojure.string/includes? (str (:name m))
                                                              "integration")))}

You could write the above code is many different ways. Whatever you write, it needs to look for the existence of integration in either the test’s name or namespace.

If you wanted to make lein test or lein test-refresh only run non-integration tests you can add a default test selector to the project.clj.

1
2
3
4
5
6
7
8
9
10

:test-selectors {:default (fn [m]
                            (not (or (clojure.string/includes? (str (:ns m))
                                                               "integration")
                                     (clojure.string/includes? (str (:name m))
                                                               "integration"))))
                 :integration (fn [m]
                                (or (clojure.string/includes? (str (:ns m))
                                                              "integration")
                                    (clojure.string/includes? (str (:name m))
                                                              "integration")))}

Enjoy! I hope this example helps you run a subset¹ of your Clojure tests through Leiningen test selectors.

Lately, I’ve been using tmux a lot. This resulted in me figuring out how to get lein-test-refresh to send notifications using tmux.

The setup linked above works great for when I’m doing work all by myself. It showed a problem when using ssh and tmux to pair with another developer. Instead of both developers receiving a notification, only one did. One is better than none but not ideal.

Below is a GIF showing the problem. Each window simulates a different developer.

This wasn’t too hard to fix. A little digging through the tmux manpage shows that tmux display-message takes an optional flag telling it which client receives the message. If we can get a list of all the clients then iterating over them and sending a message to each is straightforward.

tmux list-clients give us this list. Below is the output.

1
2
3

$ tmux list-clients
/dev/ttys002: 0 [78x41 xterm-256color] (utf8)
/dev/ttys006: 0 [78x42 xterm-256color] (utf8)

What we care about are the parts that look like /dev/ttys002. At first I used cut to grab these values but then I dug a bit deeper into the tmux manpage.

It turns out that you can specify a format to tmux list-clients. Running tmux list-clients -F "#{client_name}" gives us the output we care about.

1
2
3

$ tmux list-clients -F "#{client_name}"
/dev/ttys002
/dev/ttys006

We can combine that with xargs to send a message to multiple clients.

That command is a bit much to put into lein-test-refresh’s configuration so I shoved it in a script called notify and configured lein-test-refresh to use it. Script and GIF of that below. Now both you and your pair can get notifications.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash

USAGE="Usage: notify <message>

example: notify 'Tests passed!'"

if [ -z "$1" ]; then
    echo "$USAGE"
    exit 1
fi

message="$1"

tmux list-clients -F "#{client_name}" \
    | xargs -n1 -I{} tmux display-message -c {} "$message"

At the beginning of every year I like to take the time to reflect on my previous year’s reading. It gives me a time to correct my data and think about where I want my reading to go in the upcoming year.

Here are links to my previous end-of-year reflections: 2013, 2014, 2015, 2016, and 2017.

I’ve continued to keep track of my reading using Goodreads. My profile continues to have the full list of the books I’ve read since 2010. Here is my entire 2018 record.

I slacked off a bit when writing reviews for all of my read books in Goodreads. I often didn’t write a review until some time had passed after completing the book and, as a result, I think I did a worse job reviewing books. Some books don’t even have a written review. I’m not a fan of this and will push myself some in 2019 to do a better job.

2018 Goal

There are a few more books on writing that I’ve wanted to read for a while. I’m planning on reading at least one of them this year. I’m also want to read more Octavia Butler. - Me (in the previous reading post)

That was my goal for 2018. It breaks down into two goals:

1. Read at least one book on writing.
2. Read more Octavia Butler.

I succeeded on the Octavia Butler goal and completely failed with the other.

2018 Numbers

I read 43 books for a total of 16,213 pages. This is a bit less than last year but still a fair amount.

Highlights

Below is a list of my five star books from 2018. The book titles link to Amazon and are affiliate links. The other links are to my Goodreads review. Unfortunately, this year I didn’t do a great job of always writing a review so some of them are missing or very short.

I generally highlight a lot of passages while reading and then rarely go back to look at them. I’ve included links to my highlights. Are they worthwhile without the context of the book? I have no idea. I’ve reread them and got something out of them but many are also so far removed from my memory that they are almost useless.

• Being Mortal: Medicine and What Matters in the End by Atul Gawande (my review, my highlights)
• Sapiens: A Brief History of Humankind by Yuval Noah Harari (my review, my highlights)
• Essentialism: The Disciplined Pursuit of Less by Greg McKeown (my review, my highlights)
• Crucial Conversations Tools for Talking When Stakes Are High by Kerry Patterson, Joseph Grenny, Ron McMillan, Al Switzler (my highlights)
• Rediscovering JavaScript: Master ES6, ES7, and ES8 by Venkat Subramaniam (my review)
• CivilWarLand in Bad Decline by George Saunders (my review)
• The Obelisk Gate (The Broken Earth #2) by N.K. Jemisin
• The Stone Sky (The Broken Earth, #3) by N.K. Jemisin (my review)
• The Hate U Give by Angie Thomas (my review)
• Six of Crows (Six of Crows, #1) by Leigh Bardugo (my review)
• Crooked Kingdom (Six of Crows #2) by Leigh Bardugo

Being Mortal: Medicine and What Matters in the End by Atul Gawande

This book deals with the end of our lives. It was great. There is a lot of good insight here. Like a lot of the non-fiction books I read, I really should go back and take notes on what I highlighted.

We’re all going to deal with death and sickness. This book can help.

Sapiens: A Brief History of Humankind by Yuval Noah Harari

My entire Goodreads review is two sentences.

This is an incredible book. You should read this. - Me

I still agree with this. My friend, Steve Deobald, described this book as “the most lucid book he’s ever read.” There is a reason this book has a 4.45 rating on Goodreads. Go read the blurb about it there and then buy and read this book¹.

Essentialism: The Disciplined Pursuit of Less by Greg McKeown

If you don’t prioritize your life, someone else will. - Greg McKeown

A really great book encouraging you to focus on what matters and, as a result, make a bigger impact and be happier. It is better to make a mile of progress on one thing instead of making inches of progress in a bunch.

Tim Ferris recently published a podcast with Greg McKeown which I’d also recommend. I’ve enjoyed listening to the podcast after a bit of time away from the book. This has helped reinforce ideas from the book. If you’re hesitant to read the book, take the time to listen and pay attention to this long podcast.

I highlighted over 100 sections of this book. I plan on revisiting these notes and this book periodically.

Crucial Conversations Tools for Talking When Stakes Are High by Kerry Patterson, Joseph Grenny, Ron McMillan, Al Switzler

A crucial conversation is one where the stakes are high, opinions vary, and emotions run strong. This book provides guidance for handling those conversations better.

I enjoyed this book and thought I picked up some useful tips from it. I think this is another where doing follow up work would help solidify some of the concepts.

Rediscovering JavaScript: Master ES6, ES7, and ES8 by Venkat Subramaniam

Do you write JavaScript?

Did you write JavaScript in the past but then move on to languages like ClojureScript and miss all the changes that happened to JavaScript?

Both of those sentences apply to me. This book has been great at catching up on modern JavaScript. I find myself referencing it while writing JavaScript and it has been very helpful. It is to the point and I find myself referencing it periodically.

CivilWarLand in Bad Decline by George Saunders

I really like this book. It is a wonderful collection of short stories. This was my second time reading it and I still enjoyed it.

The Obelisk Gate (The Broken Earth #2) and The Stone Sky (The Broken Earth, #3) by N.K. Jemisin

N.K. Jemisin has won a Hugo three years in a row. Those three years line up with each release of a book in The Broken Earth series. They are really good.

This series is great. The world is interesting and the story compelling. I highly recommend it.

The Hate U Give by Angie Thomas

Reading lets you experience life from a different perspective. This book is good. It was quickly made into a movie which is also pretty good.

I read this as part of my book club and it was universally enjoyed.

Six of Crows (Six of Crows, #1) and Crooked Kingdom (Six of Crows #2) by Leigh Bardugo

I just really enjoyed this series. I enjoyed the fantasy world it was set in and have read most of Leigh Bardugo’s other books that are set in this same world.

The series is a young adult series. It isn’t complex. The reading isn’t difficult. It isn’t going to change your life and you’re not going to be blown away by the writing. It almost feels weird to include this series in the same list as CivilWarLand and The Broken Earth series. Even still, I found myself sucked into the story and didn’t mind spending the short amount of time it took to read the books.

Non-Five Star highlights

Life 3.0: Being Human in the Age of Artificial Intelligence by Max Tegmark

amazon, my highlights

I thought this was a really interesting book.

When: The Scientific Secrets of Perfect Timing by Daniel H. Pink

amazon, my review, my highlights

I really enjoyed this. Pink references other works to build a narrative about how timing matters. When should you take a nap? Is it better to go do the doctors in the morning or afternoon? How do are cognitive abilities generally change throughout the day? How should you try to end your vacations?

I did take some notes on the book while reading it and I have referenced them. It was a good book. I should have taken more notes.

Bloodchild and Other Stories by Octavia E. Butler

amazon, my review, my highlights

This is a great collection of short stories and non-fiction articles written by Octavia Butler. I really love her writing. I’ve read a few of her works and still enjoy Lilith’s Brood the most.

Below is a quote from her about science fiction that really resonated with me. It really hits home on one of the reasons I love reading science fiction.

But still I’m asked, what good is science fiction to Black people? What good is any form of literature to Black people? What good is science fiction’s thinking about the present, the future, and the past? What good is its tendency to warn or to consider alternative ways of thinking and doing? What good is its examination of the possible effects of science and technology, or social organization and political direction? At its best, science fiction stimulates imagination and creativity. It gets reader and writer off the beaten track, off the narrow, narrow footpath of what “everyone” is saying, doing, thinking—whoever “everyone” happens to be this year. And what good is all this to Black people? - Octavia Butler

Eeeee Eee Eeee by Tao Lin

amazon, my review

This book is real bizarre. For some reason I liked it.

Factfulness: Ten Reasons We’re Wrong About the World–and Why Things Are Better Than You Think by Hans Rosling

amazon, my review, my highlights

This book is great. It is very approachable and dispels some wrong common knowledge.

Stats

I struggled generating stats this year. I kept having data issues with Goodreads. There is data that is in Goodreads that is failing to export both through their export feature and API. I’m somewhat wondering what I would need to do to track reading in a different way.

Below is the reading stats per month. The numbers are based on when the book is completed. December is partially so low because the other books all carried over to January.

Electronic books continue to make up the majority of the books I’m reading.

1
2
3
4
5

|           | 2018 | 2017 | 2016 | 2015 |
|-----------+------+------+------+------|
| ebook     |   37 |   37 |   56 |   47 |
| hardcover |    1 |    7 |    0 |    1 |
| paperback |    5 |    5 |    3 |    3 |

There are two physical books not included in my read books that I started and still need to finish. They are a both books focused on fitness (climbing injuries and proper movement) and aren’t books I’m actively reading.

Nearly a third of my reading was non-fiction. For the second year in a row, only two of those were software related.

1
2
3
4

|             | Number of books |
|-------------+-----------------|
| fiction     |              29 |
| non-fiction |              14 |

2019 Goals

I have a stack of software and process books and I’d like to read through at least some of them (others are more reference books). I’m also going to bring over the 2018 goal of reading at least one book on writing.

In a more general sense, I’m hoping to put some practices together that help me gain more from the books I’m reading. I’m still thinking through what that means.

I’ve been using Emacs in a remote tmux session lately and I’ve been missing lein-test-refresh notifications when my Clojure tests pass or fail. Luckily, it only took me a little bit of searching to figure out a solution for when I’m working inside of tmux.

Below is a GIF of the notifications I get as my tests run and pass or fail.

With the above notifications, I can keep my focus on my code and only switch to the tmux window with lein test-refresh running when a test fails.

This was pretty easy to setup. You can trigger a message in tmux by running tmux display-message <MESSAGE_HERE>. To configure lein-test-refresh to send notifications to tmux simply include the following in your :test-refresh section of your project.clj or profiles.clj.

1

:test-refresh {:notify-command ["tmux" "display-message"]}

I hope you enjoy this. Its has made using a remote terminal with tmux and lein-test-refresh more enjoyable.

In an older article of mine I extolled the virtues of having unified interfaces for interacting with your projects. I recently started working at Coinbase and the group I’m working with is mostly using makefiles as that common interface. We still have some more work to do to unify the makefile targets of the various projects but I’ve picked up one tip that makes switching between projects easier.

That tip is to have the default target of your makefile be one that prints out a helpful message. This looks like the following.

1
2
3
4
5

.PHONEY: help
help:
  @grep -E '^[0-9a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
   sort | \
   awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

There is a lot going on there but it basically looks through your makefile targets and finds the ones that have a comment starting with ## after the target dependencies. Those targets are printed to the console along with the comment.

As an example, the makefile for my website looks similar to the below file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

.PHONEY: help
help:
  @grep -E '^[0-9a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
   sort | \
   awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

.PHONEY: watch
watch: ## Watch for changes and serve preview of site with drafts
  bundle exec rake clean
  bundle exec rake preview

.PHONEY: develop
develop: ## Serve a preview of the site without drafts and refresh changes
  bundle exec rake clean
  bundle exec rake develop

.PHONEY: new_adventure
new_adventure: ## Start a new adventure post
  bundle exec rake new_adventure

.PHONEY: new_post
new_post: ## Start a new post
  bundle exec rake new_post 

.PHONEY: deploy
deploy: ## deploy
  ./deploy.sh

When this file, when I run make in this websites source, I get the following output.

1
2
3
4
5
6
7

0 [last: 0s] 21:11:50 ~/src/jakemcc/blog (master)
$ make
deploy                         deploy
develop                        Serve a preview of the site without drafts and refresh changes
new_adventure                  Start a new adventure post
new_post                       Start a new post
watch                          Watch for changes and serve preview of site with drafts

This is super useful when you’re starting doing work in a new project. With this feature you can get a quick list of useful targets and a description. It allows you to quickly see what can be done in a project.

If you follow my work you probably know that I value fast feedback cycles. Most of the open-source I maintain was developed to enable faster feedback cycles. This is why lein-test-refresh and lein-autoexpect were originally created.

Leiningen supports test selectors and lein-test-refresh does as well. This lets you start-up a testing session and only run tests or namespaces with certain metadata on them. This is a super useful feature as it lets you narrow your testing scope to one (or a handful) of tests while working on solving a specific problem.

lein-test-refresh now has built-in functionality that allows you to focus your test scope without restarting the Leiningen test process. If lein-test-refresh sees a deftest or ns form marked with :test-refresh/focus true in its metadata, then it will only run tests marked with :test-refresh/focus.

Below is an example of what this looks like.

1
2

(deftest ^:test-refresh/focus test-addition
  (is (= 2 (+ 1 1))))

This functionality has only been available for a short period of time and I’ve already found it useful. I think you will too. Enjoy.

I was recently having some difficulty debugging a problem in a ClojureScript single page application. The SPA was implemented using reagent¹.

This interface stores most of its state in a global reagent.core/atom called db. To debug the problem, I thought it would be useful to track how the global state changed as I interacted with the interface. How do we do that?

For the rest of this article, pretend that (require '[reagent.core :as reagent]) has been executed.

First, let’s define db-history in the same namespace as the global reagent/atom, db. This is where we’ll collect the changes to db.

1
2
3
4
5
6

(ns ui.data
  (:require [reagent.core :as reagent]))

(defonce db (reagent/atom {:app/current-page :offer-list}))

(defonce db-history (atom []))

Next, let’s write a function called aggregate-state. This function grabs the current value in db and conjs it onto db-history. It also limits the history to the most recent 101 states.

1
2
3
4
5
6

(defn aggregate-state []
  (let [d @db]
    (swap! db-history (fn [hist]
                        (-> (take 100 hist)
                            vec
                            (conj d))))))

Now we need to invoke aggregate-state whenever db changes. We can do this using reagent/track. reagent/track takes a function and optional arguments and invokes that function whenever a reagent/atom that function depends on changes.

reagent/track! is similar except it immediately invokes the function instead of waiting for the first change. We can use it to cause aggregate-state to get called whenever db changes.

1

(defonce db-history-logger (reagent/track! aggregate-state))

Now history of the global state is being tracked. But we need a way to access it. Below is what I ended up writing. When you call ui.data.history() in Chrome’s JavaScript console, it returns an object you can click on to explore. If you pass in strings as arguments to history then it only selects some of the data from the global db and history.

1
2
3
4
5
6
7

(defn ^:export history [& args]
  (let [d @db
        k (if (seq args)
            (map keyword args)
            (keys d))]
    (clj->js {:history (mapv (fn [x] (select-keys x k)) @db-history)
              :current (select-keys d k)})))

It only took about fifteen lines of code to gain a view of our application’s state changes over time. This view helped me solve my problem. Hopefully it will help you too.

A couple months ago I was looking into a problem and noticed that there was a situation where an expensive operation could be running simultaneously multiple times. This was wasteful.

This operation happened on a timer and could also be triggered by a power user through the UI. A power user could accidentally (or purposefully) mash on a UI button and cause the instance they’re interacting with to grind to a halt¹.

It was pretty easy to prevent. All I needed to introduce was an atom and lean on compare-and-set!. compare-and-set! is a pretty neat function (and concept found in many languages). Here is the docstring:

Atomically sets the value of atom to newval if and only if the current value of the atom is identical to oldval. Returns true if set happened, else false

Basically, compare-and-set! changes the value of an atom only if it starts from a specified value and returns a boolean letting you know if it did.

To prevent an operation from running multiple times, introduce an atom and wrap calling the operation in a conditional using compare-and-set!. After doing the work, be sure to reset! your atom back to the starting value.

Below is the code.

1
2
3
4
5
6
7
8
9
10
11
12

(defonce running? (atom false))

(defn- expensive-operation!' []
  ;; do work
  )

(defn expensive-operation! []
  (when (compare-and-set! running? false true)
    (try
      (expensive-operation!')
      (finally
        (reset! running? false)))))