beau hilton


layout: post title: “Remember all the things” categories: - learning tags: - spaced repetition - desirable difficulty - education - learning

- how-to

We spend so much time reading

and forgetting.

Like most of you, I read a silly number of books, journal articles, blog posts, whatever. I forget the vast majority of what I read, almost immediately. Chances are, so do you. We're humans.

The brain is a conservative organ, ruthlessly efficient. It kills memories left and right unless it has a really good reason not to.

The upper limit of short term memory is 6-8 items, and most of what we learn is dumped within a day.1 However, there is no known upper limit on long-term memory. Let that sink in: no known upper limit. That is, memory is in-finite: we don't know how much we can know. We are still exploring the far reaches of long-term memory, and I'm excited. What will be possible after decades of purposeful retention of all the interesting and useful things?2

Here is a system cobbled together, with the help of many friends, to convince the brain to have a little mercy.3 As of October 2018, I've been doing this for a few months, and it's wonderful. It is just cumbersome enough to create conditions for learning, but not so much that I find excuses to skip the process.4 One difference between this system and what I used to do is the proposed longevity: instead of learning things to pass a class, or do a thing, my goal is to have a coherent way to gain and keep knowledge for life, in all its domains.5

In order to keep this page pretty and focus on process rather than tools, I put specific technologies and commentaries in the clickable footnotes.

Find the thing:

Hoard:

Read:

Review:

Retain:

If you have suggestions for improvement, questions, tools, or experiences, I would love to hear them.


  1. The average working memory of primates, including humans, is 4 items. The range is 1-8, ish, depending on the details of the study and participants. See EK Miller's lab page for links to recent work, especially Miller, E.K. and Buschman, T.J. (2015) Working memory capacity: Limits on the bandwidth of cognition. Daedalus, Vol. 144, No. 1, Pages 112-122. A free PDF of this paper is available on the linked website. 

  2. I have a friend who has kept up on his spaced repetition tools from the first US medical licensing exam (10-15k digital flashcards, all the things you learn in the first two years of medical school), and I positively seethe with envy at how little review he will have to do when he takes the second exam. 

    Much, if not most of my preparation time for the second exam was re-learning things I had learned for the first exam and forgotten. My proof of this was that when I missed practice questions, I could usually search my old flashcard deck and find the exact answer. How much better would life be if I could keep a larger proportion of what I learn, especially if I do it with a system that takes minutes per day?

    Of course, not everything needs to be retained, and the medical licensing exams are, unfortunately, riddled with trivia (i.e. things that are highly testable, but of little use clinically, such as the chromosomal location of a very rare genetic disorder: most doctors will never see the disease, and if they do the last thing that will matter is the location on the chromosome).

  3. This did not come out of a vacuum. I started with spaced repetition in undergrad under the guidance of a brilliant tutor, then learned from my own students, and am continually indebted to countless classmates and generous souls online. 

    I am deeply grateful to Harvard Macy Institute and Dr. Neil Mehta, in particular, for pushing me to formalize my thinking and get beyond banal “cram and regurgitate” methods for exams and classrooms. They were also the first to give me a framework, and provided many of these tools.

    These ideas are not new, and have been presented in many forms: my contribution here is a personal system for putting it all together, with my preferred tools. The key addition is spaced repetition, which makes it much more likely for ideas to be imprinted deeply and available quickly.

  4. Free PDF article on desirable difficulty from UCLA available here

  5. For the philosophic among you: I study artificial intelligence, tools that can consider trillions of variables all at once. The goal is to outsource certain types of thinking that computers are better at, for the benefit of (in my case) people with serious illness. 

    This fits with the basic story of technology: outsource a thing to amplify it (I can pick up a rock: a crane can pick up a house; I can run: an airplane can fly; I can remember some things: Google remembers all the things).

    A few technologies, however, are about insourcing, that is, developing the human element to its maximum. Spaced repetition is one of these technologies, optimized with proven algorithms to maximize how much we can learn and retain. What will be possible when highly developed human memories couple with human creativity, as well as technological support from AI and other advances? So far the research on AI suggests that the very best systems are not solely computers or solely humans, but hybrids, humans working with machines, emphasizing the strengths of both. What happens when we optimize each component? For good and for bad?

  6. In order, these are my preferred ways to find things: 1. word of mouth (in-person, ideally, but a well-crafted newsletter is also a delight. Despite all this technology, I find the best way to address the problem of “unknown unknowns” is through respected friends and colleagues). 2. purposeful search (goal-driven: help this patient, write this paper/blog post, scratch this itch). 3. automatic digests (RSS feeds of keywords, journals, authors, etc.) I find this approach less useful for students, who have usually not yet differentiated into their specific interests, because any feeds they set up will be too noisy. The link above is great for an overview and examples, but their intake funnel is too wide for my taste. For example, they include the general feed from Nature in addition to highly idiosyncratic feeds. Nature has some interesting stuff and is a blast to browse, but my goal with RSS is to be targeted and save time: keep the idiosyncracies and ditch Nature, unless you set up a filter for only the types of articles from Nature you actually care about. 

  7. Quick scan, check out the pictures, skim abstract if available, consider strength of recommendation. Certain people have given me such delicious recommendations in the past that anything they now recommend skips to the head of the reading list. 

  8. Diigo for standard web fare, Zotero for journal articles with PDFs. 

    We have free, infinitely large Google Drives (for life!) at my university, so I use the process described here to keep the PDFs available in the cloud. If I use this I can read and markup on my phone, computer, whatever, and it will automatically sync.

  9. For most web-first content, Diigo is great, and I use Xodo for PDF markup. 

    Xodo is the best free full-featured PDF tool I have found: quick, reliable, support for search, annotation (and signatures), highlighting.

    SumatraPDF is the fastest PDF reader I have found, great for pure ctrl-F search especially with huge PDF textbooks, but achieves this by being absolutely bare-bones (no handwritten annotation, highlighting, etc.).

    I also love paper and pen, and because I put the things I want to keep into spaced repetition software, I can usually recycle the paper without fear of losing something important.

  10. Highlighting is not a good method to engage with text and make it sink in. This has been studied, repeatedly. One of the studies cited in the linked review showed that highlighting impairs comprehension. 

    However, if the highlight signals something meaningful and consistent, such as only the kind of information you might want to memorize, you can use the visual cue to quickly find, review, and make decisions about your next steps.

    I still do most of my reading on paper, where highlighting makes review easy. In an app like Diigo or Kindle, your highlights are extracted into a separate document, which tightens review even further.

  11. Here's where highlighting only the things you might want to memorize shows its utility. 

    You may also find that your annotations need annotations, e.g. a question you asked was answered later in the paper.

  12. If something seems worth the 5-10 minutes it will take to make and review a flash card, do it. If not, screw it. Also, develop a low threshold for fixing or deleting cards. This makes card creation in the first place less of a daunting task. 

  13. If you keep all your digital notecards in a tool like Anki, you will have the spaced repetition part automated, provided you remember to open the app every day. 

    If you keep everything in a single Anki deck, regardless of subject, you will also gain the benefit of interleaving. Interleaving is one of a handful of techniques shown to improve memory and understanding, and basically involves mixing material freely without regard for category.

    For example, instead of homework with all quadratic equations, mix in all the math from the whole semester in random assortment. In medicine, interleaving is particularly important: if a person comes in with chest pain, heart attack is not the only possibility, and if I had studied only cardiology that month I might fail to consider panic attacks, lung problems, digestive issues, or even blunt trauma, and send the person home after a normal EKG without even looking at their chest or asking any questions. Real life is interleaved.

    So I keep my Anki cards all mixed up, such that in a single session I might see bits of useful computer code, friends’ birthdays, dosing regimens for drugs, key points of journal articles, beautiful quotes, etc., in rapid succession.