In my previous post on this topic, I tried to detail the way that I fetch and send email using my laptop using a combination of mbsync and systemd. This has been working extremely well – it runs in the background and does the right thing when I am connected to the Internet. The only issue with this setup is that I’ll occasionally need to clear a lockfile out of the .msmtp-queue directory. This is quite rare, however.

The nicest thing about using a Maildir to store your mail is that you can act on the mail using a variety of different programs – some folks prefer using Mutt to read their mail, others prefer something else. Most popular mail software on Linux seems to support storing things in Maildirs – with the apparent exception of Thunderbird.

One of the killer features of Gmail when it came out was the ability to quickly search through your email and to automatically tag it in different ways. This was one of the things that I missed moving my email to a local client – Thunderbird didn’t particularly care for searching through my 25GB mail archive. Enter Notmuch.

Notmuch is a small piece of software that wraps around the Xapian search library to index and search through a large amount of data quite easily. The name comes from the fact that the software doesn’t fetch your mail, send your mail or even really index your mail – i.e. it doesn’t do much.

Notmuch really does a few things:

• Indexes your mail using Xapian
• Associates your mail with different tags – similar to Gmail. The tags are stored in the database.
• Help search through Xapian efficiently.

It doesn’t:

• Get the mail.
• Write the mail.
• Send the mail.

That’s about it. It relies on other programs in order to actually do anything with the mail.

One important thing to note is that everything in Notmuch is a tag. The actual folder structure of your emails doesn’t really matter – all that matters are the tags in the headers.

Getting setup with Notmuch

First things first – pull your email onto your local device. I describe a working setup here. This setup includes both pulling email off of the server and sending email through your SMTP server.

Next, you will want to install notmuch and GNU Emacs. I installed both from my Linux distribution’s package manager. You will also want to install a second script called afew. Afew is a set of python scripts that assist in tagging your email. I installed afew to my ~/.local/bin directory.

You should setup the notmuch configuration file next. You can do this by using the command notmuch. This will walk you through a few configuration steps where you need to identify your own email addresses. I added all of the addresses that I’ve ever used since I tend to store lots of email.

After running the configuration wizard, you will want to edit the [new] mail configuration so that it reads:

.notmuch-config

# Configuration for "notmuch new"
#
# The following options are supported here:
#
#   tags    A list (separated by ';') of the tags that will be
#       added to all messages incorporated by "notmuch new".
#
#   ignore  A list (separated by ';') of file and directory names
#       that will not be searched for messages by "notmuch new".
#
# NOTE: *Every* file/directory that goes by one of those
# names will be ignored, independent of its depth/location
# in the mail store.


[new]
tags=new;
ignore=Trash

This configures the notmuch new command to read in and index all the mail and apply the new tag to it. We’re using this tag to let afew know what messages it needs to operate on.

Next, you want to configure afew. I’m using the following configuration file:

~/.config/afew/config

#default filter chain
[SpamFilter]
[KillThreadsFilter]
[ListMailsFilter]
[SentMailsFilter]
sent_tag = sent
[ArchiveSentMailsFilter]


[Filter.1]
message = "Facebook"
query = 'from:facebookmail.com'
tags = +facebook;+unread;-new

[Filter.2]
message = "Spam"
query = 'from:"Global Who\'sWho" OR from:Touchfire OR from:Walk-inTub OR from:"Replacement Window"'
tags = +spam;-new

[Filter.3]
message = "Get mailing lists out"
query = 'tag:lists'
tags =  +unread;-new;

[Filter.4]
message = "Get mailing lists out"
query = 'to:geda-user@delorie.com'
tags =  +lists/geda-user;-new;-inbox;


[InboxFilter]

Stepping through the file:

• SpamFilter - This filter looks throught the email headers for the header X-Spam-Flag which should be set by SpamAssasian or something similar running on your email host. This tags the message as spam – hiding it from your searches.

• KillThreadsFilter - Kill new messages to previously killed threads. Good for the ReplyAll chains from your Uncle about Obama.

• ListMailsFilter - Look through the new messages and add a the name of the list as a tag. Remove the item from your inbox. This is helpful for high volume mailing lists so you can only check them when you want to. This filter uses the ListID: header in the email. I have noticed that a large number of commericial mailing lists are using the ListID header to track their campaigns. This is a bit annoying and I haven’t found a good way to fix it yet.

• SentMailsFilter - Tag all the email that I’ve sent to others. This uses the From: header.

• ArchiveSentMailsFilter - removes the inbox tag from sent email.

• [Filter.1] - tag emails from facebook and remove them from the inbox.

• [Filter.2] - filter out some common spam that I receive. Our spam detection isn’t working too well on my email host.

• [Filter.3] - remove all mailing list messages from the inbox.

• [Filter.4] - one mailing list that I subscribe to doesn’t set the ListID header used by the ListMailsFilter. It uses “X-Mailing-List” instead. It was faster to just detect these and remove them from the mailbox than it was to do anything else.

The one thing lacking in the current setup is a way to move emails from the Inbox to subfolders. Afew has a MailMover function, but I haven’t been able it to get it working well. This is necessary to move unwanted items to the trash. I’ve been using mutt periodically to move older items into an archive folder and to the trash.

Get the mail and index it.

Now that we have notmuch and afew to index and tag the email, it’s time to automate the process. In the previous installment, we wrote a small shell script to check the internet connection state, send queued email and pull down new email. Let’s extend it to handle some new commands!

~/.local/bin/checkmail.sh

#!/bin/sh

STATE=`nmcli networking connectivity`

if [ $STATE = 'full' ]
then
    ~/.local/bin/msmtp-runqueue.sh
    mbsync nnytech
    notmuch new
    ~/.local/bin/afew -tn
    notmuch tag -inbox tag:inbox AND tag:lists
    exit 0

fi
echo "No Internets!"
exit 0

This script adds in some new commands to index new mail and to run afew to tag the mail. In addition, we run notmuch again to remove all mailing lists from the Inbox as they’re almost always not urgent.

Now, how do we read our email again?

After spending all this time getting our email downloaded, indexed and tagged we probably want to read our email. I’ve been using GNU Emacs a lot lately and had read about the Notmuch Emacs client. The neat thing about Notmuch is that it’s independent of the frontend that you read your mail with. There are setups that work well with Mutt and even Vim as well as a number of other clients.

I installed the latest version of the Notmuch frontend from the MELPA Emacs package archive. Getting starting with MELPA is described here.

Some .emacs.d/init.el-fu is required to get things working.

.emacs.d

:::lisp

;Load up Notmuch
(require 'notmuch)


; Setup some keybindings


; C-c m opens up Notmuch from any buffer
(global-set-key (kbd "C-c m") `notmuch)

;Setup Names and Directories
(setq user-mail-address "myemail@mydomain.tld"
  user-full-name "My Totally Real Name")

; stores postponed messages to the specified directory
(setq message-directory "MailLocation/Drafts") ;

;set sent mail directory
(setq notmuch-fcc-dirs "MailLocation/Sent")

;Settings for main screen
(setq notmuch-hello-hide-tags (quote ("killed")))

;A few commonly used saved searches.
(setq notmuch-saved-searches
(quote
((:name "inbox" :query "tag:inbox AND -tag:work" :key "i" :sort-order oldest-first)
 (:name "flagged" :query "tag:flagged" :key "f") ;flagged messages
 (:name "sent" :query "tag:sent -tag:work" :key "t" :sort-order newest-first)
 (:name "drafts" :query "tag:draft" :key "d")
 (:name "mailinglist" :query "tag:lists/mailinglistID" :key "c")
 (:name "all mail" :query "*" :key "a" :sort-order newest-first))))


;Message composition and sending settings

;Setup User-Agent header
(setq mail-user-agent 'message-user-agent)

(setq message-kill-buffer-on-exit t) ; kill buffer after sending mail)
(setq mail-specify-envelope-from t) ; Settings to work with msmtp

(setq send-mail-function (quote sendmail-send-it))
(setq sendmail-program "~/.local/bin/msmtp-enqueue.sh"
  mail-specify-envelope-from t
;; needed for debians message.el cf. README.Debian.gz
 message-sendmail-f-is-evil nil
  mail-envelope-from 'header
  message-sendmail-envelope-from 'header)

;Reading mail settings:

(define-key notmuch-show-mode-map "S"
    (lambda ()
    "mark message as spam"
    (interactive)
(notmuch-show-tag (list "+spam" "-inbox"))))

(define-key notmuch-search-mode-map "S"
(lambda ()
    "mark message as spam"
    (interactive)
    (notmuch-search-tag (list "-inbox" "+spam"))
    (next-line) ))

(setq notmuch-crypto-process-mime t) ; Automatically check signatures


;Crypto Settings
(add-hook 'message-setup-hook 'mml-secure-sign-pgpmime)

(setq epg-gpg-program "/usr/bin/gpg2")

;There was some problem with listing PGP keys in the Debian
;version of EPG. This magic from StackOverflow seems to resolve it.
(defun epg--list-keys-1 (context name mode)
(let ((args (append (if (epg-context-home-directory context)
          (list "--homedir"
            (epg-context-home-directory context)))
          '("--with-colons" "--no-greeting" "--batch"
        "--with-fingerprint" "--with-fingerprint")
          (unless (eq (epg-context-protocol context) 'CMS)
        '("--fixed-list-mode"))))
(list-keys-option (if (memq mode '(t secret))
              "--list-secret-keys"
            (if (memq mode '(nil public))
            "--list-keys"
              "--list-sigs")))
(coding-system-for-read 'binary)
keys string field index)
(if name
(progn
  (unless (listp name)
    (setq name (list name)))
  (while name
    (setq args (append args (list list-keys-option (car name)))
      name (cdr name))))
  (setq args (append args (list list-keys-option))))
(with-temp-buffer
  (apply #'call-process
     (epg-context-program context)
     nil (list t nil) nil args)
  (goto-char (point-min))
  (while (re-search-forward "^[a-z][a-z][a-z]:.*" nil t)
(setq keys (cons (make-vector 15 nil) keys)
      string (match-string 0)
      index 0
      field 0)
(while (and (< field (length (car keys)))
        (eq index
        (string-match "\\([^:]+\\)?:" string index)))
  (setq index (match-end 0))
  (aset (car keys) field (match-string 1 string))
  (setq field (1+ field))))
  (nreverse keys))))

How to use it?

Load up emacs, press C-c m and you’re off and running.

Next steps:

I’m still working on refining the setup. My next steps should include:

• Org-mode Capture integration
• Getting afew to actually move email out of my inbox.

Feedback

If you have any feedback, please send me an email!