Advanced IMAP - IMAP commands - atmail email experts - email hosting

Advanced IMAP

It’s been one year since I published IMAP 101, as part of our how-to tutorial series (which also includes POP 101 and SMTP 101) to help you interact with open, text-based protocols in common use within the email industry. So, this next instalment is a little overdue. My, how time flies!

If you haven’t read the original post, it might be wise to brush up on that base-level content first, because this post skips over those IMAP basics (e.g. logging in, selecting the Inbox, retrieving messages, deleting a message, and emptying the Inbox of deleted messages).

 

Advanced IMAP - IMAP commands - atmail email experts - email hosting

 

Challenge Response Authentication Mechanism (CRAM)

First, I’d like to quickly give you an overview of an advanced authentication technique called CRAM-MD5 – an implementation of the Challenge Response Authentication Mechanism. The only reason to provide this is because many years ago when I went looking for an explanation, I couldn’t find one, so I had to work it out myself. This example of CRAM-MD5 will give you the general feel for this style of authentication, and with a little investigation, you can probably figure out such mechanisms as SCRAM-SHA1 and SCRAM-SHA256 (‘S’ stands for ‘salted’ in SCRAM).

These days we are very well served by plaintext authentication mechanisms over SSL/TLS. The CRAM-MD5 (and variants) solution came about so that the plaintext password was never transmitted over the network, but rather a representation of the password that both sides could validate. The downside, however, is that both the server and the client needed the password in plaintext. Dovecot did come up with a novel solution to server-side storage, by storing an intermediate step of the CRAM-MD5 process rather than the plaintext password. You can read more about CRAM-MD5 on Wikipedia.

To perform CRAM-MD5 authentication:

  1. Connect to the server, note the “AUTH=CRAM-MD5” in the welcome banner, this means this auth type is supported.
    $ telnet imap.example.org 143
    Trying 172.16.5.101...
    Connected to imap.example.org.
    Escape character is '^]'.
    * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=CRAM-MD5] Dovecot ready.
  2. Request a challenge:
    * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=CRAM-MD5] Dovecot ready.
    a authenticate cram-md5
    + PDE4NTY1MTU5ODI1OTMyOTYuMTU3Nzc0OTgwOUBibHVlZmluLnA+
  3. Base64 decode the challenge:
    $ echo "PDE4NTY1MTU5ODI1OTMyOTYuMTU3Nzc0OTgwOUBibHVlZmluLnA+" | openssl base64 -d
    <1856515982593296.1577749809@bluefin.p>
  4. Now construct a hex-encoded HMAC MD5 of the challenge string using my password as the HMAC key:
    $ echo "PDE4NTY1MTU5ODI1OTMyOTYuMTU3Nzc0OTgwOUBibHVlZmluLnA+" | openssl base64 -d | openssl md5 -hmac "My_P@ssword1"
    (stdin)= 786abbf9d4d38e270c895b82318c2223
  5. Create a base64 encoded version of “username HMAC-MD5-hex-string”:
     $ echo -n "drichards 786abbf9d4d38e270c895b82318c2223" | openssl base64
    ZHJpY2hhcmRzIDc4NmFiYmY5ZDRkMzhlMjcwYzg5NWI4MjMxOGMyMjIz
  6. Then send this to the server:
    * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=CRAM-MD5] Dovecot ready.
    j authenticate cram-md5
    + PDE4NTY1MTU5ODI1OTMyOTYuMTU3Nzc0OTgwOUBibHVlZmluLnA+
    ZHJpY2hhcmRzIDc4NmFiYmY5ZDRkMzhlMjcwYzg5NWI4MjMxOGMyMjIz
    j OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY SPECIAL-USE] Logged in
  7. Voila, we are now authenticated and can execute normal IMAP commands:
    s select inbox
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
    * 64 EXISTS
    * 0 RECENT
    * OK [UNSEEN 1] First unseen.
    * OK [UIDVALIDITY 1548379801] UIDs valid
    * OK [UIDNEXT 72] Predicted next UID
    * OK [HIGHESTMODSEQ 82] Highest
    s OK [READ-WRITE] Select completed (0.001 + 0.000 secs).

 

Storing Flags

RFC 3501 allows clients to store flags (a type of meta-data) about messages.  There are some pre-defined flags, notably:

  • \Seen
    Message has been read
  • \Answered
    Message has been answered
  • \Flagged
    Message is “flagged” for urgent/special attention
  • \Deleted
    Message is “deleted” for removal later by EXPUNGE
  • \Draft
    Message has not completed composition (marked as a draft)
  • \Recent
    Message is “recently” arrived in this mailbox. This session is the first session to have been notified about this message. If the session is read-write, subsequent sessions will not see \Recent set for this message. This flag cannot be altered by the client.

However, some servers allow arbitrary flags to be set. Indeed, I personally have written applications that utilise flags to a great extent. This is how to set flags on an IMAP message, or a group of messages:

  1. What flags are currently stored?
    f FETCH 1:10 FLAGS
    * 1 FETCH (FLAGS ())
    * 2 FETCH (FLAGS ())
    * 3 FETCH (FLAGS ())
    * 4 FETCH (FLAGS ())
    * 5 FETCH (FLAGS ())
    * 6 FETCH (FLAGS ())
    * 7 FETCH (FLAGS ())
    * 8 FETCH (FLAGS ())
    * 9 FETCH (FLAGS ())
    * 10 FETCH (FLAGS ())
    f OK Fetch completed (0.001 + 0.000 secs).
  2. OK, so no flags stored at all. Let’s store a flag on message 2:
    g STORE 2 +FLAGS (some-flag)
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag \*)] Flags permitted.
    * 2 FETCH (FLAGS (some-flag))
  3. Let’s store a different flag on messages 5 to 8:
    k STORE 5:8 +FLAGS (a-different-flag)
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag \*)] Flags permitted.
    * 5 FETCH (FLAGS (a-different-flag))
    * 6 FETCH (FLAGS (a-different-flag))
    * 7 FETCH (FLAGS (a-different-flag))
    * 8 FETCH (FLAGS (a-different-flag))
    k OK Store completed (0.003 + 0.000 + 0.002 secs).
  4. Re-fetch the flags and ensure they stuck:
    r FETCH 1:10 FLAGS
    * 1 FETCH (FLAGS ())
    * 2 FETCH (FLAGS (some-flag))
    * 3 FETCH (FLAGS ())
    * 4 FETCH (FLAGS ())
    * 5 FETCH (FLAGS (a-different-flag))
    * 6 FETCH (FLAGS (a-different-flag))
    * 7 FETCH (FLAGS (a-different-flag))
    * 8 FETCH (FLAGS (a-different-flag))
    * 9 FETCH (FLAGS ())
    * 10 FETCH (FLAGS ())
    r OK Fetch completed (0.001 + 0.000 secs).
    

 

Let’s Discuss IMAP UID’s

IMAP supports two methods of referencing messages, sequence numbers and UID’s. Up to now I have only used sequence numbers, in both IMAP 101 and this post. Sequence numbers are relative, in that between sessions and even during the same session, the sequence number of a message may change, whereas UID’s never change. Well, that isn’t 100 percent true, UID’s of messages may be invalidated/changed if the UIDVALIDITY folder identifier is also changed. But let’s not get into that too much.

Here are some examples of using UID’s:

  1. Fetch the UIDS of the first 5 messages:
    r OK Fetch completed (0.001 + 0.000 secs).
    f fetch 1:5 UID 
    * 1 FETCH (UID 8)
    * 2 FETCH (UID 9)
    * 3 FETCH (UID 10)
    * 4 FETCH (UID 11)
    * 5 FETCH (UID 12)
    f OK Fetch completed (0.001 + 0.000 secs).
  2. Store flags, using a message UID, of the 3rd message (UID 10):
    w UID STORE 10 +FLAGS (a-funny-flag)
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag a-funny-flag)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag a-funny-flag \*)] Flags permitted.
    * 3 FETCH (UID 10 FLAGS (a-funny-flag))
    w OK Store completed (0.004 + 0.000 + 0.003 secs).
  3. Fetch the subject of a message using the UID:
    u UID FETCH 71 BODY[HEADER.FIELDS (SUBJECT)]
    * 64 FETCH (UID 71 BODY[HEADER.FIELDS (SUBJECT)] {18}
    Subject: crash
    
    )
    o OK Fetch completed (0.001 + 0.000 secs).

 

IMAP Searching

IMAP supports searching for messages on the server-side. This allows clients to offload the heavy lifting to the server to find specific messages, and therefore clients can be implemented more simply.

Whilst the searching is relatively basic, you can combine predicates to implement some very powerful search capabilities. RFC 3501 has the full story, so here I will focus on providing some search examples to get you started.

The result of the searches is a message sequence set. With this message sequence set, you can then retrieve the message parts that you want to retrieve.

  1. Let’s find all messages that are “unseen”:
    s1 SEARCH UNSEEN
    * SEARCH 1 2 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
    s1 OK Search completed (0.001 + 0.000 secs).
  2. Let’s find all messages that are “unseen” and have the subject containing the string “crash”. We will validate this by retrieving one of the messages, note the \Seen flag being set, and then re-searching and noticing the message is no longer included in the result set:
    s2 SEARCH UNSEEN HEADER SUBJECT "crash"
    * SEARCH 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
    s2 OK Search completed (0.001 + 0.000 secs).
    f FETCH 15 BODY[HEADER.FIELDS (SUBJECT)]
    * 15 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (SUBJECT)] {18}
    Subject: crash
    )
    f OK Fetch completed (0.001 + 0.000 secs).
    s3 SEARCH UNSEEN HEADER SUBJECT "crash"
    * SEARCH 14 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
    s3 OK Search completed (0.002 + 0.000 + 0.001 secs).
  3. Let’s find all messages that are smaller than 10kB or have a subject containing the string “crash”. We will validate this by retrieving the subject of a result message, noticing it is blank, and then fetch the size – showing the OR as working:
    s4 SEARCH OR SMALLER 10000 HEADER SUBJECT "crash"
    * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
    s4 OK Search completed (0.008 + 0.000 + 0.007 secs).
    r FETCH 1 BODY[HEADER.FIELDS (SUBJECT)]
    * 1 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (SUBJECT)] {2}
    
    )
    r OK Fetch completed (0.001 + 0.000 secs).
    r FETCH 1 RFC822.SIZE
    * 1 FETCH (RFC822.SIZE 447)
    r OK Fetch completed (0.001 + 0.000 secs).
  4. Let’s find all messages that do not have the subject starting with “crash”. Validate by fetching the subject, and noticing it is indeed “mary”:
    s5 SEARCH NOT HEADER SUBJECT "crash"
    * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13
    s5 OK Search completed (0.001 + 0.000 secs).
    f FETCH 10 BODY[HEADER.FIELDS (SUBJECT)]
    * 10 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (SUBJECT)] {17}
    Subject: mary
    
    )
    f OK Fetch completed (0.001 + 0.000 secs).

 

Creating IMAP Folders

To organise your messages, IMAP supports folders and folder hierarchies. As we discussed in the IMAP 101 post, the NAMESPACE command is necessary to understand the root level folder and the folder separator. In this example we will create a few new folders to store work related emails in:

  1. Get the namespace definition for this server:
    n namespace
    * NAMESPACE (("" ".")("virtual/" "/")) NIL NIL
    n OK Namespace completed (0.001 + 0.000 secs).
  2. This means all of my personal folders are stored under “” ie. in the top level and all folders are separated with a “.” character. In my professional opinion, a “.” is a very bad character in practice to use as a separator and a “/” is much better – but my test server isn’t configured this way, so let’s move on.

3. What folders do I currently have under there?

k LIST "" "*"
* LIST (\HasNoChildren \Trash) "." Trash
* LIST (\HasNoChildren) "." folder1
* LIST (\HasNoChildren) "." INBOX
k OK List completed (0.001 + 0.000 secs).

4. Let’s create a “work” folder:

c CREATE work
c OK Create completed (0.002 + 0.000 + 0.001 secs).
k2 LIST "" "*"
* LIST (\HasNoChildren \Trash) "." Trash
* LIST (\HasNoChildren) "." folder1
* LIST (\HasNoChildren) "." work
* LIST (\HasNoChildren) "." INBOX
k2 OK List completed (0.001 + 0.000 secs).

5. Now under “work” let’s create helpful folders such as “receipts”, “due today” and “follow up”:

c2 CREATE work.receipts
c2 OK Create completed (0.003 + 0.000 + 0.002 secs).
c3 CREATE "work.due today"
c3 OK Create completed (0.003 + 0.000 + 0.002 secs).
c4 CREATE "work.follow up"
c4 OK Create completed (0.002 + 0.000 + 0.001 secs).
k4 LIST "" "work*"
* LIST (\HasChildren) "." work
* LIST (\HasNoChildren) "." work.receipts
* LIST (\HasNoChildren) "." "work.due today"
* LIST (\HasNoChildren) "." "work.follow up"
k4 OK List completed (0.001 + 0.000 secs).

 

Copying Messages

IMAP does not have the concept of “moving” messages. Because messages are immutable, you must create a new version of the message and then delete the original.

In this example, we will move message 10 from the inbox into “work.due today” that was created above, and then delete message 10 from the inbox:

  1. Fetch message 10 from the inbox:
    001 select inbox
    * OK [CLOSED] Previous mailbox closed.
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag a-funny-flag)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft some-flag a-different-flag a-funny-flag \*)] Flags permitted.
    * 63 EXISTS
    * 0 RECENT
    * OK [UNSEEN 2] First unseen.
    * OK [UIDVALIDITY 1548379801] UIDs valid
    * OK [UIDNEXT 72] Predicted next UID
    * OK [HIGHESTMODSEQ 93] Highest
    001 OK [READ-WRITE] Select completed (0.003 + 0.000 + 0.002 secs).
    f fetch 10 rfc822
    * 10 FETCH (RFC822 {430}
    Return-path: <root@bluefin.p>
    Envelope-to: drichards@bluefin.p
    Delivery-date: Mon, 07 Oct 2019 23:44:00 -0700
    Received: from root by bluefin.p with local (Exim 4.91) (envelope-from <root@bluefin.p>) id 1iHjE0-00058s-8Y for drichards@bluefin.p; Mon, 07 Oct 2019 23:44:00 -0700
    Subject: mary
    Message-Id: <E1iHjE0-00058s-8Y@bluefin.p>
    From: root <root@bluefin.p>
    Date: Mon, 07 Oct 2019 23:44:00 -0700
    
    Where is Mary?
    )
    f OK Fetch completed (0.001 + 0.000 secs).
  2. Copy the message:
    c1 COPY 10 "work.due today"
    c1 OK [COPYUID 1548379804 17 1] Copy completed (0.007 + 0.000 + 0.006 secs).
  3. Set the \Deleted flag:
    s1 STORE 10 +FLAGS (\Deleted)
    * 10 FETCH (FLAGS (\Deleted \Seen))
    s1 OK Store completed (0.003 + 0.000 + 0.002 secs).
  4. Run an expunge on the inbox:
    e EXPUNGE
    * 10 EXPUNGE
    e OK Expunge completed (0.001 + 0.000 secs).
  5. Select “work.due today” and retrieve the single message, ensuring it is the same as message 10 from the inbox:
    s SELECT "work.due today"
    * OK [CLOSED] Previous mailbox closed.
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
    * 1 EXISTS
    * 1 RECENT
    * OK [UIDVALIDITY 1548379804] UIDs valid
    * OK [UIDNEXT 2] Predicted next UID
    s OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
    f2 FETCH 1 RFC822
    * 1 FETCH (RFC822 {430}
    Return-path: <root@bluefin.p>
    Envelope-to: drichards@bluefin.p
    Delivery-date: Mon, 07 Oct 2019 23:44:00 -0700
    Received: from root by bluefin.p with local (Exim 4.91) (envelope-from <root@bluefin.p>) id 1iHjE0-00058s-8Y for drichards@bluefin.p; Mon, 07 Oct 2019 23:44:00 -0700
    Subject: mary
    Message-Id: <E1iHjE0-00058s-8Y@bluefin.p>
    From: root <root@bluefin.p>
    Date: Mon, 07 Oct 2019 23:44:00 -0700
    
    Where is Mary?
    )
    f2 OK Fetch completed (0.001 + 0.000 secs).

 

Appending a Message

The final command we will look at in this post is to add a new message to a mailbox. In this case, we will add a new message to the same folder as above “work.due today”. The only tricky part of adding a message is getting the byte count of the message. To do this, create the new message in a blank text file, get the character count (assuming ASCII) which will be the same as the byte count, and then copy/paste the message into the IMAP session.

  1. Get the contents of my new message:
    $ cat /tmp/my.msg 
    Subject: Send the weekly report
    Remember to send the weekly report to THE BOSS TODAY!!
  2. What is the byte count?
    $ wc -c /tmp/my.msg 
    88 /tmp/my.msg
  3. Append the message to the inbox like so. I have added an extra byte to the literal count to account for the final carriage return I need to manually enter:
    a1 APPEND "work.due today" {89}
    + OK
    Subject: Send the weekly report
    
    Remember to send the weekly report to THE BOSS TODAY!!
    a1 OK [APPENDUID 1548379804 2] Append completed (0.007 + 4.391 + 0.005 secs).
  4. Retrieve it to make sure it is correct:
    s3 select "work.due today"
    * OK [CLOSED] Previous mailbox closed.
    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
    * 2 EXISTS
    * 1 RECENT
    * OK [UNSEEN 2] First unseen.
    * OK [UIDVALIDITY 1548379804] UIDs valid
    * OK [UIDNEXT 3] Predicted next UID
    s3 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
    f fetch 2 rfc822
    * 2 FETCH (FLAGS (\Seen \Recent) RFC822 {89}
    Subject: Send the weekly report
    
    Remember to send the weekly report to THE BOSS TODAY!)
    f OK Fetch completed (0.003 + 0.000 + 0.002 secs).

 

Find This Useful?

We’re currently taking ideas for future how-to tutorials that help newcomers to the email services industry.

If you have any other IMAP commands that you would like us to explore in a blog post, please drop us a line here and we’ll happily put something together.

 

New to atmail?

If email is not your core business and you’d like someone else to worry about IMAP, SMTP and POP – we can help.

With 20 years of global, white label, email expertise serving telecommunications and hosting providers across every continent, you can trust us to deliver white label, email solutions that are stable, secure and scalable. We power 170 million mailboxes and offer user-friendly, cloud hosted email. Or, if you want to stay in-house, we offer on-premises webmail and/or mail server options.

Talk to us today.

 

Share This Post
By Dave Richards | January 10, 2020