How Email Actually Gets Sent: A Look At SMTP
A Deep Dive Into the SMTP Protocol
Let’s take a closer look at what happens “under the hood” when you send an email. This article covers the original implementation of the SMTP protocol based off available research papers and documentation. Naturally, SMTP has evolved over time, but this article will cover the foundational elements of this communication protocol.
We’ll breakdown SMTP’s responsibilities, take a look at the communication between the client SMTP and receiver SMTP servers, and finally, we’ll use Terminal to send an email by issuing SMTP commands directly to a SMTP server.
SMTP (Secure Mail Transfer Protocol)
If you’ve ever configured an email client before, you may have seen terms like IMAP, POP, and SMTP. While IMAP and POP are protocols used for retrieving emails from a mailbox, SMTP is strictly a sending protocol. We’ll take a closer look at IMAP and POP in a separate article.
SMTP is part of the application layer of the TCP/IP protocol and traditionally operates on port 25. It utilizes a process called “store and forward” which is used to orchestrate sending your email across different networks. Within the SMTP protocol, there are smaller software services called Mail Transfer Agents that help manage the transfer of the email and its final delivery to a recipient’s mailbox. Not only does SMTP define this entire communication flow, it can also support delayed delivery of an email either at the sender site, receiver site, or at any intermediate server.
TLDR: SMTP defines a communication protocol that dictates how an email moves from your computer’s mail transfer agent, potentially across several networks, to the mail transfer agent on the destination SMTP server.
Some terminology that may be useful to understand before we take a deeper dive:
User Agent (UA): The application used to send and receive electronic mail (Outlook, Mozilla, etc.)
Mail Transfer Agent / Message Transfer Agent (MTA): A process running on an SMTP server that helps forward emails to the correct recipient and helps manage delivery of emails to a user’s mailbox.
Often MTAs help maintain a mail queue so repeat delivery attempts can be scheduled in the case a remote server is unavailable. The MTA often contains a specialized piece of software called the mail delivery agent or message delivery agent (MDA) which is responsible for the delivery of the e-mail message to a local recipient’s mailbox whereas the MTA focus more on forwarding the email from SMTP server to SMTP server.
When a user sends an email, through a user agent — for example, the Apple mail app — a client SMTP process opens a TCP connection to a SMTP process running on a server somewhere (i.e. smtp.gmail.com) on port 25. After an initial handshake, the client and server SMTP processes engage in a short request-response dialogue in order to send an email. We’ll take a closer look at that flow shortly.
Take this basic use case of the SMTP protocol (this situation is likely one in which the email is being sent within the same organization):
Notice how the MTA facilities both in sending the email and managing the recipient’s mailbox.
On most UNIX systems, the most commonly used MTA is Sendmail.
In reality, though, an email message could pass through multiple intermediary MTAs (a.k.a. mail gateways) on the way to delivering the message to the correct recipient.
In the following diagram, the local MTA is merely an intermediate step along the email’s journey.
An added benefit of email gateways is that they can handle electronic mail prepared by some other protocol and transform it to the SMTP format before forwarding it along or vice-versa.
This is a more accurate depiction of a typical SMTP flow:
We’ll take a closer look at the actual commands and payloads send between the client and server SMTP processes shortly. Before we do that, let’s look at how the contents of an email are structured.
Opening the Envelope
An email comprises of two parts — an envelope and a message/body — much like a traditional piece of mail.
The envelope contains the sender and recipient fields as well as some additional metadata (like timestamp, MIME properties, etc.) The fields in the envelope portion are often formatted like this:
field-name: field-value
Let’s look at what modern headers look like. Here are the headers from a promotional email from Caviar:
X-Apparently-To: secretemail@provider.com; Thu, 26 Mar 2020 23:31:09 +0000
Return-Path: <bounces+2178037–967c-secretemail=provider.com@email.foodie.trycaviar.com>
Authentication-Results: mta4468.mail.bf1.yahoo.com;
dkim=pass (ok) @foodie.trycaviar.com”>header.i=@foodie.trycaviar.com header.s=s1;
dkim=pass (ok) @sendgrid.info”>header.i=@sendgrid.info header.s=smtpapi;
spf=pass @email.foodie.trycaviar.com”>smtp.mailfrom=@email.foodie.trycaviar.com;
dmarc=pass(p=quarantine sp=NULL dis=none) header.from=trycaviar.com;
Received-SPF: pass (domain of email.foodie.trycaviar.com designates 167.89.63.234 as permitted sender)
X-Originating-IP: [167.89.63.234]
Received: from 10.197.32.140 (EHLO o48.email.foodie.trycaviar.com) (167.89.63.234)
by mta4468.mail.bf1.yahoo.com with SMTPS; Thu, 26 Mar 2020 23:31:08 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=foodie.trycaviar.com;
h=content-type:from:mime-version:reply-to:to:subject; s=s1;
bh=YlMg2L5M8TcplcxihVyhDyQPB40kE7Unn369HKZ3tj4=; b=ABGte+F6Qu+xT
jQ64AIt83donOVvWcvEqZrpgrY6bpK32cZ/JkneoxCH3eFAtIcccleDFyugwiQnO
e19Yc5FQwhziog7zDQi6LmpJubD9nqBJJ3N9ITXT4Rg6UYS+clpmKwNuGRdB3oMK
2RsTkYU7qpCBBCImica0rSJDnatlLU=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=sendgrid.info;
h=content-type:from:mime-version:reply-to:to:subject; s=smtpapi;
bh=YlMg2L5M8TcplcxihVyhDyQPB40kE7Unn369HKZ3tj4=; b=rjxxTnMo04Uis
aaguV1S6Ysqyh7E/VLf4vT9mHLsuF2druZ6B/0ge4Lvyr5131n34gV+o2zqpNWPr
6RlN0Qpu1jr0Af4shDf6XkAu/AnHeev4irn4d9/Tr+hFRrhxakeGddjKikdidLNr
K7a4Ie9hphqFQopJZs/p0whFUZD9Q0=
Received: by filter1323p1las1.sendgrid.net with SMTP id filter1323p1las1–31676–5E7D3B39–10
2020–03–26 23:31:05.45317912 +0000 UTC m=+85251.092327867
Received: from MjE3ODAzNw (unknown)
by ismtpd0050p1mdw1.sendgrid.net (SG) with HTTP id 4CejyYy6QrqoW-F8Gnxmew
Thu, 26 Mar 2020 23:31:05.358 +0000 (UTC)
Content-Type: multipart/alternative; boundary=eb8db8ca34fb2fa8254fd70754c309dbd5d5ba4fe01a1d742edb4ef09903
Date: Thu, 26 Mar 2020 23:31:05 +0000 (UTC)
From: “Caviar” <hungry@trycaviar.com>
Mime-Version: 1.0
Reply-to: hungry@trycaviar.com
To: secretemail@provider.com
Message-ID: <4CejyYy6QrqoW-F8Gnxmew@ismtpd0050p1mdw1.sendgrid.net>
Subject: New restaurants are here
Content-Length: 81247— eb8db8ca34fb2fa8254fd70754c309dbd5d5ba4fe01a1d742edb4ef09903
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Mime-Version: 1.0Plus, take 10% off. *******************
Oh, hey. New spots.
*******************
........
As you can see the header fields can contain far more than the sender and recipient information; those fields are merely the minimum requirement.
Any text following the envelope is classified as the body. The body is the user generated content of the email. The body section is usually terminated with a null link, blank line, or one of a few potential terminating characters (i.e. “.”)
The envelope is generally transmitted separately from the body of the email. The SMTP servers first will try and validate the sender/recipient information in the header before they care to consider and bother with transmitting the body of the email.
These fields are sent to the SMTP server using the MAIL FROM and RCPT TO commands (we’ll take a closer look at these commands shortly).
On the receiving SMTP server, it will periodically check the contents of its mailbox for new emails and forward or deliver them as necessary.
Sending the Email
As previously mentioned, SMTP is part of the application layer of the TCP/IP protocol. As a result, it should come as no surprise that SMTP uses DNS to determine the IP address of the SMTP server to contact.
Once the IP address of the recipient is acquired, the client SMTP server can connect to the remote SMTP server through a short sequence of consecutive commands.
SMTP Commands
HELO — Helps to identify the client SMTP process to the SMTP server and is usually followed by the fully qualified hostname. This is typically only sent once per session. Once this authentication step is complete, a client SMTP process can send out as many emails as they wish.
MAIL FROM — identifies to the server SMTP process the sender email address (ie. the from field in an email)
RCPT TO: After the MAIL FROM command, the client SMTP server can issue one or more RCPT TO commands to specify the email addresses of all the recipients.
DATA: This is the command that precedes the body of the email. It’s the command that notifies the SMTP server that it should prepare itself to receive some ASCII content that ends with a null line or a “.”.
We’ll take a look at these commands in practice using Terminal shortly, but it’s important to know that these commands are sent as ASCII text which allows for building mail clients and servers on any platform.
Though the list of SMTP commands is larger than this list, HELP, MAIL FROM, RCPT TO, DATA, and QUIT are the core commands.
For more information on the full scope of SMTP commands, check out this resource
Once the initial handshake is complete (the HELO step), the SMTP session begins. The client first submits the sender’s and recipient’s email addresses individually (think envelope) to the SMTP server. After every client command, the server will respond by sending “250 OK” if it received and validated the command. Otherwise, it will send an error message, generally prefixed by “550” as the error number.
After receiving confirmation of those two fields, the client SMTP process submits the email body to the server.
Here’s an example of the entire flow taken from Wikipedia, take a moment to look through it — we’ll simulate this exact flow using Telnet, Python, and Terminal next:
S: 220 smtp.example.com ESMTP Postfix
C: HELO relay.example.com
S: 250 smtp.example.com, I am glad to meet you
C: MAIL FROM:<bob@example.com>
S: 250 Ok
C: RCPT TO:<alice@example.com>
S: 250 Ok
C: RCPT TO:<theboss@example.com>
S: 250 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: From: “Bob Example” <bob@example.com>
C: To: Alice Example <alice@example.com>
C: Cc: theboss@example.com
C: Date: Tue, 15 Jan 2008 16:02:43 -0500
C: Subject: Test message
C:
C: Hello Alice.
C: This is a test message with 5 header fields and 4 lines in the message body.
C: Your friend,
C: Bob
C: .
S: 250 Ok: queued as 12345
C: QUIT
S: 221 Bye
{The server closes the connection}
Creating A Local SMTP Server / Sending An Email Using Terminal
In Terminal, we’ll use Telnet & Python to create a local SMTP server and send an email.
- Run
brew install telnet - Run
sudo python -m smtp -n -c DebuggingServer localhost:2525 - In a new Terminal window, run
telnet localhost 2525
Now, we’re ready to manually issue SMTP commands to communicate with our SMTP server.
First, let’s authenticate ourselves, type in HELO <hostname> , in this example, I’ll just use my name:
HELO <hostname>
We received a 250 confirmation from the server, so we’re ready to move on to the next step. Providing the envelope header information — the sender and the receiver.
MAIL FROM: aryamansharda@example.com
RCPT TO: steve@apple.com
Another set of 250 responses, so now we’re good to start providing the data payload. Type DATA and hit Enter.
“354 End data with <CR><LF>.<CR><LF>” tells you that in order to end the DATA section of the email, enter a “.” on a new line.
Our Terminal window with our local SMTP server should show the email information it received:
Since we don’t have any more messages to send, let’s close the connection by entering theQUIT command. Notice the 221 service closing transmission channel reply. Each SMTP command is accompanied by an acknowledgment from the SMTP server.
What we just stepped through manually in Terminal is a typical example of the SMTP protocol in action and what happens, on a much larger scale, every time you hit Send in your email client.
Hope you enjoyed this deep dive into the SMTP protocol, if there’s other algorithms or technologies you’d like me to break down, please leave a comment below!
Sources
The majority of the information and diagrams in this article were taken from this resource.
My Recent Projects
Personal Website: https://digitalbunker.dev/
http://stickerspot.digitalbunker.dev/
