SES Inbound Mail for Laravel

Receive mail through Amazon SES in Laravel: a verified SNS endpoint, takeover from the S3 transit, reconciliation, a check for stuck mail and AWS setup.

How a mail arrives

1. Amazon SES receives a mail for a recipient of your receipt rule and stores it in an S3 bucket, the transit, under the id SES gave it.
2. Amazon SNS reports the mail to the endpoint of this package. The endpoint takes a message only from an allowed topic and only with a valid signature of SNS.
3. The package records the mail, once per id, and queues its takeover.
4. The takeover copies the original byte for byte onto a disk of your application, reads the copy back and holds its SHA-256 and size against the original, and only then deletes it in the transit.
5. Your application receives the mail through ReceivesInboundMail and decides what happens to it.

Every five minutes the reconciliation takes every mail that waits in the transit longer than fifteen minutes, so a notification that never arrived costs nothing but time.

Installation

composer require pushery/ses-inbound-mail-for-laravel
php artisan migrate

The service provider is registered automatically through package discovery. The migration creates the table ses_inbound_mails, one row per mail.

The two disks

The transit is a disk on the S3 bucket SES stores into. The archive is a disk of your application, where the original of every mail stays. Configure both in config/filesystems.php:

'ses-transit' => [
    'driver' => 's3',
    'key' => env('SES_TRANSIT_KEY'),
    'secret' => env('SES_TRANSIT_SECRET'),
    'region' => 'eu-central-1',
    'bucket' => env('SES_TRANSIT_BUCKET'),
    'throw' => false,
],

The transit holds nothing but mail in transit. Its lifecycle removes whatever is older than fourteen days, so nothing is set aside there for a person to look at. The keys of the transit disk only need to read, list and delete in the bucket; the setup below writes exactly that policy.

Receiving a mail

Bind ReceivesInboundMail in a service provider of your application:

use App\Mail\OpenAConversation;
use Pushery\SesInboundMail\Contracts\ReceivesInboundMail;

$this->app->bind(ReceivesInboundMail::class, OpenAConversation::class);

use Illuminate\Support\Facades\Storage;
use Pushery\SesInboundMail\Contracts\ReceivesInboundMail;
use Pushery\SesInboundMail\ReceivedMail;

final class OpenAConversation implements ReceivesInboundMail
{
    public function receive(ReceivedMail $mail): void
    {
        $raw = Storage::disk($mail->disk)->get($mail->path);

        // $mail->messageId, $mail->sender, $mail->recipients, $mail->verdicts,
        // $mail->receivedAt, $mail->sha256 and $mail->size describe the mail.
    }
}

ReceivedMail carries the id SES gave the mail, the sender and the recipients of the envelope, the verdicts of SES on SPF, DKIM, DMARC, spam and viruses, when SES received the mail, and the disk, path, SHA-256 and size of its original. $mail->foundInTransit is true for a mail the reconciliation found because no notification reported it: its verdicts then name spam and viruses only, read from the headers SES wrote into the mail.

The takeover runs as a queued job, tried five times by default. Where your receiver throws, the original stays on your disk and the receiver is called again on the next try, without a second copy. Treat a mail you have seen before as done.

One takeover of a mail is queued at a time. Its lock lives in the default cache store of your application and outlasts every try, so the reconciliation queues no second takeover while the first waits for its next try.

The endpoint

SNS delivers to POST /ses-inbound-mail/notifications. Name the topics it may take in SES_INBOUND_MAIL_TOPICS, by their ARN and separated by commas. While the list is empty, every message is refused.

Answer	When
204	a notification of a stored mail, recorded once; a confirmed subscription; any other message of SNS
400	a message the endpoint cannot read
403	a topic outside the list, a signature that is not one of SNS, or a subscription to confirm at an address that is not one of SNS

The endpoint checks the signature of every message itself, with the certificate SNS signed it under, fetched only from an address of SNS and kept for a day. It needs no session, no CSRF token and no sign-in, so it stands outside the web group. Every refusal is logged and dispatched as Pushery\SesInboundMail\Events\NotificationRefused, with the reason, the topic and the address it came from, before anything of the message is used.

An application that answers its API on a host of its own names that host in SES_INBOUND_MAIL_DOMAIN, and the endpoint answers there alone.

The reconciliation

The package schedules php artisan ses-inbound-mail:reconcile every five minutes, on one server and without overlapping. It takes every mail that waits in the transit longer than fifteen minutes:

Mail in the transit	What happens
never reported	recorded with the envelope its headers name, then taken over
recorded, its takeover stopped	its takeover is queued once more
recorded, its takeover queued or waiting for its next try	left to that takeover
taken over before	the copy in the transit is removed
the file SES writes to test its access	removed
a name SES gives no mail	left alone and logged

The takeover deletes the original in the transit before it calls your application. A mail your application refused in every try is therefore found by its row: fifteen minutes after its takeover, the reconciliation queues it once more, so your application is called again until it takes the mail.

One mail that cannot be handled is reported and holds up none of the others. The fifteen minutes are the same for every application and are no setting.

A mail no notification reported names its envelope only in its headers: the sender from Return-Path, or From, and the recipient from the for clause of the Received header SES puts on top, which is the address the mail was delivered to, also where it was forwarded from another one. Where that clause is missing, the recipients come from Delivered-To, X-Original-To, To and Cc, and a blind copy names no recipient there. The verdicts on spam and viruses come from X-SES-Spam-Verdict and X-SES-Virus-Verdict. An address longer than 320 characters, 64 before the @ and 255 after it, is left out of the row.

To take one mail by its key, whatever its age:

php artisan ses-inbound-mail:take-over inbound/0123456789abcdef

The health check

An application that runs spatie/laravel-health registers the check that fails while a mail waits in the transit for thirty minutes, twice the time the reconciliation waits, so a mail its next run takes in the ordinary way is not reported first:

use Pushery\SesInboundMail\Health\NoMailIsStuckInTransitCheck;
use Spatie\Health\Facades\Health;

Health::checks([
    NoMailIsStuckInTransitCheck::new(),
]);

It counts a mail taken over and never handed to your application as well, by the key it had in the transit. Such a mail means the scheduler does not run, its takeover keeps failing, or your application refuses it.

Retention

The archive keeps the original of every mail, and the package's table keeps its row, which names the sender and the recipients of the envelope. Both are personal data. SES_INBOUND_MAIL_RETENTION_DAYS sets how many days they stay after the mail reached your application. With it set, the package schedules php artisan model:prune for its model once a day, on one server, and that deletes the row and the original of every mail handed over longer ago. A mail not handed over yet stays, whatever its age.

Nothing is deleted while the setting is empty, because ReceivedMail hands your application the disk and the path of the original, and an application that reads it later would lose it. Set it to no more days than your application keeps what it took from the mail. With a model of your own in the models map, the prune runs for that model, and a prunable() of its own decides instead.

Setting up AWS

php artisan ses-inbound-mail:setup-aws --check
php artisan ses-inbound-mail:setup-aws

The command sets up what the ses-inbound-mail.aws settings name and checks it with --check, which writes nothing and fails where an item is missing or differs. Every item is read first and written only where it differs, so a second run changes nothing. It runs with the credentials the AWS SDK finds on the machine, from the environment, a profile or the role of the machine, never with the keys of your application, which may only read and delete in the transit.

Item	What the setup makes sure of
bucket	the bucket of the transit exists
public access block	every public access is blocked
bucket owner enforced	the bucket owns every object, and ACLs are off
encryption	SSE-S3
bucket policy	SES may store objects under the prefix only for the receipt rule of your application
lifecycle	the fixed values below, beside any rule of another kind
topic	the SNS topic exists
topic policy	SES may publish to it only for the receipt rule of your application
subscription	the endpoint of your application is subscribed over https, and asked again while it waits for its confirmation
receipt rule	the rule of your application in the rule set inbound, with its recipients and an S3 action that reports to the topic
user policy	the IAM user of your application may only get, list and delete in the transit

The applications of an AWS account share the active receipt rule set inbound. It is created and made the active one once, by hand, and the setup stops where it does not exist.

The lifecycle is the same for every application:

Rule	Value
a mail in the transit expires, as a net under the takeover	after 14 days
a version that is no longer the current one is removed	1 day later
an upload that never finished is removed	after 1 day

Configuration

Publish the configuration file, where every option is documented inline:

php artisan vendor:publish --tag=ses-inbound-mail-config

Tag	Publishes
ses-inbound-mail	everything below, in one command
ses-inbound-mail-config	the configuration file
ses-inbound-mail-migrations	the database migrations
ses-inbound-mail-lang	the translation files

The migrations run from the package as they are, with php artisan migrate. Publishing them writes a copy into database/migrations, and the package still registers its own. Call \Pushery\SesInboundMail\SesInboundMailServiceProvider::ignoreMigrations() in the register method of one of your service providers before you publish them, or the copy fails on a table that already exists. Without the package's migrations the reconciliation is not scheduled either, since it reads the table.

Every setting reads an environment variable, so it changes without publishing the file:

Environment variable	Setting	Default
SES_INBOUND_MAIL_TOPICS	ses-inbound-mail.topics	none, every message is refused
SES_INBOUND_MAIL_PATH	ses-inbound-mail.route.path	ses-inbound-mail/notifications
SES_INBOUND_MAIL_DOMAIN	ses-inbound-mail.route.domain	every host
SES_INBOUND_MAIL_TRANSIT_DISK	ses-inbound-mail.transit_disk	ses-transit
SES_INBOUND_MAIL_ARCHIVE_DISK	ses-inbound-mail.archive.disk	local
SES_INBOUND_MAIL_ARCHIVE_DIRECTORY	ses-inbound-mail.archive.directory	inbound-mail
SES_INBOUND_MAIL_RETENTION_DAYS	ses-inbound-mail.retention.days	none, every mail is kept
SES_INBOUND_MAIL_QUEUE_CONNECTION	ses-inbound-mail.queue.connection	the default connection
SES_INBOUND_MAIL_QUEUE	ses-inbound-mail.queue.name	the default queue
SES_INBOUND_MAIL_TAKEOVER_TRIES	ses-inbound-mail.queue.tries	5
SES_INBOUND_MAIL_AWS_REGION	ses-inbound-mail.aws.region	eu-central-1
SES_INBOUND_MAIL_AWS_ACCOUNT	ses-inbound-mail.aws.account	none
SES_INBOUND_MAIL_BUCKET	ses-inbound-mail.aws.bucket	none
SES_INBOUND_MAIL_PREFIX	ses-inbound-mail.aws.prefix	inbound/
SES_INBOUND_MAIL_TOPIC	ses-inbound-mail.aws.topic	none
SES_INBOUND_MAIL_RULE	ses-inbound-mail.aws.rule	none
SES_INBOUND_MAIL_RECIPIENTS	ses-inbound-mail.aws.recipients	none
SES_INBOUND_MAIL_ENDPOINT	ses-inbound-mail.aws.endpoint	none
SES_INBOUND_MAIL_IAM_USER	ses-inbound-mail.aws.user	none

A list is separated by commas, and spaces around an entry are ignored. A number that is no number, or one below 1, keeps the default. The settings under ses-inbound-mail.aws are read by the setup alone, and it names every one that is missing.

A model of your application can take the place of the package's model through ses-inbound-mail.models, keyed by the package's class. It has to extend the package's model, or it is ignored.

Translations

The package ships its catalogs in German, English, Spanish, French, Italian, Dutch and Portuguese. A regional locale reads the catalog of its language: pt-BR and pt-PT read pt, en-GB reads en, and so does a region nobody listed, such as fr-CA or de_AT.