Receiving email attachments in AWS S3 compatible storage

CloudMailin has supported sending email attachments directly to AWS S3, Azure Blob Storage and Google Cloud Storage for some time. Our users have used this feature to offload the responsibility of storing email attachments to the cloud and reduce the overhead for their web applications.

Today, we are excited to announce a major enhancement: you can now send email attachments directly to a wide array of S3-compatible stores. This expansion includes leading providers such as Digital Ocean Spaces, Cloudflare R2, Wasabi, MinIO, and many more. In this post, we will dive deep into the benefits of this integration, explore the technical details, discuss security and privacy considerations, and walk through a practical example using Digital Ocean Spaces.

There's a lot of background in this post so feel free to skip to the section that interests you most:

• Introduction to Email-to-S3 Integration
• Why send email attachments to cloud storage? Sending Email Attachments to Digital Ocean, Cloudflare R2, Wasabi and More
• Privacy and security of Amazon S3 compatible storage
• Configuring S3-Compatible Storage: An Example with Digital Ocean Spaces

Introduction to Email-to-S3 Integration

CloudMailin added the ability to send email attachments to Amazon Web Services S3 in the early days of the service. The idea was to offload the responsibility of storing email attachments and let the web server focus on processing the email content.

We later added support for Microsoft Azure Blob Storage and Google Cloud Storage. As cloud services evolved, we then added the ability to store the full email content. This has enabled a wide range of use cases and integrations. As cloud functions like AWS Lambda have become more popular, the ability to store email attachments directly to cloud storage has become a valuable feature.

Automatically uploading received email attachments to AWS S3

If we don't offload attachments then CloudMailin will send your web application the contents of the email as either multipart/form-data or as a JSON payload. When using the JSON format this might look something like the following:

{
  "headers": {
    "from": "from@example.com",
    "to": "to@example.com",
    "subject": "Test email",
    ...
  },
  "plain": "This is a test email",
  "html": "<p>This is a test email</p>",
  "attachments": [
    {
      "content": "dGVzdGZpbGU=",
      "file_name": "file.txt",
      "content_type": "text/plain",
      "size": 8,
      "disposition": "attachment"
    }
  ]
}

There are only 8 bytes in this example but the payload could be much larger. When we offload attachments then CloudMailin will send your web application the URL of the attachment:

  ...
  "attachments": [
    {
      "url": "https://yourbucket.s3.amazonaws.com/attachments/1234567890",
      "file_name": "file.txt",
      "content_type": "text/plain",
      "size": 8,
      "disposition": "attachment"
    }
  ]

Why send email attachments to cloud storage?

As we stated, the initial reason for offloading attachments was that the overhead of receiving large files in the earlier days of the web could be problematic, it would often lead to timeouts and other issues. As the web (and cloud) evolved the benefits became greater. There are several reasons to direct email attachments cloud storage options:

• Scalability: S3-compatible stores are designed to scale effortlessly. As your volume of email attachments grows, these platforms automatically handle the increased load without requiring manual intervention.

• Reliability: Providers such as Amazon's S3 and its compatible alternatives offer high durability and availability. With multiple redundancies built into these systems, you can be confident that your data is safe and accessible when needed.

• Performance: Directly uploading attachments to cloud storage eliminates the need for intermediary steps, reducing latency and ensuring faster access to your stored files.

• Flexibility: By supporting multiple providers, CloudMailin enables you to choose the storage solution that aligns best with your technical and business requirements. This flexibility is critical in a multi-cloud strategy where diversification can lead to enhanced performance and risk mitigation.

Sending Email Attachments to Digital Ocean, Cloudflare R2, Wasabi and More

Over the years we've had a few requests from users to open up support for additional S3 compatible storage providers. Amazon's S3 is a great service, and a number of other providers have appeared on the market offering the ability to use the same API to upload files to their services. In fact Google's offering provides an S3 compatible API in addition to it's own too.

Today we're excited to announce that you can now send email attachments directly to a wide array of S3 compatible stores.

The expansion to S3 compatible storage providers will be a welcome addition for many of our users. Not everyone uses AWS S3 for their cloud storage needs; some prefer alternatives that may offer lower costs, better performance in certain regions or simply the ability to keep billing in one place.

We've tested support with Digital Ocean Spaces, Cloudflare R2 and Wasabi but expect that this will work with any S3 compatible storage provider.

Privacy and Security of Amazon S3 compatible storage

Before we look at an example of configuring Digital Ocean Spaces it's important to talk about the security aspect of this integration. When we upload attachments to Amazon or Google we use a set of credentials that belong to CloudMailin. We ask you to configure access but the account details are not shared and are owned and secured by CloudMailin.

This isn't possible with S3 compatible storage as we can't have an account with every provider. For this reason, we'll ask you to provide an access key and secret key for the provider. These will be encrypted to prevent plain text storage, and we'll explain this process in detail below. If you're not interested in the technical details, you can skip to the example below.

Limit Scope (Principle of Least Privilege)

Not all providers support the full suite of control over access keys but we think it's important to bear this in mind.

When you generate a new access key and secret we ask that you generate it specifically for CloudMailin if you can. It's also a good idea to limit the scope of the access as much as possible and generally follow best practice.

• Provide a dedicated access key and secret for CloudMailin
• Only provide access to the bucket / storage account that's needed
• Limit permissions to write only if it's possible, we don't need to read your attachments
• Rotate the access key and secret regularly
• Monitor and audit access logs regularly

How we store your credentials

When you configure your S3 compatible storage provider you'll be asked to provide an access key and secret key. The access key will be stored in plain text. The secret key will be used to validate the access to the provider and then it will be encrypted.

In order to do this our web server contains only a public key. This public key is used to perform RSA encryption on the secret key with padding and a 2048 bit key. The encrypted secret key is then converted to a base64 encoded string and stored in the database.

If you ever go to modify your configuration you'll see the secret key is stored in the database in the form enc_[***]. This is the encrypted base64 encoded key.

Because the web interface doesn't have the private key we can't decrypt the key in order to test any configuration changes that you make. You'll need to re-add the secret key so that we can test our access to the storage provider.

This can be a little annoying but we think it's a good compromise to ensure that data is stored securely in the database.

As the server receives your email it takes the encrypted secret key and decrypts it using the private key that only the server has. It then uses this to access the storage provider and upload the attachment on your behalf.

Configuring S3-Compatible Storage: An Example with Digital Ocean Spaces

Now that we've covered how the integration works, let's walk through an example, configuring Digital Ocean Spaces.

Digital Ocean Spaces

First, head to the Digital Ocean control panel and create a new Space:

Once the space is created, you'll need to create a new access key. Currently we have to select Read/Write/Delete (Objects) permissions. Hopefully in future we'll be able to specify a more restrictive set of bucket permissions.

With the access key, we will be shown our secret key. We'll need all of these to configure the integration.

The final thing we'll need is the endpoint URL for the storage provider. This is a combination of the bucket name and the region in the following format: https://<region>.digitaloceanspaces.com in our case https://lon1.digitaloceanspaces.com. CloudMailin will automatically append the bucket name to the front of the endpoint domain name so don't include it in the endpoint URL.

In summary, you'll need:

• Access key
• Secret key
• Endpoint URL (without bucket name)
• Bucket name
• Region

Configuring CloudMailin Preferences to use Digital Ocean Spaces

We can now head over to our CloudMailin address (mailbox) and select the 'Attachment Storage' option, we can then select 'S3 Compatible' and fill in the details to setup our attachment store:

Once we've filled in the details we'll see that a file is created in our Digital Ocean Spaces to test that we have access. The secret key is then immediately encrypted using the RSA public key and stored in the database.

If you ever need to modify any of the configuration details you'll note that the secret key looks like enc_.... This is the encrypted, padded and base64 encoded secret key. You'll need to re-add the secret key in order to save the changes that you make to the configuration.

That's it! You're ready to start receiving email. As we receive your email, attachments will be automatically uploaded to your Digital Ocean Spaces. The URL of each email attachment will be sent to your web application as part of the JSON payload, along with the file name, content type, size, and disposition.

Conclusion and next steps

We're excited to have expanded our support for S3 compatible storage providers. We hope this will open up new opportunities for you to use CloudMailin to offload the responsibility of storing email attachments to the cloud.

The feature is available in beta to anyone with access to the current Attachment storage feature in their plan (any paid plan).

Our next step will be to expand the feature to allow sending the entire email to these storage providers in either JSON or Raw email formats. We call this our full message storage feature.

If you have any questions or feedback please contact us.