AWS – SNS
SNS is a publisher/subscriber messaging service. SNS makes easier for you to set up, operate, and send notifications from the cloud. It pushes notifications to Apple, Google, Fire OS, and Windows devices. Besides pushing notifications to mobile devices, SNS can also deliver messages to SMS text message, email (SES), SQS, or to any HTTP endpoint.
SNS notifications can trigger Lambda functions which take the notifications as payloads.
Amazon SNS isn’t currently compatible with FIFO queues but standard queues can subscribe to a topic. When you subscribe an Amazon SQS queue to an Amazon SNS topic, Amazon SNS uses HTTPS to forward messages to Amazon SQS.
SNS provides topics where subscribers can subscribe to and receive notifications. One topic can support deliveries to multiple endpoint types. Like for example if you group together iOS, Android, and SMS into a topic, when you publish to that topic all devices will get the same message.
SNS text messaging offers 100 free messages every month. Once you have gone over 100 messages then you will be charged with $0.00645 a message. This price varies from phone company to phone company.
Create a topic
@Override
public String createTopic(String name) {
final CreateTopicRequest createTopicRequest = new CreateTopicRequest(name);
CreateTopicResult createTopicResult = amazonSNS.createTopic(createTopicRequest);
return createTopicResult.getTopicArn();
}
Subscribe to a topic
@Override
public boolean subscribePhone(String topicArn, String phone) {
final SubscribeRequest subscribeRequest = new SubscribeRequest(topicArn, "sms", "+1"+phone);
amazonSNS.subscribe(subscribeRequest);
return true;
}
@Override
public boolean subscribeEmail(String topicArn, String email) {
final SubscribeRequest subscribeRequest = new SubscribeRequest(topicArn, "email", email);
amazonSNS.subscribe(subscribeRequest);
return true;
}
@Override
public boolean subscribeQueue(String topicArn, String queueUrl) {
String subscriptionArn = Topics.subscribeQueue(amazonSNS, amazonSQS, topicArn, queueUrl);
return (subscriptionArn!=null && subscriptionArn.isEmpty()==false) ? true : false;
}
Unsubscribe from a topic
@Override
public boolean unsubscribe(String subscriptionArn) {
UnsubscribeRequest unsubscribeRequest = new UnsubscribeRequest(subscriptionArn);
UnsubscribeResult unsubscribeResult = amazonSNS.unsubscribe(unsubscribeRequest);
return (unsubscribeResult!=null) ? true : false;
}
Send a message to a topic
@Override
public boolean sendMsgToTopic(String topicArn, String msg) {
final PublishRequest publishRequest = new PublishRequest(topicArn, msg);
final PublishResult publishResponse = amazonSNS.publish(publishRequest);
return (publishResponse.getMessageId()!=null) ? true : false;
}
Send an SMS message to a phone
@Override
public boolean text(String phone, String message) {
PublishResult result = amazonSNS.publish(new PublishRequest()
.withMessage(message)
.withPhoneNumber("+1"+phone)
.withMessageAttributes(new HashMap<String, MessageAttributeValue>()));
return (result.getMessageId()!=null) ? true : false;
}
August 5, 2019 AWS – SES
Amazon Simple Email Service (Amazon SES) is a cloud-based email sending service designed to help digital marketers and application developers send marketing, notification, and transactional emails. It is a reliable, cost-effective service for businesses of all sizes that use email to keep in contact with their customers.
You can use our SMTP interface or one of the AWS SDKs to integrate Amazon SES directly into your existing applications. You can also integrate the email sending capabilities of Amazon SES into the software you already use, such as ticketing systems and email clients.
Use cases:
- Send transactional emails – Keep your customers up-to-date by sending automated emails, such as purchase confirmations, shipping notifications, order status updates, and policy change notices.
- Send notification – Keep your users informed by sending timely information, including system health reports, application alerts, and workflow status updates.
- Send marketing communication – Promote your products and services by sending advertisements, newsletters, special offers, and any other type of high-quality content your customers want to see.
- Receive incoming email – Close the loop on your email program by using Amazon SES to receive email. Incoming email can be delivered automatically to an Amazon S3 bucket. You can use AWS Lambda to execute custom code when messages are received, or use Amazon SNS to deliver notifications when you receive incoming messages that contain certain keywords.
Bounce – If your recipient’s receiver (for example, an ISP) fails to deliver your message to the recipient, the receiver bounces the message back to Amazon SES. Amazon SES then notifies you of the bounced email through email or through Amazon Simple Notification Service (Amazon SNS), depending on how you have your system set up.
- Hard bounce – A persistent email delivery failure. For example, the mailbox does not exist. Amazon SES does not retry hard bounces, with the exception of DNS lookup failures. We strongly recommend that you do not make repeated delivery attempts to email addresses that hard bounce.
- Soft bounce – A temporary email delivery failure. For example, the mailbox is full, there are too many connections (also called throttling), or the connection times out. Amazon SES retries soft bounces multiple times. If the email still cannot be delivered, then Amazon SES stops retrying it.
Amazon SES notifies you of hard bounces and soft bounces that will no longer be retried. However, only hard bounces count toward your bounce rate and the bounce metric.
Bounces can also be synchronous or asynchronous. A synchronous bounce occurs while the email servers of the sender and receiver are actively communicating. An asynchronous bounce occurs when a receiver initially accepts an email message for delivery and then subsequently fails to deliver it to the recipient.
Complaint – Most email client programs provide a button labeled “Mark as Spam,” or similar, which moves the message to a spam folder, and forwards it to the ISP. Additionally, most ISPs maintain an abuse address (e.g., abuse@example.net), where users can forward unwanted email messages and request that the ISP take action to prevent them. In both of these cases, the recipient is making a complaint. If the ISP concludes that you are a spammer, and Amazon SES has a feedback loop set up with the ISP, then the ISP will send the complaint back to Amazon SES. When Amazon SES receives such a complaint, it forwards the complaint to you either by email or by using an Amazon SNS notification, depending on how you have your system set up. We recommend that you do not make repeated delivery attempts to email addresses that generate complaints.
Suppression List – The Amazon SES suppression list is a list of recipient email addresses that have recently caused a hard bounce for any Amazon SES customer. If you try to send an email through Amazon SES to an address that is on the suppression list, the call to Amazon SES succeeds, but Amazon SES treats the email as a hard bounce instead of attempting to send it. Like any hard bounce, suppression list bounces count towards your sending quota and your bounce rate. An email address can remain on the suppression list for up to 14 days. If you are sure that the email address that you’re trying to send to is valid, you can submit a suppression list removal request.
Verification – Unfortunately, it’s possible for a spammer to falsify an email header and spoof the originating email address so that it appears as though the email originated from a different source. To maintain trust between ISPs and Amazon SES, Amazon SES needs to ensure that its senders are who they say they are. You are therefore required to verify all email addresses from which you send emails through Amazon SES to protect your sending identity. You can verify email addresses by using the Amazon SES console or by using the Amazon SES API. You can also verify entire domains. If your account is still in the Amazon SES sandbox, you also need to verify all recipient addresses except for addresses provided by the Amazon SES mailbox simulator.
Sending Limits – If an ISP detects sudden, unexpected spikes in the volume or rate of your emails, the ISP might suspect you are a spammer and block your emails. Therefore, every Amazon SES account has a set of sending limits to regulate the number of email messages that you can send and the rate at which you can send them. These sending limits help you to gradually ramp up your sending activity to protect your trustworthiness with ISPs.
When a client makes a request to Amazon SES, Amazon SES constructs an email message compliant with the Internet Message Format specification. An email consists of a header, a body, and an envelop:
- Header—Contains routing instructions and information about the message. Examples are the sender’s address, the recipient’s address, the subject, and the date. The header is analogous to the information at the top of a postal letter, though it can contain many other types of information, such as the format of the message.
- Body—Contains the text of the message itself.
- Envelope—Contains the actual routing information that is communicated between the email client and the mail server during the SMTP session. This email envelope information is analogous to the information on a postal envelope. The routing information of the email envelope is usually the same as the routing information in the email header, but not always. For example, when you send a blind carbon copy (BCC), the actual recipient address (derived from the envelope) is not the same as the “To” address that is displayed in the recipient’s email client, which is derived from the header.
The following is a simple example of an email. The header is followed by a blank line and then the body of the email. The envelope isn’t shown because it is communicated between the client and the mail server during the SMTP session, rather than a part of the email itself.
Received: from abc.smtp-out.amazonses.com (123.45.67.89) by in.example.com (87.65.43.210); Fri, 17 Dec 2010 14:26:22 From: "Andrew" <andrew@example.com>; To: "Bob" <bob@example.com> Date: Fri, 17 Dec 2010 14:26:21 -0800 Subject: Hello Message-ID: <61967230-7A45-4A9D-BEC9-87CBCF2211C9@example.com> Accept-Language: en-US Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hello, I hope you are having a good day. -Andrew
If you are sending an email message to a large number of recipients, then it makes sense to send it in both HTML and text. Some recipients will have HTML-enabled email clients, so that they can click embedded hyperlinks in the message. Recipients using text-based email clients will need you to include URLs that they can copy and open using a web browser.
import com.amazonaws.regions.Regions;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailService;
import com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClientBuilder;
import com.amazonaws.services.simpleemail.model.Body;
import com.amazonaws.services.simpleemail.model.Content;
import com.amazonaws.services.simpleemail.model.Destination;
import com.amazonaws.services.simpleemail.model.Message;
import com.amazonaws.services.simpleemail.model.SendEmailRequest;
public class AmazonSESSample {
// Replace sender@example.com with your "From" address.
// This address must be verified with Amazon SES.
static final String FROM = "sender@example.com";
// Replace recipient@example.com with a "To" address. If your account
// is still in the sandbox, this address must be verified.
static final String TO = "recipient@example.com";
// The configuration set to use for this email. If you do not want to use a
// configuration set, comment the following variable and the
// .withConfigurationSetName(CONFIGSET); argument below.
static final String CONFIGSET = "ConfigSet";
// The subject line for the email.
static final String SUBJECT = "Amazon SES test (AWS SDK for Java)";
// The HTML body for the email.
static final String HTMLBODY = "<h1>Amazon SES test (AWS SDK for Java)</h1>"
+ "<p>This email was sent with <a href='https://aws.amazon.com/ses/'>"
+ "Amazon SES</a> using the <a href='https://aws.amazon.com/sdk-for-java/'>"
+ "AWS SDK for Java</a>";
// The email body for recipients with non-HTML email clients.
static final String TEXTBODY = "This email was sent through Amazon SES "
+ "using the AWS SDK for Java.";
public static void main(String[] args) throws IOException {
try {
AmazonSimpleEmailService client =
AmazonSimpleEmailServiceClientBuilder.standard()
// Replace US_WEST_2 with the AWS Region you're using for
// Amazon SES.
.withRegion(Regions.US_WEST_2).build();
SendEmailRequest request = new SendEmailRequest()
.withDestination(
new Destination().withToAddresses(TO))
.withMessage(new Message()
.withBody(new Body()
.withHtml(new Content()
.withCharset("UTF-8").withData(HTMLBODY))
.withText(new Content()
.withCharset("UTF-8").withData(TEXTBODY)))
.withSubject(new Content()
.withCharset("UTF-8").withData(SUBJECT)))
.withSource(FROM)
// Comment or remove the next line if you are not using a
// configuration set
.withConfigurationSetName(CONFIGSET);
client.sendEmail(request);
System.out.println("Email sent!");
} catch (Exception ex) {
System.out.println("The email was not sent. Error message: "
+ ex.getMessage());
}
}
}
August 5, 2019 AWS – SQS
SQS is a messaging system similar to ActiveMQ or RabbitMQ. It is a temporary repository for messages that need to be processed but the server is busy processing other messages. SQS is fully managed by AWS. You don’t have to manage middleware or servers. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
SQS is pull-based and not push-based. Messages can live in the queue from 1 minute to 14 days. The default retention period is 4 days.
Visibility timeout is the amount of time that a message is invisible while processing. If a message is not processed between the visibility timeout then it will become available again for another reader to pick up and process. This can result in one message being processed more than once. Default visibility timeout is 30 seconds. Maximum visibility timeout is 12 hours. Increase your visibility timeout according to your needs.
Messages can contain up to 256KB of text in any format. SQS solves the issue where messages or events are more than what the server can process. It will store the messages and hand them over to the server when the server is ready for them.
There are two queue types which are Standard Queues and FIFO Queues.
Standard Queue
Standard queues guarantee that a message is delivered once but occasionally a message might get delivered twice. Messages within standard queues are generally delivered in the order they are sent.
You can set delay seconds which tells SQS to wait for a certain amount of seconds before exposing the message to the readers.
Standard queues can serve unlimited number of messages.
FIFO Queue
FIFO which stands for first-in-first-out delivers and processes messages in the order they are sent. There are no duplicates for this type of queue.
FIFO queues support up to 3,000 messages per second with batching , or up to 300 messages per second (300 send, receive, or delete operations per second) without batching. You can submit a support ticket to AWS to increase this limit.
MessageGroupId groups messages that belong together and is processed in a FIFO manner. Other messages that don’t have the same messageGroupId might not be in order.
MessageDeduplicationId is used to make sure messages are unique and that messages are not sent into the queue multiple times. If a message with a particular MessageDeduplicationId is sent successfully, any messages sent with the same MessageDeduplicationId are accepted successfully but aren’t delivered during the 5-minute deduplication interval.
DelaySeconds attribute can only be set on the queue level for FIFO.
Long Polling
Long polling doesn’t return a response unless there is a message to process or the long polling timeouts. Long polling helps you save money as your API might make less frequent calls to the AWS SQS API. The recommended timeout is 20 seconds but can be more. The longer the timeout the less likely your API call will return an empty response which more like will result in saving resources.
Using SQS Locally
At the time of writing, I have not found a way that works consistently to run SQS on my local machine. I have tried many approaches people put on the internet but none worked for me. For local development, I use a set of queues different from the ones I use for other environments like Dev, QA, or production. This has been the best solution for me. If you find a solution that works for you please leave a link in the comment section.
Spring project example for sending an email after a user has signed up on your application.
@Bean
public AWSCredentialsProvider amazonAWSCredentialsProvider() {
return new ProfileCredentialsProvider("folauk100-dev");
}
@Bean
public AmazonSQS amazonSQSMain() {
AmazonSQS sqs = AmazonSQSClientBuilder.standard().withCredentials(amazonAWSCredentialsProvider())
.withRegion(Regions.US_WEST_2).build();
return sqs;
}
How to create both FIFO queue and Standard queue
public String createQueue(String name) {
CreateQueueRequest createQueueRequest = new CreateQueueRequest().withQueueName(name);
CreateQueueResult result = amazonSQS.createQueue(createQueueRequest);
log.debug("queue url: " + result.getQueueUrl());
log.debug("queue: " + result.toString());
return result.getQueueUrl();
}
/**
* Set delaySeconds if needed as it would be able to do at message sending.
* @param name
* @return
*/
public String createFIFOQueue(String name) {
final Map<String, String> attributes = new HashMap<>();
attributes.put("FifoQueue", "true");
attributes.put("ContentBasedDeduplication", "true");
CreateQueueRequest createQueueRequest = new CreateQueueRequest()
.withQueueName(name + ".fifo")
.withAttributes(attributes);
CreateQueueResult result = amazonSQS.createQueue(createQueueRequest);
log.debug("queue url: " + result.getQueueUrl());
log.debug("queue: " + result.toString());
return result.getQueueUrl();
}
Send messages to a queue
public boolean sendMessage(String queueUrl, SQSMessage msg, int delaySeconds, Map<String, MessageAttributeValue> messageAttributes) {
if(msg==null) {
throw new RuntimeException("msg is empty");
}
SendMessageRequest sendMessageRequest = new SendMessageRequest();
sendMessageRequest.withMessageBody(msg.toJson());
sendMessageRequest.withQueueUrl(queueUrl);
if(messageAttributes!=null) {
log.debug("messageAttributes set!");
sendMessageRequest.withMessageAttributes(messageAttributes);
}
if(delaySeconds>0) {
sendMessageRequest.withDelaySeconds(delaySeconds);
}
log.debug("messageAttributes={}",ObjectUtils.toJson(sendMessageRequest.getMessageAttributes()));
SendMessageResult sendMessageResult = amazonSQS.sendMessage(sendMessageRequest);
String resultId = sendMessageResult.getMessageId();
log.info("resultId={}", resultId);
return true;
}
Receive messages
@Async
public void processAccountQueue() {
int waitingSeconds = 10;
String queueUrl = SQSQueue.ACCOUNTS_QUEUE_URL;
final ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl);
receiveMessageRequest.withWaitTimeSeconds(waitingSeconds);
while (true) {
final List<Message> messages = amazonSQS.receiveMessage(receiveMessageRequest).getMessages();
log.debug("polling {}", queueUrl);
if (messages != null && messages.size() > 0) {
for (final Message message : messages) {
log.debug("Message Received");
SQSMessage sqsMessage = SQSMessage.fromJson(message.getBody());
int retryCount = 0;
while (true) {
try {
handleAccountMessage(sqsMessage);
break;
} catch (Exception e) {
if (retryCount == 5) {
break;
}
log.debug("retryCount={}", retryCount);
retryCount++;
}
}
// delete message once it's done
DeleteMessageResult deleteMessageResult = amazonSQS
.deleteMessage(new DeleteMessageRequest(queueUrl, message.getReceiptHandle()));
log.debug("delete message response={}",
ObjectUtils.toJson(deleteMessageResult.getSdkResponseMetadata()));
}
} else {
log.debug("{} is empty", queueUrl);
}
}
}
aws sqs list-queues --profile {profile-name}
aws sqs receive-message --queue-url {queue-url} --profile {profile-name} --attribute-names All --message-attribute-names All --max-number-of-messages 10
Empty all messages from a queue
aws sqs purge-queue --queue-url {queue-url} --profile {profile-name}
aws sqs send-message --queue-url {queue-url} --message-body "test body." --delay-seconds 10 --message-attributes file://message.json
aws sqs delete-message --queue-url {queue-url} --receipt-handle {messageReceiptHandle}
Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds. The minimum is 0 seconds. The maximum is 12 hours.
aws sqs change-message-visibility --queue-url {queue-url} --receipt-handle {messageReceiptHandle} --visibility-timeout 36000
If there are two readers or listeners listening to a particular queue, only one of the readers can process a message from the queue at any given time.
AWS – KMS and Ecryption
AWS Key Management Service (KMS) makes it easy for you to create and manage keys and control the use of encryption across a wide range of AWS services and in your applications. AWS KMS is a secure and resilient service that uses hardware security modules that have been validated under FIPS 140-2, or are in the process of being validated, to protect your keys. AWS KMS is integrated with AWS CloudTrail to provide you with logs of all key usage to help meet your regulatory and compliance needs.
You can perform the following management actions on your AWS KMS master keys:
- Create, describe, and list master keys
- Enable and disable master keys
- Create and view grants and access control policies for your master keys
- Enable and disable automatic rotation of the cryptographic material in a master key
- Import cryptographic material into an AWS KMS master key
- Tag your master keys for easier identification, categorizing, and tracking
- Create, delete, list, and update aliases, which are friendly names associated with your master keys
- Delete master keys to complete the key lifecycle
With AWS KMS you can also perform the following cryptographic functions using master keys:
- Encrypt, decrypt, and re-encrypt data
- Generate data encryption keys that you can export from the service in plaintext or encrypted under a master key that doesn’t leave the service
- Generate random numbers suitable for cryptographic applications
