nopCommerce - AWS Migration

Scope

We are moving our existing websites to AWS EC2. From a high level perspective, our plan as follow.

• Move the websites to EC2
• Move the database to RDS
• Content Images upload via CMS will go to s3
• Product Images on the websites are moved to s3
• Move the console apps to a t2.micro instance

We are using nopCommerce 3.3.

Implementation

Websites and Console Apps

Moving sites and apps to cloud instances are pretty straight forward. There are no code changes required in nopCommerce.

Database

We used the SQL Database Migration Wizard tool, pretty straight forward.

S3

Migration to S3 required some code changes, but we are not getting much resistance because of the advantage of using OO and polymorphism.

Previously we already wrote a custom class that inherit from the PictureService.

public class TAPictureService : PictureService, ITAPictureService, IPictureService

Now, I can inherit my custom class by my new AWS class.

public class AwsPictureService : TAPictureService, ITAPictureService, IPictureService

We will then override the image read / write method to connect to AWS. This was mentioned previously in my other post Amazon SDK S3 with .Net

In our DependencyRegister, we now register our new AWS Service.

builder.RegisterType<AwsPictureService>().As<IPictureService>();

Now my system is able to support either local or S3 images by changing the register type via DI.

Amazon SDK S3 with .Net

Scope

We decided to move our images on our websites from local drive to Amazon S3. This includes all our product images from the e-commerce side; as well as any images or videos that are associated to the CMS side.

Setup and Installation

AWS Toolkit for Visual Studio

1. Follow the Amazon Setup Guide to install the AWS Toolkit. 
2. Download and install (or NuGet) AWS SDK for .Net.
3. Configuring AWS Credentials
4. Use the Amazon Endpoint List to configure the end point
5. The Amazon SDK API documentation is pretty easy to read.

Code

Upload an object

By using the Amazon SDK, We change our file service to put files to S3 instead of saving to local folder. It divides into 2 parts. We will prepare the PutObjectRequest and call AmazonS3Client.PutObject() to upload the file.

public override HttpStatusCode UploadPicture(string targetPath, string sourcePath)
{
    // Create a PutObject request
    var request = new PutObjectRequest
    {
        BucketName = bucket,
        Key = targetPath.ToLower(),
        FilePath = sourcePath,
        CannedACL = S3CannedACL.PublicRead
    };

    try
    {
        using (var client = new AmazonS3Client(Amazon.RegionEndpoint.APSoutheast2))
        {
            // Put object
            PutObjectResponse response = client.PutObject(request);

            return response.HttpStatusCode;
        }
    }
    catch (Exception ex)
    {
        // logging and exception handling....
    }

    return HttpStatusCode.NotFound;
}

S3CannedACL is a predefined Access Control List that allows access to be set on S3 objects.

Access an object

To access an object in s3 is much easier. You will just need to know the url and have the correct ACL setup.

The URL format is in a form of http://{mybucket}.s3.amazonaws.com/{mykey} 

mybucket is the bucket name and mykey is the key (file path) to the object.

The Quickest Way to Bulk Create AWS S3 Buckets

I am in a mission of creating a bunch of s3 buckets in bulk. I could just use the web interface and do it manually. I reckon I can get the job done in around 30 minutes, but I am looking for a more an IT way to do it.

Solution

After a quick look around, I think I can use the Command Line Interface.

Download and Setup

• http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-msi-on-windows
• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html

Configure

http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-msi-on-windows

Run the aws configure command to create a default profile. Fairly straight forward.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

A list of region name can be found here.

Commands

http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

I use the ls command to pull out a list of my buckets.

$ aws s3 ls

In order to create buckets, the command I need is mb.

$ aws s3 mb s3://mybucket

I prepared my list of commands by using Excel / Find and Replace.

$ aws s3 mb s3://sunnyw.net.dummy1
$ aws s3 mb s3://sunnyw.net.dummy2
$ aws s3 mb s3://sunnyw.net.dummy3

And I just copy and paste into the console.

PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy1
make_bucket: s3://sunnyw.net.dummy1/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy2
make_bucket: s3://sunnyw.net.dummy2/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy3
make_bucket: s3://sunnyw.net.dummy3/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell>

I didn't need any fancy API calls or build any C# console. If the requirement was more complicated, I could put those logics in PowerShell script. Simple is Good!!

Amazon Web Services - AWS101

Recently had a meeting with the Amazon expert discussing about our AWS needs. We had a brief overview of all the relevant AWS services. There were many jargon involved, I already know some of them, but not all. Mostly are just product names.

Region and Availability Zone

Availability Zone is a physically isolated location or what traditionally called data centre. In an ideal setup we would split our instances into 2 AZs for redundancy. I was told the connection between the AZs are dark fiber and latency at single digit millisecond.

Amazon region can have multiple AZs. See the difference between Region and Availability Zone.

Elastic Load Balancing

ELB is the amazon version of network load balancing. It sits in front of AZs and redirects traffics to different AZs.

CloudWatch

Amazon CloudWatch is normally what I refer to as website monitoring. It does health check on the system and sends alert to ELB for traffic control.

EC2 (Elastic Compute Cloud)

EC2 are virtual machines. We can setup IIS on EC2 instance(s) to host our websites. When we spin up an EC2 instance, we can choose from a range of predefined images with different operating system and applications.

Amazon Machine Image

I will interpret AMI as a fancy name for a machine snapshot image or known as ghost image in old fashion. We can use it to scale out our system or simply for backup / restore strategy.

Relational Database Service

RDS is what I would call a managed database instances. It is an already setup database instance and I can utilize without managing a real database instance that I need to rdp in or patch.

S3 (Simple Storage Service)

I am a little more familiar with S3 because my site is hosted by using the S3 bucket. S3 is a plain cloud file storage. Some people misinterpret S3 as CDN, because it can potentially be utilized as CDN for saving bandwidth, but there is a difference.

S3 does not utilize edge servers for faster content delivery. It only has one copy of the files in one location. It is however designed for high performance to serve static files (eg. images, video etc), and also cheaper rate as no compute hours are charged.

CloudFront

The Amazon version of CDN that does edge caching is actually called CloudFront. It is designed to deliver S3 contents to end users via edge servers.

Simple Email Service

SES is a SMTP service for sending out emails.

CloudFormation

I will describe CloudFormation as a non-UI way to create and instantiate different kind of Amazon Services. For example, we can create a EC2 instance by just writing a script in CloudFormation format and run it via the console.