Pages - Menu

Amazon SDK S3 with .Net

Scope

We decided to move our images on our websites from local drive to Amazon S3. This includes all our product images from the e-commerce side; as well as any images or videos that are associated to the CMS side.

Setup and Installation

AWS Toolkit for Visual Studio

  1. Follow the Amazon Setup Guide to install the AWS Toolkit. 
  2. Download and install (or NuGet) AWS SDK for .Net.
  3. Configuring AWS Credentials
  4. Use the Amazon Endpoint List to configure the end point
  5. The Amazon SDK API documentation is pretty easy to read.

Code

Upload an object

By using the Amazon SDK, We change our file service to put files to S3 instead of saving to local folder. It divides into 2 parts. We will prepare the PutObjectRequest and call AmazonS3Client.PutObject() to upload the file.

public override HttpStatusCode UploadPicture(string targetPath, string sourcePath)
{
    // Create a PutObject request
    var request = new PutObjectRequest
    {
        BucketName = bucket,
        Key = targetPath.ToLower(),
        FilePath = sourcePath,
        CannedACL = S3CannedACL.PublicRead
    };

    try
    {
        using (var client = new AmazonS3Client(Amazon.RegionEndpoint.APSoutheast2))
        {
            // Put object
            PutObjectResponse response = client.PutObject(request);

            return response.HttpStatusCode;
        }
    }
    catch (Exception ex)
    {
        // logging and exception handling....
    }

    return HttpStatusCode.NotFound;
}

S3CannedACL is a predefined Access Control List that allows access to be set on S3 objects.

Access an object

To access an object in s3 is much easier. You will just need to know the url and have the correct ACL setup.

The URL format is in a form of http://{mybucket}.s3.amazonaws.com/{mykey} 

mybucket is the bucket name and mykey is the key (file path) to the object.


The Quickest Way to Bulk Create AWS S3 Buckets

I am in a mission of creating a bunch of s3 buckets in bulk. I could just use the web interface and do it manually. I reckon I can get the job done in around 30 minutes, but I am looking for a more an IT way to do it.

Solution

After a quick look around, I think I can use the Command Line Interface.

Download and Setup


Configure


Run the aws configure command to create a default profile. Fairly straight forward.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

A list of region name can be found here.

Commands


http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

I use the ls command to pull out a list of my buckets.

$ aws s3 ls

In order to create buckets, the command I need is mb.

$ aws s3 mb s3://mybucket

I prepared my list of commands by using Excel / Find and Replace.

$ aws s3 mb s3://sunnyw.net.dummy1
$ aws s3 mb s3://sunnyw.net.dummy2
$ aws s3 mb s3://sunnyw.net.dummy3

And I just copy and paste into the console.

PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy1
make_bucket: s3://sunnyw.net.dummy1/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy2
make_bucket: s3://sunnyw.net.dummy2/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy3
make_bucket: s3://sunnyw.net.dummy3/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell>

I didn't need any fancy API calls or build any C# console. If the requirement was more complicated, I could put those logics in PowerShell script. Simple is Good!!

Amazon Web Services - AWS101


Recently had a meeting with the Amazon expert discussing about our AWS needs. We had a brief overview of all the relevant AWS services. There were many jargon involved, I already know some of them, but not all. Mostly are just product names.

Region and Availability Zone

Availability Zone is a physically isolated location or what traditionally called data centre. In an ideal setup we would split our instances into 2 AZs for redundancy. I was told the connection between the AZs are dark fiber and latency at single digit millisecond.

Amazon region can have multiple AZs. See the difference between Region and Availability Zone.

Elastic Load Balancing

ELB is the amazon version of network load balancing. It sits in front of AZs and redirects traffics to different AZs.

CloudWatch

Amazon CloudWatch is normally what I refer to as website monitoring. It does health check on the system and sends alert to ELB for traffic control.

EC2 (Elastic Compute Cloud)

EC2 are virtual machines. We can setup IIS on EC2 instance(s) to host our websites. When we spin up an EC2 instance, we can choose from a range of predefined images with different operating system and applications.

Amazon Machine Image

I will interpret AMI as a fancy name for a machine snapshot image or known as ghost image in old fashion. We can use it to scale out our system or simply for backup / restore strategy.

Relational Database Service

RDS is what I would call a managed database instances. It is an already setup database instance and I can utilize without managing a real database instance that I need to rdp in or patch.

S3 (Simple Storage Service)

I am a little more familiar with S3 because my site is hosted by using the S3 bucket. S3 is a plain cloud file storage. Some people misinterpret S3 as CDN, because it can potentially be utilized as CDN for saving bandwidth, but there is a difference.

S3 does not utilize edge servers for faster content delivery. It only has one copy of the files in one location. It is however designed for high performance to serve static files (eg. images, video etc), and also cheaper rate as no compute hours are charged.

CloudFront

The Amazon version of CDN that does edge caching is actually called CloudFront. It is designed to deliver S3 contents to end users via edge servers.

Simple Email Service

SES is a SMTP service for sending out emails.

CloudFormation

I will describe CloudFormation as a non-UI way to create and instantiate different kind of Amazon Services. For example, we can create a EC2 instance by just writing a script in CloudFormation format and run it via the console.

C# .Net Cannot close stream until all bytes are written.

Scenario

We are dealing with the Stream class in .Net and run into an inner exception of "Cannot close stream until all bytes are written."

Code

try
{
    using (var ms = new MemoryStream())
    {
        xElement.Save(ms);

        // Create a PutObject request
        var request = new PutObjectRequest
        {
            BucketName = bucket,
            Key = key,
            InputStream = ms
        };

        // Call S3 ....
    }
}
catch (Exception ex)
{
    _logger.Error(ex.Message, ex);
    throw ex;
}

By using the keyword using, we are expecting .Net will handle the close connection for us.

Rather than unable to close the stream, it was actually a problem where the position of the MemoryStream was at the end after the XElement.Save(). We need to manually reset the position of the stream to zero, so that it can be written to the PutObjectRequest object.

xElement.Save(ms);
ms.Position = 0;


SQL - How to do a CHARINDEX in reverse

I am a little surprise when SQL 2012 have introduced functions like TRY_CONVERT, but there are still no built-in functions to perform a CHARINDEX in reverse.

Code

Select UrlRecord.Slug
, LEN(UrlRecord.Slug) [Length]
, CHARINDEX('-', UrlRecord.Slug, 0) [Normal CHARINDEX]
, LEN(UrlRecord.Slug) - CHARINDEX('-', Reverse(UrlRecord.Slug), 0) + 1 [Reverse CHARINDEX with bug]
, IIF(CHARINDEX('-', UrlRecord.Slug, 0) = 0, 0, LEN(UrlRecord.Slug) - CHARINDEX('-', Reverse(UrlRecord.Slug), 0) + 1) [Desired CHARINDEX]
from UrlRecord
where EntityName = 'Category'
Order by Slug



In the above code, we are trying to find the last position (a charindex in reverse order) of a dash '-' from a string (UrlRecord.Slug).


  • Row 68, for a string that have only one matching character '-', either direction will return the same result.
  • Row 72, a Reverse CharIndex will return 10 for the last occurrence as opposed to a CharIndex that will return 5 for the first occurrence.
  • Row 77, there is a bug when we calculate the reverse but no matching string, it will return the full length. We will do a IIF check to return 0 when there is no match.


The final code became this.

Declare @matchChar varchar(1) = '-'
Declare @sourceString varchar(max) = 'test-test-test'

Select IIF(CHARINDEX(@matchChar, @sourceString, 0) = 0, 0, LEN(@sourceString) - CHARINDEX(@matchChar, Reverse(@sourceString), 0) + 1)

Performance

I did not pay attention to performance at all. I am sure there are rooms for improvement, but I am thinking a built-in function for such a simple operation is probably overdue by Microsoft anyway.

nopCommerce - Multi Store / Multi Currency / Multi Price for each product

Scope

We have had a few sites went live with nopCommerce. I am in the process to replicate our existing Australia sites to New Zealands sites.
  • Both sites to sell the same products, with stocks coming from the same location.
  • Products dispatch from the same warehouse.
  • Customizable Price, Old Price and Special Price per product per store and not using any currency converter.
  • etc.

What I am trying to achieve is also described here and here by other people. Or as a workitem in codeplex. No viable solution at time of writing this article.

This is base on nopCommerce v3.3.

Business Case

nopCommerce out of the box supports multiple currencies by using a currency converter.



However, this is only practical if your company is 1 entity and want to accept orders from around the world. Currency converter provides an estimate for users on how much they will be spending, but the transaction is still processed in my primary currency - AU dollars in our case.

Example of Currency Converter from my previous project - Bardot

In our business case, we own both of the AU and NZ companies as separate business entities. Our NZ company have their own price and we would like to accept local NZ credit cards in NZ currency. In this scenario, currency converter is not a feasible solution.

Technical Overview

  • Create a new ProductPrices table to store prices for different stores
  • Create an admin screen to handle Product Prices
  • Create a GetCustomPrice() to retrieve price from ProductPrices if store id match
  • Replace any existing controllers or services that use Product.Price to use GetCPrice()

Prerequisite

Code

ProductCustomPrice


We created a new SQL table to store our custom price. 


In our code, we need to create a new domain object for ProductCustomPrice and a mapping file. This is already demonstrated in previous posts in prerequisite.

PriceCalculationService


We need to override the PriceCalculationService.GetFinalPrice() in the Product Price region to use our custom price. We will need a small refactor work in core.

// decimal result = product.Price;
decimal result = GetProductInitPrice(product);

We will write a method to retrieve the custom price from our database.

public virtual ProductCustomPrice GetProductCustomPriceByProductId(int productId)
{
    if (productId == 0)
        return null;
            
    var currentStoreId = _storeContext.CurrentStore.Id;
    var query = _productCustomPriceRepository.Table
                    .Where(p => p.StoreId == currentStoreId 
                            && p.ProductId == productId);

    return query.FirstOrDefault();
}

We will override the method in our new CustomPriceCalculationService class. It will pick up custom price if available, otherwise default.

protected override decimal GetProductInitPrice(Product product)
{
    var customPrice = GetProductCustomPriceByProductId(product.Id);

    if (customPrice != null)
        return customPrice.Price;

    return base.GetProductInitPrice(product);
}

CatalogController


We then need to do the same for OldPrice, SpecialPrice, SpecialPriceStartDateTimeUtc and SpecialPriceEndDateTimeUtc. I will leave this fun exercise for my readers. As a hint, a good starting point is to refactor the CatalogController like what we did for Price.

// decimal oldPriceBase = _taxService.GetProductPrice(product, product.OldPrice, out taxRate);
decimal oldPriceBase = _taxService.GetProductPrice(product, _priceCalculationService.GetProductInitOldPrice(product), out taxRate);

We now able show different Price and SpecialPrice for multi stores.


I built an admin screen base on the TierPrice screen and ProductCustomPrice() CRUD base on Nop.Admin.Controllers.ProductController.TierPriceList().

For example, the Reebok Core Price is $169.99 and Special Price is $118.99.

By introducing Per Store Custom Price, the Reebok NZ Local store would have a customized price of $200, an old price of $180 and a special price of $130.






Other Solutions that didn't work

Tier Price

Initially I was trying to use store specific tier price with 1 unit, and in the hope that the price will be overridden by the tier price. It didn't work in 2 aspects.

Firstly, the way how tier price works is to give a better discount for the customer if they are buying multiples. The logic in nopCommerce will check if the tier price is cheaper than the original and only apply if cheaper. Unfortunately the foreign currency is not always cheaper than the primary.

Secondly, I cannot get store specific for OldPrice or SpecialPrice.

Conclusion

This is not the best solution but the one of the possible solution. If I was a core nopCommerce developer, I would have changed the domain object to work with the prices.

I am very lucky that Australia and New Zealand use the same currency symbol. For wider audience sake, I think the solution needs to be extended to cover other area as well.

Setting up nopCommerce in Azure

I am preparing a migration plan for our nopCommerce sites from our data center to cloud. Since nopCommerce is available in the Azure Gallery, I thought I would just spin one up and see how it goes.

Installation

Gallery

From gallery, I chose nopCommerce. Note that they only have the latest version.



Database

After the instance is up, I have to go thru the standard installation guide for nopCommerce. Fairly straight forward non-cloud stuff.

So far so good, in fact I don't anticipate any issues at this stage.

Store Front

And we have an e-commerce store setup on Azure comfortably easy.



Setup

Plugins

I thought there will be some resistance with the plugins folder permissions, but in fact not. I installed a commercial pack of plugins and everything work straight out of the box. ;)


Email

Unfortunately email services wouldn't work just out of the box.

Since there is no SMTP server setup in the cloud, we will need to use 3rd party service to help us sending out emails. One of the popular one you can find on the Azure Marketplace is SendGrid.

I have my connection string set as follow in nopCommerce, and email sent out and received fine.




Payment Gateway

For our test, I have setup the Payments.PayPalDirect plugin.


After ordering some sample products from our website, the fund would go thru to PayPal Sandbox.



XML Drop

In some of the old fashion system, we are still relying on an older technology that I called Xml Drop or Xml Dump.

Basically what it does is in our site, we will generate an xml file that contains some information and the 3rd party integration system will have a collector to pick up those xml files and perform their other actions. These xml files are not exposed publicly via web, but in a form of secure ftp or a share drive via vpn with authentication and authorization.

Unfortunately I cannot find such products on Azure would do that. The closest item that I found is the Azure Storage but I cannot replace Xml Drop directly.

A better solution is to replace the Xml Drop by using Web Api. If the 3rd party system is not capable to call an api, we will then need to use Queue or ServiceBus.

Conclusion

At first, I anticipated I would get much resistance at the integration points or folder permission as it is a managed IIS instance, but it turns out it was pretty straight forward.