Working with AWS S3 through C#


A while ago I needed to use AWS S3 (the Amazon’s cloud-based file storage) to store some files and then download them or get their listings through C#. As ironic as it sounds I noticed that there was no .NET implementation nor a documentation for S3 so I decided to create a file repository in C# which lets .NET developers access S3 programmatically.

Here is a rundown as to how you would work with AWS S3 through C#:

In order to use any APIs of Amazon Web Services (AWS) you will have to add the nugget package that is provided by Amazon. Simply bring up the Nuget Package Manager window and search for the keyword AWS. The first item in the search result is most likely AWS SDK for .NET which must be installed before you can access S3.

 

 

Once the SDK is installed we will have to find the properties of our S3 bucket and place it somewhere in web.config (or app.config) file. Normally these three properties of the S3 bucket is required in order to access it securely:

 

  1. Secret key
  2. Access key
  3. Region end point

These details will be provided to you by your cloud administrator. Here is a list of region end points that you can place in your configuration file (e.g. us-west-1)

 

Region name Region Endpoint Location constraint Protocol
US Standard * us-east-1 You can use one of the following two endpoints:

  • s3.amazonaws.com (Northern Virginia or Pacific Northwest)
  • s3-external-1.amazonaws.com (Northern Virginia only)
(none required) HTTP and HTTPS
US West (Oregon) region us-west-2 s3-us-west-2.amazonaws.com us-west-2 HTTP and HTTPS
US West (N. California) region us-west-1 s3-us-west-1.amazonaws.com us-west-1 HTTP and HTTPS
EU (Ireland) region eu-west-1 s3-eu-west-1.amazonaws.com EU or eu-west-1 HTTP and HTTPS
EU (Frankfurt) region eu-central-1 s3.eu-central-1.amazonaws.com eu-central-1 HTTP and HTTPS
Asia Pacific (Singapore) region ap-southeast-1 s3-ap-southeast-1.amazonaws.com ap-southeast-1 HTTP and HTTPS
Asia Pacific (Sydney) region ap-southeast-2 s3-ap-southeast-2.amazonaws.com ap-southeast-2 HTTP and HTTPS
Asia Pacific (Tokyo) region ap-northeast-1 s3-ap-northeast-1.amazonaws.com ap-northeast-1 HTTP and HTTPS
South America (Sao Paulo) region sa-east-1 s3-sa-east-1.amazonaws.com sa-east-1 HTTP and HTTPS

 

In order to avoid adding the secret key, access key and region endpoint to the <appSettings> part of your configuration file and to make this tool more organised I have created a configuration class for it. This configuration class will let you access the <configurationSection> element that is related to S3. To configure your app.config (or web.config) files you will have to add these <sectionGroup> and <section> elements to your configuration file:

 

<configSections>

<sectionGroup
name=AspGuy>

<section
name=S3Repository
type=Aref.S3.Lib.Strategies.S3FileRepositoryConfig, Aref.S3.LiballowLocation=true
allowDefinition=Everywhere />

</sectionGroup>

</configSections>

The
S3FileRepositoryConfig 
class is inherited from ConfigurationSection class and has properties that map to some configuration elements of your .config file. A sample configuration for S3 is like this:

 

<AspGuy>

<S3Repository


S3.ReadFrom.AccessKey=xxxxxxxxxx


S3.ReadFrom.SecretKey=yyyyyyyyyyyyyyyyy


S3.ReadFrom.Root.BucketName=-bucket-name-


S3.ReadFrom.RegionName=ap-southeast-2


S3.ReadFrom.RootDir=“”>

</S3Repository>

</AspGuy>

Note that <AspGuy> comes from the name property of <sectionGroup name=”AspGuy”> element. Also <S3Repository> tag comes from the name of <section> element. Each property of S3FileRepositoryConfig
is mapped to an attribute of <S3Repository> element.

Apart from SecretKey, AccessKey andBucketName you can specify a root directory name as well. This setting is there so you can begin accessing the S3 bucket from a specific folder rather than from its root, and obviously this setting is optional. For example imagine there is a bucket with the given folder structure:

  • Dir1
  • Dir1/Dir1_1
  • Dir1/Dir1_2

If you set the RootDir property to “” then when you call the GetSubDir methods of the S3 file repository if will return “Dir1” because Dir1 is the only top-level folder in the bucket. If you set the RootDir property to “Dir1” and then call the GetSubDirs method you will get two entries which are “Dir1_1” and “Dir1_2”.

Here is the code of the configuration class mentioned above:

 

For the repository class I have created an interface because of removing the dependency of clients (e.g. a web service that may need to use with various file storages) on S3. This will let you add your implementation of file system, FTP and other file storage types and use then through dependency injection. Here is the code of this interface:

 

In this interface:

  • Download: Downloads a file hosted on S3 to disk.
  • ChangeDir: Changes the current directory/folder to the given directory. If the new directory (relativePath parameter) starts with / then the path will be representing an absolute path (starting from the RootDir) otherwise it will be a relative path and will start from the current directory/folder.
  • GetFileNames: Retrieves the file names of the current folder
  • GetSubDirNames: Retrieves the name of folders in the current folder
  • AddFile: Uploads a file to S3
  • FileExists: Checks to see if a file is already on S3
  • DeleteFile: Deletes the file from S3

The implementation of these method are quiet simple using AWS SDK for .NET. The only tricky part is that S3 does not support folders. In fact in S3 everything is a key-value pair and the structure of entries is totally flat. What we do however is to use the forward slash character to represent folders and then we use this character as a delimiter to emulate a folder structure.

Untitled

 

 

 

You can clone the repository of this code which is on GitHub to play with the code. Feel free to send a pull request if you want to improve the code. The GitHub repository is located at https://github.com/aussiearef/S3FileRepository

 

 

 

 

 

 

Advertisements

3 thoughts on “Working with AWS S3 through C#

  1. Thanks for sharing superb informations. Your website is very cool. I’m impressed by the details that you have on this website. It reveals how nicely you understand this subject. Bookmarked this website page, will come back for more articles. You, my friend, ROCK! I found simply the information I already searched all over the place and simply could not come across. What a perfect website.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s