Working with AWS S3 through C#


A while ago I needed to use AWS S3 (the Amazon’s cloud-based file storage) to store some files and then download them or get their listings through C#. As ironic as it sounds I noticed that there was no .NET implementation nor a documentation for S3 so I decided to create a file repository in C# which lets .NET developers access S3 programmatically.

Here is a rundown as to how you would work with AWS S3 through C#:

In order to use any APIs of Amazon Web Services (AWS) you will have to add the nugget package that is provided by Amazon. Simply bring up the Nuget Package Manager window and search for the keyword AWS. The first item in the search result is most likely AWS SDK for .NET which must be installed before you can access S3.

 

 

Once the SDK is installed we will have to find the properties of our S3 bucket and place it somewhere in web.config (or app.config) file. Normally these three properties of the S3 bucket is required in order to access it securely:

 

  1. Secret key
  2. Access key
  3. Region end point

These details will be provided to you by your cloud administrator. Here is a list of region end points that you can place in your configuration file (e.g. us-west-1)

 

Region name

Region

Endpoint

Location constraint

Protocol

US Standard *

us-east-1

You can use one of the following two endpoints:

  • s3.amazonaws.com (Northern Virginia or Pacific Northwest)
  • s3-external-1.amazonaws.com (Northern Virginia only)

(none required)

HTTP and HTTPS

US West (Oregon) region

us-west-2

s3-us-west-2.amazonaws.com

us-west-2

HTTP and HTTPS

US West (N. California) region

us-west-1

s3-us-west-1.amazonaws.com

us-west-1

HTTP and HTTPS

EU (Ireland) region

eu-west-1

s3-eu-west-1.amazonaws.com

EU or eu-west-1

HTTP and HTTPS

EU (Frankfurt) region

eu-central-1

s3.eu-central-1.amazonaws.com

eu-central-1

HTTP and HTTPS

Asia Pacific (Singapore) region

ap-southeast-1

s3-ap-southeast-1.amazonaws.com

ap-southeast-1

HTTP and HTTPS

Asia Pacific (Sydney) region

ap-southeast-2

s3-ap-southeast-2.amazonaws.com

ap-southeast-2

HTTP and HTTPS

Asia Pacific (Tokyo) region

ap-northeast-1

s3-ap-northeast-1.amazonaws.com

ap-northeast-1

HTTP and HTTPS

South America (Sao Paulo) region

sa-east-1

s3-sa-east-1.amazonaws.com

sa-east-1

HTTP and HTTPS

 

In order to avoid adding the secret key, access key and region endpoint to the <appSettings> part of your configuration file and to make this tool more organised I have created a configuration class for it. This configuration class will let you access the <configurationSection> element that is related to S3. To configure your app.config (or web.config) files you will have to add these <sectionGroup> and <section> elements to your configuration file:

 

<configSections>

<sectionGroup
name=AspGuy>

<section
name=S3Repository
type=Aref.S3.Lib.Strategies.S3FileRepositoryConfig, Aref.S3.LiballowLocation=true
allowDefinition=Everywhere />

</sectionGroup>

</configSections>

The
S3FileRepositoryConfig
class is inherited from
ConfigurationSection
class and has properties that map to some configuration elements of your .config file. A sample configuration for S3 is like this:

 

<AspGuy>

<S3Repository


S3.ReadFrom.AccessKey=xxxxxxxxxx


S3.ReadFrom.SecretKey=yyyyyyyyyyyyyyyyy


S3.ReadFrom.Root.BucketName=-bucket-name-


S3.ReadFrom.RegionName=ap-southeast-2


S3.ReadFrom.RootDir=“”>

</S3Repository>

</AspGuy>

Note that <AspGuy> comes from the name property of <sectionGroup name=”AspGuy”> element. Also <S3Repository> tag comes from the name of <section> element. Each property of S3FileRepositoryConfig
is mapped to an attribute of <S3Repository> element.

Apart from SecretKey, AccessKey andBucketName you can specify a root directory name as well. This setting is there so you can begin accessing the S3 bucket from a specific folder rather than from its root, and obviously this setting is optional. For example imagine there is a bucket with the given folder structure:

  • Dir1
  • Dir1/Dir1_1
  • Dir1/Dir1_2

If you set the RootDir property to “” then when you call the GetSubDir methods of the S3 file repository if will return “Dir1” because Dir1 is the only top-level folder in the bucket. If you set the RootDir property to “Dir1” and then call the GetSubDirs method you will get two entries which are “Dir1_1” and “Dir1_2”.

Here is the code of the configuration class mentioned above:

using System.Configuration;

 

namespace Aref.S3.Lib.Strategies

{


public
class
S3FileRepositoryConfig : ConfigurationSection

{


private
const
string S3ReadFromAccessKey = “S3.ReadFrom.AccessKey”;


private
const
string S3ReadFromSecretKey = “S3.ReadFrom.SecretKey”;


private
const
string S3ReadFromRootBucketName = “S3.ReadFrom.Root.BucketName”;


private
const
string S3ReadFromRegionName = “S3.ReadFrom.RegionName”;


private
const
string S3ReadFromRootDir = “S3.ReadFrom.RootDir”;

 

[ConfigurationProperty(S3ReadFromAccessKey, IsRequired = true)]


public
string AccessKey

{


get { return (string) this[S3ReadFromAccessKey]; }


set { this[S3ReadFromAccessKey] = value; }

}

 

[ConfigurationProperty(S3ReadFromSecretKey, IsRequired = true)]


public
string SecretKey

{


get { return (string) this[S3ReadFromSecretKey]; }


set { this[S3ReadFromSecretKey] = value; }

}

 

[ConfigurationProperty(S3ReadFromRootBucketName, IsRequired = true)]


public
string RootBucketName

{


get { return (string) this[S3ReadFromRootBucketName]; }


set { this[S3ReadFromRootBucketName] = value; }

}

 

[ConfigurationProperty(S3ReadFromRegionName, IsRequired = true)]


public
string RegionName

{


get { return (string) this[S3ReadFromRegionName]; }


set { this[S3ReadFromRegionName] = value; }

}

 

[ConfigurationProperty(S3ReadFromRootDir, IsRequired = true)]


public
string RootDir

{


get { return (string) this[S3ReadFromRootDir]; }


set { this[S3ReadFromRootDir] = value; }

}

}

}

 

For the repository class I have created an interface because of removing the dependency of clients (e.g. a web service that may need to use with various file storages) on S3. This will let you add your implementation of file system, FTP and other file storage types and use then through dependency injection. Here is the code of this interface:

namespace Aref.S3.Lib.Interfaces

{


public
interface
IFileRepository

{


void Download(string fileName, string targetPath);


void ChangeDir(string relativePath);


IEnumerable<string> GetFileNames(string pattern);


IEnumerable<string> GetSubdirNames(string startRelativeFolder = “”);


void AddFile(string localFilePath);


bool FileExists(string relativeFileName);


void DeleteFile(string relativeFileName);

}

}

 

In this interface:

  • Download: Downloads a file hosted on S3 to disk.
  • ChangeDir: Changes the current directory/folder to the given directory. If the new directory (relativePath parameter) starts with / then the path will be representing an absolute path (starting from the RootDir) otherwise it will be a relative path and will start from the current directory/folder.
  • GetFileNames: Retrieves the file names of the current folder
  • GetSubDirNames: Retrieves the name of folders in the current folder
  • AddFile: Uploads a file to S3
  • FileExists: Checks to see if a file is already on S3
  • DeleteFile: Deletes the file from S3

The implementation of these method are quiet simple using AWS SDK for .NET. The only tricky part is that S3 does not support folders. In fact in S3 everything is a key-value pair and the structure of entries is totally flat. What we do however is to use the forward slash character to represent folders and then we use this character as a delimiter to emulate a folder structure.

Here is the source code of S3 File Repository. You can clone the repository of this code which is on GitHub to play with the code. Feel free to send a pull request if you want to improve the code. The GitHub repository is located at https://github.com/aussiearef/S3FileRepository

 

 

 

 

 

 

Advertisements

10 thoughts on “Working with AWS S3 through C#

  1. Jim West says:

    I started this but found an easier approach my needs, just access the view via the IDataSource interface i.e.

    IDataSource mySource = (IDataSource)LinqDataSource1;
    LinqDataSourceView defaultView = mySource.GetView(“DefaultView”) as LinqDataSourceView;

  2. Darwin N says:

    Is there a problem using ObjectDataSource instead?

    All you have to do in ObjectDataSource is pass the TypeName and SelectMethod and would achive the same.

  3. Urbain says:

    I am too using ObjectDataSource instead. In the partial class file to extend the DataContext type you can even use the attributes in the System.ComponentModel namespace:

    namespace NorthwindManagement.Data
    {
    [DataObject()]
    public partial class NorthwindDataContext
    {
    [DataObjectMethod(DataObjectMethodType.Select, false)]
    public List SelectCustomers()
    {
    // Code goes here
    }
    }
    }

    The definition of the ObjectDataSource in the page markup would be:

    <asp:ObjectDataSource ID="dsCustomers" runat="server"
    TypeName="Northwind.Data.NorthwindDataContext"
    SelectMethod="SelectCustomers"

    Cheers

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s