A while ago I needed to use AWS S3 (the Amazon’s cloud-based file storage) to store some files and then download them or get their listings through C#. As ironic as it sounds I noticed that there was no .NET implementation nor a documentation for S3 so I decided to create a file repository in C# which lets .NET developers access S3 programmatically.
Here is a rundown as to how you would work with AWS S3 through C#:
In order to use any APIs of Amazon Web Services (AWS) you will have to add the nugget package that is provided by Amazon. Simply bring up the Nuget Package Manager window and search for the keyword AWS. The first item in the search result is most likely AWS SDK for .NET which must be installed before you can access S3.
Once the SDK is installed we will have to find the properties of our S3 bucket and place it somewhere in web.config (or app.config) file. Normally these three properties of the S3 bucket is required in order to access it securely:
- Secret key
- Access key
- Region end point
These details will be provided to you by your cloud administrator. Here is a list of region end points that you can place in your configuration file (e.g. us-west-1)
|Region name||Region||Endpoint||Location constraint||Protocol|
|US Standard *||us-east-1||You can use one of the following two endpoints:
||(none required)||HTTP and HTTPS|
|US West (Oregon) region||us-west-2||s3-us-west-2.amazonaws.com||us-west-2||HTTP and HTTPS|
|US West (N. California) region||us-west-1||s3-us-west-1.amazonaws.com||us-west-1||HTTP and HTTPS|
|EU (Ireland) region||eu-west-1||s3-eu-west-1.amazonaws.com||EU or eu-west-1||HTTP and HTTPS|
|EU (Frankfurt) region||eu-central-1||s3.eu-central-1.amazonaws.com||eu-central-1||HTTP and HTTPS|
|Asia Pacific (Singapore) region||ap-southeast-1||s3-ap-southeast-1.amazonaws.com||ap-southeast-1||HTTP and HTTPS|
|Asia Pacific (Sydney) region||ap-southeast-2||s3-ap-southeast-2.amazonaws.com||ap-southeast-2||HTTP and HTTPS|
|Asia Pacific (Tokyo) region||ap-northeast-1||s3-ap-northeast-1.amazonaws.com||ap-northeast-1||HTTP and HTTPS|
|South America (Sao Paulo) region||sa-east-1||s3-sa-east-1.amazonaws.com||sa-east-1||HTTP and HTTPS|
In order to avoid adding the secret key, access key and region endpoint to the <appSettings> part of your configuration file and to make this tool more organised I have created a configuration class for it. This configuration class will let you access the <configurationSection> element that is related to S3. To configure your app.config (or web.config) files you will have to add these <sectionGroup> and <section> elements to your configuration file:
type=“Aref.S3.Lib.Strategies.S3FileRepositoryConfig, Aref.S3.Lib” allowLocation=“true“
S3FileRepositoryConfig class is inherited from ConfigurationSection class and has properties that map to some configuration elements of your .config file. A sample configuration for S3 is like this:
Note that <AspGuy> comes from the name property of <sectionGroup name=”AspGuy”> element. Also <S3Repository> tag comes from the name of <section> element. Each property of S3FileRepositoryConfig
is mapped to an attribute of <S3Repository> element.
Apart from SecretKey, AccessKey andBucketName you can specify a root directory name as well. This setting is there so you can begin accessing the S3 bucket from a specific folder rather than from its root, and obviously this setting is optional. For example imagine there is a bucket with the given folder structure:
If you set the RootDir property to “” then when you call the GetSubDir methods of the S3 file repository if will return “Dir1” because Dir1 is the only top-level folder in the bucket. If you set the RootDir property to “Dir1” and then call the GetSubDirs method you will get two entries which are “Dir1_1” and “Dir1_2”.
Here is the code of the configuration class mentioned above:
For the repository class I have created an interface because of removing the dependency of clients (e.g. a web service that may need to use with various file storages) on S3. This will let you add your implementation of file system, FTP and other file storage types and use then through dependency injection. Here is the code of this interface:
In this interface:
- Download: Downloads a file hosted on S3 to disk.
- ChangeDir: Changes the current directory/folder to the given directory. If the new directory (relativePath parameter) starts with / then the path will be representing an absolute path (starting from the RootDir) otherwise it will be a relative path and will start from the current directory/folder.
- GetFileNames: Retrieves the file names of the current folder
- GetSubDirNames: Retrieves the name of folders in the current folder
- AddFile: Uploads a file to S3
- FileExists: Checks to see if a file is already on S3
- DeleteFile: Deletes the file from S3
The implementation of these method are quiet simple using AWS SDK for .NET. The only tricky part is that S3 does not support folders. In fact in S3 everything is a key-value pair and the structure of entries is totally flat. What we do however is to use the forward slash character to represent folders and then we use this character as a delimiter to emulate a folder structure.
You can clone the repository of this code which is on GitHub to play with the code. Feel free to send a pull request if you want to improve the code. The GitHub repository is located at https://github.com/aussiearef/S3FileRepository