Working with S3 folders using the .Net AWS SDK

If you’ve been using S3 client in the AWS SDK for .Net you might have noticed that there are no meth­ods that let you inter­act with the fold­ers in a buck­et. As it turns out, S3 does not sup­port fold­ers in the con­ven­tion­al sense*, every­thing is still a key val­ue pair, but tools such as Cloud Berry or indeed the Ama­zon web con­sole sim­ply uses ‘/’ char­ac­ters in the key to indi­cate a fold­er struc­ture.

This might seem odd at first but when you think about it, there are no fold­er struc­ture on your hard dri­ve either, it’s a log­i­cal struc­ture the OS pro­vides for you to make it eas­i­er for us mere mor­tals to work with.

Back to the top­ic at hand, what this means is that:

  • if you add an object with key myfolder/ to S3, it’ll be seen as a fold­er
  • if you add an object with key myfolder/myfile.txt to S3, it’ll be seen as a file myfile.txt inside a myfold­er fold­er, if the fold­er object doesn’t exist already it’ll be added auto­mat­i­cal­ly
  • when you make a Lis­tO­b­jects call both myfolder/ and myfolder/myfile.txt will be includ­ed in the result

Creating folders

To cre­ate a fold­er, you just need to add an object which ends with ‘/’, like this:

public void CreateFolder(string bucket, string folder)
{
    var key = string.Format(@"{0}/", folder);
    var request = new PutObjectRequest().WithBucketName(bucket).WithKey(key);
    request.InputStream = new MemoryStream();
    _client.PutObject(request);
}

Here is a thread on the Ama­zon forum which cov­ers this tech­nique.

Listing contents of a folder

With the Lis­tO­b­jects method on the S3 client you can pro­vide a pre­fix require­ment, and to get the list of objects in a par­tic­u­lar fold­er sim­ply add the path of the fold­er (e.g. topfolder/middlefolder/) in the request:

var request = new ListObjectsRequest().WithBucketName(bucket).WithPrefix(folder);

If you are only inter­est­ed in the objects (includ­ing fold­ers) that are in the top lev­el of your folder/bucket then you’d need to do some fil­ter­ing on the S3 objects returned in the response, some­thing along the line of:

// get the objects at the TOP LEVEL, i.e. not inside any folders
var objects = response.S3Objects.Where(o => !o.Key.Contains(@"/"));

// get the folders at the TOP LEVEL only
var folders = response.S3Objects.Except(objects)
                      .Where(o => o.Key.Last() == '/' &&
                                  o.Key.IndexOf(@"/") == o.Key.LastIndexOf(@"/"));