How to Download an Entire S3 Bucket Using AWS CLI

Managing your Amazon S3 buckets often requires downloading a large swath of data, or perhaps even the entire contents. While the AWS Management Console lacks a feature to download entire buckets directly, you can achieve this easily using the AWS Command Line Interface (CLI).

Setting Up AWS CLI

First, you’ll need to make sure the AWS CLI is installed and configured on your system. The most straightforward installation method now is using pip, especially with Python 3.12, which is the latest version as of now. Execute the following command in your terminal:

pip install awscli

If you’re using a modern Python environment, pip is built-in, and this command should suffice. Once installed, you need to configure your AWS CLI with your credentials. Prompt the AWS CLI configuration using:

aws configure

You’ll need to provide your AWS Access Key ID, Secret Access Key, region, and output format preference.

Downloading the Entire S3 Bucket

Once your AWS CLI is set up, downloading the contents of an S3 bucket is straightforward. Use the following command:

aws s3 sync s3://<source_bucket> <local_destination>

Replace <source_bucket> with your bucket’s name and <local_destination> with the directory path where you want to store the files. For example, to download everything from a bucket named mybucket to your current directory, you would use:

aws s3 sync s3://mybucket .

This operation will perform a one-way sync of data from the S3 bucket to your local directory. It ensures your local directory has all the files from your S3 bucket, but it does not delete any files locally unless you add the --delete flag. This flag removes any existing files in your local directory not present in the bucket.

Note that this command will not alter your S3 bucket in any way. It merely pulls data from it.

Downloading Specific Folders Recursively

If you only need a specific folder from within a bucket, use the aws s3 cp command with the --recursive option:

aws s3 cp s3://BUCKETNAME/PATH/TO/FOLDER LocalFolderName --recursive

This command copies everything within the specified folder of your bucket to the LocalFolderName directory on your system recursively.

Further Reading

For more detailed options and examples, check out the AWS CLI Command Reference for S3.