S3Cmd Client

Owned by Gavin Biffar
Last updated: Jun 06, 2019
2 min read

S3Cmd is a common command line S3 client similar to the AWS CLI utility. More information on the S3Cmd client can be found here:

Installing S3Cmd

Linux clients can install the S3Cmd utility from common repositories. For example, CentOS hosts can install S3Cmd from the EPEL repository:

[user@localhost ~]$ sudo yum install s3cmd


Window clients can use the comparable S3Express utility or install S3Cmd, although, Python is needed to run S3Cmd:

  1. Download and install Python: https://www.python.org/downloads/windows/
  2. Download the S3Cmd source and extract the contents: https://github.com/s3tools/s3cmd/archive/master.zip 


Configuring S3Cmd

The S3Cmd utility pulls credentials, endpoint information, and other details from a configuration file. The file can be created using a wizard that is evoked when the utility is run with the '--configure' option. 

Linux clients will save the configuration file to ~/.s3cfg i.e. the user's home directory, and Windows clients will save the output of the wizard to %APPDATA%\s3cmd.ini. The location of the configuration file can be overridden with the '–config=<FILE>' option.

Linux

[user@localhost ~]$ s3cmd --configure

Windows

C:\s3cmd-master> python s3cmd --configure

For both Linux and Windows clients, the following values must be provided to allow the S3Cmd utility to connect to the USS S3 Gateway properly (the other values can be left as their default):

  • Access Key - <your username>
  • Secret Key - <your password>
  • S3 Endpoint - <your USS S3 Gateway endpoint e.g. myendpoint.s3.sdsc.edu>
  • DNS-style bucket+hostname: <this should match the value provided for the S3 Endpoint> 
  • Use HTTPS Protocol - Yes

Run S3Cmd

Examples and additional details for S3Cmd can be found here:

Create a new bucket titled 'backup':

[user@localhost ~]$ s3cmd mb s3://backup

Synchronize a local directory and it's subdirectories with the 'backup' bucket removing any files from the bucket that were deleted from the local directory:

[user@localhost ~]$ s3cmd sync /home/user s3://backup --delete-removed

Download all objects with the prefix 'user/' from the 'backup' bucket i.e. all the data synchronized to the bucket in the previous example:

[user@localhost ~]$ s3cmd get s3://backup/user/ /path/to/restore/directory/ -r

The previous examples show how the utility would have been run on a Linux client's BASH environment. Windows clients should have the same syntax. Note: Assuming the Python interpreter is in the Windows client's %PATH% environment variable, you can simply precede 's3cmd' with 'python' as seen in the Windows configuration example. If the Windows cmd shell's current working directory isn't within the extracted s3cmd folder, you must specify the full path to s3cmd.