Swift Python Script

The following installation and setup instructions are compatible with Max and Linux machines. To set up a Windows computer, please follow this guide, https://sdsc-ucsd.atlassian.net/wiki/spaces/SC/pages/1516961856.

The Command Line Examples, listed below, will work for any computer or machine after successfully installing and setting up Python Swift Client.

Install Python Swift Client

  1. Make sure you have Python version 2.7 or higher (download python)

  2. Install python-swiftclient using pip:

    1 pip install python-keystoneclient python-swiftclient
  • You may have the run the command with sudo if you receive a permission denied error:

    1 sudo pip install python-keystoneclient python-swiftclient

Setup Your Environment

  1. Login to https://dashboard.cloud.sdsc.edu

  2. Select the project you want to access (See: https://sdsc-ucsd.atlassian.net/wiki/spaces/SC/pages/110034978 )

  3. Download the OpenStack RC File (See: https://sdsc-ucsd.atlassian.net/wiki/spaces/SC/pages/110034947 )

  4. Source the OpenStack RC File in your home directory by running the command below. Replace project-openrc.sh below with the name of the file downloaded in step 3.

    1 source project-openrc.sh

You will need to repeat step 4 every time you exit from your command line prompt and log in again.

To switch between projects, repeat step 4 with the targeted project’s RC file.

Command Line Examples

post [options] [container] [object]

Updates meta information for the account, container, or object depending on the args given. If the container is not found, it will be created automatically; but this is not true for accounts and objects. Containers also allow the -r (or -read-acl) and -w (or -write-acl) options. The -m or -meta option is allowed on all and used to define the user meta data items to set in the form Name:Value. This option can be repeated.

Example to Create a Container:

1 swift post myContainer

Example to Set Meta Information:

1 swift post -m Color:Blue -m Size:Large

list [options] [container]

Lists the containers for the account or the objects for a container. -p or -prefix is an option that will only list items beginning with that prefix. -d or -delimiter is option (for container listings only) that will roll up items with the given delimiter, or character that can act as a nested directory organizer.

Example to Show Container Names:

1 swift list

Example to Show Container Contents:

1 swift list myContainer

stat [container] [object]

Displays information for the account, container, or object depending on the entry.

Example:

1 swift stat

upload [options] container file_or_directory [file_or_directory]

Uploads to the given container the files and directories specified by the remaining args. -c or -changed is an option that will only upload files that have changed since the last upload.

Example to Upload Files to the Container (you can drag and drop your file into the command prompt to get its path):

1 swift upload myContainer C:\Documents\myFolder

Example to upload only files that have been changed (saves bandwidth):

1 swift upload -c myContainer C:\Documents\myFolder

Example to upload files in a directory recursively

1 swift upload --skip-identical myContainer C:\Documents\myFolder

download -all OR download container [object] [object]

Downloads everything in the account (with -all), or everything in a container, or a list of objects depending on the args given. For a single object download, you may use the -o [-output] (filename) option to redirect the output to a specific file or if "-" then just redirect to stdout.

Example to Download a Container:

1 swift download myContainer

delete-all OR delete container [object] [object]

Deletes everything in the account (with -all), or everything in a container, or a list of objects depending on the args given.Example to Delete a File:swift.py delete myContainer C:\Documents\myFile.txt

Example to Delete a Container:

1 swift delete myContainer

Options for Swift

-version show program’s version number and exit

-h, -help show this help message and exit

-v, -verbose Print more info

-q, -quiet Suppress status output

Full List of swift Options

https://docs.openstack.org/python-swiftclient/latest/cli/index.html#swift-usage

Large Files

Note: This section pertains only to uploading files that are individually larger than 5 GB.

Files over the size of 5 GB reach the Cloud by allowing the user to split the file into pieces ("segments") then the Cloud keeps track of how they fit together (a "manifest file" will map the segments together). Once the segments are uploaded, Swift manages the updates to the segments automatically. The following is an example of how to segment, upload and download large files to and from the Cloud, using Swift.

Segment & Upload

This will upload largefile.iso to test_container in 5 segments. In this example, the 2147483648 is the number of bytes in 2 GB. So that indicates that largefile.iso is 10 GB.

1 2 3 4 5 6 7 8 9 10 input: swift upload -S 2147483648 test_container largefile.iso output: largefile.iso segment 1 largefile.iso segment 4 largefile.iso segment 3 largefile.iso segment 2 largefile.iso segment 0 largefile.iso

Example: 
swift upload -S <file size in bytes> <container name> <file name>

Now, to download the large file as a single object:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 input: swift download test_container largefile.iso output: largefile.iso You can list the segments, and download them separately if you want: swift list test_container_segments largefile.iso/1300906481.0/10485760/00000000 largefile.iso/1300906481.0/10485760/00000001 largefile.iso/1300906481.0/10485760/00000002 largefile.iso/1300906481.0/10485760/00000003 largefile.iso/1300906481.0/10485760/00000004 largefile.iso/1300906481.0/10485760/00000005 largefile.iso/1300906481.0/10485760/00000006 largefile.iso/1300906481.0/10485760/00000007 largefile.iso/1300906481.0/10485760/00000008 largefile.iso/1300906481.0/10485760/00000009

You may edit your file segments just like any other object within Cloud Files. Ensure your files are linked correctly by including your manifest file in your upload. You can change your file name by editing this manifest file as well.

Quotas

Most accounts do not have usage quotes set, however if you account does have a quota, you can view your usage and quota using the command line as shown below. The Bytes field shows your current usage and the Meta Quota-Bytes field shows your quota. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 input: swift stat output: Account: AUTH_accountid Containers: 2 Objects: 20 Bytes: 363730385 Containers in policy "policy-0": 2 Objects in policy "policy-0": 20 Bytes in policy "policy-0": 363730385 Meta Quota-Bytes: 400000000 X-Timestamp: 1518037769.10558 Content-Type: application/json; charset=utf-8 Accept-Ranges: bytes

Swift will return an HTTP 413 error when you exceed your quota, see the example below:

1 2 3 4 5 input: swift upload quota-test 0001 output: Object PUT failed: https://object.cloud.sdsc.edu/v1/AUTH_accountid/quota-test/0001 413 Request Entity Too Large b'Upload exceeds quota.'

Please email services@sdsc.edu for a new quote if you need your quota expanded.

Static Websites

First make the container public:

1 swift post -r ".r:*,.rlistings" myContainer

Next enable web listings for the container:

1 swift post -m "web-listings: true" myContainer

Stat the container with the verbose flag to find the URL to the static website. The URL is list on the first line.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 input: swift stat myContainer -v   output: URL: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer Auth Token: XXXX Account: AUTH_accountid Container: public Objects: 1 Bytes: 1024 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: Meta Web-Listings: true Accept-Ranges: bytes X-Trans-Id: XXXX X-Storage-Policy: Policy-0 X-Timestamp: 1467900000.000000 Content-Type: text/plain; charset=utf-8

For more complex websites, see https://docs.openstack.org/swift/latest/middleware.html#staticweb

Unfortunately we cannot customize the URL of your static website or assign a domain name to the website.

TempURL

Allows the creation of URLs to provide temporary access to objects.

Assign a secret temporary URL key to your project.  If you are running Linux or macOS you can run "openssl rand -base64 32" to generate a 32 character random string. 

1 swift post -m "Temp-URL-Key:Long-Random-String"

Get the StorageURL for your project.

1 2 3 4 5 input: swift stat -v | grep StorageURL output: StorageURL: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid

Create the TempURL. Run "swift tempurl" for all of the possible options. In this example, GET is the HTTP method of access, 600 is how long in seconds the TempURL is valid, https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject is the StorageURL + Container name + Object path of the object to give TempURL access, and Long-Random-String is the Temp-URL-Key from the first step.  The returned URL is the TempURL that can be distributed.  

1 2 3 4 5 input: swift tempurl GET 600 https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject Long-Random-String output: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject?temp_url_sig=29534544323cbaa2b0e6daf912ef5a15e5c8cda8&temp_url_expires=1550617412

Help

Please email support@sdsc.edu with any problems using the Swift Python script. Using --debug when running the command and providing the output to SDSC in the email will help.