| Table of Contents |
|---|
| Info |
|---|
Note: the The following installation and setup instructions are compatible with Max and Linux machines. To set up a Windows computer, please follow this guide, Swift Python Script: Windows Users. |
Install Python Swift Client
Make sure you have Python version 2.7 or higher (download python)
Install python-swiftclient using pip:
Code Block pip install python-keystoneclient python-swiftclient
You may have the run the command with sudo if you receive a permission denied error:
Code Block sudo pip install python-keystoneclient python-swiftclient
Setup Your Environment
Login to https://dashboard.cloud.sdsc.edu
Select the project you want to access (See: How to Change Current Working Project )
Download the OpenStack RC File (See: How To Download the OpenStack Project RC File )
Source the OpenStack RC File in your home directory by running the command below. Replace
project-openrc.shbelow with the name of the file downloaded in step 3.Code Block source project-openrc.sh
| Info |
|---|
You will need to repeat step 4 every time you exit from your command line prompt and log in again. |
| Info |
|---|
To switch between projects, repeat step 4 with the targeted project’s RC file. |
| Anchor | ||||
|---|---|---|---|---|
|
post [options] [container] [object]
Updates meta information for the account, container, or object depending on the args given. If the container is not found, it will be created automatically; but this is not true for accounts and objects. Containers also allow the -r (or -read-acl) and -w (or -write-acl) options. The -m or -meta option is allowed on all and used to define the user meta data items to set in the form Name:Value. This option can be repeated.
Example to Create a Container:
| Code Block | ||
|---|---|---|
| ||
swift post myContainer |
Example to Set Meta Information:
| Code Block | ||
|---|---|---|
| ||
swift post -m Color:Blue -m Size:Large |
list [options] [container]
Lists the containers for the account or the objects for a container. -p or -prefix is an option that will only list items beginning with that prefix. -d or -delimiter is option (for container listings only) that will roll up items with the given delimiter, or character that can act as a nested directory organizer.
Example to Show Container Names:
| Code Block | ||
|---|---|---|
| ||
swift list |
Example to Show Container Contents:
| Code Block | ||
|---|---|---|
| ||
swift list myContainer |
stat [container] [object]
Displays information for the account, container, or object depending on the entry.
Example:
| Code Block | ||
|---|---|---|
| ||
swift stat |
upload [options] container file_or_directory [file_or_directory]
Uploads to the given container the files and directories specified by the remaining args. -c or -changed is an option that will only upload files that have changed since the last upload.
Example to Upload Files to the Container (you can drag and drop your file into the command prompt to get its path):
| Code Block | ||
|---|---|---|
| ||
swift upload myContainer C:\Documents\myFolder |
Example to upload only files that have been changed (saves bandwidth):
| Code Block | ||
|---|---|---|
| ||
swift upload -c myContainer C:\Documents\myFolder |
Example to upload files in a directory recursively
| Code Block |
|---|
swift upload --skip-identical myContainer C:\Documents\myFolder |
download -all OR download container [object] [object]
Downloads everything in the account (with -all), or everything in a container, or a list of objects depending on the args given. For a single object download, you may use the -o [-output] (filename) option to redirect the output to a specific file or if "-" then just redirect to stdout.
Example to Download a Container:
| Code Block | ||
|---|---|---|
| ||
swift download myContainer |
delete-all OR delete container [object] [object]
Deletes everything in the account (with -all), or everything in a container, or a list of objects depending on the args given.Example to Delete a File:swift.py delete myContainer C:\Documents\myFile.txt
Example to Delete a Container:
| Code Block | ||
|---|---|---|
| ||
swift delete myContainer |
Options for Swift
-version show program’s version number and exit
-h, -help show this help message and exit
-v, -verbose Print more info
-q, -quiet Suppress status output
Full List of swift Options
https://docs.openstack.org/python-swiftclient/latest/cli/index.html#swift-usage
Large Files
Note: This section pertains only to uploading files that are individually larger than 5 GB.
Files over the size of 5 GB reach the Cloud by allowing the user to split the file into pieces ("segments") then the Cloud keeps track of how they fit together (a "manifest file" will map the segments together). Once the segments are uploaded, Swift manages the updates to the segments automatically. The following is an example of how to segment, upload and download large files to and from the Cloud, using Swift.
Segment & Upload
This will upload largefile.iso to test_container in 5 segments. In this example, the 2147483648 is the number of bytes in 2 GB. So that indicates that largefile.iso is 10 GB.
| Code Block | ||
|---|---|---|
| ||
input: swift upload -S 2147483648 test_container largefile.iso output: largefile.iso segment 1 largefile.iso segment 4 largefile.iso segment 3 largefile.iso segment 2 largefile.iso segment 0 largefile.iso |
Example:
swift upload -S <file size in bytes> <container name> <file name>
Now, to download the large file as a single object:
| Code Block | ||
|---|---|---|
| ||
input: swift download test_container largefile.iso output: largefile.iso You can list the segments, and download them separately if you want: swift list test_container_segments largefile.iso/1300906481.0/10485760/00000000 largefile.iso/1300906481.0/10485760/00000001 largefile.iso/1300906481.0/10485760/00000002 largefile.iso/1300906481.0/10485760/00000003 largefile.iso/1300906481.0/10485760/00000004 largefile.iso/1300906481.0/10485760/00000005 largefile.iso/1300906481.0/10485760/00000006 largefile.iso/1300906481.0/10485760/00000007 largefile.iso/1300906481.0/10485760/00000008 largefile.iso/1300906481.0/10485760/00000009 |
You may edit your file segments just like any other object within Cloud Files. Ensure your files are linked correctly by including your manifest file in your upload. You can change your file name by editing this manifest file as well.
Quotas
Most accounts do not have usage quotes set, however if you account does have a quota, you can view your usage and quota using the command line as shown below. The Bytes field shows your current usage and the Meta Quota-Bytes field shows your quota.
| Code Block |
|---|
input:
swift stat
output:
Account: AUTH_accountid
Containers: 2
Objects: 20
Bytes: 363730385
Containers in policy "policy-0": 2
Objects in policy "policy-0": 20
Bytes in policy "policy-0": 363730385
Meta Quota-Bytes: 400000000
X-Timestamp: 1518037769.10558
Content-Type: application/json; charset=utf-8
Accept-Ranges: bytes |
Swift will return an HTTP 413 error when you exceed your quota, see the example below:
| Code Block |
|---|
input: swift upload quota-test 0001 output: Object PUT failed: https://object.cloud.sdsc.edu/v1/AUTH_accountid/quota-test/0001 413 Request Entity Too Large b'Upload exceeds quota.' |
Please email services@sdsc.edu for a new quote if you need your quota expanded.
| Anchor | ||||
|---|---|---|---|---|
|
First make the container public:
| Code Block | ||
|---|---|---|
| ||
swift post -r ".r:*,.rlistings" myContainer |
Next enable web listings for the container:
| Code Block | ||
|---|---|---|
| ||
swift post -m "web-listings: true" myContainer |
Stat the container with the verbose flag to find the URL to the static website. The URL is list on the first line.
| Code Block | ||
|---|---|---|
| ||
input:
swift stat myContainer -v
output:
URL: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer
Auth Token: XXXX
Account: AUTH_accountid
Container: public
Objects: 1
Bytes: 1024
Read ACL: .r:*,.rlistings
Write ACL:
Sync To:
Sync Key:
Meta Web-Listings: true
Accept-Ranges: bytes
X-Trans-Id: XXXX
X-Storage-Policy: Policy-0
X-Timestamp: 1467900000.000000
Content-Type: text/plain; charset=utf-8 |
For more complex websites, see https://docs.openstack.org/swift/latest/middleware.html#staticweb
Unfortunately we cannot customize the URL of your static website or assign a domain name to the website.
| Anchor | ||||
|---|---|---|---|---|
|
Allows the creation of URLs to provide temporary access to objects.
Assign a secret temporary URL key to your project. If you are running Linux or macOS you can run "openssl rand -base64 32" to generate a 32 character random string.
| Code Block | ||
|---|---|---|
| ||
swift post -m "Temp-URL-Key:Long-Random-String" |
Get the StorageURL for your project.
| Code Block | ||
|---|---|---|
| ||
input:
swift stat -v | grep StorageURL
output:
StorageURL: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid |
Create the TempURL. Run "swift tempurl" for all of the possible options. In this example, GET is the HTTP method of access, 600 is how long in seconds the TempURL is valid, https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject is the StorageURL + Container name + Object path of the object to give TempURL access, and Long-Random-String is the Temp-URL-Key from the first step. The returned URL is the TempURL that can be distributed.
| Code Block | ||
|---|---|---|
| ||
input: swift tempurl GET 600 https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject Long-Random-String output: https://object.cloud.sdsc.edu:443/v1/AUTH_accountid/myContainer/myObject?temp_url_sig=29534544323cbaa2b0e6daf912ef5a15e5c8cda8&temp_url_expires=1550617412 |
Help
Please email support@sdsc.edu with any problems using the Swift Python script. Using --debug when running the command and providing the output to SDSC in the email will help.