SDSC Cloud Object Storage User Guide

Overview

SDSC Cloud Object Storage provides S3-compatible object storage across two availability zones:

Zone	Endpoint	Location

Zone	Endpoint	Location
AZ1 (Primary)	`https://az1.cloud.sdsc.edu`	San Diego
AZ2 (Secondary)	`https://az2.cloud.sdsc.edu`	Phoenix

Both zones are fully independent. Data stored in one zone is not automatically replicated to the other — replication must be explicitly enabled per bucket. This gives you full control over where your data lives and your storage costs.

Getting Started

Prerequisites

You will need:

An active account in the SDSC OpenStack environment
EC2 credentials (access key and secret key) — see below
An S3-compatible client (AWS CLI, s3cmd, boto3, Cyberduck, etc.)

Generating EC2 Credentials

EC2 credentials are the access key and secret key used to authenticate with the object storage system.

Via OpenStack CLI:

openstack ec2 credentials create

Save the access and secret values from the output — the secret is only shown once.

Via OpenStack Dashboard (Horizon):

Log in to the OpenStack dashboard
Navigate to Project → API Access
Click Download OpenStack RC File or go to Project → Compute → Access & Security → API Access
Select EC2 Credentials → Create EC2 Credentials

Configuring Your S3 Client

AWS CLI

Install AWS CLI and configure a profile:

aws configure --profile sdsc

Enter when prompted:

AWS Access Key ID:     <your-ec2-access-key>
AWS Secret Access Key: <your-ec2-secret-key>
Default region name:   sdsc-object
Default output format: json

Test your configuration against AZ1:

aws s3 ls --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Test against AZ2:

aws s3 ls --profile sdsc \
  --endpoint-url https://az2.cloud.sdsc.edu

To save the default endpoint-url for the profile, so you don’t have to always enter --endpoint-url https://az1.cloud.sdsc.edu or --endpoint-url https://az2.cloud.sdsc.edu for every command:

aws configure set endpoint_url https://az1.cloud.sdsc.edu --profile sdsc

or

aws configure set endpoint_url https://az2.cloud.sdsc.edu --profile sdsc

s3cmd

Configure s3cmd:

s3cmd --configure

Or create ~/.s3cfg manually:

[default]
access_key = <your-ec2-access-key>
secret_key = <your-ec2-secret-key>
host_base = az1.cloud.sdsc.edu
host_bucket = az1.cloud.sdsc.edu/%(bucket)s
use_https = True

Test:

s3cmd ls

boto3 (Python)

import boto3

s3 = boto3.client(
    's3',
    endpoint_url='https://az1.cloud.sdsc.edu',
    aws_access_key_id='<your-ec2-access-key>',
    aws_secret_access_key='<your-ec2-secret-key>',
    region_name='sdsc-object'
)

# List buckets
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

Working with Buckets

Creating a Bucket

Buckets are created in the zone you specify via the endpoint URL. A bucket created in AZ1 is accessible from AZ1's endpoint; a bucket created in AZ2 is accessible from AZ2's endpoint.

AWS CLI:

# Create in AZ1
aws s3 mb s3://my-bucket \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# Create in AZ2
aws s3 mb s3://my-bucket \
  --profile sdsc \
  --endpoint-url https://az2.cloud.sdsc.edu

boto3:

s3.create_bucket(Bucket='my-bucket')

Listing Buckets

aws s3 ls \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Deleting a Bucket

Buckets must be empty before deletion:

# Remove all objects first
aws s3 rm s3://my-bucket --recursive \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# Delete the bucket
aws s3 rb s3://my-bucket \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Working with Objects

Uploading Objects

# Upload a single file
aws s3 cp myfile.txt s3://my-bucket/ \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# Upload a directory recursively
aws s3 cp ./mydata/ s3://my-bucket/mydata/ \
  --recursive \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Downloading Objects

# Download a single file
aws s3 cp s3://my-bucket/myfile.txt ./myfile.txt \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# Download all objects in a bucket
aws s3 sync s3://my-bucket/ ./local-copy/ \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Listing Objects

aws s3 ls s3://my-bucket/ \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# List recursively with sizes
aws s3 ls s3://my-bucket/ \
  --recursive \
  --human-readable \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Deleting Objects

# Delete a single object
aws s3 rm s3://my-bucket/myfile.txt \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

# Delete all objects matching a prefix
aws s3 rm s3://my-bucket/mydata/ \
  --recursive \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

Bucket Replication

By default, data stored in one zone stays in that zone. You can enable cross-zone replication on a per-bucket basis using the S3 Replication API. Once enabled, objects written to a bucket in either zone will automatically be copied to the other zone.

When to Use Replication

Replication is recommended when you need:

High availability — data accessible even if one zone is unavailable (disk system unavailable, SDSC network outage will cause both zones to be unavailable)
Disaster recovery — protection against zone-level failures (disk system failure, as az2 still relies on SDSC network for authentication)

Replication is not needed for every bucket — it increases storage consumption across both zones and should be used intentionally.

Enabling Replication on a Bucket

Replication must be enabled after the bucket is created. The bucket must exist before you configure replication.

AWS CLI:

aws s3api put-bucket-replication \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --bucket my-bucket \
  --replication-configuration '{
    "Role": "",
    "Rules": [{
      "ID": "replicate-all",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "Destination": {
        "Bucket": "arn:aws:s3:::my-bucket"
      }
    }]
  }'

You can run this from either zone's endpoint. Replication will flow in both directions once enabled.

boto3:

replication_config = {
    'Role': '',
    'Rules': [{
        'ID': 'replicate-all',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Destination': {
            'Bucket': 'arn:aws:s3:::my-bucket'
        }
    }]
}

s3.put_bucket_replication(
    Bucket='my-bucket',
    ReplicationConfiguration=replication_config
)

Checking Replication Status

aws s3api get-bucket-replication \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --bucket my-bucket

Disabling Replication

aws s3api delete-bucket-replication \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --bucket my-bucket

Note: Disabling replication does not delete objects that have already been replicated to the other zone. Both copies remain until explicitly deleted.

Replication Behavior

Replication is asynchronous — objects typically appear in the second zone within 60 seconds
Objects written to either zone will replicate to the other (bidirectional)
Deletions are replicated — deleting an object in one zone will delete it in both
Objects uploaded before replication was enabled are not automatically replicated
Replication applies to the entire bucket — partial replication by prefix is not currently supported

Choosing Which Zone to Use

Both zones offer equivalent functionality. General guidance:

Use AZ1 (San Diego) as your primary zone for new buckets
Use AZ2 (Phoenix) If you need an offsite copy of data
Enable replication on critical buckets to protect against zone outages
For large datasets where replication cost is a concern, store in one zone and back up selectively

Access Control

By default, all buckets and objects are private — only the owner can access them.

Making an Object Public

aws s3api put-object-acl \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --bucket my-bucket \
  --key myfile.txt \
  --acl public-read

Make a Bucket Public

Create a text file named policy.json (or whatever you prefer) and add the following JSON. Make sure to replace my-bucket with your actual bucket name:

Public access to all objects in the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

Note: This policy explicitly grants read access (s3:GetObject) to anyone ("*") for every object inside that bucket (my-bucket/*).

Public access to all objects in the bucket and to list all objects in the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicListBucket",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"
    },
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

Note: This policy explicitly grants anyone ("*") permission to list the bucket's contents (s3:ListBucket) and read or download every individual object inside it (s3:GetObject).

Run the following AWS CLI command to attach the policy to your bucket. Make sure to replace my-bucket with your actual bucket name:

aws s3api put-bucket-policy \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --bucket my-bucket \
  --policy file://policy.json

The URL to access the objects in the bucket is:

https://az1.cloud.sdsc.edu/project-id:my-bucket/path/to/object

If you don’t know your project-id, you can run:

aws s3api list-buckets \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu

In the output, the project-id is the Owner.ID results, it will be listed with the user-id with a $ between them, so in the example below, da4962d3368042ac8337e2dfdd3e7bf3 is the project-id and da4962d3368042ac8337e2dfdd3e7bf3 is the user-id.

{
    "Buckets": [
        {
            "Name": "my-bucket",
            "CreationDate": "2026-06-30T17:24:50.223000+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "project-name",
        "ID": "da4962d3368042ac8337e2dfdd3e7bf3$da4962d3368042ac8337e2dfdd3e7bf3"
    },
    "Prefix": null
}

Generating a Presigned URL

Share temporary access to a private object without changing permissions:

aws s3 presign s3://my-bucket/myfile.txt \
  --profile sdsc \
  --endpoint-url https://az1.cloud.sdsc.edu \
  --expires-in 3600

This generates a URL valid for 1 hour (3600 seconds).

boto3:

url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'myfile.txt'},
    ExpiresIn=3600
)
print(url)

Common Issues

"Access Denied" when listing buckets Verify your EC2 credentials are correct and your AWS CLI region is set to sdsc-object.

Bucket not visible in other zone If you created a bucket in AZ1, it will appear in AZ2's bucket list (metadata is synced) but objects will not be there unless replication is enabled.

Objects not appearing after replication is enabled Replication only applies to objects written after replication was enabled. Previously uploaded objects will not be replicated automatically. Re-upload or copy them to trigger replication.

Replication delay Replication is asynchronous and typically completes within 60 seconds. During high load it may take longer. Check both endpoints if an object appears missing immediately after upload.

SSL certificate warnings The endpoints use valid SSL certificates. If you see certificate warnings, ensure your system's CA bundle is up to date.

Getting Help

Please email support@sdsc.edu with any problems accessing SDSC Cloud Storage

Ceph S3 object storage system guide

SDSC Cloud Object Storage User Guide

Overview

Getting Started

Prerequisites

Generating EC2 Credentials

Configuring Your S3 Client

AWS CLI

s3cmd

boto3 (Python)

Working with Buckets

Creating a Bucket

Listing Buckets

Deleting a Bucket

Working with Objects

Uploading Objects

Downloading Objects

Listing Objects

Deleting Objects

Bucket Replication

When to Use Replication

Enabling Replication on a Bucket

Checking Replication Status

Disabling Replication

Replication Behavior

Choosing Which Zone to Use

Access Control

Making an Object Public

Make a Bucket Public

Generating a Presigned URL

Common Issues

Getting Help