How to Set Up Storage Providers for Notebooks

Notebooks are a web-based Jupyter IDE with shared persistent storage for long-term development and inter-notebook collaboration, backed by accelerated compute.

Storage providers let you connect various storage resources to Gradient. Once connected, you can use this storage to store and access data, such as models, artifacts, and datasets, to use in Gradient.

Gradient uses storage providers with versioned data to ensure that your data is verified and immutable. Gradient creates a folder with the same name as your Paperspace team ID within the storage provider. Gradient storage providers do not provide general S3 capabilities through the storage provider interface. However, if you define additional storage providers, you can use the tools compatible with your storage provider to interact with the data stored by Gradient.

Your Gradient account automatically comes with a storage provider named Gradient Managed. You can use this storage provider without additional configuration for storing data in Gradient’s hosted S3-compatible object storage.

Gradient Managed storage has a default persistent storage quota, based on your Gradient subscription level, which you can use for no additional charges. After the default quota is consumed, you may need to upgrade your subscription plan to have access to more storage. See your Gradient Subscription plan details for more information.

Gradient supports the following additional storage providers:

• DigitalOcean Spaces Object Storage
• AWS S3
• S3-compatible storage buckets

Add DigitalOcean Spaces Object Storage

To add DigitalOcean Spaces Object Storage as a storage provider, follow these steps:

1. In the Paperspace console, click the profile icon in the top right and then select Team Settings from the dropdown menu.

2. In the Storage tab, click Add a Storage Provider.

   

3. In the Create a new storage provider window, select the DigitalOcean Spaces option.

   For more information on how to create a Spaces bucket, see Create Spaces Bucket

4. Enter the following information:

   • Display Name: Name of the Spaces bucket
   • Endpoint: Endpoint URL of the bucket. To find the URL, go to the DigitalOcean Control Panel, click your bucket, and click the Settings tab. Copy the value shown in Origin Endpoint dropdown. For example, https://ps-spaces.nyc3.digitaloceanspaces.com.
   • Access Key ID and Secret Access Key: Access key & secret for the bucket. For more information on how to generate the access and secret key, see Sharing Access to Buckets with Access Keys.

   By default, CORS is automatically configured for any DigitalOcean Spaces integration for use by Gradient. If you want to modify your CORS config, navigate to the Settings tab of your Spaces bucket and make modifcations in the CORS Configurations section.

5. Click Create.

Once the Spaces bucket is added, you can use it for storing your datasets and other artifacts.

Add Other Storage Providers

To add AWS S3 or other S3-compatible storage, use the following steps:

1. Create a bucket within your storage provider with read/write credentials, such as an access key and secret key, for accessing the data.

   Note the bucket name, endpoint URL, and the access key and secret key.

2. Configure your storage bucket. You may need to configure CORS rules for your bucket.

3. In the Paperspace console, click the profile icon in the top right and then select Team Settings from the dropdown menu. In the Storage tab, click Add a Storage Provider. In the Create a new storage provider window, add the storage provider.