Dataset for a Run

​

1. Download with Run commmand

The first method involves downloading your dataset directly using a run command. This method works well with tools such as wget or the kaggle API. Here is an example of how to use this approach:

​

1.1 Using YAML

run:
  - workdir: /root
    command: |-
      wget [your-wget-command]
      kaggle datasets download -d [dataset]

In the provided example, replace [your-wget-command] and [dataset] with your actual wget command and the kaggle dataset identifier respectively.

​

1.2 Using Web Console

In the VESSL Web Console, locate the Start commands section and simply type in the command.

​

2. Import a VESSL Dataset

VESSL Dataset supports multiple data sources. Register your dataset in VESSL Web Console and reference it when you create a run.

​

2.1 Using YAML

You can download data from a VESSL Dataset into your run container with import field.

import: 
  /input: vessl-dataset://{organizationName}/{datasetName}

​

2.2 Using Web Console

Select your dataset from the list of VESSL Dataset under Import > Dataset option.

​

3. Import a Object Storage

If you haven’t register your dataset as a VESSL Dataset, you can still import AWS S3 bucket or Google Cloud Storage.

​

3.1 Using YAML

Download the dataset from the cloud provider bucket into the run container with import field.

• Import from AWS S3

• Import from Google Cloud Storage

  import:
    /input: s3://{bucketName}/{prefix}

​

3.2 Using Web Console

Choose the cloud provider option and specify the Bucket Name and Prefix.

​

4. Import the Uploaded Local Files

You can manually upload local files via the Web Console as follows, and then refer to them in your run.

​

5. Mount a VESSL Dataset

Among the registered VESSL Dataset, the On-Premise Storage type can be directrly mount to your run container without donwloading a copy of it.

​

5.1 Using YAML

Directly mount and access the dataset in the run container with mount field.

mount:
  /input: vessl-dataset://{organizationName}/{datasetName}

​

5.2 Using Web Console

Select your dataset from the list of VESSL Dataset under Mount > Dataset option.

​

6. Mount an On-Premise Storage

Similar to importing an Object Storage, you can mount the On-Premise Storage type of data without registering it as a VESSL Dataset.

​

6.1 Using YAML

With mount field, you can mount hostpath or nfs dataset and directly access it in the run container.

• Hostpath

• NFS

  mount:
    /input: hostpath://{path}

​

6.2 Using Web Console

Choose the dataset mount option(hostpath or nfs) and specify the Server and Path.

​

What’s Next

For more detailed YAML reference and definitions, please visit: