To create a dataset from AWS S3 buckets, you will need to first launch a Workspace. Make sure to select a storage size that is at least as big as your dataset.
Once the Workspace is launched, set your AWS credentials:
Then, download the bucket into a local folder:
aws s3 sync s3://<s3-bucket-name>/ /onepanel/input/datasets/<your-dataset-name>
cd into that folder and create the Onepanel dataset:
onepanel datasets init
onepanel push -b
Note that by setting the
-b option on
onepanel push , your upload will run in the background. You will receive an email when the upload is complete.