To create a dataset from Google Cloud Storage (GCS) buckets, you will need to first launch a Workspace. Make sure to select a storage size that is at least as big as your dataset.
Once the Workspace is launched, install
curl https://sdk.cloud.google.com | bash
Once installation is complete, restart shell:
exec -l $SHELL
Then, you need to either login with your personal account:
gcloud auth login
or you can alternatively login using a service account.
Once logged in, you can download the bucket into a local folder:
gsutil cp gs://<gs-bucket-name>/ /onepanel/input/datasets/<your-dataset-name>
cd into that folder and create the Onepanel dataset as follows:
onepanel datasets init
onepanel datasets push -b
Note that by setting the
-b option on
onepanel datasets push , your upload will run in the background. You will receive an email when the upload is complete.