Uploading Data¶
Making Directories¶
You can make directories within your project with the mkdir
command.
(pilot1-env) $ pilot mkdir tabular
Uploading Datasets to a Project¶
Prerequisites: Make sure your project is set! You can check this with the pilot project
command.
Upload datasets to your project with the upload
subcommand.
Given the file example.tsv:
Numbers Title
5 foo
6 foo
We can upload example.tsv
to our project with:
(pilot1-env) $ pilot upload example.tsv /
If you want to place your file inside a folder, such as after running pilot mkdir tabular
,
you can provide the relative path instead:
(pilot1-env) $ pilot upload example.tsv tabular
The above command will upload a file to the root directory of your project.
It will now be visible in the portal, and will show up when doing a pilot list
or pilot describe example.tsv
.
You may notice some fields are missing from the metadata. Pilot will attempt to
gather as much metadata as possible about the file you are uploading, but you can
supplement the data by providing a JSON document example_metadata.json
:
{
"data_type": "Metadata",
"dataframe_type": "List,
}
You can add a metadata JSON document with the -j
flag.
(pilot1-env) $ pilot upload -j my_metadata.json example.tsv /
You can find more info about what to include in my_metadata.json
in the Reference Guide.
Uploading Directories¶
In addition to files, you can also upload entire directories. In this mode, every file within the directory is treated as the same record. pilot describe and pilot download will behave differently to show/act on the files contained in the result instead of the result itself, but all command invocations remain the same.
(pilot1-env) $ pilot upload my_folder /
The above results in the my_folder record containing metadata on all files inside the local folder. All files in the folder are then uploaded to the destination.
Deleting Datasets¶
Deleting datasets removes both the file and the search record. Like the describe
command,
you will refer to the search record by its relative path within the project.
Delete the above example file example.tsv
with the following:
(pilot1-env) $ pilot delete tabular/example.tsv