Datasets and Dataset Collections

nova-galaxy provides abstractions for handling individual files (Dataset) and collections of files (DatasetCollection) within Galaxy.

dataset = Dataset("tests/test_files/test_text_file.txt")
dataset.upload(store)

my_collection = DatasetCollection("path/to/my/collection")

By default Datasets will take their name from the filepath given, but they can be given unique names by passing a string into the constructor in the name parameter.

Datasets can be marked as a remote file if you don’t want to upload them from your local machine. Remote files are files that your upstream Galaxy instance will have access to. For example, if your upstream Galaxy instance has access to a directory named /SNS, you can load a file from there as a dataset:

data = Dataset(path=REMOTE_FILE_PATH, remote_file=True)

Datasets can be uploaded to a store by calling the upload method.

with nova_instance.connect() as connection:
    store = connection.get_data_store(name="nova_galaxy_testing")
    store.mark_for_cleanup()
    test_tool = Tool(TEST_TOOL_ID)
    test_data = Dataset(path="tests/test_files/test_text_file.txt", force_upload=False)
    test_data.upload(store=store)

Note, when the remote_files flag is set to true, the files are not actually “uploaded”. Instead, they will be ingested into Galaxy as a link to the actual file, so file size should not slow down the system.

When running tools, any Dataset that is used as an input parameter will be automatically uploaded/ingested, unless that dataset has already been uploaded. In order to force the dataset to be uploaded when a tool runs, even if it has been uploaded before, the dataset can be marked with force_upload by passing in a boolean value to that parameter in the constructor.

By default force_upload is actually True.

If instead of loading a file from disk or ingesting a file, you want to directly upload some text or some other serializable python value, you can set the dataset content directly:

data.set_content(content="this is some content, that I'm setting", file_type=".txt")

The file_type argument is optional and will default to a text file.

In order to fetch the content of a dataset you can either download the dataset to a path using download() or fetch the content and store it directly in memory using get_content() (be careful using this with large files.)

DatasetCollections currently have less functionality than individual Datasets, as most collections will come from tool outputs. The get_content() method will return a list of info on each element in the collection rather than the content of each element. The download() method will save the collection (with all content included) as a zip archive to the given path.