Using Google Cloud Storage Inputs

The Imputation Service supports input files stored in Google Cloud Storage (GCS). If a pipeline accepts multiple input files and supports both local file upload and cloud-based input files, all inputs must come from the same location - either all local files or all cloud-based files. Please note that some pipelines only support cloud-based input files.
 

Getting Started with Google Cloud Storage

If you're new to Google Cloud Storage, you'll first need to create a Google Cloud project and upload your data.

Follow Google Cloud's Quickstart Guide to:

• Create a Google Cloud project
• Enable billing
• Create a Cloud Storage bucket
• Upload your input files

Note: Google Cloud Storage charges are separate from the Imputation Service. Google Cloud includes 5 GiB of Cloud Storage at no cost. See Google's Cloud Storage pricing documentation for current pricing details.

Once your files are uploaded, you'll need to grant the Imputation Service permission to access them.
 

Granting Access to Your Data

Before your job can begin, both the Imputation Service and your Terra account must be able to read your input files.

Grant Storage Legacy Object Reader (or another role that includes equivalent read permissions) on the bucket containing your files to the following:

• [e-mail]
  Allows the Imputation Service to read your input files during processing.
• Your Terra proxy group (formatted like PROXY_{user-specific-id}@firecloud.org)
  This can be found on your profile. It allows the service to verify that you have permission to access the data you're submitting, preventing unauthorized users from submitting jobs against data they cannot access.

Before starting your job, the service verifies that both it and your Terra proxy group have read access to each input file (or its parent bucket). If either check fails, the submission will not start.
 

Specifying Cloud Input Files

When submitting a job, provide each cloud input using its Google Cloud Storage URI, for example:

gs://my-bucket/path/to/sample.cram

All cloud inputs should be specified using the gs:// URI format.

Low Pass Imputation requires a manifest input file formatted to include the Google Cloud Storage URIs for each of your input CRAMs. You can learn more about how to format your manifest file here.

Helpful Tips

• The permissions described above only need to remain in place while your job is running. Because the Imputation Service reads your files directly from your bucket rather than copying them elsewhere, access must remain available until the job completes.
• After your job finishes, you can revoke the permissions if you no longer wish to share the data.
• To minimize Google Cloud network egress charges, consider storing your input files in a bucket located in us-central1, where the Imputation Service runs. Using buckets in other regions may incur data transfer fees as files are read during processing. Visit the GCS documentation to learn more about bucket regions.