Parallel composite uploads

One strategy for uploading large files is called parallel composite uploads. In such an upload, a file is divided into up to 32 chunks, the chunks are uploaded in parallel to temporary objects, the final object is recreated using the temporary objects, and the temporary objects are deleted.

Parallel composite uploads can be significantly faster if network and disk speed are not limiting factors; however, the final object stored in your bucket is a composite object, which only has a crc32c hash and not an MD5 hash. As a result, you must use crcmod to perform integrity checks when downloading the object with Python applications. You should only perform parallel composite uploads if the following apply:

• You don't need the uploaded objects to have an MD5 hash.

• Any Python user, including gsutil users, who needs to download your objects has either google-crc32c or crcmod installed.

  For example, if you use Python to upload video assets that are only served by a Java application, parallel composite uploads are a good choice because there are efficient CRC32C implementations available in Java.

Required roles

To get the permissions that you need to perform parallel composite uploads, ask your administrator to grant you the Storage Object User (roles/storage.objectUser) IAM role on the bucket.

You might also be able to get the required permissions through custom roles or other predefined roles.

For instructions on granting roles on buckets, see Set and manage IAM policies on buckets.

How tools and APIs use parallel composite uploads

Depending on how you interact with Cloud Storage, parallel composite uploads might be managed automatically on your behalf. This section describes parallel composite upload behavior for different tools and provides information for how you can modify the behavior.

TEMPORARY_PREFIX/RANDOM_VALUE_HEX_DIGEST_COMPONENT_ID

import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;
import java.io.IOException;
import java.nio.file.Path;
import java.util.List;

class AllowParallelCompositeUpload {

  public static void parallelCompositeUploadAllowed(String bucketName, List<Path> files)
      throws IOException {
    TransferManager transferManager =
        TransferManagerConfig.newBuilder()
            .setAllowParallelCompositeUpload(true)
            .build()
            .getService();
    ParallelUploadConfig parallelUploadConfig =
        ParallelUploadConfig.newBuilder().setBucketName(bucketName).build();
    List<UploadResult> results =
        transferManager.uploadFiles(files, parallelUploadConfig).getUploadResults();
    for (UploadResult result : results) {
      System.out.println(
          "Upload for "
              + result.getInput().getName()
              + " completed with status "
              + result.getStatus());
    }
  }
}

Troubleshooting

This section contains information on how to troubleshoot gcloud CLI errors that can occur with parallel composite uploads.

Can't check if the destination bucket is compatible with parallel composite uploads

Issue: When you upload a file that's 150 MiB or larger by using gcloud storage cp or gcloud storage mv, the command logs the following message once, but the upload completes successfully with exit code 0:

ERROR: Cannot check if the destination bucket is compatible for running
parallel composite uploads as the user does not permission to perform GET
operation on the bucket. The operation will be performed without parallel
composite upload feature and hence might perform relatively slower.

Solution: The gcloud CLI is attempting to run the parallel composite upload compatibility check. The principal doesn't have the storage.buckets.get permission on the destination bucket, so the request for the bucket's metadata is denied. The gcloud CLI handles the error internally and doesn't display an HTTP status code. Instead, it logs the message and falls back to a regular upload.

If you want parallel composite uploads, ensure that the principal has the storage.buckets.get, storage.objects.get, and storage.objects.delete permissions on the destination bucket. These permissions can be granted through the Storage Admin (roles/storage.admin) role. Alternatively, if the principal already has the storage.objects.get and storage.objects.delete permissions through a role like Storage Object User (roles/storage.objectUser), they can set the storage/parallel_composite_upload_compatibility_check property to False to disable the compatibility check.

403: does not have storage.objects.get access

Issue: Uploads of files smaller than 150 MiB succeed, but uploads of files that are 150 MiB or larger fail with exit code 1 and an error similar to the following:

HTTPError 403: PRINCIPAL does not have storage.objects.get access to
the Google Cloud Storage object.

Solution: storage/parallel_composite_upload_compatibility_check is set to False and storage/parallel_composite_upload_enabled is either set to None or True. In this configuration, the gcloud CLI skips the bucket compatibility check and proceeds directly with a parallel composite upload.

The temporary objects upload successfully because the principal has the storage.objects.create permission. However, composing the temporary objects into the final object requires the storage.objects.get permission, and deleting the temporary objects afterward requires the storage.objects.delete permission.

To resolve this issue, choose one of the following options:

• If you don't need parallel composite uploads, disable them:

  gcloud config set storage/parallel_composite_upload_enabled False
  

  All files are uploaded as regular uploads, and the principal needs only the permissions included in the Storage Object Creator (roles/storage.objectCreator) role. You can leave storage/parallel_composite_upload_compatibility_check at any value.

• If you need parallel composite uploads, grant the principal the storage.objects.get and storage.objects.delete permissions on the destination bucket. These permissions can be granted through the Storage Object User (roles/storage.objectUser) role.

If the failed upload left temporary objects in your bucket, delete them. By default, temporary objects use the prefix gcloud/tmp/parallel_composite_uploads/see_gcloud_storage_cp_help_for_details. To delete temporary objects that resulted from a failed upload, you need the storage.objects.delete permission.

Note that the gcloud storage rsync command behaves differently than gcloud storage cp and gcloud storage mv. When the destination is a bucket root, such as gs://my-bucket, gcloud storage rsync always requires the storage.buckets.get permission in order to sync to the bucket, regardless of your parallel composite upload settings. When the destination is a bucket subdirectory, such as gs://my-bucket/dir/, gcloud storage rsync doesn't require the storage.buckets.get permission, as long as the compatibility check is disabled in your parallel composite upload settings. In this scenario, gcloud storage rsync requires the storage.objects.get permission instead of storage.buckets.get.