Add Storage.create() method which takes a restartable input stream #201

Activity

1. product-auto-label

   added

   api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.

   on Mar 21, 2020
2. frankyn

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.

   and removed

   api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.

   on Mar 21, 2020
3. dmitry-fa commented on Mar 23, 2020

   

   dmitry-fa

   on Mar 23, 2020

   Last edited by dmitry-fa

   Contributor

   More actions

   @frankyn
   Will it be enough to implement upload based on WriterChannel?
4. dmitry-fa

   self-assigned this

   on Apr 29, 2020
5. dmitry-fa commented on May 6, 2020

   

   dmitry-fa

   on May 6, 2020

   Last edited by dmitry-fa

   Contributor

   More actions

   Hi @frankyn,

   Is this functionality really needed?
   Justification for that request (googleapis/google-cloud-java#2620) sounds like: so that it is easier for users to create blobs without loading the entire content into memory.
   But Storage.create(IntputStream) doesn't load the entire content into memory. I successfully uploaded 1.5Gb on the JVM with the heap of 50M (-Xmx50m). Upload is implemented in:

   com.google.api.client.http.AbstractInputStreamContent {
       public void writeTo(OutputStream out) throws IOException {
           IOUtils.copy(this.getInputStream(), out, this.closeInputStream);
           out.flush();
       }
   }
   

   IOUtils.copy() uses buffer of 4096 bytes to read from and write to the given streams, so no problem with memory.

   As you probably remember, #179 will introduce a number of upload() methods which read the content portion by portion and applies resumable approach.

   Even if upload with a restartable input stream is still needed for some reason, it could be implemented without introducing a new class representing a restartable input stream (as it implemented in googleapis/google-cloud-java#2682), no need to expose internal details to public. Upload details can be reflected in the method name:

     private class RestorableInputStream  {...}
     public Blob createBlobUsingRestorableInputStream(blobInfo, inputStream, options) {
        RestorableInputStream  restorable = new RestorableInputStream (inputStream);
      ...
     }
   

   I personally do not see any benefits from adding such method. Do you?
6. product-auto-label

   added

   api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.

   on Jul 10, 2020
7. frankyn

   closed this as completedon Oct 7, 2020