Image upload on Rackspace gets stuck in a loop

Hello,
I'm using the latest version of pkgcloud (version 2.1.1), and after trying to upload an image to Rackspace, it looks like the upload gets stuck in a loop, and keeps uploading the same data over and over. My function for file upload is down below:

uploadFile(filePath): Promise<string> {
        return new Promise((resolve, reject)=>{
            let client = pkgcloud.storage.createClient(myAuthentication);
            let readStream = fs.createReadStream(filePath);
            const total = fs.statSync(filePath).size;

            let upload = {
                container: myContainer
                remote: "test.png"
                size: total,
                contentType: 'image/png',
            };

            let writeStream = client.upload(upload);

            writeStream.on('error', function (err) {
                reject(new Error(err));
            });

            let uploaded = 0;
            let i = 0;
            writeStream.on('data', function (data) {
                let chunk = data.length;
                uploaded += chunk;
                console.log(`Uploading chunk ${i++} data ${uploaded / 1000000.0}/${total / 1000000.0} MB`);
            })

            writeStream.on('success', function (file) {
                resolve(file);
                console.log("File uploaded!");
            });

            console.log("Uploading file......." + filePath);
            readStream.pipe(writeStream);
        });
}

Based on the logs, I noticed that the upload size gets a lot bigger than the actual size of the image (ex. uploads 1.5+ GB , when we try to upload a 5 MB image). However this never throws an error, or successfully uploads the file, it just gets stuck in this loop.
I initially tried uploading more images at once from a directory, and this happened. However, when there was a hidden empty file present (which gets uploaded as well, since my code uploads all the files from a directory), then it worked properly.
After doing a lot of testing, I found out that this bug is not present in the version 1.7.0 or earlier. It starts occurring from version 2.0.0. And also this is the version in which you update the version of through2, the wrapper around Node.js streams.Transform.
To fix this issue temporarily I switched to the 1.7.0 version of pkgcloud, but I'm looking forward to hearing from you regarding this issue.
Thank you

Hello @emilijacc .. You just saved me from a lot of frustration.

I had the same problem uploading larger file > 500KB, it never returns an error or anything, it just keeps uploading and consumed more than > 400MB of my internet data in the endless loop.

Thanks a lot, switching to 1.7.0 as you mentioned seemed to work for files > 500KB and more.

I'm a huge fan of Rackspace Cloud (files) but I almost switched to aws s3 because of this bug.

I really hope the @pkgcloud team sorts this out, it's really frustrating.

We got the same issue here, but using Openstack's upload with the latest version of pkgcloud.

The upload blocks the node process, and ended with a 413 status (Over Limit) from openstack after a while.

// username, password, tenantName, region, authUrl, container, openStackId are set before

const client = pkgcloud.storage.createClient({
    provider: 'openstack',
    username: username,
    password: password,
    version: 'v2.0',
    keystoneAuthVersion: 'v3',
    region,
    domainName: 'default',
    projectDomainName: 'default',
    tenantName,
    authUrl
});

const writeStream = client.upload({
    container,
    remote: openStackId,
    headers: {
        'content-type': 'image/jpeg'
    }
});

writeStream.on('error', function (err) {
    log.error({
        err, openStackId
    }, "[filesService/uploadOpenStack] Erreur when store to openStack");
    return cb(err);
});

writeStream.on('success', function (file) {
    if (file.etag !== md5) {
        log.error({ openStackId }, "[filesService/uploadOpenStack] md5 check failed");

        client.removeFile(container, openStackId, function (err) {
            if (err) {
                log.error({ err }, "[filesService/uploadOpenStack] erreur on removeFile");
            }
        });

        return cb(new Error("[filesService/uploadOpenStack] md5 check failed"));
    }

    return cb(null);
});

2021 and had to downgrade to 1.7.0 to get the upload to work...

End of 2021 and had to downgrade to 1.7.0...

FYI. It is extremely rare, but when Rackspace is experiencing issues - and isn't just return 503 errors - it's possible even 1.7.0 will hang forever. If you are using this use a timeout after a while! You might not notice this on a small system, but with a lot of uploads it will happen randomly. Would want to cancel the write stream and such.

I am not clear on what the actual underlying condition is that triggers the success/error callbacks to just never fire. The network connection is disconnected though and error isn't called which is a huge issue and might just be the main problem. Essentially rackspace disconnects and that message never reaches the error handler.

I can't seem to downgrade to 1.7.0 anymore, doing so causes the cli to freeze up for a little while and then spit out a whole lot of errors. Seems I'm stuck with either a broken upload, or having to find an alternative

I can't seem to downgrade to 1.7.0 anymore, doing so causes the cli to freeze up for a little while and then spit out a whole lot of errors. Seems I'm stuck with either a broken upload, or having to find an alternative

So after a bit of trial and error, i figured out that just updating the through2 package from 3.0.1 to 4.0.2 fixes this issue. I've made an MR (#702), but for anyone stumbling on this with the same issue, my work-around for now is the following:

run npm i patch-package

go into node_modules -> pkgcloud -> node_modules -> through2 -> through2.js and copy the 4.0.2 code from the through2 github into this file, overwriting everything

run npx patch-package pkgcloude/through2 --exclude 'nothing' (the --exclude 'nothing' may not be necessary)

add "postinstall": "patch-package" to your scripts in your own package.json

Navigation Menu

Repository navigation

Image upload on Rackspace gets stuck in a loop #676

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Image upload on Rackspace gets stuck in a loop #676

Description

Activity

nperegrine commented on Oct 8, 2019

Getz85 commented on Jan 24, 2020

iomturner commented on Feb 11, 2021

RayNawara commented on Dec 17, 2021

sirisian commented on Mar 4, 2022

BeauTaapken commented on May 16, 2024

BeauTaapken commented on May 16, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions