[ BUG ] generated sitemaps are truncated #362

Activity

1. omar-dulaimi

   added

   needs investigation

   on Jun 5, 2021
2. omar-dulaimi commented on Jun 12, 2021

   

   omar-dulaimi

   on Jun 12, 2021

   Author

   More actions

   Any ideas @derduher
3. omar-dulaimi commented on Jun 12, 2021

   

   omar-dulaimi

   on Jun 12, 2021

   Author

   More actions

   I did this as a workaround for now, until I get a better solution:

   
   const data = await generateSiteMap();
   const logger = createWriteStream(resolve("/tmp/all-urls.json.txt"), {
     flags: "a",
   });
   data.forEach((el) => {
     logger.write(JSON.stringify(el));
     logger.write("\n");
   });
   logger.end();
   
   const stream = lineSeparatedURLsToSitemapOptions(
     createReadStream(resolve("/tmp/all-urls.json.txt"))
   )
     .pipe(sms)
     .pipe(createWriteStream(resolve("/tmp/sitemap-index.xml")));
   
   await new Promise((fulfill) => stream.on("finish", fulfill));
   await uploadToS3();
   await clearGeneratedSitemapsFromTmpDir();
   
4. huntharo commented on Sep 13, 2021

   

   huntharo

   on Sep 13, 2021

   Contributor

   More actions

   @omar-dulaimi - What i'm thinking is that when ending the stream, it doesn't end immediately, so files are uploaded partially. - Yes that is correct.

   Unfortunately most of the documentation examples are contrived and do not show how to wait for a file to be completely loaded or written as they just pipe the input to the output and don't have any code that waits for that to finish, so this is an area that needs some documentation improvements.

   Here is how to most easily wait for the local sitemap file to be written and closed before reading it and uploading it to S3:

   import { promisify } from 'util';
   import { finished, Readable } from 'stream';
   const finishedAsync = promisify(finished);
   
   // Break out the write stream creation so we can access it
   const sitemapFile = createWriteStream(resolve("/tmp/sitemap-index.xml"));
   // Your code
   sms.pipe(sitemapFile); // Your code pointing to new variable
   // Your code
   sms.end();
   
   // This is the magic bit - We don't want to wait just for the the SMS to emit finish... we want to wait until the file stream has received that and closed the file
   await finishedAsync(sitemapFile);
   
   // Now the file should have all the contents and sending it to S3 will result in a non-truncated file

   Note: if you pipe the sitemap through gzip you still need to await the file being finished not the gzip stream being finished.
5. derduher

   added

   documentation needed

   and removed

   needs investigation

   on Dec 31, 2021