main.dart.js caching problem for web still is not solved

After releasing a new version and uploading it to the server, clients who already have their main.dart.js cached receive the update with a huge delay.

Simplest solution is already mentioned in #57964 - just add a hash to main.dart.js like main.dart.buildDir_hash_for_example.js

The "solution" of the original issue doesn't fix stated problem.

@sachemmark

@sachemmark without any details in your issue it is impossible to resolve. For starters, it is not possible to tell whether this is due to the service worker behavior or the hosting implementation of caching. Can you provide evidence that your proposed solution will solve the problem (i.e. have you tried this yourself?) If so I have no issues implementing it

Running this script from directory build/web

uuid=$(uuidgen) && sed -i "s/main.dart.js/$uuid.main.dart.js/g" index.html && mv main.dart.js "$uuid.main.dart.js"

this works for me. It would be great if this could be done automatically by builder

this works for me. It would be great if this could be done automatically by builder

But I don't know what "works" means. Do you mean this resolve the issue on hosting? or locally. If on hosting, could you include network/application tab logs that show the new version not activating when it should?

The same issue, had to write the same script that was adding build hash prefix to the main.dart.js to update changes.
Similar issue (#57964) was closed even though it wasn't fixed. Check the comment section please, a lot of people are facing this too.

i.e. this could just be bypassing the service worker cache, which is sort of defeating the purpose

It resolves issue on hosting. We are using Nginx and CloudFlare.

@MarcinusX

@MarcinusX many people have reported the bug with no details to show for it. I'm eager to fix it but I do not want to make large sweeping changes without more evidence (I have never repo'd a problem locally)

@jonahwilliams

@jonahwilliams I am not sure that it is possible to provide you such evidence. I don't have CloudFlare's logs obviously.
All we could see is that new version updates only by TTL or after adding build hash prefix to the main.dart.js.
I guess this caching mechanism is common in several systems and changing file names enforces them to update caches.
And that may be one of the reasons why other front-end frameworks such as Vue.js and React.js include build hash in *.js files.
So it is not a bug you could reproduce locally, it's more about specifics of hosting platforms.

The way the service worker functions, after a new version is uploaded you should expect some delay (with the offline first strategy). First the browser must download the new service worker and activate. Then when the user fully leaves the site (not just a refresh). the browser will activate the new service worker, and subsequent visits will use the new version.

What the version hash could do, is punch though a CDN/aggressive cache so that the new service worker is downloaded before the previously cached artifact expires. This is fundamentally a workaround for the specific cache hosting behavior you have configured, and IMO does not belong in the template as a default - otherwise we would by default be bypassing caches which is sort of contrary the goal, no?

@jonahwilliams

@jonahwilliams Sure. I think there should be a cli option for this. It could be much useful for so hot fixes wow (%
Our dev team is inspired to add this option and If you're ok with it we could make a PR in a week or so.

@jonahwilliams

@jonahwilliams

I believe a lot of caching solutions (Cloudflare, nginx with caching enabled, local browser cache) use filenames as cache keys. Randomizing the filename will resolve the issues with the aforementioned caches (e.g. when a web client sees a new release a few hours after it's published, depends on a browser).

I've come to work with Flutter after some other JS frameworks, and every one of them (it may be some default builder feature though) adds a random hex string to js script filenames. It was quite a disappointment not to see this feature in Flutter Web.

Do you have the time to try out #64976 ?

I'm not sure if this is sufficient to bypass a CDN

I noticed that you change version number in #64976. I've already tried changing version number before and it didn't work. It seemed to be helpful for somebody on stackoverflow though not for anyone. But I will try this out and come back in a few minutes with results

Waiting for a new version with this merge to be launched 🤔

Adding a query param to main.dart.js solves the caching problem for the main file but the assets will still not update until the new service worker is activated.

Using the sample code below I get the following result after deploying a new build on Firebase hosting.

This may cause unexpected behavior and users will still see old assets.

Sample code

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key key, this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Future<String> assetVersion;

  @override
  void initState() {
    super.initState();

    assetVersion =
        DefaultAssetBundle.of(context).loadString('assets/asset.txt');
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'Main.dart version 1.0',
            ),
            FutureBuilder<String>(
              initialData: 'Loading asset version',
              future: assetVersion,
              builder: (context, snapshot) =>
                  Text("Asset version ${snapshot.data}"),
            ),
          ],
        ),
      ),
    );
  }
}

@jonahwilliams

@jonahwilliams how do we force the cache to be busted? Do I need to make any changes to the index.js such as fingerprinting main.dart.js in the <script src=" tag or should things work out of the box with #64976

Edit
I can confirm that if I use the index.html file out of the box, there is no cache busting when I deploy to netlify. I have to do a hard refresh. I have some logic in my build process that fingerprints main.dart.js. When I comment that out, the cache doesn't get cleared. I'm also not sure how to modify service_worker.js because they reference main.dart.js so some functionality may be breaking if I rename main.dart.js

When I build my index.html file, I currently get the following generated:

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <meta content="IE=Edge" http-equiv="X-UA-Compatible">
  <meta name="description" content="A new Flutter project.">

  <!-- iOS meta tags & icons -->
  <meta name="apple-mobile-web-app-capable" content="yes">
  <meta name="apple-mobile-web-app-status-bar-style" content="black">
  <meta name="apple-mobile-web-app-title" content="chat">

  <meta name="google-signin-client_id" content="...">

  <link rel="apple-touch-icon" href="icons/Icon-192.png">

  <!-- Favicon -->
  <link rel="shortcut icon" type="image/png" href="favicon.ico"/>

  <title>chat</title>
  <link rel="manifest" href="manifest.json">
</head>
<body>
  <!-- This script installs service_worker.js to provide PWA functionality to
       application. For more information, see:
       https://developers.google.com/web/fundamentals/primers/service-workers -->
  <script>
    if ('serviceWorker' in navigator) {
      window.addEventListener('load', function () {
        navigator.serviceWorker.register('flutter_service_worker.js');
      });
    }
  </script>
  <script src="main.dart.js" type="application/javascript"></script>
</body>
</html>

And my flutter doctor output:

[✓] Flutter (Channel unknown, 1.23.0-7.0.pre, on Mac OS X 10.14.6 18G6020 x86_64, locale en-US)
 
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 11.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 3.4)
[✓] IntelliJ IDEA Community Edition (version 2020.1.4)
[✓] VS Code (version 1.49.3)
[✓] Connected device (3 available)

• No issues found!

I would like to force the cache to be cleared whenever the contents of main.dart.js is built.

@venkatd

@venkatd
main.dart.js version hash was removed in #66069. It seems you need a page load to update the service worker and another one to get the latest main.dart.js file. This is because main.dart.js is loaded using the script tag before the new service worker has time to activate.

@jonahwilliams
Shouldn't main.dart.js still have the 'randomHash' param? Or maybe it should be fetched by the service worker using 'onlineFirst' like '/'?

No, because the randomHash unfortunately broke in combination with deferred imports, since we cannot insert hashes into those requests (as they are generated by the dart compiler).

In terms of # of refreshes to update, I am working on (though I don't know when it will be finished) a package to make it easy to show a "new version available" message: flutter/packages#206

@azbuky

@azbuky @jonahwilliams

My experience has been so far:

I deploy a new version to production via CI
I have to clear the cache in order to view the updated version

I haven't yet checked if a double refresh fixes this, but it's important for our use case that a single refresh yields the new version.

We have an internal Flutter web netlify build plugin and I've been adding my own fingerprinting as a workaround. I rename main.dart.js and update references in the service worker. I'm sure there are downsides which I'm not considering.

Source code here:
https://gist.github.com/venkatd/8f5bb02365bcbf01a19f19a661cd6e8f#file-flutter-netlify-js-L72-L91

Any feedback on this workaround?

We have an internal Flutter web netlify build plugin and I've been adding my own fingerprinting as a workaround. I rename main.dart.js and update references in the service worker. I'm sure there are downsides which I'm not considering.

If you use (or depend on libraries) that use deferred imports this will break. You will not need to clear the cache on the latest dev, just a page reresh.

I was using 1.23.0-7.0.pre and, for whatever reason, a page refresh didn't result in the the new version getting served. Is there any additional info that could help you understand the issue? (I'm heading out of work now but will be revisiting this tomorrow.)

My steps were:

Deploy a new version using netlify
Refresh the page
Old version is still there
Clear the Chrome browser cache
New version is there

I'll double check this later to confirm this is the behavior I'm experiencing.

If you use (or depend on libraries) that use deferred imports this will break. You will not need to clear the cache on the latest dev, just a page reresh.

The problem is it is not enough a page refresh, but two are needed. I mean you have to load (or refresh) the page after the update and do one more refresh after that to get the latest version. One for the service worker to activate and one to load the new main.dart.js file. So the user does not load the updated version the first time they navigate to the web page.

Moreover when you load the page the first time, main.dart.js will load assets from the old version at first but after the new service worker is activated, it will load asset files from the new version.

Would loading main.dart.js after the service worker activates, work? There would be a small load delay when the browser loads the new update but it would work on first load.

Is there further reading on why two refreshes are needed or how caching is currently handled via the service worker? The high level issue makes sense but I'm not following exactly why two refreshes are needed.

I'll play around with it tonight. Ideally we want to keep the load as fast as possible since our build is already pretty large.

@jonahwilliams

@jonahwilliams I've been looking at reducing data transfer for our flutter web app. I noticed that the install is caching the main.dart.js with ?revision= and as ? v=<randomHash> in PR fix. This causes the largest asset to be downloaded twice.

I very well could be missing something but couldn't the random hash be applied to the RESOURCES in the service worker too? Or maybe a map to make the relationship that it's the same file and doesn't need to be loaded twice?

@azbuky

Adding a query param to main.dart.js solves the caching problem for the main file but the assets will still not update until the new service worker is activated.

Using the sample code below I get the following result after deploying a new build on Firebase hosting.

This may cause unexpected behavior and users will still see old assets.

Sample code

Did you find a solution to this @azbuky ? I have appended a buildnumber to main.dart.js with a custom script in my build pipeline so I always get the latest version. This is fine but I'm loading localized texts from a file called "en.json" using rootBundle.loadString('assets/locale/en.json) and this file is cached in the browser. I cannot append a hash to this path, at least not with url syntax like this: rootBundle.loadString('assets/locale/en.json?this_does=not_work)

Any ideas on how I can cache bust this file? I'm fine with my "logo.png" to be cached but I have some assets that will change every release and I need to cache bust these.

@mikeesouth

Hi @mikeesouth,

I am loading main.dart.js in index.html after the service worker is installed and activated using the following script.
It's not perfect but works for my use case.

Sample code

  <script>
    function loadMain() {
      var script = document.createElement('script');
      script.src = "main.dart.js";
      document.body.appendChild(script);
    }
    if ('serviceWorker' in navigator) {
      window.addEventListener('DOMContentLoaded', function () {
        navigator.serviceWorker.register('flutter_service_worker.js')
          .then(function (serviceWorkerRegistration) {
            if (serviceWorkerRegistration.installing == null) {
              loadMain();
            } else {
              serviceWorkerRegistration.onupdatefound = function () {
                var newServiceWorker = serviceWorkerRegistration.installing;
                newServiceWorker.onstatechange = function () {
                  if (newServiceWorker.state === 'installed') {
                    newServiceWorker.postMessage('skipWaiting');
                  } else if (newServiceWorker.state === 'activated') {
                    loadMain();
                  }
                }
              }
            }
          }).catch(function (error) {
            loadMain();
          });
      });
    } else {
      loadMain();
    }
  </script>

Another option would be to modify the generated service worker. You can edit the fetch event handler to check for a specific request and ~~append the revision hash to make sure you have the right version~~ fetch it online first like the '/' key is.
In flutter_service_worker.js I would try something like the code below, somewhere in self.addEventListener("fetch", (event) => {...}

Sample code

if (key == 'assets/locale/en.json') {
  return onlineFirst(event);
}

Edit: I realized my second suggestion would not work using the revision hash. The only other option I can think of is fetching the resource online first. I have updated the sample code to reflect that.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

main.dart.js caching problem for web still is not solved #64968

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

main.dart.js caching problem for web still is not solved #64968

Description

Activity

jonahwilliams commented on Aug 31, 2020

sachemmark commented on Aug 31, 2020

jonahwilliams commented on Aug 31, 2020

MihanixA commented on Aug 31, 2020

jonahwilliams commented on Aug 31, 2020

MihanixA commented on Aug 31, 2020

jonahwilliams commented on Aug 31, 2020

MihanixA commented on Aug 31, 2020

jonahwilliams commented on Aug 31, 2020

MihanixA commented on Aug 31, 2020

pomo-mondreganto commented on Aug 31, 2020

jonahwilliams commented on Aug 31, 2020

MihanixA commented on Aug 31, 2020

17 remaining items

hyiso commented on Sep 10, 2020

azbuky commented on Sep 10, 2020

venkatd commented on Oct 6, 2020

azbuky commented on Oct 7, 2020

jonahwilliams commented on Oct 7, 2020

venkatd commented on Oct 7, 2020

jonahwilliams commented on Oct 7, 2020

venkatd commented on Oct 7, 2020

azbuky commented on Oct 7, 2020

venkatd commented on Oct 7, 2020

kmcgill88 commented on Oct 9, 2020

mikeesouth commented on Dec 16, 2020

azbuky commented on Dec 16, 2020

github-actions commented on Aug 8, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions