Repository navigation
Article edit page makes many queries #675
Description
Activity
Maybe you have mediafiles in a dropdown somewhere? It sounds like
raw_id_fields = ("mediafile",)might help.Where would this go? We don't currently customise the page admin tools, or use the translation system. We don't have any Admin classes that I could see to put this on.
I've pointed my local Django at memcache now (was using the local memory cache in development), and that has reduced the number of database queries, but a similar number of cache hits are happening, and still a lot of CPU usage and very long page loads. This basically maxes out an i7 core for several minutes.
Is there no way to prefetch the necessary data here for the translations here: https://github.com/feincms/feincms/blob/master/feincms/module/medialibrary/modeladmins.py#L229
- This is strange, if you are not doing anything special. I've not seen such an admin behaviour even on our larger sites (around 700 pages); while the admin site is not really totally snappy, it's okay to work with. Is your side very large, has lots of translations, anything you feel is out of the ordinary?…I've pointed my local Django at memcache now (was using the local memory cache in development), and that has reduced the number of database queries, but a similar number of cache hits are happening, and still a lot of CPU usage and very long page loads. This basically maxes out an i7 core for several minutes.
We have 358 pages, and don't do any translation at all. We also don't use the admin site for anything except FeinCMS (although the site has a lot of other functionality outside FeinCMS).
Some tracebacks, I've trimmed all of our middleware apart from the last frame, just to keep the size of these down a little. We have a fair few middlewares, but they don't cause us performance problems elsewhere so I don't think they are relevant.
/Users/dan/Code/styleme/styleme/core/middleware.py in __call__(71) return self.get_response(request) /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/module/medialibrary/models.py in __str__(150) trans = self.translation /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in translation(242) self._cached_translation = self.get_translation() /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in get_translation(227) self.translations.all(), language_code) /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in _get_translation_object(196) return queryset.all()[0]/Users/dan/Code/styleme/styleme/core/middleware.py in __call__(71) return self.get_response(request) /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/module/medialibrary/models.py in __str__(150) trans = self.translation /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in translation(242) self._cached_translation = self.get_translation() /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in get_translation(227) self.translations.all(), language_code) /Users/dan/.virtualenvs/styleme/lib/python3.6/site-packages/feincms/translations.py in _get_translation_object(186) ).order_by('-language_code')[0]These seem to be the hot code paths that are getting called many times. These are still contributing to around 2700 queries with caching turned on.
It might be that some queryset internals changed again, and feincms.utils.queryset_transform stopped working because of that.
(Historical note: The internals use started long before Django invented .prefetch_related() -- the queryset transform feature is mostly redundant but has never been removed since at the time it was still more capable than Django's prefetch mechanism. Maybe this changed with the introduction of Prefetch objects.)
Testing locally it does not look like it. A cache is almost a requirement for a manageable media library as implemented in FeinCMS. Maybe set
FEINCMS_THUMBNAIL_CACHE_TIMEOUTto some long value, e.g. 30 * 6400 or so?Ah yes, it did look like it would require a Prefetch object style prefetch_related. That makes sense.
@matthiask hey, any update on this, it would be good to get this fixed by porting FeinCMS to use new Prefetch objects, and dropping all the old prefetching machinery that seems to be broken on recent Django versions.
I'm a bit surprised that this code is run so often. The translations code doesn't even use timeouts, it caches translations indefinitely. It sometimes happens on really slow (or big) sites that we have to warm the cache a bit so that loading translations doesn't slow everything down too much but that's mostly an issue when the cache is still empty.
We are unfortunately not using the
feincms.translationsmachinery too much in new projects anymore so fixing this isn't a priority for us. Maybe #679 helps? You could "just" setFEINCMS_MEDIAFILE_TRANSLATIONS = Falseif you do not need the translations at all.
On my install, I find basic article pages take about 5500 queries to produce, and take around 200 seconds to load.
Specifically, this query is repeated about 2000 times:
This one around 2000 times as well:
And this one around 1500 times: