Repository navigation
snapshot pages take a long time to fully populate when initially published #348
Description
I regularly fetch https://www.stackage.org/nightly file for Repology project to track haskell package versions in Stackage, and it turns out that the file is generated incompletely some times, which can be seen on the graph:
https://repology.org/graph/repo/stackage_nighly/projects_total.svg
Reacted by Simon Michael
Activity
juhp commentedon May 12, 2020
Thank you for reporting this.
I know for a while there were some stability problems behind the CloudFlare, not sure if that has improved on the stackage server. Maybe @snoyberg or others can comment. (Otherwise retrying might be the best workaround for now.)
Repology is a very cool project, thank you
AMDmi3 commentedon May 12, 2020
It seems to me that the issue is the file generation rather than distribution, because XML parser doesn't fail on my side, which would mean the file is not truncated, but a valid complete html with incomplete set of entries. That is why retrying doesn't help as file parsing always succeeds.
juhp commentedon May 14, 2020
Btw would you be better off downloading the json from https://github.com/commercialhaskell/stackage-snapshots/tree/master/nightly/ ? That repo is actually the source of truth for Stackage snapshot package versions.
Anyway looking at the data more carefully I finally understood what you are seeing:
So we don't guarantee there is a snapshot every single day(night) - it is best effort. :-)
ie Some days there is no new snapshot generated due to broken dependencies etc.
So for example with https://www.stackage.org/nightly-2020-05-09 :
there was no snapshot on 2020-05-09! That page could really be a 404 or something.
So on those days it would be better just to fallback to the previous snapshot.
Hope that helps.
AMDmi3 commentedon May 15, 2020
Hmm, this make sense, will rewrite parser for it.Nah, it's not suitable as it takes too much time to download both as a repository and as a snapshot. Would be nice if there were symlinks pointing to the latest nightly and lts snapshots, as long as github follows these and allows to download the latest snapshot as a blob this way.
Or I could stay on parsing HTML, but it would be nice if html was never generated for incomplete snapshot.
juhp commentedon May 16, 2020
I believe https://www.stackage.org/nightly and https://www.stackage.org/lts both redirect to the latest nightly and lts snapshots.
AMDmi3 commentedon May 16, 2020
That's what we began with - if snapshot creation fails, as you say, these are generated with incomplete set of packages (instead of staying at the previous snapshot data).
juhp commentedon May 17, 2020
AFAIK that is not true, https://www.stackage.org/nightly should always redirect to the latest available nightly snapshot. If you can show HTTP output to the contrary we will surely look into it more.
AMDmi3 commentedon May 17, 2020
I've shown the link to graph in the first comment which shows fluctuation of number of packages. To demonstrate the problem we'll have to wait for it to repeat itself.
juhp commentedon May 18, 2020
Sure we can leave this open until you are able to provide more detailed information, thanks
AMDmi3 commentedon May 19, 2020
Here it is: right now, https://www.stackage.org/nightly contains 1298 packages.
Before, there was 2483 packages: https://www.stackage.org/nightly-2020-05-16
juhp commentedon May 19, 2020
Hmm I currently see https://www.stackage.org/nightly-2020-05-19 with 2494 packages.
www.stackage.org is a Yesod webapp - I believe the pages are generated by the server direct from stackage snapshot data in a db.
AMDmi3 commentedon May 19, 2020
This is what I see:
Similar from any browser:
I've tried to capture screenshot with https://www.screenshotmachine.com, but it shows 2494, I believe the cause of this is CDN caching which is different depending for different regions.
Regardless, the original issue is that the page with incomplete data is generated.
juhp commentedon May 20, 2020
@snoyberg maybe you can have a look at this when you find time, thanks
snoyberg commentedon May 20, 2020
I think your message earlier is completely correct @juhp: using the web interface for this is the wrong thing to do. Taking the YAML files is the correct approach.
The CRON job that uploads the information is probably inserting packages one at a time into the database. Could that be improved? Sure, I guess, and I'd be happy to receive a PR. But it's not something I'd want to spend time on myself.
juhp commentedon May 20, 2020
(Sure, I was about to suggest moving this to https://github.com/fpco/stackage-server/issues)
snoyberg commentedon May 20, 2020
Honestly, I'd probably close it as WONTFIX.
AMDmi3 commentedon May 20, 2020
As I've said, there's no easy way to get current yajl file - downloading repository either with git or as a snapshot requires too much traffic and time for the purpose, and there's no way to download latest yajl as a blob as the path cannot be predicted and there's no symlink to the latest snapshot.
Summarizing, either fixing html generation or adding symlinks to the latest lts and nighlty snapshots in the repository would do. Otherwise I'll have to drop support for stackage in Repology.
snoyberg commentedon May 20, 2020
The path can definitely be predicted. Nightly for 2020-05-19 would, for example, be https://github.com/commercialhaskell/stackage-snapshots/blob/master/nightly/2020/5/19.yaml. The file https://s3.amazonaws.com/haddock.stackage.org/snapshots.json provides information on the latest nightly and minor release for each LTS major release.
AMDmi3 commentedon May 20, 2020
It cannot, as there can be no snapshot for the current date yet, or at all, or the latest snapshot could be for the next day depending on the timezone. lts is not tied to a date at all.
This is better, still it would require writing a custom fetcher which looks into two likely not synchronized locations for such a simple thing as downloading a single file. Would adding symlinks in the repo be too hard? E.g.
lts/latest→15/13.yaml. I've already checked, it would allow latest snapshot to be downloaded ashttps://github.com/commercialhaskell/stackage-snapshots/blob/master/lts/latest. May benefit other consumers as well.snoyberg commentedon May 21, 2020
I don't want to add the symlinks, since that will change the repo from a write-only repo to one with modifications.
I think what I've provided here is pretty straightforward: a JSON file with a clear mapping to a URL with YAML files. I'm not even sure how the symlink approach would be easier to work with than that.
AMDmi3 commentedon May 22, 2020
Which is the actual problem with that?
Because with symlink, I'll be able to download latest snapshot as a regular file. Otherwise I'll have to write custom script which first fetches json, parses it to determine snapshot name, then only then fetches the snapshot.
juhp commentedon Nov 2, 2020
A lot of time passed...
If you prefer not to parse the json you could check the redirect from stackage.org/nightly
(eg on command-line something like
curl -I -L https://www.stackage.org/nightly| grep location:)and then download that location.
A git symlink would also require you to download 2 files I believe (the symlink file and the actual yaml).
AMDmi3 commentedon Nov 2, 2020
I'd prefer to parse JSON, but I'd also prefer to fetch it with a single request, without having to write custom code to determine which JSON to fetch.
What is this supposed to achieve? I just use HTTP client which follows redirects.
Implement "incomplete" repository flag
[-]Cases of incomplete https://www.stackage.org/nightly[/-][+]snapshot pages take a long time to fully populate when initially published[/+]juhp commentedon Dec 13, 2024
I think the uploading is a bit faster now on the "new" server (about one year old now) - but the windows do still happen
chreekat commentedon May 2, 2025
I'm moving this to stackage-server. I'd like to make this page atomic eventually.