Visitar URL original
Add endpoint for querying time info for lsn by arpad-m · Pull Request #5497 · neondatabase/neon · GitHub
Skip to content

Add endpoint for querying time info for lsn - #5497

Merged
arpad-m merged 28 commits into
mainfrom
arpad/ts_by_lsn
Oct 19, 2023
Merged

arpad-m merged 28 commits into
mainfrom
arpad/ts_by_lsn

Conversation

@arpad-m

@arpad-m arpad-m commented Oct 7, 2023 •

Copy link
Copy Markdown
Contributor

Problem

See #5468.

Summary of changes

Add a new get_timestamp_of_lsn endpoint, returning the timestamp
associated with the given lsn.

Fixes #5468.

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@arpad-m
arpad-m requested review from a team as code owners October 7, 2023 07:51
@arpad-m
arpad-m requested review from save-buffer and shanyp and removed request for a team October 7, 2023 07:51
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
it isnt really an issue as it's in a test but whatever,
couldn't find ways to disable it for the tests dir.
Comment thread test_runner/regress/test_lsn_mapping.py
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
Comment thread test_runner/regress/test_lsn_mapping.py Outdated
Comment thread pageserver/src/http/openapi_spec.yml Outdated
@github-actions

github-actions Bot commented Oct 10, 2023 •

Copy link
Copy Markdown

2322 tests run: 2207 passed, 0 failed, 115 skipped (full report)


Flaky tests (2)

Postgres 16

  • test_pageserver_recovery: release

Postgres 14

  • test_competing_branchings_from_loading_race_to_ok_or_err: debug

Code coverage (full report)

  • functions: 52.9% (8339 of 15753 functions)
  • lines: 80.7% (48552 of 60182 lines)

The comment gets automatically updated with the latest test results
f6946e9 at 2023-10-19T02:08:35.574Z :recycle:

Comment thread pageserver/src/http/routes.rs Outdated
@xenomote

Copy link
Copy Markdown

Is this what you were asking for and does it fit your use cases?

yea seems good to me, I can update the go client once this is merged in 👍

@jcsp jcsp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern about this is how large that vector of timestamps will grow, since I don't know how many SLRU blocks we can expect to see. I've asked for some guidance in #postgres -- maybe this is no issue at all, but from the existing code it isn't obvious.

We could avoid storing the whole list if we didn't output the median, so if this becomes a problem in practice we might need to do that.

@shanyp

shanyp commented Oct 18, 2023

Copy link
Copy Markdown
Contributor

@arpad-m if the median is not a requirement and imposes issues, lets drop it

@arpad-m
arpad-m requested a review from knizhnik October 18, 2023 15:37
@arpad-m

arpad-m commented Oct 18, 2023

Copy link
Copy Markdown
Contributor Author

This test failure is interesting:

2023-10-18T16:58:07.0716962Z [gw6] linux -- Python 3.9.2 /github/home/.cache/pypoetry/virtualenvs/neon-_pxWMzVK-py3.9/bin/python
2023-10-18T16:58:07.0717704Z test_runner/regress/test_lsn_mapping.py:160: in test_ts_of_lsn_api
2023-10-18T16:58:07.0718462Z     timestamp = datetime.fromisoformat(result).replace(tzinfo=timezone.utc)
2023-10-18T16:58:07.0719183Z E   ValueError: Invalid isoformat string: '2023-10-18T16:50:28.990633000Z'

With:

2023-10-18T16:58:07.0754908Z 2023-10-18 16:50:31.684 INFO [test_lsn_mapping.py:158] result: 2023-10-18T16:50:28.990633000Z, after_ts: 2023-10-18 16:50:28.993508+00:00

On the other hand, on my local box it works, with:

2023-10-18 19:12:50.341 INFO [test_lsn_mapping.py:158] result: 2023-10-18T17:12:49.492124000Z, after_ts: 2023-10-18 17:12:49.501023+00:00

Apparently parsing time strings with Z in their name is a feature that's available since Python 3.11, which I have on my local box, but CI has Python 3.9... python/cpython#92177

Comment thread pageserver/src/http/openapi_spec.yml Outdated
Comment thread pageserver/src/http/openapi_spec.yml Outdated
@arpad-m

arpad-m commented Oct 19, 2023 •

Copy link
Copy Markdown
Contributor Author

For rerun attempt 2 for the latest commit "merge branch 'main' into arpad/ts_by_lsn", of the 6 regress tests, 2 failed.

logs for regress-tests (debug v15):

2023-10-18T19:35:53.6388101Z E   requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://localhost:30597/v1/tenant/02e588451d194f16bec8efc9679d0b0e/timeline/7c89ab12c4c81339777a1b96d99fe8c2/get_timestamp_of_lsn?lsn=0/14B6000
2023-10-18T19:35:53.6389524Z 
2023-10-18T19:35:53.6389820Z The above exception was the direct cause of the following exception:
2023-10-18T19:35:53.6390421Z test_runner/regress/test_lsn_mapping.py:153: in test_ts_of_lsn_api
2023-10-18T19:35:53.6390842Z     result = client.timeline_get_timestamp_of_lsn(
2023-10-18T19:35:53.6391315Z test_runner/fixtures/pageserver/http.py:461: in timeline_get_timestamp_of_lsn
2023-10-18T19:35:53.6391737Z     self.verbose_error(res)
2023-10-18T19:35:53.6392057Z test_runner/fixtures/pageserver/http.py:167: in verbose_error
2023-10-18T19:35:53.6392616Z     raise PageserverApiException(msg, res.status_code) from e
2023-10-18T19:35:53.6393259Z E   fixtures.pageserver.http.PageserverApiException

and regress-tests (release v15):

2023-10-18T19:33:42.2903323Z E   requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://localhost:25062/v1/tenant/51e028d4977dcedcef9e40f63a1e4919/timeline/be64fdd2c046133cfe0dbe31255d95a3/get_timestamp_of_lsn?lsn=0/14B6000
2023-10-18T19:33:42.2904239Z 
2023-10-18T19:33:42.2904444Z The above exception was the direct cause of the following exception:
2023-10-18T19:33:42.2904944Z test_runner/regress/test_lsn_mapping.py:153: in test_ts_of_lsn_api
2023-10-18T19:33:42.2905507Z     result = client.timeline_get_timestamp_of_lsn(
2023-10-18T19:33:42.2906091Z test_runner/fixtures/pageserver/http.py:461: in timeline_get_timestamp_of_lsn
2023-10-18T19:33:42.2906608Z     self.verbose_error(res)
2023-10-18T19:33:42.2907010Z test_runner/fixtures/pageserver/http.py:167: in verbose_error
2023-10-18T19:33:42.2907719Z     raise PageserverApiException(msg, res.status_code) from e
2023-10-18T19:33:42.2908244Z E   fixtures.pageserver.http.PageserverApiException

So the same error two times.

The rerun attempt 1 for the same commit showed different failures.

For regress-tests (debug v14):

2023-10-18T18:46:14.1431409Z [gw4] linux -- Python 3.9.2 /github/home/.cache/pypoetry/virtualenvs/neon-_pxWMzVK-py3.9/bin/python
2023-10-18T18:46:14.1432251Z test_runner/regress/test_lsn_mapping.py:168: in test_ts_of_lsn_api
2023-10-18T18:46:14.1432984Z     assert timestamp >= before_timestamp, "before_timestamp before timestamp"
2023-10-18T18:46:14.1433643Z E   AssertionError: before_timestamp before timestamp
2023-10-18T18:46:14.1434736Z E   assert datetime.datetime(2023, 10, 18, 18, 39, 6, 879910, tzinfo=datetime.timezone.utc) >= datetime.datetime(2023, 10, 18, 18, 39, 6, 908394, tzinfo=datetime.timezone.utc)

For regress-tests (debug v15), regress-tests (release v15) we have the 404 error again, and for regress-tests (release v16) we have the assertion failure again:

2023-10-18T18:45:06.5601640Z [gw6] linux -- Python 3.9.2 /github/home/.cache/pypoetry/virtualenvs/neon-_pxWMzVK-py3.9/bin/python
2023-10-18T18:45:06.5606381Z test_runner/regress/test_lsn_mapping.py:168: in test_ts_of_lsn_api
2023-10-18T18:45:06.5608107Z     assert timestamp >= before_timestamp, "before_timestamp before timestamp"
2023-10-18T18:45:06.5608482Z E   AssertionError: before_timestamp before timestamp
2023-10-18T18:45:06.5609027Z E   assert datetime.datetime(2023, 10, 18, 18, 38, 8, 467197, tzinfo=datetime.timezone.utc) >= datetime.datetime(2023, 10, 18, 18, 38, 8, 469206, tzinfo=datetime.timezone.utc)

We can get this PR probably merged by pressing on the "retry" button a few times but the test would still be extremely flaky.

Therefore, I have added a commit that adds a few sleeps and don't consider the first lsn (where we likely encounter the 404). let's see if this helps.

@arpad-m
arpad-m merged commit f842b22 into main Oct 19, 2023
@arpad-m
arpad-m deleted the arpad/ts_by_lsn branch October 19, 2023 02:50
arpad-m added a commit that referenced this pull request Nov 9, 2023
The `get_timestamp_of_lsn` pageserver endpoint has been added in #5497,
but the yml it added was wrong: the lsn is expected in hex format, not
in integer (decimal) format.
zeekay pushed a commit to hanzosql/neon that referenced this pull request Sep 10, 2026
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.0 to
3.7.4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/aio-libs/aiohttp/releases">aiohttp's
releases</a>.</em></p>
<blockquote>
<h2>aiohttp 3.7.3 release</h2>
<h2>Features</h2>
<ul>
<li>Use Brotli instead of brotlipy
<code>[neondatabase#3803](aio-libs/aiohttp#3803)
&lt;https://github.com/aio-libs/aiohttp/issues/3803&gt;</code>_</li>
<li>Made exceptions pickleable. Also changed the repr of some
exceptions.
<code>[neondatabase#4077](aio-libs/aiohttp#4077)
&lt;https://github.com/aio-libs/aiohttp/issues/4077&gt;</code>_</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Raise a ClientResponseError instead of an AssertionError for a blank
HTTP Reason Phrase.
<code>[neondatabase#3532](aio-libs/aiohttp#3532)
&lt;https://github.com/aio-libs/aiohttp/issues/3532&gt;</code>_</li>
<li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior
for patch without slash.
<code>[neondatabase#3669](aio-libs/aiohttp#3669)
&lt;https://github.com/aio-libs/aiohttp/issues/3669&gt;</code>_</li>
<li>Fix overshadowing of overlapped sub-applications prefixes.
<code>[neondatabase#3701](aio-libs/aiohttp#3701)
&lt;https://github.com/aio-libs/aiohttp/issues/3701&gt;</code>_</li>
<li>Make <code>BaseConnector.close()</code> a coroutine and wait until
the client closes all connections. Drop deprecated &quot;with
Connector():&quot; syntax.
<code>[neondatabase#3736](aio-libs/aiohttp#3736)
&lt;https://github.com/aio-libs/aiohttp/issues/3736&gt;</code>_</li>
<li>Reset the <code>sock_read</code> timeout each time data is received
for a <code>aiohttp.client</code> response.
<code>[neondatabase#3808](aio-libs/aiohttp#3808)
&lt;https://github.com/aio-libs/aiohttp/issues/3808&gt;</code>_</li>
<li>Fixed type annotation for add_view method of UrlDispatcher to accept
any subclass of View
<code>[neondatabase#3880](aio-libs/aiohttp#3880)
&lt;https://github.com/aio-libs/aiohttp/issues/3880&gt;</code>_</li>
<li>Fixed querying the address families from DNS that the current host
supports.
<code>[neondatabase#5156](aio-libs/aiohttp#5156)
&lt;https://github.com/aio-libs/aiohttp/issues/5156&gt;</code>_</li>
<li>Change return type of MultipartReader.<strong>aiter</strong>() and
BodyPartReader.<strong>aiter</strong>() to AsyncIterator.
<code>[neondatabase#5163](aio-libs/aiohttp#5163)
&lt;https://github.com/aio-libs/aiohttp/issues/5163&gt;</code>_</li>
<li>Provide x86 Windows wheels.
<code>[neondatabase#5230](aio-libs/aiohttp#5230)
&lt;https://github.com/aio-libs/aiohttp/issues/5230&gt;</code>_</li>
</ul>
<h2>Improved Documentation</h2>
<ul>
<li>Add documentation for <code>aiohttp.web.FileResponse</code>.
<code>[neondatabase#3958](aio-libs/aiohttp#3958)
&lt;https://github.com/aio-libs/aiohttp/issues/3958&gt;</code>_</li>
<li>Removed deprecation warning in tracing example docs
<code>[neondatabase#3964](aio-libs/aiohttp#3964)
&lt;https://github.com/aio-libs/aiohttp/issues/3964&gt;</code>_</li>
<li>Fixed wrong &quot;Usage&quot; docstring of
<code>aiohttp.client.request</code>.
<code>[neondatabase#4603](aio-libs/aiohttp#4603)
&lt;https://github.com/aio-libs/aiohttp/issues/4603&gt;</code>_</li>
<li>Add aiohttp-pydantic to third party libraries
<code>[neondatabase#5228](aio-libs/aiohttp#5228)
&lt;https://github.com/aio-libs/aiohttp/issues/5228&gt;</code>_</li>
</ul>
<h2>Misc</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's
changelog</a>.</em></p>
<blockquote>
<h1>3.7.4 (2021-02-25)</h1>
<h2>Bugfixes</h2>
<ul>
<li>
<p><strong>(SECURITY BUG)</strong> Started preventing open redirects in
the
<code>aiohttp.web.normalize_path_middleware</code> middleware. For
more details, see
<a
href="https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg</a>.</p>
<p>Thanks to <code>Beast Glatisant
&lt;https://github.com/g147&gt;</code>__ for
finding the first instance of this issue and <code>Jelmer Vernooij
&lt;https://jelmer.uk/&gt;</code>__ for reporting and tracking it down
in aiohttp.
<code>[neondatabase#5497](aio-libs/aiohttp#5497)
&lt;https://github.com/aio-libs/aiohttp/issues/5497&gt;</code>_</p>
</li>
<li>
<p>Fix interpretation difference of the pure-Python and the Cython-based
HTTP parsers construct a <code>yarl.URL</code> object for HTTP
request-target.</p>
<p>Before this fix, the Python parser would turn the URI's absolute-path
for <code>//some-path</code> into <code>/</code> while the Cython code
preserved it as
<code>//some-path</code>. Now, both do the latter.
<code>[neondatabase#5498](aio-libs/aiohttp#5498)
&lt;https://github.com/aio-libs/aiohttp/issues/5498&gt;</code>_</p>
</li>
</ul>
<hr />
<h1>3.7.3 (2020-11-18)</h1>
<h2>Features</h2>
<ul>
<li>Use Brotli instead of brotlipy
<code>[neondatabase#3803](aio-libs/aiohttp#3803)
&lt;https://github.com/aio-libs/aiohttp/issues/3803&gt;</code>_</li>
<li>Made exceptions pickleable. Also changed the repr of some
exceptions.
<code>[neondatabase#4077](aio-libs/aiohttp#4077)
&lt;https://github.com/aio-libs/aiohttp/issues/4077&gt;</code>_</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Raise a ClientResponseError instead of an AssertionError for a blank
HTTP Reason Phrase.
<code>[neondatabase#3532](aio-libs/aiohttp#3532)
&lt;https://github.com/aio-libs/aiohttp/issues/3532&gt;</code>_</li>
<li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior
for patch without slash.
<code>[neondatabase#3669](aio-libs/aiohttp#3669)
&lt;https://github.com/aio-libs/aiohttp/issues/3669&gt;</code>_</li>
<li>Fix overshadowing of overlapped sub-applications prefixes.
<code>[neondatabase#3701](aio-libs/aiohttp#3701)
&lt;https://github.com/aio-libs/aiohttp/issues/3701&gt;</code>_</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/0a26acc1de9e1b0244456b7881ec16ba8bb64fc3"><code>0a26acc</code></a>
Bump aiohttp to v3.7.4 for a security release</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/021c416c18392a111225bc7326063dc4a99a5138"><code>021c416</code></a>
Merge branch 'GHSA-v6wp-4m6f-gcjg' into master</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/4ed7c25b537f71c6245bb74d6b20e5867db243ab"><code>4ed7c25</code></a>
Bump chardet from 3.0.4 to 4.0.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5333">#5333</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/b61f0fdffc887df24244ba7bdfe8567c580240ff"><code>b61f0fd</code></a>
Fix how pure-Python HTTP parser interprets <code>//</code></li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5c1efbc32c46820250bd25440bb7ea96cb05abe9"><code>5c1efbc</code></a>
Bump pre-commit from 2.9.2 to 2.9.3 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5322">#5322</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/007507580137efcc0a20391a0792f39b337d9c1a"><code>0075075</code></a>
Bump pygments from 2.7.2 to 2.7.3 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5318">#5318</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5085173d947e6cc01b6daf1aa48fe7698834c569"><code>5085173</code></a>
Bump multidict from 5.0.2 to 5.1.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5308">#5308</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5d1a75e68d278c641c90021409f4eb5de1810e5e"><code>5d1a75e</code></a>
Bump pre-commit from 2.9.0 to 2.9.2 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5290">#5290</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/6724d0e7a944fd7e3a710dc292d785fa8fe424fd"><code>6724d0e</code></a>
Bump pre-commit from 2.8.2 to 2.9.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5273">#5273</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/c688451ce31b914c71b11d2ac6c326b0c87e6d1f"><code>c688451</code></a>
Removed duplicate timeout parameter in ClientSession reference docs. (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5262">#5262</a>)
...</li>
<li>Additional commits viewable in <a
href="https://github.com/aio-libs/aiohttp/compare/v3.7.0...v3.7.4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.7.0&new-version=3.7.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/neondatabase/neon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Vadim Kharitonov <vadim@neon.tech>
zeekay pushed a commit to hanzosql/neon that referenced this pull request Sep 10, 2026
## Problem

See neondatabase#5468.

## Summary of changes

Add a new `get_timestamp_of_lsn` endpoint, returning the timestamp
associated with the given lsn.

Fixes neondatabase#5468.

---------

Co-authored-by: Shany Pozin <shany@neon.tech>
zeekay pushed a commit to hanzosql/neon that referenced this pull request Sep 10, 2026
…ase#5840)

The `get_timestamp_of_lsn` pageserver endpoint has been added in neondatabase#5497,
but the yml it added was wrong: the lsn is expected in hex format, not
in integer (decimal) format.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API to request the timestamp for given LSN

6 participants