Visitar URL original
fix(session): recover MTProto connections safely · hydrogram/hydrogram@8da10a7 · GitHub
Skip to content

Commit 8da10a7

Browse files
committed
fix(session): recover MTProto connections safely
1 parent e15d874 commit 8da10a7

13 files changed

Lines changed: 1580 additions & 278 deletions

File tree

‎docs/source/faq/socket-send-oserror-timeouterror-connection-lost-reset.rst‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,70 @@ You can consider the following:
1010
- Use shorter non-asynchronous processing loops.
1111
- Use ``asyncio.sleep()`` instead of ``time.sleep()``.
1212
- Use a stable network connection.
13+
14+
Connection recovery
15+
-------------------
16+
17+
Hydrogram coordinates recovery through one task per MTProto session. A closed
18+
transport, connection reset, failed write, request timeout, or missing ping reply
19+
starts recovery. Concurrent requests wait for that same task. Recovery preserves
20+
the authorization key and never logs out, revokes a session, or deletes a session
21+
file.
22+
23+
Connection attempts use exponential backoff with jitter. The session allows five
24+
consecutive initialization attempts, including connections that fail again within
25+
30 seconds. Backoff starts between 0.5 and 1 second, doubles between attempts, and
26+
is capped at 30 seconds. Each initialization can make up to three transport
27+
connection attempts. A connection that remains healthy for 30 seconds restores the
28+
recovery budget. Exhaustion leaves the session in the ``failed`` state and raises
29+
``ConnectionError`` to callers. Further RPCs fail promptly instead of silently
30+
starting another unlimited retry loop; an explicit session start or client
31+
reconnection is needed to try again.
32+
33+
A lack of updates is normal for an idle client. Periodic pings require a pong and
34+
provide the transport liveness check. The update watchdog can probe update state
35+
after an idle period, but inactivity alone does not trigger a reconnect.
36+
37+
Request outcomes
38+
----------------
39+
40+
Requests that have not been sent can proceed after recovery. Requests already
41+
sent either return their available response or fail promptly with
42+
:class:`~hydrogram.errors.ConnectionLostError`, a ``ConnectionError`` subclass.
43+
A request whose response times out raises
44+
:class:`~hydrogram.errors.RequestTimeoutError`, a ``TimeoutError`` subclass.
45+
Both expose ``delivery_uncertain``: when true, Telegram may have processed the
46+
request. Check its outcome before deciding whether to retry a write operation.
47+
Hydrogram does not automatically replay these requests or RPCs returning server
48+
errors, since repeating an arbitrary method could duplicate side effects.
49+
50+
An explicit bad-server-salt rejection permits a bounded resend with the corrected
51+
salt. Sequence-number errors rebuild only the ephemeral MTProto session and
52+
surface the rejected request to its caller. Authentication failures such as
53+
``AUTH_KEY_DUPLICATED`` or ``SESSION_REVOKED`` stop recovery and surface the error
54+
without changing stored credentials. Clock-related bad-message errors require
55+
correcting the local clock; they are not retried automatically. Two-factor login
56+
prompts remain available on the current connection.
57+
58+
Connection diagnostics
59+
----------------------
60+
61+
``app.connection_health`` returns a snapshot without making a network request:
62+
63+
.. code-block:: python
64+
65+
health = app.connection_health
66+
print(health["state"], health["is_connected"], health["last_error"])
67+
68+
The snapshot contains ``state`` (``stopped``, ``connecting``, ``reconnecting``,
69+
``connected`` or ``failed``), ``is_connected``, ``dc_id``, ``reconnect_attempt``,
70+
``pending_requests``, ``last_received_at`` and ``last_error``. The last-received
71+
timestamp includes protocol traffic such as pongs, not just updates.
72+
``app.is_connected`` continues to describe the client lifecycle; use the health
73+
snapshot to inspect transport readiness during recovery.
74+
75+
The ``hydrogram.session.session`` logger records connection loss, retry delays,
76+
successful recovery, and exhausted attempts. Retry records include ``dc_id``,
77+
``connection_state``, ``reconnect_attempt`` and ``reconnect_delay`` as structured
78+
logging fields. Stopping the client cancels its recovery, ping, receive, update
79+
watchdog, and owned update-processing tasks.

‎hydrogram/client.py‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
BadRequest,
4949
CDNFileHashMismatch,
5050
ChannelPrivate,
51+
RPCError,
5152
SessionPasswordNeeded,
5253
VolumeLocNotFound,
5354
)
@@ -352,6 +353,27 @@ def __init__(
352353
async def __aenter__(self):
353354
return await self.start()
354355

356+
@property
357+
def connection_health(self) -> dict:
358+
"""Snapshot of the main session's connection health.
359+
360+
Includes ``state``, ``is_connected``, ``dc_id``, ``reconnect_attempt``,
361+
``pending_requests``, ``last_received_at`` and ``last_error``.
362+
This property performs no network requests. Use it to inspect transient
363+
recovery independently of the client lifecycle flag ``is_connected``.
364+
"""
365+
if self.session is None:
366+
return {
367+
"state": "stopped",
368+
"is_connected": False,
369+
"dc_id": None,
370+
"reconnect_attempt": 0,
371+
"pending_requests": 0,
372+
"last_received_at": None,
373+
"last_error": None,
374+
}
375+
return self.session.health
376+
355377
async def __aexit__(self, *args):
356378
with contextlib.suppress(ConnectionError):
357379
await self.stop()
@@ -382,7 +404,12 @@ async def updates_watchdog(self):
382404
if self.update_manager is not None:
383405
self.update_manager.request_recovery()
384406
else:
385-
await self.invoke(raw.functions.updates.GetState())
407+
try:
408+
await self.invoke(raw.functions.updates.GetState())
409+
except (OSError, asyncio.TimeoutError, RPCError):
410+
# An idle period is not a transport failure. Session pings
411+
# own liveness checks and coordinate any necessary recovery.
412+
log.warning("Update watchdog probe failed", exc_info=True)
386413

387414
async def authorize(self) -> User:
388415
if self.bot_token:

‎hydrogram/connection/connection.py‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
import asyncio
2323
import logging
24+
import random
2425
from typing import TYPE_CHECKING
2526

2627
from hydrogram.session.internals import DataCenter
@@ -56,16 +57,21 @@ def __init__(
5657
self.protocol: TCP | None = None
5758

5859
async def connect(self) -> None:
59-
for i in range(Connection.MAX_CONNECTION_ATTEMPTS):
60+
for i in range(self.MAX_CONNECTION_ATTEMPTS):
6061
self.protocol = self.protocol_factory(ipv6=self.ipv6, proxy=self.proxy)
6162

6263
try:
6364
log.info("Connecting...")
6465
await self.protocol.connect(self.address)
65-
except OSError as e:
66+
except (OSError, asyncio.TimeoutError) as e:
6667
log.warning("Unable to connect due to network issues: %s", e)
6768
await self.protocol.close()
68-
await asyncio.sleep(1)
69+
if i + 1 == self.MAX_CONNECTION_ATTEMPTS:
70+
raise ConnectionError("Transport connection attempts exhausted") from e
71+
await asyncio.sleep(random.uniform(0.5, 1.0) * 2**i)
72+
except BaseException:
73+
await self.protocol.close()
74+
raise
6975
else:
7076
log.info(
7177
"Connected! %s DC%s%s - IPv%s",
@@ -76,15 +82,19 @@ async def connect(self) -> None:
7682
)
7783
break
7884
else:
79-
log.warning("Connection failed! Trying again...")
80-
raise ConnectionError
85+
raise ConnectionError("No transport connection attempts configured")
8186

8287
async def close(self) -> None:
83-
await self.protocol.close()
88+
if self.protocol is not None:
89+
await self.protocol.close()
8490
log.info("Disconnected")
8591

8692
async def send(self, data: bytes) -> None:
93+
if self.protocol is None:
94+
raise ConnectionError("Transport is not connected")
8795
await self.protocol.send(data)
8896

8997
async def recv(self) -> bytes | None:
98+
if self.protocol is None:
99+
return None
90100
return await self.protocol.recv()

‎hydrogram/connection/transport/tcp/tcp.py‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,11 @@ def __init__(self, ipv6: bool, proxy: Proxy) -> None:
6161
@property
6262
def closed(self) -> bool:
6363
return (
64-
self._closed or self.writer is None or self.writer.is_closing() or self.reader is None
64+
self._closed
65+
or self.writer is None
66+
or self.writer.is_closing()
67+
or self.reader is None
68+
or self.reader.at_eof()
6569
)
6670

6771
async def _connect_via_proxy(self, destination: tuple[str, int]) -> None:
@@ -93,11 +97,13 @@ async def _connect_via_proxy(self, destination: tuple[str, int]) -> None:
9397
)
9498
sock.settimeout(TCP.TIMEOUT)
9599

96-
await self.loop.sock_connect(sock=sock, address=destination)
97-
98-
sock.setblocking(False)
99-
100-
self.reader, self.writer = await asyncio.open_connection(sock=sock)
100+
try:
101+
await self.loop.sock_connect(sock=sock, address=destination)
102+
sock.setblocking(False)
103+
self.reader, self.writer = await asyncio.open_connection(sock=sock)
104+
except BaseException:
105+
sock.close()
106+
raise
101107

102108
async def _connect_via_direct(self, destination: tuple[str, int]) -> None:
103109
host, port = destination
@@ -136,13 +142,12 @@ async def close(self) -> None:
136142
self._closed = True
137143

138144
async def send(self, data: bytes) -> None:
139-
if self.writer is None or self._closed:
140-
raise OSError("Connection is closed")
141-
142145
async with self.lock:
146+
if self.closed:
147+
raise OSError("Connection is closed")
143148
try:
144149
self.writer.write(data)
145-
await self.writer.drain()
150+
await asyncio.wait_for(self.writer.drain(), self.TIMEOUT)
146151
except Exception as e:
147152
log.info("Send exception: %s %s", type(e).__name__, e)
148153
self._closed = True
@@ -157,7 +162,10 @@ async def recv(self, length: int = 0) -> bytes | None:
157162
while len(data) < length:
158163
try:
159164
chunk = await asyncio.wait_for(self.reader.read(length - len(data)), TCP.TIMEOUT)
160-
except (OSError, asyncio.TimeoutError):
165+
except asyncio.TimeoutError:
166+
# A quiet socket is normal. Session pings detect unresponsive peers.
167+
continue
168+
except OSError:
161169
self._closed = True
162170
return None
163171
else:

‎hydrogram/errors/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,20 @@
2626
from .rpc_error import UnknownError
2727

2828

29+
class ConnectionLostError(ConnectionError):
30+
"""The connection was lost; a sent request's delivery may be uncertain."""
31+
32+
def __init__(self, message: str, *, delivery_uncertain: bool = True) -> None:
33+
self.delivery_uncertain = delivery_uncertain
34+
super().__init__(message)
35+
36+
37+
class RequestTimeoutError(TimeoutError):
38+
"""A sent request timed out. It may have been processed by Telegram."""
39+
40+
delivery_uncertain: bool = True
41+
42+
2943
class BadMsgNotification(Exception): # noqa: N818
3044
descriptions: ClassVar = {
3145
16: "The msg_id is too low, the client time has to be synchronized.",
@@ -74,8 +88,10 @@ def __init__(self, msg: str | None = None):
7488
__all__ = [
7589
"BadMsgNotification",
7690
"CDNFileHashMismatch",
91+
"ConnectionLostError",
7792
"ListenerStopped",
7893
"ListenerTimeout",
94+
"RequestTimeoutError",
7995
"SecurityCheckMismatch",
8096
"SecurityError",
8197
"UnknownError",

‎hydrogram/methods/advanced/invoke.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,8 @@ async def invoke(
5959
The API Schema function filled with proper arguments.
6060
6161
retries (``int``):
62-
Number of retries.
62+
Attempt limit for requests not yet sent and explicit flood waits.
63+
Requests with uncertain delivery and server errors are not replayed.
6364
6465
timeout (``float``):
6566
Timeout in seconds.
@@ -72,6 +73,10 @@ async def invoke(
7273
7374
Raises:
7475
RPCError: In case of a Telegram RPC error.
76+
ConnectionLostError: If the transport fails. Check ``delivery_uncertain``
77+
before deciding whether the request can be safely retried.
78+
RequestTimeoutError: If Telegram does not respond in time. The request
79+
may have been processed; check its outcome before retrying.
7580
"""
7681
if not self.is_connected:
7782
raise ConnectionError("Client has not been started yet")

‎hydrogram/methods/auth/connect.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,12 @@ async def connect(
5151
await self.storage.test_mode(),
5252
)
5353

54-
await self.session.start()
54+
try:
55+
await self.session.start()
56+
except BaseException:
57+
await self.session.stop()
58+
await self.storage.close()
59+
raise
5560

5661
self.is_connected = True
5762

‎hydrogram/methods/auth/terminate.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@
1717
# You should have received a copy of the GNU Lesser General Public License
1818
# along with Hydrogram. If not, see <http://www.gnu.org/licenses/>.
1919

20+
import asyncio
2021
import logging
2122

2223
import hydrogram
2324
from hydrogram import raw
25+
from hydrogram.errors import RPCError
2426

2527
log = logging.getLogger(__name__)
2628

@@ -41,8 +43,13 @@ async def terminate(
4143
raise ConnectionError("Client is already terminated")
4244

4345
if self.takeout_id:
44-
await self.invoke(raw.functions.account.FinishTakeoutSession())
45-
log.info("Takeout session %s finished", self.takeout_id)
46+
try:
47+
await self.invoke(raw.functions.account.FinishTakeoutSession())
48+
except (OSError, asyncio.TimeoutError, RPCError):
49+
log.warning("Could not finish takeout session during shutdown", exc_info=True)
50+
else:
51+
log.info("Takeout session %s finished", self.takeout_id)
52+
self.takeout_id = None
4653

4754
if self.update_manager is not None:
4855
await self.update_manager.stop()
@@ -58,7 +65,9 @@ async def terminate(
5865
self.updates_watchdog_event.set()
5966

6067
if self.updates_watchdog_task is not None:
61-
await self.updates_watchdog_task
68+
self.updates_watchdog_task.cancel()
69+
await asyncio.gather(self.updates_watchdog_task, return_exceptions=True)
70+
self.updates_watchdog_task = None
6271

6372
self.updates_watchdog_event.clear()
6473

‎hydrogram/methods/utilities/stop.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,14 @@ async def main():
5959
"""
6060

6161
async def do_it():
62+
# Do not make worker shutdown wait for an in-flight reconnect budget.
63+
# Pending RPCs must be released before waiting on their handlers.
64+
if (
65+
self.session is not None
66+
and self.session.recovery_task is not None
67+
and not self.session.recovery_task.done()
68+
):
69+
await self.session.stop()
6270
await self.terminate()
6371
await self.disconnect()
6472

‎hydrogram/session/auth.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,8 @@ def unpack(b: BytesIO):
6666
async def invoke(self, data: TLObject):
6767
data = self.pack(data)
6868
await self.connection.send(data)
69-
response = BytesIO(await self.connection.recv())
69+
# Authentication runs before the session's ping monitor is available.
70+
response = BytesIO(await asyncio.wait_for(self.connection.recv(), 15))
7071

7172
return self.unpack(response)
7273

0 commit comments

Comments
 (0)