Visitar URL original
bpo-42213: Remove redundant cyclic GC hack in sqlite3 by erlend-aasland · Pull Request #26462 · python/cpython · GitHub
Skip to content

bpo-42213: Remove redundant cyclic GC hack in sqlite3 #26462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
erlend-aasland wants to merge 13 commits into from
Closed

bpo-42213: Remove redundant cyclic GC hack in sqlite3 #26462

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
70511f3
bpo-42213: Remove redundant cyclic GC hack in sqlite3
May 29, 2021
6cb435b
Split open tests into their own test case, and cleanup more explicitly
Jun 2, 2021
1e3f589
clean up test_trace_callback_content explicitly
Jun 2, 2021
5ff1a80
Document workarounds
Jun 2, 2021
866e226
Use test.support.gc_collect iso. gc.collect
Jun 2, 2021
063878c
Revert gc workarounds
Jun 2, 2021
b53aacb
Address review: Explicitly close opened database files
Jun 2, 2021
e0ffe76
Try to harden connection close
Jun 2, 2021
4ebb593
sqlite3_close_v2() always returns SQLITE_OK
Jun 2, 2021
7d8014a
Check connection on __enter__
Jun 2, 2021
1b23df1
Add tests that exercise stuff against a closed database
Jun 2, 2021
d31403e
Harden test clean up
Jun 2, 2021
aba8dd7
Add managed resource helper
Jun 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Try to harden connection close 
- add wrapper for sqlite3_close_v2()
- explicitly free pending statements before close()
  • Loading branch information
Erlend E. Aasland
Erlend E. Aasland committed Jun 2, 2021
commit e0ffe7664f6da6ce7f8e91f786a4d2157f6b9ca2
30 changes: 20 additions & 10 deletions Modules/_sqlite/connection.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,19 @@ connection_clear(pysqlite_Connection *self)
return 0;
}

static int
connection_close(pysqlite_Connection *self)
{
int rc = SQLITE_OK;

if (self->db) {
rc = sqlite3_close_v2(self->db);
self->db = NULL;
}

return rc;
}

static void
connection_dealloc(pysqlite_Connection *self)
{
Expand All @@ -258,9 +271,7 @@ connection_dealloc(pysqlite_Connection *self)
tp->tp_clear((PyObject *)self);

/* Clean up if user has not called .close() explicitly. */
if (self->db) {
sqlite3_close_v2(self->db);
}
(void)connection_close(self);

tp->tp_free(self);
Py_DECREF(tp);
Expand Down Expand Up @@ -345,22 +356,21 @@ static PyObject *
pysqlite_connection_close_impl(pysqlite_Connection *self)
/*[clinic end generated code: output=a546a0da212c9b97 input=3d58064bbffaa3d3]*/
{
int rc;

if (!pysqlite_check_thread(self)) {
return NULL;
}

pysqlite_do_all_statements(self, ACTION_FINALIZE, 1);

if (self->db) {
rc = sqlite3_close_v2(self->db);
/* Free pending statements before closing. This implies also cleaning
* up cursors, as they may have strong refs to statements. */
Py_CLEAR(self->statement_cache);
Py_CLEAR(self->statements);
Py_CLEAR(self->cursors);

@pablogsal pablogsal Jun 2, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if you use the connection after you call close()? Are we checking for NULL for these veriables everywhere else? (Cannot check myself as I'm on my phone)

@erlend-aasland erlend-aasland Jun 2, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll get a sqlite3.ProgrammingError. There's a lot of sanity checks in the code for this. The sqlite3 test suite runs fine, but of course, we do not have 100% code coverage yet. (Digression: see issue 43553 for improving sqlite3 code coverage.)

After close, if you try to use a cursor, fetch from a pending statement, or manipulate the connection, you'll get a sqlite3.ProgrammingError:

>>> cx = sqlite3.connect(":memory:")
>>> cu = cx.cursor()
>>> res = cu.execute("select 1")
>>> cx.close()
>>> res.fetchall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.ProgrammingError: Cannot operate on a closed database.
>>> cu.execute("select 1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.ProgrammingError: Cannot operate on a closed database.
>>> cx.create_function("test", 1, lambda x: x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.ProgrammingError: Cannot operate on a closed database.
>>>

@erlend-aasland erlend-aasland Jun 2, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some more tests wrt. this: 1b23df1
Also, there's some regression tests that exercise operations on closed connections (and closed cursors).

Let me know if you want me to add more tests.

@erlend-aasland erlend-aasland Jun 3, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a strange observation:
I did a ref leak test on Windows yesterday, and it segfaulted on test_bpo31770. After some testing, I reverted the change highlighted in this conversation; I removed the three Py_CLEAR, added pysqlite_do_all_statements(self, ACTION_FINALIZE, 1) back, but kept connection_close(). That fixed the ref leak test on Windows, and the test suite still worked; it did not "leak" open test database files. So, for the new PR, I'll keep this modification. We'll see how the CI fares :)

@erlend-aasland erlend-aasland Jun 3, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc. @vstinner

@erlend-aasland erlend-aasland Jun 3, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI seems to be fine with this: erlend-aasland#9


int rc = connection_close(self);
if (rc != SQLITE_OK) {
_pysqlite_seterror(self->db);
return NULL;
} else {
self->db = NULL;
}
}

Expand Down