Visitar URL original
bpo-27645: Supporting native backup facility of SQLite by lelit · Pull Request #377 · python/cpython · GitHub
Skip to content
Closed
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d93391d
Expose the SQLite Online Backup API as Connection.backup()
lelit Jul 28, 2016
7b1277a
Preliminary documentation for sqlite3.Connection.backup()
lelit Jul 28, 2016
1258d47
Reduce code nesting depth by using a finally label, as other function…
lelit Mar 1, 2017
d8ea457
Wrap SQLite calls between BEGIN_ALLOW_THREADS and END_ALLOW_THREADS
lelit Mar 1, 2017
35fa0ad
Add an entry in Misc/NEWS related to issue #27645
lelit Mar 1, 2017
c1af5a0
Remove extraneous entry introduced in d4439d6c5b2a67682f3db74d395c603…
lelit Mar 1, 2017
c07d982
Attribute the patch to myself
lelit Mar 1, 2017
cdc7eda
Make pages and progress keyword-only args in sqlite3.Connection.backup()
lelit Mar 1, 2017
6d3eb69
Remove reference to SQLite docs in Connection.backup() documentation
lelit Mar 1, 2017
6fab17f
Use f-string in the sqlite3.Connection.backup() example
lelit Mar 1, 2017
6449f3d
Explicitly mark sqlite3.Connection.backup() as added in v3.7
lelit Mar 1, 2017
f84d375
Reduce chances of cluttering future “git blame”
lelit Mar 1, 2017
e748942
Parametrize the name of the database involved in the backup
lelit Mar 3, 2017
426dad4
Test propagation of exception raised in backup's progress callback
lelit Mar 3, 2017
895726b
Use a better name for variable
lelit Mar 3, 2017
77dfa39
Do not delay next iteration if the result was OK
lelit Mar 3, 2017
71726ba
Omit the backup method when underlying SQLite library is older than 3…
lelit Mar 3, 2017
c4fbc20
Assert that the non-mandatory arguments are keyword-only
lelit Mar 3, 2017
ff1609b
Pass also the current status of the ongoing backup to the progress ca…
lelit Mar 4, 2017
d688f20
Slightly different way handling backup step's error state
lelit Mar 4, 2017
4ed6c31
When an error occurs while the backup is going on, remove the target …
lelit Mar 4, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions Doc/library/sqlite3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -521,6 +521,34 @@ Connection Objects
f.write('%s\n' % line)


.. method:: backup(filename, *, pages=0, progress=None)

This method makes a backup of a SQLite database into the mandatory argument
*filename*, even while it's being accessed by other clients, or concurrently by
the same connection.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe mention that backups work for on-disk and in-memory databases like the sqlite3 C api docs mention?


By default, or when *pages* is either ``0`` or a negative integer, the entire

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we really need to expose this. The default value should cover most of the use cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that I should remove the pages argument? But if I do that, also the progress one becomes pointless, because with the default value there will be at most one callback, when the engine completes the backup. What am I missing?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to keep pages for huge databases.

database is copied in a single step; otherwise the method performs a loop
copying up to the specified *pages* at a time.

If *progress* is specified, it must either ``None`` or a callable object that

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either be ``None``

will be executed at each iteration with two integer arguments, respectively the
*remaining* number of pages still to be copied and the *total* number of pages.

Example::

# Copy an existing database into another file
import sqlite3

def progress(remaining, total):
print(f"Copied {total-remaining} of {total} pages...")

con = sqlite3.connect('existing_db.db')
con.backup('copy_of_existing_db.db', 1, progress)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs:

.. versionadded:: 3.7

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

.. versionadded:: 3.7


.. _sqlite3-cursor-objects:

Cursor Objects
Expand Down
104 changes: 104 additions & 0 deletions Lib/sqlite3/test/backup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
import sqlite3 as sqlite
from tempfile import NamedTemporaryFile
import unittest

class BackupTests(unittest.TestCase):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a @unittest.skipIf(...) decorator to skip the whole test case if SQLite version is not 3.6.11 or newer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add tests to make sure that keyword-only parameters are really keyword-only:

self.cx.backup(bckfn.name, 1, lambda x, y: None)  # should raise TypeError

def setUp(self):
cx = self.cx = sqlite.connect(":memory:")
cx.execute('CREATE TABLE foo (key INTEGER)')
cx.executemany('INSERT INTO foo (key) VALUES (?)', [(3,), (4,)])
cx.commit()

def tearDown(self):
self.cx.close()

def testBackup(self, bckfn):
cx = sqlite.connect(bckfn)
result = cx.execute("SELECT key FROM foo ORDER BY key").fetchall()
self.assertEqual(result[0][0], 3)
self.assertEqual(result[1][0], 4)

def CheckSimple(self):
with NamedTemporaryFile(suffix='.sqlite') as bckfn:
self.cx.backup(bckfn.name)
self.testBackup(bckfn.name)

def CheckProgress(self):
journal = []

def progress(remaining, total):
journal.append(remaining)

with NamedTemporaryFile(suffix='.sqlite') as bckfn:
self.cx.backup(bckfn.name, pages=1, progress=progress)
self.testBackup(bckfn.name)

self.assertEqual(len(journal), 2)
self.assertEqual(journal[0], 1)
self.assertEqual(journal[1], 0)

def CheckProgressAllPagesAtOnce_0(self):
journal = []

def progress(remaining, total):
journal.append(remaining)

with NamedTemporaryFile(suffix='.sqlite') as bckfn:
self.cx.backup(bckfn.name, progress=progress)
self.testBackup(bckfn.name)

self.assertEqual(len(journal), 1)
self.assertEqual(journal[0], 0)

def CheckProgressAllPagesAtOnce_1(self):
journal = []

def progress(remaining, total):
journal.append(remaining)

with NamedTemporaryFile(suffix='.sqlite') as bckfn:
self.cx.backup(bckfn.name, pages=-1, progress=progress)
self.testBackup(bckfn.name)

self.assertEqual(len(journal), 1)
self.assertEqual(journal[0], 0)

def CheckNonCallableProgress(self):
with NamedTemporaryFile(suffix='.sqlite') as bckfn:
with self.assertRaises(TypeError) as cm:
self.cx.backup(bckfn.name, pages=1, progress='bar')
self.assertEqual(str(cm.exception), 'progress argument must be a callable')

def CheckModifyingProgress(self):
journal = []

def progress(remaining, total):
if not journal:
self.cx.execute('INSERT INTO foo (key) VALUES (?)', (remaining+1000,))
self.cx.commit()
journal.append(remaining)

with NamedTemporaryFile(suffix='.sqlite') as bckfn:
self.cx.backup(bckfn.name, pages=1, progress=progress)
self.testBackup(bckfn.name)

cx = sqlite.connect(bckfn.name)
result = cx.execute("SELECT key FROM foo"
" WHERE key >= 1000"
" ORDER BY key").fetchall()
self.assertEqual(result[0][0], 1001)

self.assertEqual(len(journal), 3)
self.assertEqual(journal[0], 1)
self.assertEqual(journal[1], 1)
self.assertEqual(journal[2], 0)

def suite():
return unittest.TestSuite(unittest.makeSuite(BackupTests, "Check"))

def test():
runner = unittest.TextTestRunner()
runner.run(suite())

if __name__ == "__main__":
test()
5 changes: 3 additions & 2 deletions Lib/test/test_sqlite.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import sqlite3
from sqlite3.test import (dbapi, types, userfunctions,
factory, transactions, hooks, regression,
dump)
dump, backup)

def load_tests(*args):
if test.support.verbose:
Expand All @@ -18,7 +18,8 @@ def load_tests(*args):
userfunctions.suite(),
factory.suite(), transactions.suite(),
hooks.suite(), regression.suite(),
dump.suite()])
dump.suite(),
backup.suite()])

if __name__ == "__main__":
unittest.main()
3 changes: 3 additions & 0 deletions Misc/NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ What's New in Python 3.7.0 alpha 1?
Core and Builtins
-----------------

- bpo-27645: sqlite3.Connection now exposes a backup() method.
Patch by Lele Gaifax.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "Patch by Your Name." to the next line.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

- bpo-28598: Support __rmod__ for subclasses of str being called before
str.__mod__. Patch by Martijn Pieters.

Expand Down
88 changes: 88 additions & 0 deletions Modules/_sqlite/connection.c
Original file line number Diff line number Diff line change
Expand Up @@ -1477,6 +1477,92 @@ pysqlite_connection_iterdump(pysqlite_Connection* self, PyObject* args)
return retval;
}

static PyObject *
pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject* kwds)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I read SQLite release notes correctly, sqlite3_backup_init has been added in 3.6.11 so we need to expose this only if SQLite version is 3.6.11 or newer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, is it better to omit the whole function or rather raise an NotImplementedError?

@berkerpeksag berkerpeksag Mar 3, 2017 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's omit the whole function. This is what we do for other version specific features in the stdlib. Please also update the documentation to note that this feature is available from SQLite 3.6.11.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, done. Thank you!

{
char* filename;
int pages = -1;
PyObject* progress = Py_None;
PyObject* retval = NULL;
int rc;
sqlite3 *bckconn;
sqlite3_backup *bckhandle;
static char *keywords[] = {"filename", "pages", "progress", NULL};

if (!PyArg_ParseTupleAndKeywords(args, kwds, "s|$iO:backup", keywords,
&filename, &pages, &progress)) {
goto finally;
}

if (progress != Py_None && !PyCallable_Check(progress)) {
PyErr_SetString(PyExc_TypeError, "progress argument must be a callable");
goto finally;
}

if (pages == 0) {
pages = -1;
}

Py_BEGIN_ALLOW_THREADS
rc = sqlite3_open(filename, &bckconn);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#359 could be useful here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you are suggesting: should I take the same approach of conditionally using sqlite3_open_v2() instead?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to say that this should be merged after #359.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This don't actually have any effect. sqlite3_open(filename, db) is the same as sqlite3_open_v2(filename, db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL)

Py_END_ALLOW_THREADS

if (rc != SQLITE_OK) {
goto finally;
}

Py_BEGIN_ALLOW_THREADS
bckhandle = sqlite3_backup_init(bckconn, "main", self->db, "main");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't database name be a parameter?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not really the database name: accordingly to the documentation that name denotes the target, that can either be "main" for the main storage, "temp" for the temporary storage or the name of the attached database.
Of course we can easily expose that as a (default) argument, but I did not see as necessary when I coded the feature.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a source_name and a dest_name parameters.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhm, further investigation seems to reveal that the dest_name is pointless: attached databases are not a persistent property of a database, so that we cannot allow the following:

bck = sqlite.connect('/tmp/thebackup')
bck.execute(f"ATTACH DATABASE '/tmp/theattachment' AS attached_db")
bck.commit()

db = sqlite.connect('/tmp/database')
db.backup('/tmp/thebackup', dest_name='attached_db')

Changed back to a single name argument to specify the source database...

Py_END_ALLOW_THREADS

if (bckhandle) {
do {
Py_BEGIN_ALLOW_THREADS
rc = sqlite3_backup_step(bckhandle, pages);
Py_END_ALLOW_THREADS

if (progress != Py_None) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should call progress only when sqlite3_backup_step return SQLITE_OK

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about that: maybe a better option would be pass the result status to the progress callback, so that it can eventually signal the error and do some cleanup (think about closing a progress dialog).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now the progress callback receives three arguments.

if (!PyObject_CallFunction(progress, "ii",
sqlite3_backup_remaining(bckhandle),
sqlite3_backup_pagecount(bckhandle))) {
/* User's callback raised an error: interrupt the loop and
propagate it. */
rc = -1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add a test for the case that progress raise an error

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will do!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}
}

/* Sleep for 250ms if there are still further pages to copy */
if (rc == SQLITE_OK || rc == SQLITE_BUSY || rc == SQLITE_LOCKED) {
Py_BEGIN_ALLOW_THREADS
sqlite3_sleep(250);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure we need to sleep when SQLITE_OK is returned? I know this is what is done in the example but we do allow other threads to take the lock between calls to sqlite3_backup_step.
In addition we might want to have some kind of timeout here. We can block forever if someone holds the db lock.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should: the step function returns OK when it “successfully copies N pages and there are still more pages to be copied”.
Not sure about the timeout: maybe we could check whether there is no progress in, say, X calls to the step function?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconsidering, I think you're right here: there's no reason to delay next iteration when the result was SQLITE_OK.

Py_END_ALLOW_THREADS
}
} while (rc == SQLITE_OK || rc == SQLITE_BUSY || rc == SQLITE_LOCKED);

Py_BEGIN_ALLOW_THREADS
sqlite3_backup_finish(bckhandle);
Py_END_ALLOW_THREADS
}

if (rc != -1) {
rc = _pysqlite_seterror(bckconn, NULL);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this but from the documentation sqlite3_backup_step will not set an error code on error but will return one. I can see that in the example they do the same as here but I am not sure if this is the right behavior. Would we receive SQLITE_NOMEM error from sqlite3_backup_step?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will try to understand the point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to address this concern in my latest patches.

}

Py_BEGIN_ALLOW_THREADS
sqlite3_close(bckconn);
Py_END_ALLOW_THREADS

if (rc == SQLITE_OK) {
Py_INCREF(Py_None);
retval = Py_None;
} else {
/* TODO: should the (probably incomplete/invalid) backup be removed here? */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should delete the backup

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will do that.

Thanks a lot for the review!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need some help here: I expected to find an abstract/portable PyOS_UnlinkFile(), but couldn't find it.
What is the recommended idiom to remove a file from C? Is the only way to "import" the os module and then PyObject_CallFunction() on its unlink member?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the simplest way for now, using unlink(2) from unistd.h. Any hint for a better way is welcome.

}

finally:
return retval;
}

static PyObject *
pysqlite_connection_create_collation(pysqlite_Connection* self, PyObject* args)
{
Expand Down Expand Up @@ -1649,6 +1735,8 @@ static PyMethodDef connection_methods[] = {
PyDoc_STR("Abort any pending database operation. Non-standard.")},
{"iterdump", (PyCFunction)pysqlite_connection_iterdump, METH_NOARGS,
PyDoc_STR("Returns iterator to the dump of the database in an SQL text format. Non-standard.")},
{"backup", (PyCFunction)pysqlite_connection_backup, METH_VARARGS | METH_KEYWORDS,
PyDoc_STR("Makes a backup of the database. Non-standard.")},
{"__enter__", (PyCFunction)pysqlite_connection_enter, METH_NOARGS,
PyDoc_STR("For context manager. Non-standard.")},
{"__exit__", (PyCFunction)pysqlite_connection_exit, METH_VARARGS,
Expand Down