Visitar URL original
bpo-27645: Supporting native backup facility of SQLite by lelit · Pull Request #4238 · python/cpython · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
0c9a4e5
Expose the SQLite Online Backup API as Connection.backup()
lelit Jul 28, 2016
624849a
Preliminary documentation for sqlite3.Connection.backup()
lelit Jul 28, 2016
b17e572
Reduce code nesting depth by using a finally label, as other function…
lelit Mar 1, 2017
3a64654
Wrap SQLite calls between BEGIN_ALLOW_THREADS and END_ALLOW_THREADS
lelit Mar 1, 2017
8220ed6
Make pages and progress keyword-only args in sqlite3.Connection.backup()
lelit Mar 1, 2017
f7c6fc5
Remove reference to SQLite docs in Connection.backup() documentation
lelit Mar 1, 2017
df607df
Use f-string in the sqlite3.Connection.backup() example
lelit Mar 1, 2017
169369b
Explicitly mark sqlite3.Connection.backup() as added in v3.7
lelit Mar 1, 2017
7ee5341
Reduce chances of cluttering future “git blame”
lelit Mar 1, 2017
f88cd12
Parametrize the name of the database involved in the backup
lelit Mar 3, 2017
7ee018c
Test propagation of exception raised in backup's progress callback
lelit Mar 3, 2017
21bfc82
Use a better name for variable
lelit Mar 3, 2017
9574b91
Do not delay next iteration if the result was OK
lelit Mar 3, 2017
09407c8
Omit the backup method when underlying SQLite library is older than 3…
lelit Mar 3, 2017
f99e65e
Assert that the non-mandatory arguments are keyword-only
lelit Mar 3, 2017
aaa1508
Pass also the current status of the ongoing backup to the progress ca…
lelit Mar 4, 2017
13de3a1
Slightly different way handling backup step's error state
lelit Mar 4, 2017
9b2f47a
When an error occurs while the backup is going on, remove the target …
lelit Mar 4, 2017
ce55873
Add NEWS entry for issue 27645
lelit Oct 5, 2017
5a08168
New argument "sleep" to specify a different number of milliseconds
lelit Oct 24, 2017
960303f
Allow to copy the database to either an external file or to another C…
lelit Oct 24, 2017
4bd0b3e
Check that target backup DB is not in transaction
lelit Oct 24, 2017
7dc53f0
Use DeleteFileW() under MS Windows to delete incomplete backup
lelit Dec 5, 2017
a333639
Second attempt at properly calling DeleteFileW()
lelit Dec 5, 2017
4ea5ca4
Setup exception from sqlite error condition
lelit Dec 5, 2017
48fd04b
Close the backup connection in case of error
lelit Dec 5, 2017
3256d52
Attempt to chain the exception raised by incomplete backup deletion
lelit Dec 5, 2017
a04a86e
Fix double call to PyMem_Free()
lelit Dec 5, 2017
a4334a6
Remove leftover comment
lelit Dec 5, 2017
69d8996
Drop const qualifier to avoid compiler warning
lelit Dec 5, 2017
ac5c64b
Avoid using NamedTemporaryFile in test cases
lelit Dec 5, 2017
09671e4
Make the sleep parameter a double, accepting a value in seconds
lelit Dec 5, 2017
b5260e0
Replace gotos with explicit return, given that no cleanup is needed
lelit Dec 5, 2017
37f316f
Rename test methods using snake_style instead of CamelCase
lelit Dec 5, 2017
b7fcf9e
Use one single pointers declaration style in the new function
lelit Dec 6, 2017
6566166
Fix glitch in example code
lelit Dec 8, 2017
e6f8950
Rectify explanation of the pages parameter
lelit Dec 8, 2017
5b06a74
Add missing ending dot
lelit Dec 8, 2017
5c20723
Simplify code
lelit Dec 8, 2017
2155feb
Add missing const qualifier
lelit Dec 8, 2017
d21c9cb
Fix leak by DECREFing the result of progress callback call
lelit Dec 8, 2017
8e096c2
Restore the suite() function, used by outer Lib/test/test_sqlite.py
lelit Dec 8, 2017
a2f15bc
Simplify backup() implementation
lelit Dec 12, 2017
66df8b3
Remove now useless includes
lelit Dec 12, 2017
814ef4e
Recognize common error codes for old SQLite3 missing sqlite3_errstr()
lelit Dec 12, 2017
8fe5c30
Move argument type check to PyArg_ParseTupleAndKeywords()
lelit Dec 12, 2017
acc2f37
Use better names for local variables
lelit Dec 12, 2017
32285b1
Merge remote-tracking branch 'upstream/master' into pr/4238
berkerpeksag Mar 10, 2018
7943561
Cosmetic fixes
berkerpeksag Mar 10, 2018
824d5f8
Fix segfault in test_bad_target_closed_connection
berkerpeksag Mar 10, 2018
6fd854a
Relax an assert
berkerpeksag Mar 10, 2018
57f49c6
This will be included in 3.7
berkerpeksag Mar 10, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Allow to copy the database to either an external file or to another C…
…onnection
  • Loading branch information
lelit committed Oct 24, 2017
commit 960303f9eb394e2ba91d10e5e674997a48811ac2
18 changes: 13 additions & 5 deletions Doc/library/sqlite3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -521,11 +521,12 @@ Connection Objects
f.write('%s\n' % line)


.. method:: backup(filename, *, pages=0, progress=None, name="main", sleep=250)
.. method:: backup(target, *, pages=0, progress=None, name="main", sleep=250)

This method makes a backup of a SQLite database into the mandatory argument
*filename*, even while it's being accessed by other clients, or concurrently by
the same connection.
*target*, even while it's being accessed by other clients, or concurrently by
the same connection. The *target* can be either a string, the path to the file
where the backup will be written, or another :class:`Connection` instance.

By default, or when *pages* is either ``0`` or a negative integer, the entire
database is copied in a single step; otherwise the method performs a loop
Expand All @@ -544,9 +545,8 @@ Connection Objects
The *sleep* argument specifies the number of milliseconds to sleep by between

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost all CPython functions and methods use seconds as double for timeout. Milliseconds is going to confuse people.

successive attempts to backup remaining pages.

Example::
Example 1, copy an existing database into another file::

# Copy an existing database into another file
import sqlite3

def progress(status, remaining, total):
Expand All @@ -555,6 +555,14 @@ Connection Objects
con = sqlite3.connect('existing_db.db')
con.backup('copy_of_existing_db.db', 1, progress)

Example 2, copy an existing database into a transient copy::

import sqlite3

source = sqlite3.connect('existing_db.db')
dest = sqlite3.connect(':memory')

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

':memory:'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

source.backup(dest)

.. note:: This is available only when the underlying SQLite library is at
version 3.6.11 or higher.

Expand Down
11 changes: 11 additions & 0 deletions Lib/sqlite3/test/backup.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ def testBackup(self, bckfn):
self.assertEqual(result[0][0], 3)
self.assertEqual(result[1][0], 4)

def CheckBadTarget(self):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't follow the example of the other sqlite tests and use standard test names test_foo.

with self.assertRaises(TypeError):
self.cx.backup(None)

def CheckKeywordOnlyArgs(self):
with self.assertRaises(TypeError):
self.cx.backup('foo', 1)
Expand Down Expand Up @@ -124,6 +128,13 @@ def CheckDatabaseSourceName(self):
self.cx.backup(bckfn.name, name='attached_db')
self.testBackup(bckfn.name)

def CheckBackupToOtherConnection(self):
dx = sqlite.connect(':memory:')
self.cx.backup(dx)
result = dx.execute("SELECT key FROM foo ORDER BY key").fetchall()
self.assertEqual(result[0][0], 3)
self.assertEqual(result[1][0], 4)

def suite():
return unittest.TestSuite(unittest.makeSuite(BackupTests, "Check"))

Expand Down
50 changes: 37 additions & 13 deletions Modules/_sqlite/connection.c
Original file line number Diff line number Diff line change
Expand Up @@ -1456,7 +1456,8 @@ pysqlite_connection_iterdump(pysqlite_Connection* self, PyObject* args)
static PyObject *
pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject* kwds)
{
char* filename;
PyObject* target = NULL;
const char* filename;
int pages = -1;
PyObject* progress = Py_None;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP 7, the * come before the var name, not after the type. It's PyObject *progress. Same for char *name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used the same style as the whole unit, do you think I should follow PEP7 in only new function?
I personally prefer that style, but seemed better to follow the code around... the same reasoning I had with the backup.py (but that's completely new...)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure? :)

+    sqlite3 *bckconn;
+    sqlite3_backup *bckhandle;

I guess muscle memory is hard to beat...

Your remark is valid. I don't care that much about PEP 7, as long as you stick to one one style. It's really up to you, either obey PEP 7 for new code or stick to the existing style.

char* name = "main";
Expand All @@ -1466,10 +1467,16 @@ pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject*
int sleep = 250;
sqlite3 *bckconn;
sqlite3_backup *bckhandle;
static char *keywords[] = {"filename", "pages", "progress", "name", "sleep", NULL};
static char *keywords[] = {"target", "pages", "progress", "name", "sleep", NULL};

if (!PyArg_ParseTupleAndKeywords(args, kwds, "s|$iOsi:backup", keywords,
&filename, &pages, &progress, &name, &sleep)) {
if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|$iOsi:backup", keywords,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See other comment, Format d accepts ints and floats and converts them to double. Multiple the double with 1000 and cast it to int sqlite3_sleep(sleep)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will do!

&target, &pages, &progress, &name, &sleep)) {
goto finally;
}

if (!PyUnicode_Check(target) && !PyObject_TypeCheck(target, &pysqlite_ConnectionType)) {
PyErr_SetString(PyExc_TypeError,
"target argument must be either a string or a Connection instance");
goto finally;
}

Expand All @@ -1482,12 +1489,27 @@ pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject*
pages = -1;
}

Py_BEGIN_ALLOW_THREADS
rc = sqlite3_open(filename, &bckconn);
Py_END_ALLOW_THREADS
if (PyUnicode_Check(target)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth to support path-like objects?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not indeed... but at that point, should I also change the other functions accepting a path name?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On other hand, it may be better to remove support of a file name. The code will be simpler and more reliable if accept only a Connection instance.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhm, yes, slightly simpler indeed, but that would shift the complication to the caller site: in all my use cases I'd use the file name form, so if that gets dropped I'd need to set up another connection instead...
I added the connection form to deal with Cédric's opposite use case, to restore the content of a test database.
I'm -0 on removing such support, but leave the final decision to you.

filename = PyUnicode_AsUTF8(target);
if (filename == NULL) {
goto finally;
}

if (rc != SQLITE_OK) {
goto finally;
Py_BEGIN_ALLOW_THREADS
rc = sqlite3_open(filename, &bckconn);
Py_END_ALLOW_THREADS

if (rc != SQLITE_OK) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pysqlite_connection_init() uses a different code for handling errors. Is there an error in pysqlite_connection_init() or the code here can be simpler?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding of the first paragraph, I would say yes: sqlite3_open() always return a new database connection, with only one exception, OOM.
Do you agree?

If yes, what's the best approach:
a. fix pysqlite_connection_init() now
b. defer the fix to a different PR
?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a separate issue should be open for fixing pysqlite_connection_init(). Since this is a bug fix, it needs to be backported to other branches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #32274: I was wrong about that, the database connection will be released by the pysqlite_connection_dealloc() function in any case.

goto finally;
}
}
else {
if ((pysqlite_Connection*)target == self) {
PyErr_SetString(PyExc_ValueError, "target cannot be the same connection instance");
goto finally;
}
filename = NULL;
bckconn = ((pysqlite_Connection*)target)->db;
}

Py_BEGIN_ALLOW_THREADS
Expand Down Expand Up @@ -1538,16 +1560,18 @@ pysqlite_connection_backup(pysqlite_Connection* self, PyObject* args, PyObject*
}
}

Py_BEGIN_ALLOW_THREADS
sqlite3_close(bckconn);
Py_END_ALLOW_THREADS
if (filename != NULL) {
Py_BEGIN_ALLOW_THREADS
sqlite3_close(bckconn);
Py_END_ALLOW_THREADS
}

if (cberr == 0 && rc == SQLITE_OK) {
Py_INCREF(Py_None);
retval = Py_None;
} else {
/* Remove the probably incomplete/invalid backup */
if (unlink(filename) < 0) {
if (filename != NULL && unlink(filename) < 0) {
/* FIXME: this should probably be chained to the outstanding
exception */
return PyErr_SetFromErrno(PyExc_OSError);
Expand Down