Visitar URL original
Allow incremental I/O to blobs in sqlite3 · Issue #69093 · python/cpython · GitHub
Skip to content

Allow incremental I/O to blobs in sqlite3 #69093

Closed
Closed
Allow incremental I/O to blobs in sqlite3#69093
Labels
extension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

Description

@jimminter
jimminter
mannequin
opened on Aug 21, 2015
BPO 24905
Nosy @berkerpeksag, @serhiy-storchaka, @palaviv, @eamanu, @erlend-aasland, @nightlark
PRs
  • bpo-24905: Support BLOB incremental I/O in sqlite module #271
  • gh-69093: Add incremental I/O to blobs support in sqlite3 #30356
  • gh-69093: Support basic incremental I/O to blobs in sqlite3 #30680
    		•
    Files
  • blob.patch
  • blob2.patch
  • blob3.patch
  • blob4.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2015-08-21.08:05:21.911>
labels = ['extension-modules', 'type-feature', '3.8']
title = 'Allow incremental I/O to blobs in sqlite3'
updated_at = <Date 2022-01-20.12:42:10.486>
user = 'https://bugs.python.org/jimminter'

    bugs.python.org fields:

    activity = <Date 2022-01-20.12:42:10.486>
actor = 'erlendaasland'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2015-08-21.08:05:21.911>
creator = 'jim_minter'
dependencies = []
files = ['41798', '44142', '46531', '46549']
hgrepos = []
issue_num = 24905
keywords = ['patch']
message_count = 28.0
messages = ['248945', '259262', '259520', '272873', '272883', '273052', '286957', '287040', '287065', '287081', '287162', '288515', '288517', '288519', '288724', '288725', '297205', '315450', '329457', '372225', '399813', '409494', '409580', '409584', '409591', '409592', '410920', '411021']
nosy_count = 9.0
nosy_names = ['ghaering', 'berker.peksag', 'serhiy.storchaka', 'jim_minter', 'waldhol', 'palaviv', 'eamanu', 'erlendaasland', 'rmast']
pr_nums = ['271', '30356', '30680']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24905'
versions = ['Python 3.8']

    Activity

    1. jimminter commented on Aug 21, 2015

      @jimminter
      jimmintermannequin
      on Aug 21, 2015
      MannequinAuthor

      SQLite supports incremental I/O to blobs, i.e. the capability to stream reads and writes to blobs without having to load the entire blob into memory first. See https://www.sqlite.org/c3ref/blob_open.html for more details on the C API.

      It'd be nice if it were possible to do this in Python using sqlite3 (it is already possible with apsw).

    2. jimminter
      added
      type-featureA feature request or enhancement
      on Aug 21, 2015
    3. SilentGhost
      added
      extension-modulesC modules in the Modules dir
      on Jan 16, 2016

    4. palaviv commented on Jan 30, 2016

      @palaviv
      palavivmannequin
      on Jan 30, 2016
      Mannequin

      I opened a pull request for blob support in the pysqlite github repository:
      ghaering/pysqlite#93

      I will do the needed changes for python3 and will post a patch soon.

    5. palaviv commented on Feb 3, 2016

      @palaviv
      palavivmannequin
      on Feb 3, 2016
      Mannequin

      I did the needed changes for the pull request at pysqlite for porting to python3. I will continue updating the patch if there will be changes in pysqlite and vice versa.

    6. palaviv commented on Aug 16, 2016

      @palaviv
      palavivmannequin
      on Aug 16, 2016
      Mannequin

      Pinging as mentioned in the devguide.

    7. serhiy-storchaka commented on Aug 16, 2016

      @serhiy-storchaka
      serhiy-storchaka
      on Aug 16, 2016
      Member

      Added comments on Rietveld. The patch contains typos and violates PEP-7 and PEP-12. And there are questions about API.

    8. palaviv commented on Aug 18, 2016

      @palaviv
      palavivmannequin
      on Aug 18, 2016
      Mannequin

      Thanks for the review Serhiy. Attached is the updated patch after the changes.

    9. palaviv commented on Feb 4, 2017

      @palaviv
      palavivmannequin
      on Feb 4, 2017
      Mannequin

      Pinging again. I think this would be a great enhancement to the sqlite module.

    10. serhiy-storchaka commented on Feb 5, 2017

      @serhiy-storchaka
      serhiy-storchaka
      on Feb 5, 2017
      Member

      Sorry Aviv, I just forget about this issue.

      Added new comments on Rietveld. Many lines in sqlite3.rst still are too long.

      It would be worth to ask other developers about wanted interface.

    11. serhiy-storchaka
      added
      3.7 (EOL)end of life
      on Feb 5, 2017

    12. palaviv commented on Feb 5, 2017

      @palaviv
      palavivmannequin
      on Feb 5, 2017
      Mannequin

      Thanks for the CR Serhiy. Attached is a new patch after the fixes from the CR.

      What other developers should I ask? The interface is file like and is the same as apsw.

    13. serhiy-storchaka commented on Feb 6, 2017

      @serhiy-storchaka
      serhiy-storchaka
      on Feb 6, 2017
      Member

      apsw has different order of arguments for blobopen(), all arguments are mandatory, and the parameter for read/write mode is called "writeable" rather than "readonly". This part of API needs to be discussed. Ask for discussion on mailing lists: Python-Ideas and maybe Python-List.

    14. palaviv commented on Feb 6, 2017

      @palaviv
      palavivmannequin
      on Feb 6, 2017
      Mannequin

      Uploading patch after fixes from berker CR.

      The blob_open API can can have the following options:

      1. The table, column and row must be mandatory parameters.
      2. The read/write permissions can have the following options:
        a. No default (mandatory parameter).
        b. default read-only
        c. default write-only
      3. The dbname can be without a default of "main" and then it will be a mandatory parameter.

      I don't think that there is enough differences between the possible API's to justify sending a message to the mailing lists.

    15. palaviv commented on Feb 24, 2017

      @palaviv
      palavivmannequin
      on Feb 24, 2017
      Mannequin

      I opened a PR in github. I tagged some other developers if that will not start a discussion on the API I will post in the python-ideas.

    16. 33 remaining items

    17. JelleZijlstra
      added 3 commits that reference this issue on Apr 15, 2022

      gh-69093: Don't allow instantiation of sqlite3.Blob objects (GH-91570)

      d104f4d

      gh-69093: improve sqlite3.Connection.blobopen() error handling (GH-91571

      c06a4ff

      gh-69093: Add context manager support to sqlite3.Blob (GH-91562)

      a861756
    18. JelleZijlstra
      added 2 commits that reference this issue on Apr 16, 2022

      gh-69093: sqlite3 blob doc amendments (GH-91561)

      95573ad

      gh-69093: Add indexing and slicing support to sqlite3.Blob (#91599)

      29afb7d

    19. erlend-aasland commented on Apr 22, 2022

      @erlend-aasland
      erlend-aasland
      on Apr 22, 2022
      Contributor

      Incremental I/O for blobs is now supported in the sqlite3 extension module in the upcoming Python 3.11 release. Thanks a lot to @palaviv, @JelleZijlstra, and everyone else who contributed by commenting and reviewing.

      IMO, the implementation can benefit from further tweaks, but let's do that in separate issues if needed.

    20. erlend-aasland
      closed this as completedon Apr 22, 2022

    21. erlend-aasland commented on Apr 22, 2022

      @erlend-aasland
      erlend-aasland
      on Apr 22, 2022
      Contributor

      cc. @eamanu, @nightlark

    22. erlend-aasland
      removed
      3.8 (EOL)end of life
      on Apr 22, 2022
    23. tiran
      added 2 commits that reference this issue on May 31, 2022

      pythongh-69093: Fix Setup.local.in rule for _sqlite3

      51b90e4

      gh-69093: Fix Setup.local.in rule for _sqlite3 (GH-93380)

      8a5e3c2
    24. miss-islington
      added 2 commits that reference this issue on May 31, 2022

      pythongh-69093: Fix Setup.local.in rule for _sqlite3 (pythonGH-93380)

      a30c855

      gh-69093: Fix Setup.local.in rule for _sqlite3 (GH-93380)

      29bfabe
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      extension-modulesC modules in the Modules dirtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions