Repository navigation
Please bump SONAME when removing library symbols. #488
Description
At the moment the library SONAME version is 0. To work well in a Linux environment with programs linked to the shared library, the library should change SONAME version every time a non-backwards compatible change is done. One example like this is the removal of symbols.
While working on updating sqlcipher in Debian, I discovered that several times between the current Debian release 3.4.1 and the latest version 4.5.5 backwards compatibility was broken without changing the SONAME.
To update the SONAME, the -version-info "8:6:8" statement in Makefile.in need to be updated, and the diff between the first and third number need to increase. At the moment the SONAME version is 8-8=0, while in the new version in Debian I plan to move to 9:6:8 to get 9-8=1.
Activity
sjlombardo commentedon Nov 13, 2023
Hello @petterreinholdtsen - you mentioned changes between 3.4.1 and 4.5.5 that affected backwards compatibility, could you provide some more details on what those changes were? We have historically inherited the version-info tag from SQLite, since it defines the public API.
petterreinholdtsen commentedon Nov 13, 2023
hosiet commentedon Sep 30, 2024
Hi,
The release of sqlcipher 4.6.1 comes with another round of API breakage due to the refactoring of crypto.h merging. Some public (aka exported) APIs from
crypto.hare removed for good. To @sjlombardo : you asked about what are the changes and now this is the perfect example and good excuse of bumping SONAME.Please let me know if any SONAME bumping will take place soon, and I can coordinate that in Debian.
sjlombardo commentedon Mar 25, 2025
Hello @hosiet, we've just released SQLCipher 4.7.0 and I wanted to give you a heads up that it contains some modifications and some breaking changes detailed in the release announcement related to source builds. This includes the complete rework of the build system from upstream SQLite. Consistent with the new build system and the upstream project, we will not be adjusting SONAME with our releases. However, the new build system does make it easier for you as a package maintainer to set SONAME for your packages on updates via the
--sonameconfigure parameter. Please see the following posts for more details:https://sqlite.org/src/forumpost/9e1dd7ea
https://sqlite.org/forum/info/2ea56de68de030ed869c
petterreinholdtsen commentedon Mar 25, 2025
If the intention is that every Linux distribution should provide incompatible sqlcipher libraries with different SONAME and ensure that binaries linked on one Linux distribution do not work on others, this seem like a good approach.
hosiet commentedon Mar 25, 2025
For Debian and Ubuntu recently: as the Debian 13 development freeze and release is imminent, I will not pursue releasing Debian 13 with sqlcipher 4.7.0 unless someone suggest otherwise.
For long-term SONAME handling: I see no benefit confusing SONAME of libsqlite with libsqlcipher, or entangling with libsqlite. However if you don't mind downstream packages to choose a random SONAME (e.g. 2 for v4.7.0 here), we can do it.
But there are more important issues. I see from https://www.zetetic.net/blog/2025/03/25/sqlcipher-4.7.0-release/ that it "Renamed default executable and library build outputs from sqlcipher and libsqlcipher to sqlite3 and libsqlite3". Now, how do you expect downstream Linux distributions to ship your library and executable? Distros definitely need a clear message from upstream developers.
sjlombardo commentedon Mar 26, 2025
Hello @hosiet @petterreinholdtsen - I am re-opening this issue so that we can continue to discuss outstanding concerns.
In retrospect, I realize that my comment when closing this issue may have seemed curt or dismissive. That was not my intent. I was at the tail end of release activities, processing a number of GitHub issues and pull requests, and was trying to finish up.
This post is my attempt to rectify that by providing background context and addressing some of these comments (though it might err on the side of being much longer).
I think that the statement above was intended as sarcasm, though if I'm misreading the tone or implication, apologies for misinterpreting. That said, I'm not sure whether it is actually a bad thing for every Linux distribution to provide different SQLCipher libraries. They may be compiled with different settings, use different versions of the software, or have different requirements. In other words, if Debian's packages are expecting the "Debian SQLCipher" flavor, then maybe it is appropriate for them to have a different name then those for RedHat?
To be clear, I'm not stating that is the right course of action. I have no experience as a package maintainer for a major Linux distribution, so this is well outside of my wheelhouse. I am just suggesting that from an outside perspective it doesn't seem like a foregone conclusion.
Further, to provide some insight into our possible non-alignment, the intent of this project is to provide a long term, well maintained fork of SQLite with encryption and security extensions. The primary use case for this library is to support specific applications that want to, on a case-by-case basis, replace a dependency on SQLite with a SQLCipher as an API-compatible library that provides encryption services.
It has not historically been a goal of the SQLCipher project to provide an operating system component or system-level shared library. Of course this ship sailed a long time ago, and SQLCipher is now included in many Linux distributions. We want to try our best to support those situations in a comfortable way, where possible. But it is not our primary goal.
Finally, I wonder how SONAME values are being handled for SQLite in major Linux distributions. To put this in perspective, the SQLite developers, from which SQLCipher inherits the build system, do not think that SONAME should be a concern of the upstream project. SQLite has made ABI changes in the past, yet they've never managed the SONAME in downstream distributions. Recent forum posts indicate that they have no intention of doing so in the future and the new build system seems to prohibit it (i.e. SONAME can only be adjusted at build time). SQLite is used so much more extensively than SQLCipher, so how is this being handled between different Linux distributions?
I've reviewed all of the symbol changes mentioned earlier in this thread. They are all changes to SQLCipher's "private" API. I acknowledge that, due to the nature of our patches on SQLite, there has been "leakage" of intended private symbols up into the public space. Some of this was avoidable in hindsight so we've been trying very hard in recent releases to clean that up by refactoring. In other words, these represent ABI changes, but they aren't public API changes for consuming applications that would exclusively use the SQLite API (i.e.
sqlite3_*) defined insqlite3.h. Perhaps there is some middle ground here.Agreed; there is no benefit to conflating these. SQLCipher provides the same API, but there are some changes that may be ABI compatible within the scope of
sqlite3_*symbols yet are not strictly backward compatible. The classic case is major upgrades to the encryption settings, which typically occur at major version bumps (i.e. SQLCipher 3 to SQLCipher 4, eventually to SQLCipher 5).We don't have any specific objection to downstream packages choosing appropriate or random SONAME values for their needs. As noted above, it may even be desirable to change the SONAME when the ABI and API don't change, e.g. on major version bumps. I'm just not sure we can handle this on the behalf of package maintainers, especially with the aforementioned changes to the build process introduced by SQLite.
Some additional context might be warranted here. For much of SQLCipher's early development, the output of builds were identical to SQLite's (
sqlite3,libsqlite3.so, andsqlite3.h). Unfortunately, many years ago, we made the regretful decision of applying a very large patch from an earlier contributor that changed everything (configure, autoconf, makefiles), etc to usesqlcipher,libsqlcipher.so, andsqlcipher.h.This was a bad call and we have suffered through the repercussions of that change in the interest of backwards compatibility for a decade. It has caused source conflicts on every single SQLite merge, introduced divergence between multi-platform builds, created confusion in documentation, and caused problems for users who want to drop in SQLCipher to replace SQLite without re-building/re-linking.
Recently, SQLite made the major decision of completely rewriting their build system by eliminating autoconf in favor of autosetup in SQLite 3.48.0. This change already introduced some unavoidable breaking changes to the build process. This presented us with an important decision and potential opportunity. Option 1 would have been to take the upstream changes, then reintroduce another big diff on top of them for the new build system. Option 2 was to take advantage of this rare opportunity (i.e. SQLite's breaking changes) to drastically simplify things and re-align with the SQLite outputs. We chose the latter, and I strongly believe it is the right decision at the right time for the project.
We definitely would not expect you to change your package artifacts to use
sqlite3. It would cause conflicts and break the expectations of your dependent packages. Since all of your build outputs were using the name sqlcipher, you want to preserve that, e.g. by renaming the outputs downstream:-Wl,-soname,libsqlcipher.so.XtoLDFLAGS(according to whatever decisions are made about SONAME)libsqlite3.sotolibsqlcipher.sosqlite3tosqlciphersqlite3.htosqlcipher.hThat is probably oversimplifying a bit and I may be missing some steps, but you get the idea. I am hopeful that it should be feasible to accomplish without too much difficulty. We successfully completed a similar process ourselves as part of the migration to 4.7.0 on 20+ of our own commercial and enterprise packages.
I agree that this is a prudent choice. SQLCipher 4.7.0 includes some very nice improvements, but they are not absolutely critical in the context of typical usage on Linux. In light of the major changes here waiting is definitely the conservative move.
Finally, I wanted to mention that we have a lot of respect and appreciation for what you do as maintainers. The majority of SQLCipher 4.7.0 was written and tested on Debian, and we use Ubuntu extensively as well. You have a difficult job as volunteer intermediaries between projects like SQLCipher and end users. Situations like this put you in the unenviable position of making changes to patches, re-testing, etc. We're definitely not trying to make things difficult, and will help support the process as best as possible while working within the constraints of this projects goals.
Please let me know what you think or if you have any feedback.
hosiet commentedon Mar 26, 2025
Hi @sjlombardo , thanks for the detailed elaboration. It really explained the current condition well.
For library and binary names: Being a drop-in replacement to the original libsqlite3 library seems to be one of the major focus of sqlcipher, but that is not the case in Linux distros. I get that the distro-shipped files will keep the
(lib)sqlciphernames. It will involve some downstream patching, but I believe it won't be too difficult. We have enough time to handle it before packaging sqlcipher 4.7.0.My personal opinion on SONAME here: I believe we should get to the very beginning on the purpose of SONAME. SONAME was introduced to avoid binaries from linking to a library with missing or incompatible symbols. Some projects don't care compatibility and bump SONAME with each release, and that is fine. With upstream project staying up to prevent ABI/API breakage, the SONAME should not change, which is also good. Any ABI/API breakage should be considered bugs, which seems to be the case of current sqlcipher and sqlite 3.x up till now as you claimed.
Nevertheless, the setting of SONAME is often recommended by upstream project, or only by the downstream distro in the lack thereof. It is fine if you as upstream do not wish to recommend any SONAME value, and we now get it. Looking into libsqlcipher in different Linux distros, having different libsqlcipher SONAME across different Linux distros is already the status quo. This breaks cross-distro binary compatibility, which actually never existed. That being said, it is still very important to notify downstream (e.g., via release note) if sqlcipher decide to deliberately break library API or ABI in any future versions so that downstream can be aware of it and make necessary changes.
SQLite developers seems to claim that there is no compatibility breakage since SQLite 3.x and thus no SONAME bump in the meanwhile. As I am not deeply involved in sqlite development, I do not know if this claim is true or not. The transition from sqlite 2.x to 3.x indeed happened back in 2000s to handle the breakage.
(Side note: Controlling the symbol visibility and avoiding leaking private symbols specific to libsqlcipher would be important anyway.)
sjlombardo commentedon Mar 31, 2025
Hello @hosiet - I'm glad to hear that you don't think it will be too difficult to adjust names for the downstream packaging for a future update.
With regard to the SONAME bumps, we maintain a CHANGELOG.md file in the repository to track updates. We could make a note in there for a release where we believe there to be a ABI or API change. Something like this:
Would something like that meet the need?
petterreinholdtsen commentedon Mar 31, 2025
Change SONAME version from 0 to 1 with version 4.0.0.
Syed-Shahrukh-OSSRevival commentedon Sep 10, 2025
@hosiet @petterreinholdtsen there is a RFS for sqlcipher 4.10.0
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1113930
hosiet commentedon Nov 18, 2025
We are now at Nov 2025, and things have settled down. For future records:
/usr/lib/*/libsqlcipher.so.2as a symlink to/usr/lib/*/libsqlcipher.so.3.50.4shared library. Users shall see and uselibsqlcipher.so.2. The SONAME will get bumped in the future on incompatible changes./usr/include/sqlcipher/sqlite3.hand/usr/include/sqlcipher/sqlite3ext.has usual. The pkgconfig file is named and shipped assqlcipher.pcas usual with-lsqlcipherused in theLibs:line.We would appreciate explicit records in future ChangeLog whenever a new release is made and any APIs get removed or changed in an incompatible way.
robertkirkman commentedon Aug 18, 2026
my distribution is not even ABI compatible with debian and I can run no Debian binaries, nor can Debian run mine, but despite that, I would still find a library filename that automatically matches between all distributions of the same library convenient and helpful, specifically because it allows people who have memorized which library filename of which library has which version of sqlcipher and/or version of sqlite to calculate whether a library in another distribution whose library filename they are looking at most likely has the same version of sqlcipher and/or sqlite or not. Even without the need or ability to actually copy a build of the library from one environment to another and link it there, a default that is predictable and uniform would avoid confusion and save time for distributors.
For a very clear example, hypothetically, if my distribution, which is not ABI compatible with Debian, had FFmpeg 8, but it had a library with library filename
libavcodec.so.60, while a version of Debian I was simultaneously looking at also had FFmpeg 8, but had a library with library filenamelibavcodec.so.62, that would be extremely confusing for me and it would take me a very long time to figure out what was happening, especially if something I was trying to fix had to do with other behavior inconsistent between Debian and my distribution, which might be related to whatever had caused the library filename to be different in my distribution, but just as easily might not.By using the phrase "and/or" in my example, I emphasize my recognition of sqlcipher's unusual dedication to the soft fork concept of eliminating all possible differences from sqlite, by leaving it open to interpretation whether the sqlcipher library filename number should be different from, or consistently exactly match the library filename number of builds of sqlite from the same source code that was used to build sqlcipher - since for my case, even if sqlcipher isn't 100% exactly ABI-compatible with sqlite, knowing that it was built from code that implemented the same versions of the same sqlite features and symbols originally, despite the sqlcipher-specific changes, is reassuring and time-saving for me.
unfortunately, this is difficult for my distribution Termux due to technical limitations. Unlike the majority of distributions, I rely on the software's
make installcommand generating the correct files the first time and not overriding or corrupting the files that belong to other software. I recognize that other distribution maintainers have often been trained to aim for the goal of installing the software usingmake installinto a temporary directory that does not pollute the environment or conflict with other software, and that is even a good thing, however I hope it can be understood that not every distribution build system was designed exactly the same way, and that to the same extent that each distribution brings the software to a platform distant from Debian in technical architecture, the distribution may lack the build consistency and scalability safeguards that other distributions rely on to help them paper over pitfalls like sqlcipher<->sqlite.Here is a question relevant to the aspect of the problem that impacts me:
There is a line here added on June 2nd, 2021 to the github.com/rusqlite/rusqlite repository that, in a certain edge case, is explicitly searching for a
libsqlcipher.soinstalled prior on the system where the Rust build script is running:https://github.com/rusqlite/rusqlite/blob/bc267609a4547e0c7bc156f424662ebaf05a06a7/libsqlite3-sys/build.rs#L372
however, upstream sqlcipher has here expressed a sentiment that their library should not be a system-level shared library (or at least was not originally intended to be)
Should an issue be opened in the github.com/rusqlite/rusqlite repository instructing them to respect the wishes of sqlcipher by not treating it as a system-level shared library and removing this option from their software? I think probably not, so this situation has probably always been inevitable.
In my opinion, it seems only a matter of time, if it has not already occurred somewhere, before explicit references to
sqlcipher.h,sqlcipherext.h, and/orsqlcipher.pcbecome hardcoded in software written in C or C++, where.hand.pcfiles are sometimes more heavily relied on than they are in Rust software.robertkirkman commentedon Aug 18, 2026
I suggest the implementation of a boolean "default library filename number" build option in sqlcipher that in its default mode, makes no changes, but in its "distribution" mode, automatically provides this symbolic link and sets the library filename to a library filename that tracks the ABI of sqlcipher-specific symbols.
If that were done, it would satisfy both the wish of upstream sqlcipher to "drop in on"
libsqlite.so.3.50.4by default when no build options are toggled, but when both the option to rename the library and the option to change the number at the end of the library filename are toggled on using build arguments, the "distribution" mode would be enabled that would instead provide a library filenamelibsqlcipher.so.2, and the 2 value should be maintained by sqlcipher, tracked in the github.com/sqlcipher/sqlcipher repository, and incremented when sqlcipher implements ABI-breaking changes to sqlcipher-specific symbols.In my opinion, asking Debian to track the sqlcipher-specific symbols themselves and increment their own specific number only when they can tell it's necessary is contributing to the fragmentation, not only because I believe that the developers who have the most accurate knowledge about whether they are changing the ABI of sqlcipher-specific symbols in each update are the developers of sqlcipher, but also because it is not at all clear to other distributors what library filename they should choose unless they spend at least as much time as I just have researching the difference (and lack thereof) between sqlcipher and sqlite. (for transparency and to emphasize the point, I believe I've spent about 100 hours cumulatively researching this so far, spread over a period of 7 months from January 2026 to now - I would not have been able to write at this length on this subject if I had not been studying the situation for some time)
If relevant options were all implemented and documented in sqlcipher's upstream documentation, including an example showing how to enable all of the "distribution" options simultaneously to implement a build of sqlcipher that could be described in the documentation as "not conflicting with sqlite", that would really reduce the amount of research and work that would be necessary for all distributors to provide consistent distributions of sqlcipher.
robertkirkman commentedon Aug 18, 2026
that link does not work for me, but I build
sqlitewithout explicitly specifying any number, yet out comeslibsqlite3.so.3.53.4, just as sqlcipher does by default (once sqlcipher rebases on sqlite version 3.53.4). however, I do notice thatobjdump -p libsqlite3.so.3.53.4 | grep SONAMEdoes not print anything, so that must be what is referred to here; that the library file name contains a number, but in both sqlcipher and sqlite, no SONAME is actually marked inside the library file, only a number in the name, so I will edit my above posts to avoid this confusion. What I am concerned with is mainly the library file name (and other file names) conflicting directly with those of sqlite.robertkirkman commentedon Aug 18, 2026
so if I understand correctly, you duplicated the same renaming logic 20+ times across 20+ different repositories, similarly to the list of distributions I have showed, each one of which has a slightly different variant of roughly similar (but gradually diverging) renaming logic,
but you didn't think at any point, "this isn't DRY"?
I love DRY, which probably shows in my posts, but if DRY is not a primary objective for the sqlcipher project, that would be understandable and would clarify the situation.