DLL Hell on Linux, but not Solaris; also: POSIX file locks are insane
There was a very interesting thread recently on the MIT kerberos list. It involves DLL hell on Linux. Tom Yu and I continued this on #krbdev for a while. I’ve long been a fan of the Solaris RTLD, and the linker aliens (the engineers that work on it). Features like direct binding and RTLD_GROUP have been incredibly useful at preventing all sorts of linking issues. But it was a surprise to me that the Solaris RTLD had changed in a way that greatly reduces the risk of DLL hell; I should have been aware of this change, but I wasn’t, probably precisely because it had been made.
I’m going to plug a few things about the Solaris link-editor and RTLD:
- Direct binding rocks (see also, and much else). Direct binding improves performance, much like the Linux linker’s pre-linking, though not quite as much. But more than anything it makes dynamic linking have semantics that are much closer to static linking as far as accidental symbol interposition goes: accidental symbol interposition can’t happen. Pre-linking doesn’t get you that. This is *huge*. IMO -B direct needs to be made the default, at least some day.
- The search-for-SONAMEs-in-RPATH-first-then-the-link-map feature that I just learned of is simply awesome. I do think it violates expectations in some ways, but it’s a violation that significantly improves the system.
- RTLD_GROUP, and generally all the RTLD_* flags that Solaris has but Linux lacks (this has varied over time, as Linux has adopted some Solaris-isms).
- The documentation on the Solaris linker is simply one of the best things about it. I first happened upon the Solaris linker guide in 1999 or 2000. It has been an incredibly valuable tool to me.
- The Solaris linker source from the last OpenSolaris build is available, and relatively easy to understand.
Now, DLL Hell arises (at least on Unix systems) when you have multiple versions of the same software installed and somehow two or more of those versions end up getting referenced and/or loaded into the same process. Typically this happens when there are pluggable frameworks involved, such as PAM (but also name services, PKCS#11, and many other frameworks). A typical example might be an sshd that uses the GSS-API, and through it, Kerberos, while also using PAM, and through it a PAM Kerberos module that uses a different Kerberos library version. If only one version of the affected dependency gets loaded and it’s compatible with both of its dependents then all will be fine, else all hell breaks loose. If both versions get loaded and each dependent links with the correct dependency, then most likely everything works out.
And here we come to the part of this post that is about POSIX file locking. What might go wrong if two versions of a library are loaded and used in the same process? Well, there’s not a lot of obvious things that might go wrong — after all, each version will have its own global variable namespace and since the typical DLL Hell cases by their nature result n “never the twain shall meet”, it stands to reason that having two versions of a library loaded in the same process should be no different than having two different implementations of the same feature loaded in the same process.
But here’s the rub: POSIX file locks are insane. And if two shared objects in one process use POSIX file locking to synchronize access to the same files without knowing about each other, then they’ll step all over each other’s toes, likely resulting in corruption of some sort. This is because the first close(2) of a file descriptor drops all POSIX file locks held by the process on that file, even when those locks were acquired on a different file descriptor that references the same file. Insane, right?
Take a look at the way SQLite3 handles POSIX file locking. Pretty gross.
Now suppose you have two versions of libsqlite3 loaded in the same process. If they access different DB files (and journals, and WAL files) then all will be fine. But if they should try to access the same DB files concurrently then each instance of libsqlite3 will fail to know about the other’s file locks! The result may well be DB corruption (I haven’t tried it). Note that the SQLite3 developers have at times recommended static linking… (sorry, no link; search the sqlite-users list archive)…
Now, suppose that the Kerberos library were to use POSIX file locking for synchronizing access to, say, credentials cache files (ccache), keytab files, replay cache files (rcache), etcetera… (it does, actually). Or OpenSSL, perhaps, might use POSIX file locks to control write access to trust anchors (unlikely to change often), pre-shared certs, and private keys, say (OpenSSL does no such thing, I checked, I’m asking you to imagine that it did just because OpenSSL has been frequently involved in DLL Hell in my experience). Yeah, that could be bad.
So the biggest source of problems with having multiple instances of a library in the same process, on Unix and Unix-like systems, is likely going to be POSIX file locking.
My friend Simo tells me that POSIX file locking should just be changed to not drop all locks on first close, that it should drop only those locks that were obtained on the file descriptor being closed, and that this should be done unilaterally (and that it can already be done on Linux with a mount option). I suspect that there are a handful of applications that depend on first-close-drops-all-locks, but not enough to make such a change too risky, so I’m coming around to his view on this. Particularly if the RTLDs make it easier to survive DLL Hell, as Solaris’ does.
I plead with the Linux RTLD powers that be: please add RTLD_GROUP, -Bdirect, and search the rpath before the link map like Solaris does.
As for POSIX file locks, we should start a conversation about fixing them. Replacing POSIX FIle locks with something similar but not broken is probably a non-starter, but I’m open to arguments to the contrary.
Comments?