Older versions of cmake (e.g. in Ubuntu 18.04) did not provide cmake modules for these libraries that also created a target. Fixes #2393
This makes the code a bit less messy. We barely need if(BUILD_...) guards inside the cmake files, just the root CMakeLists.txt.