UP | HOME

[7oct2023] pybind11 transitive library deps

Table of Contents

Experiencing link difficulties with transitive library dependencies.

1. Update

My original invesgation (below) mistaken. See update cmake headeronly deps.

2. Setup (before Update)

  • cmake version 3.25.3
  • pybind11 version ???
  • nix build (see https://github.com:rconybea/xo-nix2) Consequences of nix build:
    • Each package installed to a separate directory – no "common swimming pool" like /usr/lib
    • Implies install directory always distinct from any directory containing build inputs
    • Tends to reveal oversights in toolchain, as we'll see below
  • pybind library (xo-pyreflect) with dependency on a separate library (xo-reflect), that in turn has secondary dependencies (xo-refcnt, xo-indentlog). Note that xo-indentlog is header-only.
  • Expect this cmake script to work:

    find_package(pybind11)
    pybind11_add_module(pyreflect pyreflect.cpp)
    
    find_package(reflect CONFIG REQUIRED)
    target_link_libraries(pyreflect PUBLIC reflect)
    

3. Problem (before Update)

  • Instead, link fails. Link line something like:

    g++ -fPIC ... -o pyreflect.cpython-311-x86_64-linux-gnu.so /path/to/libreflect.so -lrefcnt -lindentlog
    

    Two problems here:

    1. directory containing librefcnt.so isn't on the link line (no -L/path/to/refcnt/dir for example).
    2. libindentlog.so does not exist, since indentlog is header-only
  • Looked into intermediate outputs like lib/cmake/reflectTargets.cmake, excerpt:

    set_target_properties(reflect PROPERTIES
        INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
        INTERFACE_LINK_LIBRARIES "indentlog;refcnt"
    )
    

    It's not obvious how xo_pyreflect can know that indentlog is header-only, while refcnt isn't (though could presumably extract the relevant libdir from find_package() with some work).

4. Workaround (before Update)

  • Recognize that pyreflect link shouldn't need refcnt on the link line, since libreflect.so has a DT_NEEDED entry for it.

    $ readelf -d /path/to/libreflect.so
    
    Dynamic section at offset 0x17860 contains 34 entries:
      Tag        Type                         Name/Value
     0x0000000000000001 (NEEDED)             Shared library: [librefcnt.so.1]
    ...
    
  • When building pyreflect, suppress transitive dependencies For example:

    # xo_cxx.cmake
    macro(xo_pybind11_dependency target dep)
        find_package(${dep} CONFIG REQUIRED)
        set_property(TARGET ${dep} PROPERTY INTERFACE_LINK_LIBRARIES "")
        target_link_libraries(${target} PUBLIC ${dep})
    endmacro()
    

    Then in .cmake for pyreflect, something equivalent to:

    pybind11_add_module(pyreflect pyreflect.cpp)
    xo_pybind11_dependency(pyreflect reflect)
    

Author: Roland Conybeare

Created: 2024-09-08 Sun 18:33

Validate