Linkers and loaders

Why you should care

Most applications have dependencies, most dependencies are external to and separate from the applicationi, which is therefore incomplete.
All those dependencies are the "environment" of the application, and usually are written and managed by someone else.
Linkers and loaders, which include import and similar tools from Python etc., are those that put together the application so it is complete and can run.
Lots of bugs and maintainability issues are the consequences of insufficient care about how linkers/loaders/importers are configured.

Binary format

Data sections: RO code, RO initialized data, RW code, RW initialized data, RW unitialized data.
Metadata sections: RO sizes of data sections, RO symbol table, RW relocation links, RW external links.
Linker: merge several binary files, merging and thinning the external links table.
Loader: copy into memory a binary file fixing the relocation links.

Typical runtime layout:

|tttttttt|XXXXXXXX|DDDDDDDDDDD|ddddddddddd|_____________| ... |_______|
  tables    code     RO data     RW data     variables          stack

RO symbol table

List of names and addresses that refer to code or data locations in the binary; names of exported addresses. Can be huge, especially if long symbol names.
A fully linked binary does not need a symbol table, so it can be "stripped", but it can still be useful for debugging.
During linking these remain relative addresses, but change.
During loading they become absolute addresses.

RW relocation links

Usually: jump targets, subroutine calls, references to non-stack data locations. Often quite small.
Usually: indirect, so code remains RO.
During linking these remain relative addresses, but change.
During loading they become absolute addresses.

RW external links

Lists of addresses that use external names, and the name they refer to. Can be huge especially if long symbol tables.
Usually: indirect, so code remains RO. Sometimes "trampolines" (RW code).
During linking they turn from names into relative addresses.
During loading the become absolute addresses.

Additional features

Pre-loading: relocate all addresses to some specific absolute address, so no relocation is needed during loading if the binary is loaded at that addresses. This also allows relocation tables to be RO.
Position Independent Code: if CPU architecture allows it, use only relative addresses within a binary, so there is no need for relocation tables of that binary. Slightly less efficient usually.
Static linking: all linking is done before loading. This allows for relocation and external references to be direct instead of indirect.
Dynamic linking: linking is done after loading. Pre-loading and Position Independent Code make that faster.

Advanced features

Link time profiling: add to the indirect relocation or external address some "counting" trampolines.
Link-time checking: verify aspects of code other than addresses, for example type checking of external variables and procedures.
Link-time optimizations: after the program is linked, re-scan the code to optimnize it, for example more code simplification, inlining, loop unrolling, dynamic overload resolution. It helps if linking is done on a virtual code, but it can also be done on actual CPU code.

Linux details

Loader name: /lib64/ld-linux-x86-64.so.2, special path: $LD_PRELOAD.
Static linker name: ld, static linker path: $LD_RUN_PATH.
Dynamic linker name: ld.so (actually same as the loader), dynamic linker path: $LD_LIBRARY_PATH.

Python details

The Python module linker is called import" The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope.
Its paths are the environment variable PYTHON_PATH and the Python variable sys.path.
Python also has specialized module loaders but the details usually do not matter.

The set of executable, shared object, and module paths constitute a Python environment:

$ python3 -m venv /tmp/venv-demo
$ source /tmp/venv-demo/bin/activate

(venv-demo)$ printenv | grep /tmp/venv-demo
VIRTUAL_ENV=/tmp/venv-demo
PATH=/tmp/venv-demo/bin::/usr/local/sbin:/usr/local/bin:/usr/local/share/sbin:/usr/local/share/bin:/usr/sbin:/usr/bin:/usr/bin/X11::/usr/games

(venv-demo)$ type python3
python3 is /tmp/venv-demo/bin/python3

(venv-demo)$ python3 -c "import sys; print(sys.path)"
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/venv-demo/lib/python3.12/site-packages']

Linkers and loaders and environments