sphinx-codeautolink
sphinx-codeautolink makes code examples clickable by inserting links from individual code elements to the corresponding reference documentation. We aim for a minimal setup assuming your examples are already valid Python. Click any names in the example below for a demonstration:
import lib
knight = lib.Knight()
while knight.limbs >= 0:
print(knight.taunt())
knight.scratch()
A directive to create a table of references from code examples to a single
definition is also provided, which can also be integrated with autodoc entries.
For example, .. autolink-examples:: lib.Knight produces:
Integration with intersphinx is seamless:
import numpy as np
from matplotlib import pyplot as plt
x = np.linspace(0, 2 * np.pi, 100)
plt.plot(x, np.sin(x))
plt.show()
Quick start
sphinx-codeautolink can be installed from the following sources:
$ pip install sphinx-codeautolink
# or, alternatively:
$ conda install -c conda-forge sphinx-codeautolink
To enable sphinx-codeautolink, modify the extension list in conf.py.
Note that the extension name uses an underscore rather than a hyphen.
extensions = [
...,
"sphinx_codeautolink",
]
That’s it! Now your code examples are linked. For ways of concatenating multiple examples and setting default import statements among other things, have a look at the Reference documentation.
sphinx-codeautolink elsewhere:
Caveats
For a more thorough explanation, see About.
Only works with HTML documentation
Only processes literal blocks, not inline code
Doesn’t run example code
Parsing and type hint resolving is incomplete
Thanks
The inspiration for sphinx-codeautolink came from seeing similar awesome docs generated by Sphinx-Gallery! Their source was also immensely helpful to read when I was stumbling through Sphinx and docutils. If you have a folder full of example Python scripts you’d like to include in your documentation, you’ll not be disappointed in their offer.
Release notes
These release notes are based on Keep a Changelog. sphinx-codeautolink adheres to Semantic Versioning.
0.15.1 (2024-04-17)
0.15.0 (2023-02-05)
0.14.1 (2023-01-30)
Fix added debug info on failed resolving crashing the build (#134)
0.14.0 (2023-01-28)
Add configuration for mapping inventory locations (
codeautolink_inventory_map) (#131)Improve warning messages to include debugging hints (#131)
Fix AnnAssigns with no links not overwriting values (#133)
0.13.2 (2023-01-26)
Fix parsing IPython blocks that had a leading comment (#130)
0.13.1 (2023-01-16)
Fix IPython block parsing where output is not prefixed with Out (#129)
0.13.0 (2023-01-08)
0.12.1 (2022-11-05)
0.12.0 (2022-09-14)
0.11.0 (2022-06-08)
Support Python 3.10 (#33)
Include the expected location of a type in
codeautolink_warn_on_failed_resolvefor debugging (#106)Define extension environment version for Sphinx (#107)
Merge environments only when the extension is active (#107)
Link arguments and annotated assignment with type hints (#108)
0.10.0 (2022-01-25)
Don’t try to link empty name between two subsequent calls (#96)
Introduce
codeautolink_warn_on_missing_inventoryandcodeautolink_warn_on_failed_resolveto issue additional warnings when linking or resolving an object fails (#97)Support callable classes (#98)
0.9.0 (2022-01-13)
0.8.0 (2021-12-16)
Correctly test for optional types in annotations (#72)
Don’t check for
notranslateCSS class, allowing for additional classes (#75)Allow to specify block parsers as importable references (#76)
Allow parallel builds (#77)
Automatic support for
ipython3code blocks (#79)Correctly produce links for
pycode blocks (#81)
0.7.0 (2021-11-28)
0.6.0 (2021-11-21)
Remove text decoration from produced links (#3)
Turn autodoc integration off by default (#58)
Avoid index error when handling syntax errors (#60)
Construct fully-qualified names more strictly to avoid hiding other issues (#61)
Resolve string annotations in the module scope (#62)
Correctly ensure that return annotations are valid types (#63)
Resolve imported functions to their original location if a documentation entry is not found in the used location (#64)
Fix multi-target assignment and unpacked assignment (#66)
Correctly accept
Noneas a custom block transformer (#67)Document support for
sphinx.ext.doctestblocks (#67)
0.5.1 (2021-11-20)
Fix intersphinx links in documents inside folders (#56)
0.5.0 (2021-11-07)
This release changes an internal API. Please delete the cache file before building documentation.
Link import statements (#42)
Gracefully handle functions that don’t have an annotations dict (#47)
Enable configurations without autodoc (#48)
Support custom code block syntax (#49)
Fix crash on annotation-only assignment (#50)
Fix issue with filenames that have dots (#52)
Correctly remove extension when building non-HTML documentation (#53)
Support searching extra CSS classes for code example matching (#54)
Add configuration for global default concatenation state (#55)
0.4.0 (2021-10-08)
0.3.0 (2021-10-05)
Treat optional types as their underlying type (#21)
Improve
autolink-examplesargument structure and provide an option making a collapsible table (#25)Rename directives for consistency (#27)
Correctly link decorators (#28)
Move cache to Sphinx doctree directory (#29)
Support Python console blocks (#30)
Add configuration for default import statements (#31)
Support star imports (#32)
Accept multiline prefaces (#35)
Fix autodoc injection on one-line docstrings (#36)
0.2.1 (2021-10-01)
Fix type resolving for class instances (#24)
0.2.0 (2021-10-01)
0.1.1 (2021-09-22)
0.1.0 (2021-09-22)
Initial release
Reference
The public API of sphinx-codeautolink consists mostly of the configuration
and directives made available to Sphinx.
The extension is enabled with the name sphinx_codeautolink.
During the build phase, a cache containing code example information is saved
to the Sphinx doctree directory to track references during partial builds.
Configuration
- codeautolink_autodoc_inject
Type:
bool. Inject aautolink-examplestable to the end of all autodoc definitions. Defaults toFalse.
- codeautolink_global_preface
Type:
str. Include aautolink-prefacebefore all blocks. When other prefaces or concatenated sources are used in a block, the global preface is included first and only once.
- codeautolink_concat_default
Type:
bool. Default behavior for code block concatenation (seeautolink-concat). Value corresponds to the “on” and “off” settings in the directive. Defaults toFalse.
- codeautolink_custom_blocks
Type:
Dict[str, None | str | Callable[[str], Tuple[str, str]]]. Register custom parsers for lexers of unknown types of code blocks. They are registered as a dict mapping a block lexer name to a function possibly cleaning up the block content to valid Python syntax. If none is specified, no transformations are applied. A string is interpreted as an importable transformer function. The transformer must return two strings: the code appearing in documentation (often just the original source) and the cleaned Python source code. The transformer must preserve line numbers for correct matching. The transformer may raise a syntax error, which is caught automatically and a corresponding Sphinx warning using subtype “parsing_error” is issued.
- codeautolink_search_css_classes
Type:
List[str]. Extra CSS classes used to search for code examples when matching the final HTML. May contain multiple values separated by spaces as they would be passed tobs4.BeautifulSoup.find_all.
- codeautolink_inventory_map
Type:
Dict[str, str]. Remap the final location of any inventory entry. Useful when objects are imported and documented somewhere else than their original location as advertised by__module__.
- codeautolink_warn_on_missing_inventory
Type:
bool. Issue warning when an object cannot be found in the inventory (autodoc or intersphinx). Defaults toFalse.
- codeautolink_warn_on_failed_resolve
Type:
bool. Issue warning when failing to resolve the canonical location of an object that a code element references. Defaults toFalse.
Directives
- .. autolink-examples:: object
Insert a table containing links to sections that reference
objectin their code examples. The table is removed if it would be empty.Options
- :type: (object's reference type, single value)
The object’s reference type as used in other RST roles, e.g.
:func:`function`.typeis “class” by default, which seems to work for other types as well.
- :collapse: (no value)
Make the table collapsible (using a “details” HTML tag).
- .. autolink-concat:: [mode]
Toggle literal block concatenation. Concatenated code blocks are treated as a continuous source, so that imports and statements in previous blocks affect later blocks. Concatenation is begun at the directive, not applied retroactively. The directive also resets concatenation state. Until this directive is encountered,
codeautolink_concat_defaultis used as the default behavior.mode, if specified, must be one of:“on” - concatenate all blocks in the current file (default value)
“off” - stop concatenation
“section” - concatenate until the next title, then reset to the previous value (“on” or “off”) also resetting concatenation state
- .. autolink-preface:: [code]
Include a hidden preface in the next code block. The next block consumes this directive even if it is not processed (e.g. non-Python blocks) to avoid placement confusion. A multiline preface can be written in the content portion of the directive. Prefaces are included in block concatenation.
- .. autolink-skip:: [level]
Skip sphinx-codeautolink functionality.
level, if specified, must be one of:“next” - next block (default)
“section” - blocks until the next title
“file” - all blocks in the current file
“off” - turn skipping off
If “next” was specified, the following block consumes this directive even if it is not processed (e.g. non-Python blocks) to avoid placement confusion. Skipped blocks are ignored in block concatenation as well, and concatenation is resumed without breaks after skipping is over.
CSS class
The CSS class used in all code block links is sphinx-codeautolink-a.
Cleanup functions
The functions below are usable for cleaning
pycon and ipython code blocks.
They are intended to be used with codeautolink_custom_blocks.
- sphinx_codeautolink.clean_pycon(source)
Clean up Python console syntax to pure Python.
Warning types
Sphinx logging machinery is used to issue warnings during documentation builds.
All warning subtypes below are in the codeautolink.* namespace
and can be ignored with configuring suppress_warnings.
invalid_argument: issued when a directive is used incorrectlyclean_block: issued when cleaning a block fails with aSyntaxErrorparse_block: issued when parsing a block fails with aSyntaxErrorimport_star: issued when a library cannot be imported to determine the names that animport *would introducematch_block: issued when a block cannot be matchedmatch_name: issued when a code snippet cannot be matched
The following warnings are only issued depending on configuration:
missing_inventory: issued when an object cannot be found in the inventoryfailed_resolve: issued when an object’s canonical location in a module cannot be determined
About
sphinx-codeautolink is built with a few major components: code analysis,
import and type hint resolving, and HTML injection.
Code analysis is performed with the builtin ast parsing tool to generate
a set of reference chains to imported modules.
That information is fed to the name resolver, which attempts to match a series
of attributes and calls to the concrete type in question by following
type hints and other information accessible via imports of the library.
If a match is found, a link to the correct reference documentation entry
is injected after the ordinary Sphinx build is finished.
Caveats
Only works with HTML documentation, disabled otherwise. If the extension is off, it silently removes directives that would produce output.
Only processes literal blocks, not inline code. Sphinx has great tools for linking definitions inline, and longer code should be in a block anyway.
Doesn’t run example code. Therefore all possible resolvable types are not found, and the runtime correctness of code cannot be validated. Nonsensical operations that would result in errors at runtime are possible. However, syntax errors are caught while parsing!
Parsing and type hint resolving is incomplete. While all Python syntax is supported, some ambiguous cases might produce unintuitive results or even incorrect results when compared to runtime behavior. We try to err on the side of caution, but here are some of the compromises and limitations:
Only simple assignments of names, attributes and calls to a single name are tracked and used to resolve later values.
Only simple return type hints that consist of a single, possibly optional type are tracked through call and attribute access chains.
Type hints of intersphinx-linked definitions are not necessarily available. Resolving names using type hints is only possible if the package is installed, but simple usage can be tracked via documentation entries alone.
Deleting or assigning to a global variable from an inner scope is not recognised in outer scopes. This is because the value depends on when the function is called, which is not tracked. Additionally, variable values are assumed to be static after leaving an inner scope, i.e. a function referencing a global variable. This is not the case in Python: values may change after the definition and impact the function. Encountering this should be unlikely, because it only occurs in practice when a variable shadows or overwrites an imported module or its part.
These cases are subject to change when the library matures. For more details on the expected failures, see our test suite on GitHub. Please report any unexpected failures!
Sphinx semantics
Clean build
For correct partial builds, code reference information is saved to a file which is updated when parsing new or outdated files. It shouldn’t become outdated, but a clean build can be achieved with sphinx-build -E or by deleting the build directory.
Sphinx cache
A function specified in codeautolink_custom_blocks prevents Sphinx
from caching documentation results. Consider using an importable instead.
For more information, see the discussion in #76.
Parallel build and custom parsers
Locally defined custom block parsers in codeautolink_custom_blocks
cannot be passed to Pickle, which prevents parallel Sphinx builds.
Please consider using an importable function instead.
Copying code blocks
If you feel like code links make copying code a bit more difficult, sphinx-copybutton is a fantastic extension to use. It adds a button to copy an entire code block to the clipboard. So give it a go, perhaps even if you don’t think links make copying harder!
Matching failures
Matching can fail on two levels, for a whole code example or a specific line.
Firstly, failing to match an entire code example is almost always considered
a bug, which you can report on GitHub.
If third-party code blocks are in use, matching may fail because of
inconsistent or unrecognised CSS classes. The class related to the block lexer
name is automatically added to the list of CSS classes that are searched when
matching code examples as highlight-{lexer}.
If the class has another value, codeautolink_search_css_classes
can be used to extend the search. To find out which classes should be added,
build your documentation, locate the code example and use the class of the
outermost div tag. For example:
codeautolink_search_css_classes = ["highlight-default"]
Secondly, matching can fail on a specific line or range of lines. This is often a bug, but the known expected failure cases are presented here:
Multiline statements cannot be matched on Python versions before 3.8. This is because the builtin AST parser does not supply the necessary line number information to construct the proper search range.
Debugging missing links
There are multiple potential reasons for missing links. Here are some common causes and ways to debug and resolve the issue. First, please enable all warning messages found in Configuration to see information about known link misses.
Missing Sphinx inventory entry
Links cannot be resolved, because the documentation entry for a particular object cannot be found in the Sphinx inventory. Likely causes:
The autodoc (or equivalent) entry is missing entirely. To resolve, add the corresponding entry to your documentation.
The object has been relocated and is documented elsewhere, i.e. the
__module__attribute and Sphinx location are out of sync. To resolve, provide the correct location incodeautolink_inventory_map.
Failed link resolving
Determining the canonical location of an object failed. Likely causes:
Missing type hints in function returns or class attributes. To resolve, add appropriate type hints. See Caveats for limitations.
Highly dynamic or runtime-dependent code which is not possible to parse only via imports. To resolve, consider simplifying or filing an issue.
Examples
Short examples about how to achieve certain tasks with sphinx-codeautolink.
Basic use
Once sphinx-codeautolink has been enabled, code in all Python code blocks will be analysed and linked to known reference documentation entries.
import lib
knight = lib.Knight()
while knight.limbs >= 0:
print(knight.taunt())
knight.scratch()
Different import styles are supported, along with all Python syntax.
Star imports might be particularly handy in code examples.
Doctest and console blocks using .. code:: pycon work too.
Including code via literalinclude requires using a
:language: py parameter.
>>> from lib import *
>>> def visit_town(time_spent: int, budget: int) -> Shrubbery:
... return Shrubbery(time_spent > 20, budget > 300)
>>> visit_town(35, 200)
Shrubbery(looks_nice=True, too_expensive=False)
A list of all code examples where a particular definition is used is handy particularly in the reference documentation itself:
Such a table is generated with autolink-examples:
.. autolink-examples:: lib.Knight
Invisible imports
When writing lots of small snippets of code, having the same import at the beginning of every example becomes quite repetitive. The import can be hidden instead.
lib.Knight().taunt()
The previous block is produced with autolink-preface:
.. autolink-preface:: import lib
.. code:: python
lib.Knight().taunt()
A multiline preface can be written in the content portion of the directive:
.. autolink-preface::
import lib
from lib import Knight
A global preface can be set in codeautolink_global_preface
to avoid writing the same imports repeatedly.
Concatenating examples
Examples interlaced with explanations can make for more comprehensible docs.
import lib
knight = lib.Knight()
After explaining some details, the following block may continue where the previous left off.
while knight.limbs >= 0:
print(knight.taunt())
knight.scratch()
This was achieved with autolink-concat:
.. autolink-concat:: section
.. code:: python
import lib
knight = lib.Knight()
.. code:: python
while knight.limbs >= 0:
print(knight.taunt())
knight.scratch()
Now all Python code blocks within the same section will be concatenated.
See autolink-concat for more information and options.
Skipping blocks
If needed, Python blocks can be skipped, resulting in no links for that block and preventing it from being included in further sources with concatenation.
import lib
lib.Knight()
Which is done via autolink-skip:
.. autolink-skip::
.. code:: python
import lib
lib.Knight()
Skipping is supported for single blocks, sections and entire files.
See autolink-skip for more information and options.
Autodoc integration
A backreference table of the code examples that use a definition is handy for example in reference documentation. sphinx-codeautolink provides an autodoc integration for that purpose, which injects the appropriate table to each autodoc entry.
- lib.Knight.scratch(self)
Scratch the knight.
- Return type:
None
To enable the integration, set codeautolink_autodoc_inject.
If you’d like to place the directive manually, implement a small
Sphinx extension
with a listener for the autodoc-process-docstring
event.
An object type “class” seems to work for other types as well.
def process_docstring(app, what, name, obj, options, lines):
lines.append("")
lines.append(".. autolink-examples:: " + name)
lines.append(" :type: class")
lines.append(" :collapse:")
def setup(app):
app.connect("autodoc-process-docstring", process_docstring)
Intersphinx integration
When writing code examples that use builtins or other libraries, intersphinx
can be used to enable links to documentation on other Sphinx-generated sites.
Intersphinx is integrated seamlessly, linking objects as long as the correct
intersphinx_mapping is defined.
if __debug__:
print(...)
else:
raise RuntimeError(f"Could not debug!")
import numpy as np
from matplotlib import pyplot as plt
x = np.linspace(0, 2 * np.pi, 100)
plt.plot(x, np.sin(x))
plt.show()
Reference tables across intersphinx work too:
It seems that the reference type information is more important
for Sphinx when dealing with external modules,
likely because the references cannot be resolved dynamically.
Please specify a type in autolink-examples:
.. autolink-examples:: numpy.linspace
:type: func
Doctest code blocks
Using the sphinx.ext.doctest extension for code examples requires
setting up codeautolink_custom_blocks.
To help in that, clean_pycon
is provided as a ready-made transformer.
extensions = [
...,
"sphinx.ext.doctest",
]
codeautolink_custom_blocks = {
"python3": None,
"pycon3": "sphinx_codeautolink.clean_pycon",
}
doctest and testcode blocks now work as expected.
However, any test setup and teardown code is not taken into account.
>>> import lib
>>> lib.Knight()
IPython blocks and notebooks
Code blocks using ipython or ipython3 lexers are supported by default.
The function clean_ipython() is used to handle
IPython-specific syntax like magic functions and console prefixes.
%reset
import lib
lib.Knight().taunt()
IPython’s .. ipython:: directive is also supported:
In [1]: import lib
In [2]: lib.Knight().taunt()
Out[2]: 'None shall pass!'
They are also useful for integrating Jupyter notebooks and similar source code,
which is possible with separate Sphinx extensions like nbsphinx or MyST-NB.
Enabling codeautolink_concat_default with notebooks is recommended.
IPython processing is enabled if the ipython library is installed.
It is also included in the ipython extra of sphinx-codeautolink.
pip install sphinx-codeautolink[ipython]
Third-party code blocks
Third-party code blocks that use the basic Pygments lexers for Python
are supported out of the box.
The example below uses matplotlib’s plot_directive
to automatically run the code and include a plot in the documentation:
import numpy as np
from matplotlib import pyplot as plt
x = np.linspace(0, 2 * np.pi, 100)
plt.plot(x, np.cos(x))
Code blocks with special highlighting or syntax are supported
with custom transformer functions in codeautolink_custom_blocks.
For example, a transformer could be implemented as follows:
def transform(source):
"""Ignore lines starting with `!--`."""
lines = []
for line in source.split("\n"):
if line.strip().startswith("!--"):
line = ""
lines.append(line)
return source, "\n".join(lines)
codeautolink_custom_blocks = {"python": transform}
Sometimes links with third-party code blocks are broken. See About for a potential solution.
Custom link styles
If you want a specific style to be applied to code block links,
you may add your own CSS file to the Sphinx build.
All code block links use the sphinx-codeautolink-a class.
For example, you can add dotted lines to all links and change the hover colour:
# conf.py
html_static_path = ['static']
html_css_files = ['custom.css']
/* static/custom.css */
.sphinx-codeautolink-a{
border-bottom-color: rgb(0, 0, 0);
border-bottom-style: dotted;
border-bottom-width: 1px;
}
.sphinx-codeautolink-a:hover{
color: rgb(255, 139, 139);
}
Example library
This document contains the reference documentation of a dummy library used in sphinx-codeautolink’s documentation. Besides providing valid hyperlink targets, it also demonstrates the default autodoc integration.