fchollet/namex

Namex: clean up the public namespace of your package

Namex is a simple utility to separate the implementation of your Python package and its public API.

Instead of letting users access every symbol in your .py files, Namex lets you create an allowlist of public symbols. You have fully control of what they are named and under what path they are exposed, without having to change where the code is actually located.

Why use Namex?

1. Explicit control of the namespace your users see. Don't mistakenly expose a private utility because you didn't prefix it with an underscore!
2. Easy refactoring without having to worry where the code actually lives. Want to move something to the legacy/ folder but keep it in the same place in the API? No worries.
3. Easy symbol aliasing without having to manually import an object in a different file and having to route around circular imports.
4. Easy to spot when your public API has changed (e.g. in PR) and easy to setup programmatic control over who can make changes (API owners approval).

Example usage

Step 0: format your codebase

Make sure your codebase is correctly structured. Your file structure should look like this (here we use the keras_tuner package as an example):

keras_tuner_repo/
... setup.py
... keras_tuner/
...... src/  # This is your code
......... __init__.py
......... (etc)

If instead, it currently looks like this:

keras_tuner_repo/
... setup.py
... keras_tuner/
...... __init__.py
...... (etc)

Then you can convert your codebase by calling the following command (your working directory must be keras_tuner_repo/):

namex.convert_codebase(package="keras_tuner", code_directory="src")

Step 1: export your public APIs

Add @export() calls in your code:

import namex

@namex.export(package="keras_tuner", path="keras_tuner.applications.HyperResNet")
class HyperResNet:
    ...

You can also pass a list of paths as path, to make the same symbol visible under various aliases:

@namex.export(
    package="keras_tuner",
    path=[
        "keras_tuner.applications.HyperResNet",
        "keras_tuner.applications.resnet.HyperResNet",
    ])
class HyperResNet:
    ...

Step 2: generate API __init__.py files

Call the following command to generate API export files (your working directory must be keras_tuner_repo/):

namex.generate_api_files(package="keras_tuner", code_directory="src")

Step 3: build your package

You can now build your package as usual -- your users will only see the symbols that you've explicitly exported, e.g. keras_tuner.applications.HyperResNet.

The original symbols are "hidden away" in keras_tuner.src.

Tips

Using a custom API export decorator

While optional, we recommend that each project create its own API export decorator. Here's an example:

class keras_tuner_export(namex.export):
    def __init__(self, path):
        super().__init__(package="keras_tuner", path=path)

This has several advantages:

1. It makes your decorator calls shorter since you won't have to respecify the package name each time.
2. It provides sanity checking for your export paths, since the base decorator will be checking that your export paths all start with the package name (keras_tuner.).

In addition, since you only need to actually run the decorators at build time, you might want to allow users to import your package without having namex installed. You can do this via the following pattern:

try:
    import namex
except ImportError:
    namex = None

if namex:
    class keras_tuner_export(namex.export):
        def __init__(self, path):
            super().__init__(package="keras_tuner", path=path)
else:
    class keras_tuner_export:
        def __init__(self, path):
            pass

        def __call__(self, symbol):
            return symbol

Using a custom build script

Do you find it confusing and annoying that your source directory (e.g. the keras_tuner/ directory) is getting taken over by a bunch of autogenerated __init__.py files and directories, while the actual code your care about is relegated in keras_tuner/src/? Me too.

Don't worry, it's easy to use namex while keeping your regular project file structure. Remember, you only need to run namex when you're building your package. So you can simply write up a custom build script that restructures your codebase on the fly during package building.

We don't provide this functionality out of the box because everyone's build process is different (it's Python we're talking about). Write your own. Here's an example build script that should generalize to many codebases:

import os
import shutil
import namex
import glob

package = "keras_tuner"
build_directory = "build"
if os.path.exists(build_directory):
    raise ValueError(f"Directory already exists: {build_directory}")

# Copy sources (`keras_tuner/` directory and setup files) to build directory
os.mkdir(build_directory)
shutil.copytree(package, os.path.join(build_directory, package))
shutil.copy("setup.py", os.path.join(f"{build_directory}", "setup.py"))
shutil.copy("setup.cfg", os.path.join(f"{build_directory}", "setup.cfg"))
os.chdir(build_directory)

# Restructure the codebase so that source files live in `keras_tuner/src`
namex.convert_codebase(package, code_directory="src")
# Generate API __init__.py files in `keras_tuner/`
namex.generate_api_files(package, code_directory="src", verbose=True)

# Build the package
os.system("python3 -m build")

# Save the dist files generated by the build process
os.chdir("..")
if not os.path.exists("dist"):
    os.mkdir("dist")
for filename in glob.glob(os.path.join(build_directory, "dist", "*.*")):
    shutil.copy(filename, "dist")

# Clean up: remove the build directory (no longer needed)
shutil.rmtree(build_directory)