nvbn/py-backwards

Py-backwards

Python to python compiler that allows you to use some Python 3.6 features in older versions, you can try it in the online demo.

Requires Python 3.3+ to run, can compile down to 2.7.

Supported features

Target 3.5:

• formatted string literals like f'hi {x}'
• variables annotations like x: int = 10 and x: int
• underscores in numeric literals like 1_000_000 (works automatically)

Target 3.4:

• starred unpacking like [*range(1, 5), *range(10, 15)] and print(*[1, 2], 3, *[4, 5])
• dict unpacking like {1: 2, **{3: 4}}

Target 3.3:

• import pathlib2 instead of pathlib

Target 3.2:

• yield from
• return from generator

Target 2.7:

• functions annotations like def fn(a: int) -> str
• imports from __future__
• super without arguments
• classes without base like class A: pass
• imports from six moves
• metaclass
• string/unicode literals (works automatically)
• str to unicode
• define encoding (not transformer)
• dbm => anydbm and dbm.ndbm => dbm

For example, if you have some python 3.6 code, like:

def returning_range(x: int):
    yield from range(x)
    return x


def x_printer(x):
    val: int
    val = yield from returning_range(x)
    print(f'val {val}')


def formatter(x: int) -> dict:
    items: list = [*x_printer(x), x]
    print(*items, *items)
    return {'items': items}


result = {'x': 10, **formatter(10)}
print(result)


class NumberManager:
    def ten(self):
        return 10

    @classmethod
    def eleven(cls):
        return 11


class ImportantNumberManager(NumberManager):
    def ten(self):
        return super().ten()

    @classmethod
    def eleven(cls):
        return super().eleven()


print(ImportantNumberManager().ten())
print(ImportantNumberManager.eleven())

You can compile it for python 2.7 with:

➜ py-backwards -i input.py -o output.py -t 2.7

Got some ugly code and ensure that it works:

➜ python3.6 input.py
val 10
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
{'x': 10, 'items': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}
10
11
➜ python2 output.py                           
val 10
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
{'x': 10, 'items': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}
10
11

Usage

Installation:

pip install py-backwards

Compile code:

py-backwards -i src -o compiled -t 2.7

Testing compiled code

For testing compiled code with each supported python version you can use tox and tox-py-backwards. You need to install them:

pip install tox tox-py-backwards

Fill tox.ini (py_backwards = true in testenv section enables py-backwards), like:

[tox]
envlist = py27,py33,py34,py35,py36

[testenv]
deps = pytest
commands = py.test
py_backwards = true

And run tests with:

tox

Distributing compiled code

For distributing packages compiled with py-backwards you can use py-backwards-packager. Install it with:

pip install py-backwards-packager

And change setup import in setup.py to:

try:
    from py_backwards_packager import setup
except ImportError:
    from setuptools import setup

By default all targets enabled, but you can limit them with:

setup(...,
      py_backwards_targets=['2.7', '3.3'])

After that your code will be automatically compiled on bdist and bdist_wheel.

Running on systems without Python 3.3+

You can use docker for running py-backwards on systems without Python 3.3+, for example for testing on travis-ci with Python 2.7:

docker run -v $(pwd):/data/ nvbn/py-backwards -i example -o out -t 2.7

Development

Setup:

pip install .
python setup.py develop
pip install -r requirements.txt

Run tests:

 py.test -vvvv --capture=sys --enable-functional

Run tests on systems without docker:

 py.test -vvvv

Writing code transformers

First of all, you need to inherit from BaseTransformer, BaseNodeTransformer (if you want to use NodeTransfromer interface), or BaseImportRewrite (if you want just to change import).

If you use BaseTransformer, override class method def transform(cls, tree: ast.AST) -> TransformationResult, like:

from ..types import TransformationResult
from .base import BaseTransformer


class MyTransformer(BaseTransformer):
    @classmethod
    def transform(cls, tree: ast.AST) -> TransformationResult:
        return TransformationResult(tree=tree,
                                    tree_changed=True,
                                    dependencies=[])

If you use BaseNodeTransformer, override visit_* methods, for simplification this class have a whole tree in self._tree, you should also set self._tree_changed = True if the tree was changed:

from .base import BaseNodeTransformer


class MyTransformer(BaseNodeTransformer):
    dependencies = []  # additional dependencies

    def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.FunctionDef:
        self._tree_changed = True  # Mark that transformer changed tree
        return self.generic_visit(node)

If you use BaseImportRewrite, just override rewrites, like:

from .base import BaseImportRewrite


class MyTransformer(BaseImportRewrite):
    dependencies = ['pathlib2']

    rewrites = [('pathlib', 'pathlib2')]

After that you need to add your transformer to transformers.__init__.transformers.

It's hard to write code in AST, because of that we have snippets:

from ..utils.snippet import snippet, let, extend


@snippet
def my_snippet(class_name, class_body):
    class class_name:  # will be replaced with `class_name`
        extend(class_body)  # body of the class will be extended with `class_body`
        
        def fn(self):
            let(x)  # x will be replaced everywhere with unique name, like `_py_backwards_x_1`
            x = 10
            return x

And you can easily get content of snippet with:

my_snippet.get_body(class_name='MyClass',
                    class_body=[ast.Expr(...), ...])

Also please look at tree utils, it contains such useful functions like find, get_parent and etc.

Related projects

• py-backwards-astunparse
• tox-py-backwards
• py-backwards-packager
• pytest-docker-pexpect

License MIT