MagicStack/MagicPython

Magic Python

This is a package with preferences and syntax highlighter for cutting edge Python 3, although Python 2 is well supported, too. The syntax is compatible with Sublime Text, Atom and Visual Studio Code. It is meant to be a drop-in replacement for the default Python package.

Attention VSCode users: MagicPython is used as the default Python highlighter in Visual Studio Code. Don't install it unless you want or need the cutting edge version of it. You will likely see no difference because you're already using MagicPython.

MagicPython correctly highlights all Python 3 syntax features, including type annotations, f-strings and regular expressions. It is built from scratch for robustness with an extensive test suite.

Type hints in comments require support by the color scheme. The one used in the screenshot is Chromodynamics.

Installation Instructions

This is meant to be a drop-in replacement for the default Python package.

In Atom, install the MagicPython package and disable the built-in language-python package.

In Sublime Text, install MagicPython package via "Package Control" and disable the built-in Python package (using Package Control -> Disable Package, or directly by adding "Python" to "ignored_packages" in the settings file).

In VSCode, starting with version 0.10.1, install MagicPython with Install Extension command.

Alternatively, the package can be installed manually in all editors:

• copy the MagicPython package into the Sublime/Atom/VSCode user packages directory;
• disable Python package;
• enjoy.

Changes and Improvements

The main motivation behind this package was the difficulty of using modern Python with other common syntax highlighters. They do a good job of the 90% of the language, but then fail on the nuances of some very useful, but often overlooked features. Function annotations tend to freak out the highlighters in various ways. Newly introduced keywords and magic methods are slow to be integrated. Another issue is string highlighting, where all raw strings are often assumed to be regular expressions or special markup used by .format is completely ignored. Bumping into all of these issues on daily basis eventually led to the creation of this package.

Overall, the central idea is that it should be easy to notice something odd or special about the code. Odd or special doesn't necessarily mean incorrect, but certainly worth the explicit attention.

Annotations

Annotations should not break the highlighting. They should be no more difficult to read at a glance than other code or comments.

A typical case is having a string annotation that spans several lines by using implicit string concatenation. Multi-line strings are suboptimal for use in annotations as it may be highly undesirable to have odd indentation and extra whitespace in the annotation string. Of course, there is no problem using line continuation or even having comments mixed in with implicit string concatenation. All of these will be highlighted as you'd expect in any other place in the code.

def some_func(a,                        # nothing fancy here, yet

              b: 'Annotation: '         # implicitly
                 '"foo" for Foo, '      # concatenated
                 '"bar" for Bar, '      # annotation
                 '"other" otherwise'='otherwise'):

A more advanced use case for annotations is to actually have complex expressions in them, such as lambda functions, tuples, lists, dicts, sets, comprehensions. Admittedly, all of these might be less frequently used, but when they are, you can rely on them being highlighted normally in all their glorious details.

# no reason why this should cause the highlighter to break
#
def some_func(a:
                 # annotation starts here
                 lambda x=None:
                    {key: val
                        for key, val in
                            (x if x is not None else [])
                    }
                    # annotation ends here and below is the default for 'a'
                    =42):

Result annotations are handled as any other expression would be. No reason to worry that the body of the function would look messed up.

# no reason why this should cause the highlighter to break
#
def some_func() -> {
                     'Some',        # comments
                     'valid',       # are
                     'expression'   # good
                   }:

Strings

Strings are used in many different ways for processing and presenting data. Making the highlighter more friendly towards these uses can help you concentrate your efforts on what matters rather than visual parsing.

Raw strings are often interpreted as regular expressions. This is a bit of a problem, because depending on the application this may actually not be the most common case. Raw strings can simply be the input to some other processor, in which case regexp-specific highlighting is really hindering the overall readability. MagicPython follows a convention that a lower-case r prefix means a regexp string, but an upper-case R prefix means just a raw string with no special regexp semantics. This convention holds true for all of the legal combinations of prefixes. As always the syntax is biased towards Python 3, thus it will mark Python-2-only prefixes (i.e. variations of ur) as deprecated.

String formatting is often only supported for '%-style formatting', however, the recommended and more readable syntax used by .format is ignored. The benefits of using simple and readable {key} replacement fields are hindered by the fact that in a complex or long string expression it may not be easily apparent what parameters will actually be needed by .format. This is why MagicPython highlights both kinds of string formatting syntax within the appropriate string types (bytes don't have a .format method in Python 3, so they don't get the special highlighting for it, raw and unicode strings do). Additionally, the highlighter also validates that the formatting is following the correct syntax. It can help noticing an error in complex formatting expressions early.

Python 3.6 f-strings are supported in both the raw and regular flavors. The support for them is somewhat more powerful than what can be done in regular strings with .format, because the f-string spec allows to highlight them with less ambiguity.

Numeric literals

Most numbers are just regular decimal constants, but any time that octal, binary, hexadecimal or complex numbers are used it's worth noting that they are of a special type. Highlighting of Python 2 'L' integers is also supported.

Underscores in numeric literals are also supported (PEP 515, introduced in Python 3.6):

100_000_000_000         0b_1110_0101        0x_FF_12_A0_99

Python 3.5 features

New keywords async and await are properly highlighted. Currently, these keywords are new and are not yet reserved, so the Python interpreter will allow using them as variable names. However, async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7. It is very important that the highlighter shows their proper status when they are used as function parameter names, as that could otherwise be unnoticed and lead to very messy debugging down the road.

Built-ins and Magic Methods

Various built-in types, classes, functions, as well as magic methods are all highlighted. Specifically, they are highlighted when they appear as names in user definitions. Although it is not an error to have classes and functions that mask the built-ins, it is certainly worth drawing attention to, so that masking becomes a deliberate rather than accidental act.

Highlighting built-ins in class inheritance list makes it slightly more obvious where standard classes are extended. It is also easier to notice some typos (have you ever typed Excepiton?) a little earlier.

Parameters and Arguments

MagicPython highlights keywords when they are used as parameter/argument names. This was mentioned for the case of async and await, but it holds true for all other keywords. Although the Python interpreter will produce an appropriate error message when reserved keywords are used as identifier names, it's still worth showing them early, to spare even this small debugging effort.

Development

You need npm and node.js to work on MagicPython.

• clone the repository
• run make to build the local development environment
• run make release to build the syntax packages for Sublime Text and Atom (running make test also generates the "release" packages)

Please note that we have some unit tests for the syntax scoping. We will be expanding and updating our test corpus. This allows us to trust that tiny inconsistencies will not easily creep in as we update the syntax and fix bugs. Use make test to run the tests regularly while updating the syntax spec. Currently the test files have two parts to them, separated by 3 empty newlines: the code to be scoped and the spec that the result must match.

If you intend to submit a pull request, please follow the following guidelines:

• keep code lines under 80 characters in length, it improves readability

• please do use multi-line regular expressions for any non-trivial cases like:

  • the regexp contains a mix of escaped and unescaped braces/parentheses
  • the regexp has several | in it
  • the regexp has several pairs of parentheses, especially nested ones
  • or the regexp is simply longer than 35 characters

• always run make test to ensure that your changes didn't have unexpected side effects

• update unit tests and add new ones if needed, keeping the test cases short whenever possible

Multiple scopes

It is sometimes necessary to assign multiple scopes to the same matched group. It is very important to keep in mind that the order of these scopes is apparently treated as significant by the engines processing the grammar specs. However, it is equally important to know that different specification formats seem to have different order of importance (most important first vs. last). Since we try to create grammar that can be compiled into several different formats, we must chose one convention and then translate it when necessary during compilation step. Our convention is therefore that most important scope goes first.

Color Scheme

If you want to write your own color scheme for MagicPython you can find a list of all the scopes that we use in misc/scopes. The file is automatically generated based on the syntax grammar, so it is always up-to-date and exhaustive.