######### Pyflyby #########

.. image:: https://badge.fury.io/py/pyflyby.svg :target: https://pypi.org/project/pyflyby/

.. image:: https://travis-ci.org/deshaw/pyflyby.png?branch=master :target: https://travis-ci.org/deshaw/pyflyby

Pyflyby is a set of Python programming productivity tools for Python.

For command-line interaction:

• py: command-line multitool

For IPython interaction:

• autoimporter: automatically imports symbols when needed.

For editing python source code:

• tidy-imports: adds missing 'import's, removes unused 'import's, and also reformats import blocks.
• find-import: prints to stdout how to import a particular symbol.
• reformat-imports: reformats import blocks
• collect-imports: prints out all the imports in a given set of files.
• collect-exports: prints out definitions in a given set of modules, in the form of import statements.
• transform-imports: renames imported modules/functions.

Learn more about Pyflyby <https://www.deshaw.com/library/desco-quansight-improving-jupyter-efficiency>_ in this blog post.

Installation

.. code:: bash

$ pip install pyflyby

This creates an alias for your ipython named py which runs the pyflyby plug internally. pyflyby has a dependency on ipython, if it isn't already installed do install it with:

.. code:: bash

$ pip install ipython

Quick start: Autoimporter + IPython

.. code:: bash

$ py In [1]: re.search("[a-z]+", "....hello...").group(0) [PYFLYBY] import re Out[1]: 'hello'

In [2]: chisqprob(arange(5), 2) [PYFLYBY] from numpy import arange [PYFLYBY] from scipy.stats import chisqprob Out[2]: [ 1. 0.6065 0.3679 0.2231 0.1353]

To load pyflyby into an existing IPython session as a 1-off:

.. code:: bash

$ ipython In [1]: %load_ext pyflyby

To configure IPython/Jupyter Notebook to load pyflyby automatically:

.. code:: bash

$ py pyflyby.install_in_ipython_config_file

or

.. code:: bash

$ echo 'c.InteractiveShellApp.extensions.append("pyflyby")'
>> ~/.ipython/profile_default/ipython_config.py

$ ipython In [1]: b64decode('aGVsbG8=') [PYFLYBY] from base64 import b64decode Out[1]: 'hello'

Auto importer lazy variables

It is possible to use the autoimporter to lazily define variables.

To use, put the following in your IPython startup files (~/.ipython/profile_default/startup/autoimp.py), or in your IPython configuration file:

.. code:: python

from pyflyby import add_import

add_import("foo", "foo = 1")

add_import(
    "df, data as dd",
    '''
    import pandas as pd
    data = [1,2,3]
    df =  pd.DataFrame(data)
''')

You can add the keyword strict=False to not fail if not in IPython or of the pyflyby extensions is not loaded.

Quick start: py command-line multi-tool

.. code:: bash

$ py b64decode aGVsbG8= [PYFLYBY] from base64 import b64decode [PYFLYBY] b64decode('aGVsbG8=', altchars=None) 'hello'

$ py log2 sys.maxint [PYFLYBY] from numpy import log2 [PYFLYBY] import sys [PYFLYBY] log2(9223372036854775807) 63.0

$ py 'plot(cos(arange(30)))' [PYFLYBY] from numpy import arange [PYFLYBY] from numpy import cos [PYFLYBY] from matplotlib.pyplot import plot [PYFLYBY] plot(cos(arange(30))) <plot>

$ py 38497631 / 13951446 2.7594007818257693

$ py foo.py

Quick start: tidy-imports

To use tidy-imports, just specify the filename(s) to tidy.

For example:

.. code::

$ echo 're.search("[a-z]+", "....hello..."), chisqprob(arange(5), 2)' > foo.py

$ tidy-imports foo.py --- /tmp/foo.py +++ /tmp/foo.py @@ -1 +1,9 @@ +from future import absolute_import, division, with_statement + +from numpy import arange +from scipy.stats import chisqprob +import re + re.search("[a-z]+", "....hello..."), chisqprob(arange(5), 2)

Replace /tmp/foo.py? [y/N]

To exclude a file, use --exclude <pattern>.

Local imports

By default, tidy-imports only touches top-level import statements. To also tidy imports inside function and class bodies (removing unused local imports), use the --tidy-local-imports flag:

.. code:: bash

$ tidy-imports --tidy-local-imports foo.py

You can make this the default for a project via pyproject.toml (see Per-Project configuration of tidy-imports_).

Ignoring specific imports

To keep a particular import that would otherwise be removed as unused, add the # tidy-imports: ignore-import comment pragma to that line:

.. code:: python

import os # tidy-imports: ignore-import

This works for both top-level and local imports, and for from style imports:

.. code:: python

from mypackage import _setup_hooks # tidy-imports: ignore-import

def foo(): import pdb # tidy-imports: ignore-import ...

Quick start: import libraries

Create a file named .pyflyby with lines such as

.. code:: python

from mypackage.mymodule import MyClass, my_function import anotherpackage.anothermodule

You can put this file in your home directory or in the same directory as your *.py files.

Details: automatic imports

AUTOMATIC IMPORTS - never type "import" again!

This module allows your "known imports" to work automatically in your IPython interactive session without having to type the 'import' statements (and also without having to slow down your Python startup with imports you only use occasionally).

Example::

In [1]: re.search("[a-z]+", "....hello...").group(0) [PYFLYBY] import re Out[1]: 'hello'

In [2]: chisqprob(arange(5), 2) [PYFLYBY] from numpy import arange [PYFLYBY] from scipy.stats import chisqprob Out[2]: [ 1. 0.6065 0.3679 0.2231 0.1353]

In [3]: np.sin(arandom(5)) [PYFLYBY] from numpy.random import random as arandom [PYFLYBY] import numpy as np Out[3]: [ 0.0282 0.0603 0.4653 0.8371 0.3347]

In [4]: isinstance(42, Number) [PYFLYBY] from numbers import Number Out[4]: True

It just works

Tab completion works, even on modules that are not yet imported. In the following example, notice that numpy is imported when we need to know its members, and only then::

$ ipython In [1]: nump<TAB> In [1]: numpy In [1]: numpy.arang<TAB> [PYFLYBY] import numpy In [1]: numpy.arange

The IPython "?" magic help (pinfo/pinfo2) automatically imports symbols first if necessary::

$ ipython In [1]: arange? [PYFLYBY] from numpy import arange ... Docstring: arange([start,] stop[, step,], dtype=None) ...

Other IPython magic commands work as well::

$ ipython In [1]: %timeit np.cos(pi) [PYFLYBY] import numpy as np [PYFLYBY] from numpy import pi 100000 loops, best of 3: 2.51 us per loop

$ echo 'print arange(4)' > foo.py $ ipython In [1]: %run foo.py [PYFLYBY] from numpy import arange [0 1 2 3]

.. warning::

Auto-import on ``Tab`` completion requires IPython 9.3 or newer.

Implementation details

The automatic importing happens at parse time, before code is executed. The namespace never contains entries for names that are not yet imported.

This method of importing at parse time contrasts with previous implementations of automatic importing that use proxy objects. Those implementations using proxy objects don't work as well, because it is impossible to make proxy objects behave perfectly. For example, instance(x, T) will return the wrong answer if either x or T is a proxy object.

Details: import libraries

Pyflyby uses "import libraries" that tell how to import a given symbol.

An import library file is simply a python source file containing 'import' (or 'from ... import ...') lines. These can be generated automatically with collect-imports and collect-exports.

Known imports

Find-imports, tidy-imports, and autoimport consult the database of known imports to figure out where to get an import. For example, if the imports database contains::

from numpy import arange, NaN

then when you type the following in IPython::

print(arange(10))

the autoimporter would automatically execute from numpy import arange.

The database can be one file or multiple files. This makes it easy to have project-specific known_imports along with global and per-user defaults.

The PYFLYBY_PATH environment variable specifies which files to read. This is a colon-separated list of filenames or directory names. The default is::

PYFLYBY_PATH=/etc/pyflyby:~/.pyflyby:.../.pyflyby

If you set::

PYFLYBY_PATH=/foo1/bar1:/foo2/bar2

then this replaces the default.

You can use a hyphen to include the default in the path. If you set::

PYFLYBY_PATH=/foo1/bar1:-:/foo2/bar2

then this reads /foo1/bar1, then the default locations, then /foo2/bar2.

In $PYFLYBY_PATH, .../.pyflyby (with three dots) means that all ancestor directories are searched for a member named ".pyflyby".

For example, suppose the following files exist::

/etc/pyflyby/stuff.py /u/quarl/.pyflyby/blah1.py /u/quarl/.pyflyby/more/blah2.py /proj/share/mypythonstuff/.pyflyby /proj/share/mypythonstuff/foo/bar/.pyflyby/baz.py /.pyflyby

Further, suppose:

• /proj is on a separate file system from /.
• $HOME=/u/quarl

Then tidy-imports /proj/share/mypythonstuff/foo/bar/quux/zot.py will by default use the following::

/etc/pyflyby/stuff.py /u/quarl/.pyflyby/blah1.py /u/quarl/.pyflyby/more/blah2.py /proj/share/mypythonstuff/foo/bar/.pyflyby/baz.py /proj/share/mypythonstuff/.pyflyby (a file)

.. note::

• /.pyflyby is not included, because traversal stops at file s