calmjs.parse

A collection of parsers and helper libraries for understanding ECMAScript; a near feature complete fork of |slimit|. A CLI front-end for this package is shipped separately as |crimp|.

.. image:: https://github.com/calmjs/calmjs.parse/actions/workflows/build.yml/badge.svg?branch=master :target: https://github.com/calmjs/calmjs.parse/actions/workflows/build.yml?query=branch:master .. image:: https://ci.appveyor.com/api/projects/status/5dj8dnu9gmj02msu/branch/master?svg=true :target: https://ci.appveyor.com/project/metatoaster/calmjs-parse/branch/master .. image:: https://coveralls.io/repos/github/calmjs/calmjs.parse/badge.svg?branch=master :target: https://coveralls.io/github/calmjs/calmjs.parse?branch=master

.. |calmjs.parse| replace:: calmjs.parse .. |crimp| replace:: crimp .. |ply| replace:: ply .. |slimit| replace:: slimit .. _crimp: https://pypi.python.org/pypi/crimp .. _ply: https://pypi.python.org/pypi/ply .. _slimit: https://pypi.python.org/pypi/slimit

Introduction

For any kind of build system that operates with JavaScript code in conjunction with a module system, the ability to understand what modules a given set of sources require or provide is paramount. As the Calmjs project provides a framework that produces and consume these module definitions, the the ability to have a comprehensive understanding of given JavaScript sources is a given. This goal was originally achieved using |slimit|, a JavaScript minifier library that also provided a comprehensive parser class that was built using Python Lex-Yacc (i.e. |ply|).

However, as of mid-2017, it was noted that |slimit| remained in a minimum state of maintenance for more than four years (its most recent release, 0.8.1, was made 2013-03-26), along with a number of serious outstanding issues have left unattended and unresolved for the duration of that time span. As the development of the Calmjs framework require those issues to be rectified as soon as possible, a decision to fork the parser portion of |slimit| was made. This was done in order to cater to the interests current to Calmjs project at that moment in time.

The fork was initial cut from another fork of |slimit| (specifically lelit/slimit <https://github.com/lelit/slimit>_), as it introduced and aggregated a number of bug fixes from various sources. To ensure a better quality control and assurance, a number of problematic changes introduced by that fork were removed. Also, new tests were created to bring coverage to full, and issues reported on the |slimit| tracker were noted and formalized into test cases where applicable. Finally, grammar rules were updated to ensure better conformance with the ECMA-262 (ES5) specification.

The goal of |calmjs.parse| is to provide a similar API that |slimit| had provided, except done in a much more extensible manner with more correctness checks in place. This however resulted in some operations that might take longer than what |slimit| had achieved, such as the pretty printing of output.

A CLI front-end that makes use of this package is provided through |crimp|_.

Installation

The following command may be executed to source the latest stable version of |calmjs.parse| wheel from PyPI for installation into the current Python environment.

.. code:: console

$ pip install calmjs.parse

As this package uses |ply|, it requires the generation of optimization modules for its lexer. The wheel distribution of |calmjs.parse| does not require this extra step as it contains these pre-generated modules for |ply| up to version 3.11 (the latest version available at the time of previous release), however the version of |ply| that is installed is beyond the supported version, the following caveats will apply.

If a more recent release of |ply| becomes available and the environment upgrades to that version, those pre-generated modules may become incompatible, which may result in a decreased performance and/or errors. A corrective action can be achieved through a manual optimization_ step if a newer version of |calmjs.parse| is not available, or |ply| may be downgraded back to version 3.11 if possible.

Alternatively, install a more recent version of |calmjs.parse| wheel that has the most complete set of pre-generated modules built.

Once the package is installed, the installation may be tested_ or be used directly_.

Manual installation and packaging requirements

*This section is for developers and advanced users; contains important
information for package maintainers for OS distributions (e.g. Linux)
that will prevent less than ideal experiences for downstream users.*

Development is still ongoing with |calmjs.parse|, for the latest
features and bug fixes, the development version may be installed through
git like so:

.. code:: console

    $ pip install ply setuptools  # this MUST be done first; see below for reason
    $ pip install -e git+https://github.com/calmjs/calmjs.parse.git#egg=calmjs.parse

Note that all dependencies MUST be pre-installed ``setup.py build`` step
to run, otherwise the build step required to create the pre-generated
modules will result in failure.

If |ply| isn't installed:

.. code:: console

    $ python -m pip install -e .
    ...
    running egg_info
    ...
    WARNING: cannot find distribution for 'ply'; using default value,
    assuming 'ply==3.11' for pre-generated modules
    ERROR: cannot find pre-generated modules for the assumed 'ply'
    version from above and/or cannot `import ply` to build generated
    modules, aborting build; please either ensure that the source
    archive containing the pre-generate modules is being used, or that
    the python package 'ply' is installed and available for import
    before attempting to use the setup.py to build this package; please
    refer to the top level README for further details

If ``setuptools`` isn't installed:

.. code:: console

    $ python -m pip install -e .
    ...
    running egg_info
    ...
    Traceback (most recent call last):
      ...
    ModuleNotFoundError: No module named 'pkg_resources'

Naturally, the git repository can be cloned directly and execute
``python setup.py develop`` while inside the root of the source
directory; again, both |ply| AND ``setuptools`` MUST already have be
available for import.

As the git repository does NOT contain any pre-generated modules or
code, the above message is likely to be seen by developers or distro
maintainers who are on their first try at interacting with this
software.  However, the zip archives released on PyPI starting from
version 1.3.0 do contain these modules fully pre-generated, thus they
may be used as part of a standard installation step, i.e. without
requiring |ply| be available for import before usage of the ``setup.py``
for any purpose.  While the same warning message about |ply| being
missing may be shown, the pre-generated modules will allow the build
step to proceed as normal.

Manual optimization
~~~~~~~~~~~~~~~~~~~

As lex and yacc require the generation of symbol tables, a way to
optimize the performance is to cache the results.  For |ply|, this is
done using an auto-generated module.  However, the generated file is
marked with a version number, as the results may be specific to the
installed version of |ply|.  In |calmjs.parse| this is handled by giving
them a name specific to the version of |ply| and the major Python
version, as both together does result in subtle differences in the
outputs and expectations of the auto-generated modules.

Typically, the process for this optimization is automatic and a correct
symbol table will be generated, however there are cases where this will
fail, so for this reason |calmjs.parse| provide a helper module and
executable that can be optionally invoked to ensure that the correct
encoding be used to generate that file.  Other reasons where this may be
necessary is to allow system administrators to do so for their end
users, as they may not have write privileges at that level.

To execute the optimizer from the shell, the provided helper script may
be used like so:

.. code:: console

    $ python -m calmjs.parse.parsers.optimize

If warnings appear that warn that tokens are defined but not used, they
may be safely ignored.

This step is generally optionally for users who installed this package
from PyPI via a Python wheel, provided the caveats as outlined in the
installation section are addressed.

.. _tested:

Testing the installation
~~~~~~~~~~~~~~~~~~~~~~~~

To ensure that the |calmjs.parse| installation is functioning correctly,
the built-in testsuite can be executed by the following:

.. code:: console

    $ python -m unittest calmjs.parse.tests.make_suite

If there are failures, please file an issue on the `issue tracker
<https://github.com/calmjs/calmjs.parse/issues>`_ with the full
traceback, and/or the method of installation.  Please also include
applicable information about the environment, such as the version of
this software, Python version, operating system environments, the
version of |ply| that was installed, plus other information related to
the issue at hand.


Usage
-----

.. _used directly:

As this is a parser library, no executable shell commands are provided.
There is however a helper callable object provided at the top level for
immediate access to the parsing feature.  It may be used like so:

.. code:: pycon

    >>> from calmjs.parse import es5
    >>> program_source = '''
    ... // simple program
    ... var main = function(greet) {
    ...     var hello = "hello " + greet;
    ...     return hello;
    ... };
    ... console.log(main('world'));
    ... '''
    >>> program = es5(program_source)
    >>> # for a simple repr-like nested view of the ast
    >>> program  # equivalent to repr(program)
    <ES5Program @3:1 ?children=[
      <VarStatement @3:1 ?children=[
        <VarDecl @3:5 id