PJRmi

Overview

PJRmi is an API for performing Remote Method Invocation (RMI, aka RPC) in a Java process from a Python one. The principle it works by is to create shim objects in Python which the user will transparently treat as they would most Python objects, but which actually cause things to happen inside the Java process.

PJRmi does everything by reflection so all you need to implement, as a service-provider, is the getObjectInstance() method for it (see below).

As well as the examples below, you can try out the Jupyter notebook for live versions.

We have also written about PJRmi, including how it relates to our other open source efforts, in our external document library.

Highlights

Some headline features of PJRmi are:

Seamless interoperation between Java and Python types.
Bidirectional calling; Python to Java, or Java to Python.
Support for lambdas and duck-typed Python implementations of Java interfaces.
Versatile and extensible connectivity options, ranging from in-process to network-based.
Thread-safe execution, with built-in locking support and asynchronous execution via futures.
Realtime code injection.
A numpy equivalent math library for Java, which is directly interoperable with Python. This includes ndarray implementations in native Java which duck-type as their Python equivalents.

Use-case examples:

Scriptification of Java applications.
Exposing Java service interfaces to Python clients.
Command and control.
On-the-fly debugging and/or dev-ops.

Other work

PJRmi isn't alone in implementing a bridge between Python and Java; some other implementations are:

JEP embeds CPython in Java through JNI allowing Java to call down into Python. It's purely in-process.
py4j allows Python to call into Java. It also supports Java calling back into Python so that Python clients can implement Java interfaces. It works by communicating over a socket.
jpy is another in-process implementation. One of its key features is support for fast pass-by-value operations with arrays by use of pointer hand-off.
jpype is another in-process implementation. Since it also uses internal C-based handoff it's highly performant.
PyJNIus is another implementation which accesses Java classes using JNI.

As well as the feature sets of the above, PJRmi supports complex Java constructs, has smooth integration of the two languages' type systems, and can be used in different modes of operation transparently to the user.

Installing

If you want to try out PJRmi then you have a couple of options:

pip install pjrmi
Clone this repository, run ./gradlew wheel in it, and pip install the resultant wheel file.

Note that the version on PyPI is does not contain any C++ extensions, since it does not include any platform-specific binaries. A locally built version will have these. These extensions are only required if you want to leverage certain optimizations, or if you want to run the in-process JVM, and it will work fine without them.

A Simple Example

The framework can be brought up in a number of different ways for clients to connect to:

A PJRmi thread is added to an existing Java process, and Python clients may connect over the network
A Java JVM is instantiated inside the Python process, with the Python process being the only connection
A Java child process is launched from within the Python process, with the parent process being the only connection
A Java JVM spawned within the Python process.

Here is an example of the last of these.

$ python
>>> import pjrmi
>>> c = pjrmi.connect_to_child_jvm()

Now c is a PJRmi instance and can be used to request information from the server. There are basically two ways to get information: you can get a reference to a Java object, or a reference to a Java class. See also the various other connect_to_blah() methods in the Python interface.

With that connection we can get a class definition, create an instance of it and call methods on that instance:

>>> ArrayList = c.javaclass.java.util.ArrayList
>>> a = ArrayList([1,2,3])
>>> a.size()
3
>>> a.toString()
'[1, 2, 3]'
>>> a.get(1)
2
>>> a.hashCode()
30817
>>> a.contains(1)
True

And you can treat the Java objects much like Python ones:

>>> str(a)
'[1, 2, 3]'
>>> hash(a)
30817
>>> sum(a)
6
>>> 1 in a
True

Accessing Existing Object Instances

>>> foo = c.object_for_name('Foo')

You can get a reference to a java object with the command c.object_for_name(<string>). This calls into a Java method on the server, and that method will see the string and decide what object to pass back. In this example it will simply return None (or null, from Java's perspective). The method is left unimplemented in the PJRmi class -- each server will have to implement it and decide which objects you want to expose to Python. The stand-alone test server example in com.deshaw.PJRmi.main simply does this:

// Create a simple instance which just echoes back the name it's given
final PJRmi pjrmi =
    new PJRmi("PJRmi", provider) {
        @Override protected Object getObjectInstance(CharSequence name) {
            return name.toString().intern();
        }
    };

Most of the time Java objects are returned as they are but, in this example, foo will be a special subclass of a Python string; so if we want the Java String we need to pull it out. Java Objects are generally returned as objects, only Strings and primitive types (int, double, boolean, ...) have special boxed handling.

>>> foo = c.object_for_name('Foo')
>>> foo
u'Foo'
>>> foo = foo.java_object
>>> foo
<pjrmi.java.lang.String at 0x39e7c10>
>>> foo.<tab>
foo.CASE_INSENSITIVE_ORDER  foo.endsWith                foo.lastIndexOf             foo.startsWith
foo.charAt                  foo.equals                  foo.length                  foo.subSequence
foo.codePointAt             foo.equalsIgnoreCase        foo.matches                 foo.substring
foo.codePointBefore         foo.format                  foo.notify                  foo.toCharArray
foo.codePointCount          foo.getBytes                foo.notifyAll               foo.toLowerCase
foo.compareTo               foo.getChars                foo.offsetByCodePoints      foo.toString
foo.compareToIgnoreCase     foo.getClass                foo.regionMatches           foo.toUpperCase
foo.concat                  foo.hashCode                foo.replace                 foo.trim
foo.contains                foo.indexOf                 foo.replaceAll              foo.valueOf
foo.contentEquals           foo.intern                  foo.replaceFirst            foo.wait
foo.copyValueOf             foo.isEmpty                 foo.split
>>> foo.getBytes?
Type:       function
String Form:<function getBytes at 0x5fdeb90>
File:       .../pjrmi.py
Definition: foo.getBytes(*args, **kwargs)
Docstring:
A wrapper for the Java method:
    java.lang.String#getBytes()
taking the following forms:
    [B getBytes()
    [B getBytes(java.lang.String)
    [B getBytes(java.nio.charset.Charset)
    void getBytes(int, int, [B, int)

When the Python client gets the reply, it gets an id, which is a handle to the object, and the object's type. If it hasn't seen that type before, it will ask the Java server to get type information, including the class hierarchy and all available methods. It will use this to provide ipython tab completion and documentation.

>>> foo._<tab>
foo.__add__           foo.__doc__           foo.__len__           foo.__repr__          foo._bases            foo._is_immutable
foo.__class__         foo.__eq__            foo.__module__        foo.__setattr__       foo._classname        foo._is_primitive
foo.__cmp__           foo.__format__        foo.__ne__            foo.__sizeof__        foo._handle           foo._pjrmi
foo.__del__           foo.__getattribute__  foo.__new__           foo.__str__           foo._hash_code        foo._type_id
foo.__delattr__       foo.__hash__          foo.__reduce__        foo.__subclasshook__  foo._instance_of
foo.__dict__          foo.__init__          foo.__reduce_ex__     foo.__weakref__       foo._is_array
>>> foo._is_immutable
True
>>> str(foo)
'Foo'
>>> foo._str
'Foo'

Using Classes To Create New Object Instances

Two different ways to get a handle on a Java class as a Python one:

>>> byte_array = c.class_for_name('[B')
>>> String     = c.javaclass.java.lang.String

c.class_for_name(<string>), and its syntactic-sugar equivalent, return a handle to a Java class object, which can be used to call static methods. This will return any class it knows about without any special handling (unlike object_for_name, for which the server has to have a way of mapping from each input string to an object).

You can then use the result to create new instances:

>>> ArrayList = c.javaclass.java.util.ArrayList
>>> ArrayList([1,2,3])
<pjrmi.java.util.ArrayList at 0x378a650>
>>> str(_)
'[1, 2, 3]'

Note that these classes are fully populated with methods and so forth, as such tab-completion works on them and reflection-generated docstrings exist also:

>>> ArrayList.<tab>
ArrayList.add             ArrayList.containsAll     ArrayList.hashCode        A

Pjrmi

Install / Use

README