Interoperability.md

layout	toc_group	link_title	permalink
docs-experimental	python	Interoperability	/reference-manual/python/Interoperability/

Interoperability

The best way to embed GraalPy is to use the GraalVM SDK Polyglot API.

The Polyglot API

Since GraalVM supports several other programming languages including JavaScript, R, Ruby, and those that compile to LLVM bitcode, it also provides a Python API to interact with them. In fact, GraalVM uses this API internally when executing Python native extensions using the GraalVM LLVM runtime.

Other languages are only available for the GraalPy Java distributions. To install other languages into a GraalPy, use libexec/graalpy-polyglot-get from the distribution's root directory. To install Ruby, for example:

libexec/graalpy-polyglot-get ruby

You can import the polyglot module to interact with other languages:

>>> import polyglot

You can evaluate some inlined code from another language:

>>> polyglot.eval(string="1 + 1", language="ruby")
2

You can evaluate some code from a file, by passing the path to it:

>>> with open("./my_ruby_file.rb", "w") as f:
...    f.write("Polyglot.export('RubyPolyglot', Polyglot)")
41
>>> polyglot.eval(path="./my_ruby_file.rb", language="ruby")
<foreign object at ...>

You can import a global value from the entire polyglot scope:

>>> ruby_polyglot = polyglot.import_value("RubyPolyglot")

This global value should then work as expected:

• Accessing attributes assumes it reads from the members namespace.

  >>> ruby_polyglot.to_s
  <foreign object at ...>

• Calling methods on the result tries to do a straight invoke and falls back to reading the member and trying to execute it.

  >>> ruby_polyglot.to_s()
  Polyglot

• Accessing items is supported both with strings and numbers.

  >>> ruby_polyglot.methods()[10] is not None
  True

You can export some object from Python to other supported languages so they can import it:

>>> foo = object()
>>> polyglot.export_value(value=foo, name="python_foo")
<object object at ...>
>>> jsfoo = polyglot.eval(language="js", string="Polyglot.import('python_foo')")
>>> jsfoo is foo
True

The export function can be used as a decorator. In this case the function name is used as the globally exported name:

>>> @polyglot.export_value
... def python_method():
...     return "Hello from Python!"

Here is an example of how to use the JavaScript regular expression engine to match Python strings:

>>> js_re = polyglot.eval(string="RegExp()", language="js")

>>> pattern = js_re.compile(".*(?:we have (?:a )?matching strings?(?:[!\\?] )?)(.*)")

>>> if pattern.exec("This string does not match"):
...    raise SystemError("that shouldn't happen")

>>> md = pattern.exec("Look, we have matching strings! This string was matched by Graal.js")

>>> "Here is what we found: '%s'" % md[1]
"Here is what we found: 'This string was matched by Graal.js'"

This program matches Python strings using the JavaScript regular expression object. Python reads the captured group from the JavaScript result and prints it.

As a more complex example, see how you can read a file using R, process the data in Python, and use R again to display the resulting data image, using both the R and Python libraries in conjunction. To run this example, first install the required R library:

R -e 'install.packages("https://www.rforge.net/src/contrib/jpeg_0.1-8.tar.gz", repos=NULL)'

This example also uses image_magix.py and works on a JPEG image input (you can try with this image). These files have to be in the same directory that the script below is located in and run from.

import polyglot
import sys
import time
sys.path.insert(0, ".")
from image_magix import Image

load_jpeg = polyglot.eval(string="""function(file.name) {
    library(jpeg)
    jimg <- readJPEG(file.name)
    jimg <- jimg*255
    jimg
}""", language="R")

raw_data = load_jpeg("python_demo_picture.jpg")

# the dimensions are R attributes; define function to access them
getDim = polyglot.eval(string="function(v, pos) dim(v)[[pos]]", language="R")

# Create object of Python class 'Image' with loaded JPEG data
image = Image(getDim(raw_data, 2), getDim(raw_data, 1), raw_data)

# Run Sobel filter
result = image.sobel()

draw = polyglot.eval(string="""function(processedImgObj) {
    require(grDevices)
    require(grid)
    mx <- matrix(processedImgObj$`@data`/255, nrow=processedImgObj$`@height`, ncol=processedImgObj$`@width`)
    grDevices:::awt()
    grid.raster(mx, height=unit(nrow(mx),"points"))
}""", language="R")

draw(result)
time.sleep(10)

The Java Host Interop API

Finally, to interoperate with Java (only when running on the JVM), you can use the java module:

>>> import java
>>> BigInteger = java.type("java.math.BigInteger")
>>> myBigInt = BigInteger.valueOf(42)
>>> # public Java methods can just be called
>>> myBigInt.shiftLeft(128)
<JavaObject[java.math.BigInteger] at ...>
>>> # Java method names that are keywords in Python can be accessed using `getattr`
>>> getattr(myBigInt, "not")()
<JavaObject[java.math.BigInteger] at ...>
>>> byteArray = myBigInt.toByteArray()
>>> # Java arrays can act like Python lists
>>> list(byteArray)
[42]

For packages under the java package, you can also use the normal Python import syntax:

>>> import java.util.ArrayList
>>> from java.util import ArrayList
>>>
>>> java.util.ArrayList == ArrayList
True
>>> al = ArrayList()
>>> al.add(1)
True
>>> al.add(12)
True
>>> al
[1, 12]

In addition to the type built-in method, the java module exposes the following methods as well:

Built-in	Specification
instanceof(obj, class)	returns True if obj is an instance of class (class must be a foreign object class)
is_function(obj)	returns True if obj is a Java host language function wrapped using Truffle interop
is_object(obj)	returns True if obj if the argument is Java host language object wrapped using Truffle interop
is_symbol(obj)	returns True if obj if the argument is a Java host symbol, representing the constructor and static members of a Java class, as obtained by java.type

>>> ArrayList = java.type('java.util.ArrayList')
>>> my_list = ArrayList()
>>> java.is_symbol(ArrayList)
True
>>> java.is_symbol(my_list)
False
>>> java.is_object(ArrayList)
True
>>> java.is_function(my_list.add)
True
>>> java.instanceof(my_list, ArrayList)
True

See Polyglot Programming and Embed Languages for more information about interoperability with other programming languages.

The Behavior of Types

The interop protocol defines different "types" which can overlap in all kinds of ways and have restrictions on how they can interact with Python.

Interop Types to Python

Most importantly and upfront: all foreign objects passing into Python have the Python type foreign. There is no emulation of i.e., objects that are interop booleans to have the Python type bool. This is because interop types can overlap in ways that the Python built-in types cannot, and it would not be clear what should take precedence. Instead, the foreign type defines all of the Python special methods for type conversion that are used throughout the interpreter (methods such as __add__, __int__, __str__, __getitem__, etc.) and these try to do the right thing based on the interop type (or raise an exception.)

Types not listed in the below table have no special interpretation in Python right now.

Interop type	Python interpretation
Null	It is like None. Important to know: interop null values are equal, but not identical! This was done because JavaScript defines two "null-like" values; undefined and null, which are not identical
Boolean	Behaves like Python booleans, including the fact that in Python, all booleans are also integers (1 and 0 for true and false, respectively)
Number	Behaves like Python numbers. Python only has one integral and one floating point type, but it cares about the ranges in some places such as typed arrays.
String	Behaves like Python strings.
Buffer	Buffers are also a concept in Python's native API (albeit a bit different). Interop buffers are treated like Python buffers in some places (such as memoryview) to avoid copies of data.
Array	Arrays can be used with subscript access like Python lists, with integers and slices as indices.
Hash	Hashes can be used with subscript access like Python dicts, with any hashable kind of object as key. "Hashable" follows Python semantics, generally all interop types with identity are deemed "hashable". Note that if an interop object is both Array and Hash, the behavior of the subscript access is undefined.
Members	Members can be read using normal Python . notation or the getattr etc. functions.
Iterable	Iterables are treated like Python objects with an __iter__ method, that is, they can be used in loops and other places that accept Python iterables.
Iterator	Iterators are treated like Python objects with a __next__ method.
Exception	Interop exceptions can be caught in generic except clauses.
MetaObject	Interop meta objects can be used in subtype and isinstance checks
Executable	Executable objects can be executed as functions, but never with keyword arguments.
Instantiable	Instantiable objects behave like executable objects (similar to how Python treats this)

Python to Interop Types

Interop type	Python interpretation
Null	Only None.
Boolean	Only subtypes of Python bool. Note that in contrast to Python semantics, Python bool is never also an interop number.
Number	Only subtypes of int and float.
String	Only subtypes of str.
Array	Any object with a __getitem__ and a __len__, but not if it also has keys, values, and items (like dict does.)
Hash	Only subtypes of dict.
Members	Any Python object. Note that the rules for readable/writable are a bit ad-hoc, since checking that is not part of the Python MOP.
Iterable	Anything that has an __iter__ method or a __getitem__ method.
Iterator	Anything with a __next__ method.
Exception	Any Python BaseException subtype.
MetaObject	Any Python type.
Executable	Anything with a __call__ method.
Instantiable	Any Python type.

The Truffle Interoperability Extension API

It is possible to extend the Truffle Interoperability protocol directly from python via a simple API defined in the polyglot module. The purpose of this API is to allow custom / user defined types to take part in the interop ecosystem. This is particularly useful for external types which are not compatible by default with the interop protocol. An example in this sense are the numpy numeric types (e.g., numpy.int32) which are not supported by default by the interop protocol.

The API

Function / Type	Description
register_interop_behavior	Takes the receiver type as first argument. The remainder keyword arguments correspond to the respective Truffle Interop messages. Not All interop messages are supported.
get_registered_interop_behavior	Takes the receiver type as first argument. Returns the list of extended Truffle Interop messages for the given type.
@interop_behavior	Class decorator, takes the receiver type as only argument. The interop messages are extended via static methods defined in the decorated class (supplier).
UnsupportedMessage	builtin exception to be raised by the user as dictated by the Truffle Interop Protocol. Typically, if a is_X method returns False the corresponding as_X method raises UnsupportedMessage

Supported messages

The majority (with some exceptions) of the Truffle Interop messages are supported by the interop behavior extension API, as seen in the table below.
The naming convention for the register_interop_behavior keyword arguments follows the snake_case naming convention, i.e.: the Truffle Interop fitsInLong message becomes fits_in_long and so on. Each message can be extended with a pure python function (default keyword arguments, free vars and cell vars are not allowed) or a boolean constant. Following is the list of currently supported interop messages:

Truffle Message	Extension argument name	Expected return type
isBoolean	is_boolean	bool
isDate	is_date	bool
isDuration	is_duration	bool
isIterator	is_iterator	bool
isNumber	is_number	bool
isString	is_string	bool
isTime	is_time	bool
isTimeZone	is_time_zone	bool
isExecutable	is_executable	bool
fitsInBigInteger	fits_in_big_integer	bool
fitsInByte	fits_in_byte	bool
fitsInDouble	fits_in_double	bool
fitsInFloat	fits_in_float	bool
fitsInInt	fits_in_int	bool
fitsInLong	fits_in_long	bool
fitsInShort	fits_in_short	bool
asBigInteger	as_big_integer	int
asBoolean	as_boolean	bool
asByte	as_byte	int
asDate	as_date	3-tuple with the following elements: (year: int, month: int, day: int)
asDouble	as_double	float
asDuration	as_duration	2-tuple with the following elements: (seconds: long, nano_adjustment: long)
asFloat	as_float	float
asInt	as_int	int
asLong	as_long	int
asShort	as_short	int
asString	as_string	str
asTime	as_time	4-tuple with the following elements: (hour: int, minute: int, second: int, microsecond: int)
asTimeZone	as_time_zone	a string (the timezone) or int (utc delta in seconds)
execute	execute	object
readArrayElement	read_array_element	object
getArraySize	get_array_size	int
hasArrayElements	has_array_elements	bool
isArrayElementReadable	is_array_element_readable	bool
isArrayElementModifiable	is_array_element_modifiable	bool
isArrayElementInsertable	is_array_element_insertable	bool
isArrayElementRemovable	is_array_element_removable	bool
removeArrayElement	remove_array_element	NoneType
writeArrayElement	write_array_element	NoneType
hasIterator	has_iterator	bool
hasIteratorNextElement	has_iterator_next_element	bool
getIterator	get_iterator	a python iterator
getIteratorNextElement	get_iterator_next_element	object
hasHashEntries	has_hash_entries	bool
getHashEntriesIterator	get_hash_entries_iterator	a python iterator
getHashKeysIterator	get_hash_keys_iterator	a python iterator
getHashSize	get_hash_size	int
getHashValuesIterator	get_hash_values_iterator	a python iterator
isHashEntryReadable	is_hash_entry_readable	bool
isHashEntryModifiable	is_hash_entry_modifiable	bool
isHashEntryInsertable	is_hash_entry_insertable	bool
isHashEntryRemovable	is_hash_entry_removable	bool
readHashValue	read_hash_value	object
writeHashEntry	write_hash_entry	NoneType
removeHashEntry	remove_hash_entry	NoneType

Usage Example

the simple register_interop_behavior API

import polyglot

class MyType(object):
    data = 10

polyglot.register_interop_behavior(MyType,
    is_string=True,
    as_string=lambda t: f"MyType({t.data})",
)

the decorator @interop_behavior API

This API is sugar for the simpler register_interop_behavior API and requires that interop message extension is achieved via static methods of the decorated class.
The names of the static methods are identical to the keyword names expected by register_interop_behavior.

import polyglot

class MyType(object):
    data = 10

@polyglot.interop_behavior(MyType)
class MyTypeInteropBehaviorSupplier:
    @staticmethod
    def is_string(t): 
        return True

    @staticmethod
    def as_string(t):
        return f"MyType({t.data})"