Skip to content

Example python package using juliacall #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cjdoris opened this issue Oct 2, 2021 · 10 comments
Open

Example python package using juliacall #49

cjdoris opened this issue Oct 2, 2021 · 10 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@cjdoris
Copy link
Collaborator

cjdoris commented Oct 2, 2021

No description provided.

@cjdoris cjdoris added enhancement New feature or request good first issue Good for newcomers labels Oct 2, 2021
@msundvick
Copy link

Possibly related: I was just looking into trying to make a combined python-julia project (with the python and julia code in the same git repository, and neither are published). Any ideas for how this should work? I've got an example where I manually modify the "juliacalldeps.json" file in a build step, but is there something smarter that could be done?

@cjdoris
Copy link
Collaborator Author

cjdoris commented Nov 5, 2021

What exactly does the build step do? I imagine you need to refer to the directory of the package. I've been thinking about adding some templating so that ${__DIR__} expands to the directory. Would that suffice?

@msundvick
Copy link

msundvick commented Nov 7, 2021

Ok, to make this easier I thought I'd actually publish an example (and also contribute to this issue a little more directly). You can see what I'm currently doing here https://github.com/msundvick/ffi_examples.

Specifically, the relevant build step is in build.py:

    with open("juliacalldeps.json", "r") as f:
        jl = json.load(f)

    jl["packages"]["FfiExamples"]["path"] = os.getcwd()
    with open("juliacalldeps.json", "w") as f:
        json.dump(jl, f)

    import juliacall

As for whether ${__DIR__} will work... Maybe? I'm guessing this will basically do the same thing as the replacement that I've got going here. Is there any reason that relative paths couldn't be used instead?

Edit: I've tried to strip it down a bit; there's a bunch of other stuff in that repo: https://github.com/msundvick/ffi_examples/tree/just-julia.

@cjdoris
Copy link
Collaborator Author

cjdoris commented Nov 9, 2021

Oh right yeah, the path should definitely be evaluated relative to the directory containing the file. I'll make a separate issue for that.

@cjdoris
Copy link
Collaborator Author

cjdoris commented Nov 11, 2021

I've fixed the relative paths issue on the master branch.

@henry2004y
Copy link

I'm also expecting an example python package that uses JuliaCall, i.e. a demonstration of Python wrapper over a Julia package.
More specifically, I'm expecting demos of

  • type conversions, e.g. ways to check and convert Julia structs into native Python objects. I see many methods for converting Julia types into Python from the Julia side Wrap-Julia_values, but I haven't found methods for converting Julia types into Python from the Python side (I guess like jlconvert from the naming conventions?)
  • general ways of handling Julia codes in Python. (There is now a seval, as the correspondence of pyeval and pyexec.)

Maybe I can also help on these as well.

@cjdoris
Copy link
Collaborator Author

cjdoris commented Feb 13, 2022

There is no jlconvert because that wouldn't be idiomatic - Python has no equivalent of convert.

Instead, JuliaCall takes advantage of multiple inheritance so that a Julia Dict is wrapped as a DictValue, which is both an AnyValue (i.e. a Julia object) and a Mapping (i.e. behaves like a Python dict). If you want an actual Python dict you can do dict(x) which is idiomatic.

This system can be extended by defining new subtypes of AnyValue and overloading pyjl to wrap particular Julia types as the new Python type.

It would be great if you could contribute an example package.

@henry2004y
Copy link

henry2004y commented Feb 13, 2022

I learn something more! So here is a situation where I want to pass a Julia object (currently shown as AnyValue) to a Python function. I have a Julia package which defines a struct in MyPkg like

struct MyType
   name::String
   dir::String
   fid::IOStream
   variable::Vector{String}
   #...
end

function load(...)
   # some operations...
   return MyType(...)
end

and in Python

from juliacall import Main as jl
jl.seval("using MyPkg")
file = "example_file_name"
meta = jl.load(file) # <class 'juliacall.AnyValue'>, shown by type(meta)

Now I may need some Python functions which operates on this Julia type

def plot(meta: MyType):
    var = jl.readvariable(meta, "myvar")
    x = np.arange(meta.coordmin[0], meta.coordmax[0], meta.dcoord[0])
    plt.plot(x, var)
    plt.show()

Without the type annotation in Python, this can work, since all the methods for meta lives in the Julia package which understands MyType and JuliaCall does the automatic conversion for me. However, say if I do need a concrete type even for the Python function, it seems that I need new subtypes of AnyValue, as your said

This system can be extended by defining new subtypes of AnyValue and overloading pyjl to wrap particular Julia types as the new Python type.

If you can quickly provide a concrete example of how to do this, I think I may easily come up with a Python interface for my data processing package that uses Matplotlib for plotting 😄 .

@fzeiser
Copy link

fzeiser commented Apr 4, 2022

I'd be great to have some example(s) on this. I started with a very basic repo, but still I have difficulties. Maybe someone here has a good idea?

https://github.com/fzeiser/juliacall_test

I have a very simple routine I want to call in julia from python:

module Foo
export foo

using PythonCall

function foo(a)
	println(typeof(a))
	b = pyconvert(Array, a)
	println(typeof(b))
end

end

If I try to call this script with juliacall and jl.seval(f'include("[...]/Foo.jl")') it will work! If i use using Foo it will not.

I think that it would be nice to have some basic examples of this kind. and of course -- any help on my particular example is appreciated :).

The issue with using ... arises when compiling a julia modules that uses using PythonCall; one can see that the Bar module can be run with using Bar.... For python call_using.py I get following problem:

   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] + Foo v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] + Foo v0.0.0 `~/repos/tmp/julia/Foo`

signal (11): Segmentation fault
in expression starting at /home/u54671/repos/tmp/julia/Foo/src/Foo.jl:4
_dl_lookup_symbol_x at /usr/src/debug/glibc-2.34-29.fc35.x86_64/elf/dl-lookup.c:850
do_sym at /usr/bin/../lib64/libc.so.6 (unknown line)
dlsym_doit at /usr/bin/../lib64/libc.so.6 (unknown line)
_dl_catch_exception at /usr/bin/../lib64/libc.so.6 (unknown line)
_dl_catch_error at /usr/bin/../lib64/libc.so.6 (unknown line)
_dlerror_run at /usr/bin/../lib64/libc.so.6 (unknown line)
dlsym at /usr/bin/../lib64/libc.so.6 (unknown line)
jl_dlsym at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
#dlsym#1 at ./libdl.jl:59 [inlined]
dlsym at ./libdl.jl:57 [inlined]
init_pointers at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:283
init_pointers at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:283 [inlined]
init_context at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/context.jl:40
__init__ at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/CPython.jl:21
unknown function (ip: 0x7efd19fa4ac3)
unknown function (ip: 0x7efd7e6020af)
jl_init_restored_modules at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5ef37ce2)
unknown function (ip: 0x7efd5f02e51c)
unknown function (ip: 0x7efd5f036c66)
unknown function (ip: 0x7efd5efca6d0)
unknown function (ip: 0x7efd5eae5ec6)
unknown function (ip: 0x7efd7e6e9363)
unknown function (ip: 0x7efd7e6053e0)
unknown function (ip: 0x7efd7e604cd1)
unknown function (ip: 0x7efd7e604fae)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5efbc9ba)
unknown function (ip: 0x7efd5ef3743a)
unknown function (ip: 0x7efd5ebea44e)
unknown function (ip: 0x7efd5ebea76b)
unknown function (ip: 0x7efd7e5e6b7b)
unknown function (ip: 0x7efd7e5e63dd)
unknown function (ip: 0x7efd7e5e73ae)
unknown function (ip: 0x7efd7e5e7ac1)
unknown function (ip: 0x7efd7e604591)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5e6394e7)
unknown function (ip: 0x7efd7e5e6b7b)
unknown function (ip: 0x7efd7e5e63dd)
unknown function (ip: 0x7efd7e5e73ae)
unknown function (ip: 0x7efd7e5e7ac1)
unknown function (ip: 0x7efd7e604591)
unknown function (ip: 0x7efd7e604fae)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5f0e5897)
unknown function (ip: 0x7efd5ec03ea7)
unknown function (ip: 0x7efd5ec04018)
unknown function (ip: 0x7efd7e6283e9)
jl_repl_entrypoint at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
main at /usr/bin/julia (unknown line)
__libc_start_call_main at /usr/bin/../lib64/libc.so.6 (unknown line)
__libc_start_main at /usr/bin/../lib64/libc.so.6 (unknown line)
_start at /usr/bin/julia (unknown line)
Allocations: 552910 (Pool: 552426; Big: 484); GC: 1
Traceback (most recent call last):
  File "/home/u54671/repos/tmp/python/call_using.py", line 8, in <module>
    jl.seval("using Foo")
  File "/home/u54671/.julia/packages/PythonCall/XgP8G/src/jlwrap/module.jl:19", line 7, in seval
juliacall.JuliaError: Failed to precompile Foo [fdd1085a-9f79-4a83-9a23-f2e3e5a2036c] to /home/u54671/.julia/compiled/v1.7/Foo/jl_UWZHpi.
Stacktrace:
 [1] pyjlmodule_seval(self::Module, expr::PythonCall.Py)
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/module.jl:13
 [2] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/base.jl:62
 [3] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
   @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/jlwrap.jl:47

[I tried to add a juliapkg.json, but that did not help / or I did it wrong]

@fzeiser
Copy link

fzeiser commented Apr 4, 2022

Big surprise for me: I got it working -- but I'm not sure what the general message of this should be / what I did wrong in the first place.

I added a juliapkg.json, but that didn't solve the issue:

> python call_using.py
[juliapkg] Locating Julia ^1.7
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] Using Julia 1.7.2 at /usr/bin/julia
[juliapkg] Using Julia project at /home/u54671/pyenv/algo/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")])
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
 [...]
signal (11): Segmentation fault
[...]
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/base.jl:62
 [3] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
   @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/jlwrap.jl:47

. I then tried to run the same commands as juliapkg seemingly tries to run in the terminal

subprocess.CalledProcessError: Command '['/usr/bin/julia', '--project=/home/u54671/pyenv/algo/julia_env', '-e', 'import Pkg; Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")]); Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()']'
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Fedora 35 build
|__/                   |

julia> import Pkg; Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")]);
   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`

julia> using Foo
[ Info: Precompiling Foo [fdd1085a-9f79-4a83-9a23-f2e3e5a2036c]
^[[    CondaPkg Found dependencies: /home/u54671/.julia/packages/PythonCall/XgP8G/CondaPkg.toml
    CondaPkg Resolving changes
             + python
    CondaPkg Installing packages
  Package               Version  Build               Channel                    Size
──────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64     Cached
  + _openmp_mutex           4.5  1_gnu               conda-forge/linux-64     Cached
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     Cached
  + ca-certificates   2021.10.8  ha878542_0          conda-forge/linux-64     Cached
  + ld_impl_linux-64     2.36.1  hea4e1c9_2          conda-forge/linux-64     Cached
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64     Cached
  + libgcc-ng            11.2.0  h1d223b6_14         conda-forge/linux-64     Cached
  + libgomp              11.2.0  h1d223b6_14         conda-forge/linux-64     Cached
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64     Cached
  + libuuid              2.32.1  h7f98852_1000       conda-forge/linux-64     Cached
  + libzlib              1.2.11  h166bdaf_1014       conda-forge/linux-64     Cached
  + ncurses                 6.3  h9c3ff4c_0          conda-forge/linux-64     Cached
  + openssl               3.0.2  h166bdaf_1          conda-forge/linux-64     Cached
  + pip                  22.0.4  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + python               3.10.4  h2660328_0_cpython  conda-forge/linux-64     Cached
  + python_abi             3.10  2_cp310             conda-forge/linux-64     Cached
  + readline                8.1  h46c0cb4_0          conda-forge/linux-64     Cached
  + setuptools           61.3.1  py310hff52083_0     conda-forge/linux-64     Cached
  + sqlite               3.37.1  h4ff8645_0          conda-forge/linux-64     Cached
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64     Cached
  + tzdata                2022a  h191b570_0          conda-forge/noarch       Cached
  + wheel                0.37.1  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + xz                    5.2.5  h516909a_1          conda-forge/linux-64     Cached
  + zlib                 1.2.11  h166bdaf_1014       conda-forge/linux-64     Cached

  Summary:

  Install: 24 packages

  Total download: 0 B

──────────────────────────────────────────────────────────────────────────────────────


julia> using Foo

julia> Foo.foo("asd")
String
Vector{String}

julia> 
> python call_using.py
   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
PythonCall.PyList{PythonCall.Py}
Vector{Int64}
None

Suddenly, now I can run call_using.py just fine:

$ python call_using.py
   Resolving package versions...
  No Changes to `~/pyenv/algo/julia_env/Project.toml`
  No Changes to `~/pyenv/algo/julia_env/Manifest.toml`
PythonCall.PyList{PythonCall.Py}
Vector{Int64}
None

Maybe someone can try to reproduce this? Maybe it was just a confusion on my side due to the environments? Still, I think the error message was not quite telling...

@github-actions github-actions bot added the stale Issues about to be auto-closed label Sep 20, 2023
@JuliaPy JuliaPy deleted a comment from github-actions bot Sep 22, 2023
@cjdoris cjdoris removed the stale Issues about to be auto-closed label Sep 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

4 participants