feat(images): add support for Core Lightning v23.08.2, v24.02.2 & v24.05

[jamaljsr]

Closes #772
Replaces #774

Description

Adds support for Core Lightning v23.08.2 & v24.02.2.

I've migrated to using the Core Lightning embedded REST API plugin clnrest from previously using the third-party c-lightning-REST plugin. Since older versions of CLN do not support the clnrest-port flag, I have dropped support in Polar for versions prior to these two.

Screenshots

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (2a8b97e) to head (f5b8580).

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #879   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          141       141           
  Lines         4609      4663   +54     
  Branches       897       902    +5     
=========================================
+ Hits          4609      4663   +54     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jamaljsr]

I'm going to hold off on merging this until right before shipping the next release. The removal of older CLN nodes from nodes.json would break existing Polar users that could apply the out-of-band node version updates.

[toneloc]

Awesome! I would like to help test building the CLN Docker images.

Please let me know how I can best help test.

[jamaljsr]

Hey @toneloc, the images have already been built and pushed to Docker Hub. The instructions for how I build the images are in the docker readme.

If you want to test them out in Polar, you'll need to be running the the code in this PR branch because the new images won't work in v2.2.0. You'll need to setup your dev environment to run Polar from source code. The instructions are in the CONTRIBUTING doc. Just be sure to checkout this branch.

[toneloc]

Awesome thanks. FYI, I had to remove --openssl-legacy-provider in package.json to get it to work on my Mac M2 ... but it worked.

After that I was able to start up a Polar network with two v24.02.2 CLN nodes and create a channel and send payments back and forth. All via UI.

If you want me to test anything in particular please lmk. Will let you know if I see anything else notable.

Nice work. Next I want to try to create custom Docker image so I can add in a CLN plugin. Any tips on that please lmk.

[jamaljsr]

@toneloc Thanks so much for testing and confirming it works for you. That is very helpful.

FYI, I had to remove --openssl-legacy-provider in package.json to get it to work on my Mac M2 ... but it worked.

What version of NodeJS are you running? I've seen issues related to this flag when you aren't running v20.

Next I want to try to create custom Docker image so I can add in a CLN plugin. Any tips on that please lmk.

Getting your plugin into the docker image should be pretty easy. This is how Polar currently installs the c-lightning-REST plugin into the /opt/c-lightning-rest/ dir in the image.

polar/docker/clightning/Dockerfile

Lines 170 to 183 in 927459e

	# install nodejs
	RUN apt-get update -y \
	&& apt-get install -y curl gosu git \
	&& curl -sL https://deb.nodesource.com/setup_14.x | bash - \
	&& apt-get install -y nodejs \
	&& apt-get clean \
	&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
	# install c-lightning-REST API plugin
	RUN git clone https://github.com/Ride-The-Lightning/c-lightning-REST.git /opt/c-lightning-rest/ \
	&& cd /opt/c-lightning-rest \
	&& npm install \
	&& chmod -R a+rw /opt/c-lightning-rest \
	&& mv /opt/c-lightning-rest/clrest.js /opt/c-lightning-rest/plugin.js

You'd just need to get your plugin's files into the image via git clone or COPY. Once the files are inside of the image, you can update the node's startup command in Polar to include the path to your plugin. Similar to this:

--plugin=/opt/c-lightning-rest/plugin.js

[toneloc]

What version of NodeJS are you running? I've seen issues related to this flag when you aren't running v20.

I am running Node v20.0.0.

Awesome thanks and cheers. I was able to build the image and test. This is further then I've ever been so kudos to you.

In the Dockerfile I had to remove the line COPY .bashrc /home/clightning/.bashrc and manually change v${CLN_VERSION} to v24.02.2 (and of course also add git clone and CP and MV install lines to Dockerfile, as you mention).

Note that when creating channels the green lines didn't show up between custom nodes, not sure why.

Did not have error without custom nodes. Nothing weird in CLN logs.

Anyway, error and screenshot pasted below. Cheers

[electron] 22:39:20.379 › Failed to sync the network TypeError: Cannot convert undefined or null to object
[electron]     at Function.keys (<anonymous>)
[electron]     at http://localhost:3000/static/js/main.chunk.js:39919:12
[electron]     at Array.forEach (<anonymous>)
[electron]     at updateChartFromNodes (http://localhost:3000/static/js/main.chunk.js:39918:24)
[electron]     at Object.fn (http://localhost:3000/static/js/main.chunk.js:35937:91)
[electron]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[electron]     at async http://localhost:3000/static/js/main.chunk.js:5693:7

[jamaljsr]

Oh interesting. From looking at the code involved in that stacktrace, I suspect that this error is not the root cause, but a result of the chart links object being undefined, which shouldn't be possible. Is this the first error displayed in the logs immediately after opening a channel?

Would you mind sharing logs?

• The files in ~/.polar/logs. Please stop Polar, delete the existing files, then start Polar and reproduce the issue so the logs are isolated.
• The DevTools console logs would be helpful as well. If you run yarn dev, the DevTools should automatically open. If you go to the Console tab then right click anywhere on the logs and choose "Save as..." to save the logs to a file that you can share.

[toneloc]

Look like on restart the line showed up!

Then I created a channel and it is working now. Not sure what happened.

FYI:

Polar logs:

polarlogs.txt

And front end:

localhost-1714237419178.log

The Polar logs look okay. The front end logs may show you something! I'll let you known as I play around.

[michaelWuensch]

I can also confirm that it works with the new CLN versions. Didn't encounter any issues so far. 👍

[jamaljsr]

@michaelWuensch Thanks for the feedback. 👍

[darioAnongba]

Hi @jamaljsr and thanks for this awesome features. Any idea when this will be merged ? Otherwise, how to use this version of Polar even before the release? Currently struggling a lot to run CLN 24.02 with clnrest locally, using Polar would help a lot.

[jamaljsr]

Hi @darioAnongba, I'm planning on packaging up a release sometime next week. I'm trying to wait for a few of the currently open PRs to get merged.

If you'd like to locally package a build of Polar with the changes from this PR, I mentioned how to do this in #879 (comment).

[darioAnongba]

thanks! I'll try to run it on my own with your comment.

Edit: just letting you know that this PR no longer compiles because the version of the package tapd-api that is used here has been removed in npm. I tried running the version in npm 0.3.99-alpha.pre1 but it has breaking changes so the code doesn't build anymore.

I'll better wait patiently.

[michaelWuensch]

Can you also add CLN 24.05 with the upcoming release?

[jamaljsr]

@michaelWuensch I added support for v24.05 here as well.

[niteshbalusu11]

Can this be merged? :)

[niteshbalusu11]

Actually I built this branch myself and i am getting these errors for cln and lnd and can't get the containers to start.

lightningd: --log-file=/home/clightning/.lightning/debug.log: Failed to open: No such file or directory

failed to load config: ValidateConfig: log rotation setup failed: failed to create file rotator: open /home/lnd/.lnd/logs/bitcoin/regtest/lnd.log: no such file or directory

[jamaljsr]

@niteshbalusu11 What OS and docker version are you running. I tested on Mac, Windows, and Linux and didn't have any issues.

[niteshbalusu11]

@niteshbalusu11 What OS and docker version are you running. I tested on Mac, Windows, and Linux and didn't have any issues.

It's only happening with latest versions of LND and CLN btw.

MacOS Arm chips

docker compose version
Docker Compose version v2.27.0-desktop.2

docker version
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[jamaljsr]

Ok, I'm running on an ARM Mac. Those are filesystem errors you're getting. My hunch is that Docker is misbehaving. Try restarting Docker and/or your computer. That will likely resolve the issue.

[niteshbalusu11]

Ok looks like it was docker misbehaving. Seems to be working fine now. Thank you!