Ray Autoscaler - updating docker image in cluster config does not update docker image used in cluster

[uazafar]

What happened + What you expected to happen

Hi,

I'm submitting this issue after discussion with Jiajun Yao on Slack.

I notice that when I update the docker image in my cluster configuration and run ray up..., the cluster continues to run the older image. I see a message in the logs that states:

"A container with name X is running image A instead of B (which was provided in the YAML)"

I see from the source code there is a function _check_if_container_restart_is_needed, which has the section:

        if running_image != image:
            cli_logger.error(
                "A container with name {} is running image {} instead "
                + "of {} (which was provided in the YAML)",
                self.container_name,
                running_image,
                image,
            )

It only sets re_init_required to True when differences in the mounts are detected. I'm wondering why this is not set to True when a new image is detected? It seems like specifying a new image should allow a cluster to use that new image, thereby requiring a container restart. Curious as to why this is the case and if there is a workaround that does not involve running ray down... and ray up....

Please let me know if you require more information.

Thanks
Usman

Versions / Dependencies

Python: 3.11
Ray: 2.43.0

Reproduction script

To reproduce this, create a cluster (I did so on AWS) which points to a specific docker image. Then change the docker image (i.e. with a newer tag) and run ray up.

Issue Severity

Low: It annoys or frustrates me.