[RLlib] Basic PPO script throws obscure error when building RLModule

[ArturNiederfahrenhorst]

What happened + What you expected to happen

Using the repro script below, I get the following error:

Traceback (most recent call last):
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 102, in build
module = self.module_class(
^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/ppo/torch/default_ppo_torch_rl_module.py", line 24, in init
super().init(*args, **kwargs, catalog_class=catalog_class)
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/torch/torch_rl_module.py", line 50, in init
RLModule.init(self, *args, **kwargs)
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 467, in init
self.setup()
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/ppo/default_ppo_rl_module.py", line 31, in setup
self.catalog.actor_critic_encoder_config.base_encoder_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'actor_critic_encoder_config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/artur/code/footsies/main.py", line 14, in
algo = config.build()
^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/utils/deprecation.py", line 128, in _ctor
return obj(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/algorithm_config.py", line 5417, in build
return self.build_algo(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/algorithm_config.py", line 958, in build_algo
return algo_class(
^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/algorithm.py", line 528, in init
super().init(
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/tune/trainable/trainable.py", line 157, in init
self.setup(copy.deepcopy(self.config))
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/algorithm.py", line 631, in setup
self.env_runner_group = EnvRunnerGroup(
^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/env_runner_group.py", line 198, in init
self._setup(
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/env_runner_group.py", line 293, in _setup
self._local_env_runner = self._make_worker(
^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/env_runner_group.py", line 1207, in _make_worker
return self.env_runner_cls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/single_agent_env_runner.py", line 118, in init
self.make_module()
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/single_agent_env_runner.py", line 691, in make_module
self.module = module_spec.build()
^^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 113, in build
module = self.module_class(module_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/ppo/torch/default_ppo_torch_rl_module.py", line 24, in init
super().init(*args, **kwargs, catalog_class=catalog_class)
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/torch/torch_rl_module.py", line 50, in init
RLModule.init(self, *args, **kwargs)
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 408, in init
deprecation_warning(
File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/utils/deprecation.py", line 48, in deprecation_warning
raise ValueError(msg)
ValueError: RLModule(config=[RLModuleConfig]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..) instead.
(SingleAgentEnvRunner pid=66294) A.L.E: Arcade Learning Environment (version 0.10.1+6a7e0ae)
(SingleAgentEnvRunner pid=66294) [Powered by Stella]
(SingleAgentEnvRunner pid=66294) 2025-03-13 13:58:13,761 WARNING rl_module.py:430 -- Didn't create a Catalog object for your RLModule! If you are not using the new API stack yet, make sure to switch it off in your config: config.api_stack(enable_rl_module_and_learner=False, enable_env_runner_and_connector_v2=False). All algos use the new stack by default. Ignore this message, if your RLModule does not use a Catalog to build its sub-components.
(SingleAgentEnvRunner pid=66294) 2025-03-13 13:58:13,761 WARNING deprecation.py:50 -- DeprecationWarning: RLModule(config=[RLModuleConfig object]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., model_config=.., catalog_class=..) instead. This will raise an error in the future!
(SingleAgentEnvRunner pid=66294) 2025-03-13 13:58:13,761 WARNING deprecation.py:50 -- DeprecationWarning: get_rl_module_config has been deprecated. Use RLModule(*, observation_space=.., action_space=.., ....) instead. This will raise an error in the future!
(SingleAgentEnvRunner pid=66294) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SingleAgentEnvRunner.init() (pid=66294, ip=127.0.0.1, actor_id=c3440375d741e12b59d7f85501000000, repr=<ray.rllib.env.single_agent_env_runner.SingleAgentEnvRunner object at 0x38717e6c0>)
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^
(SingleAgentEnvRunner pid=66294) File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 408, in init [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(SingleAgentEnvRunner pid=66294) super().init(*args, **kwargs, catalog_class=catalog_class) [repeated 2x across cluster]
(SingleAgentEnvRunner pid=66294) RLModule.init(self, *args, **kwargs) [repeated 2x across cluster]
(SingleAgentEnvRunner pid=66294) self.setup()
(SingleAgentEnvRunner pid=66294) File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/algorithms/ppo/default_ppo_rl_module.py", line 31, in setup
(SingleAgentEnvRunner pid=66294) self.catalog.actor_critic_encoder_config.base_encoder_config,
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(SingleAgentEnvRunner pid=66294) AttributeError: 'NoneType' object has no attribute 'actor_critic_encoder_config'
(SingleAgentEnvRunner pid=66294) During handling of the above exception, another exception occurred:
(SingleAgentEnvRunner pid=66294) ray::SingleAgentEnvRunner.init() (pid=66294, ip=127.0.0.1, actor_id=c3440375d741e12b59d7f85501000000, repr=<ray.rllib.env.single_agent_env_runner.SingleAgentEnvRunner object at 0x38717e6c0>)
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 2x across cluster]
(SingleAgentEnvRunner pid=66294) self.make_module()
(SingleAgentEnvRunner pid=66294) File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/env/single_agent_env_runner.py", line 691, in make_module
(SingleAgentEnvRunner pid=66294) self.module = module_spec.build()
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^^
(SingleAgentEnvRunner pid=66294) File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/core/rl_module/rl_module.py", line 113, in build
(SingleAgentEnvRunner pid=66294) module = self.module_class(module_config)
(SingleAgentEnvRunner pid=66294) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(SingleAgentEnvRunner pid=66294) deprecation_warning(
(SingleAgentEnvRunner pid=66294) File "/Users/artur/miniforge3/envs/footsies/lib/python3.12/site-packages/ray/rllib/utils/deprecation.py", line 48, in deprecation_warning
(SingleAgentEnvRunner pid=66294) raise ValueError(msg)
(SingleAgentEnvRunner pid=66294) ValueError: RLModule(config=[RLModuleConfig]) has been deprecated. Use RLModule(observation_space=.., action_space=.., inference_only=.., learner_only=.., model_config=..) instead.

Versions / Dependencies

Tested on 2.42.1 and 2.43.0

Reproduction script

import gymnasium as gym
from ray.rllib.algorithms.ppo import PPOConfig

config = PPOConfig()

config.api_stack(
    enable_rl_module_and_learner=1,
    enable_env_runner_and_connector_v2=1
)

config.environment(env="ale_py:ALE/Pong-v5")

algo = config.build()

algo.train()

Issue Severity

None

[ArturNiederfahrenhorst]

Found out that this won't happen for Cartpole, but for Pong.

[Phirefly9]

as someone who also ran into this, I have a guess your pong environment returns a complex observation space such as a dict, and you didn't add the flatten observation connector

[Phirefly9]

ok I checked and turns out it is a box obs space but multi dimensional depending what mode you are in.

Im thinking tune/rllib is failing to automatically determine what connector to use for anything other than a 1d box/discrete

my env had a dict obs space and had the exact same error

[ArturNiederfahrenhorst]

Just returning to this.
The logs contain the following line for 2d inputs:

Didn't create a Catalog object for your RLModule! If you are not using the new API stack yet, make sure to switch it off in your config: config.api_stack(enable_rl_module_and_learner=False, enable_env_runner_and_connector_v2=False). All algos use the new stack by default. Ignore this message, if your RLModule does not use a Catalog to build its sub-components.

The error occurs because I'm not using the atari wrapper.
The atari wrapper will recompute the observation space to be 84x84 for atari envs, for which we have a detault CNN configuration and therefore we can automatically generate a Catalog object.

[ArturNiederfahrenhorst]

Therefore, the following script runs fine:

import gymnasium as gym
import ray
from ray.rllib.algorithms.ppo import PPOConfig
from ray.rllib.env.wrappers.atari_wrappers import wrap_atari_for_new_api_stack
from ray.tune.registry import register_env

ray.init(runtime_env={"env_vars": {"RAY_DEBUG": "legacy"}})

config = PPOConfig()

def _env_creator(cfg):
return wrap_atari_for_new_api_stack(
gym.make("ale_py:ALE/Pong-v5", **cfg, **{"render_mode": "rgb_array"}), # "CartPole-v1"
dim=64,
framestack=None,
)

register_env("env", _env_creator)

config = (
PPOConfig()
.api_stack(
enable_rl_module_and_learner=True,
enable_env_runner_and_connector_v2=True,
)
.environment("env")
)

algo = config.build()

algo.train()

[ArturNiederfahrenhorst]

I've opened a PR to quickly check if we can resolve this easily