.. include:: /_includes/rllib/announcement.rst .. include:: /_includes/rllib/we_are_hiring.rst .. include:: /_includes/rllib/rlmodules_rollout.rst .. note:: Interacting with Catalogs mainly covers advanced use cases. Catalog (Alpha) =============== Catalogs are where `RLModules `__ primarily get their models and action distributions from. Each :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` has its own default :py:class:`~ray.rllib.core.models.catalog.Catalog`. For example, :py:class:`~ray.rllib.algorithms.ppo.ppo_torch_rl_module.PPOTorchRLModule` has the :py:class:`~ray.rllib.algorithms.ppo.ppo_catalog.PPOCatalog`. You can override Catalogs’ methods to alter the behavior of existing RLModules. This makes Catalogs a means of configuration for RLModules. You interact with Catalogs when making deeper customization to what :py:class:`~ray.rllib.core.models.Model` and :py:class:`~ray.rllib.models.distributions.Distribution` RLlib creates by default. .. note:: If you simply want to modify a :py:class:`~ray.rllib.core.models.Model` by changing its default values, have a look at the ``model config dict``: .. dropdown:: **``MODEL_DEFAULTS`` dict** :animate: fade-in-slide-down This dict (or an overriding sub-set) is part of :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` and therefore also part of any algorithm-specific config. You can override its values and pass it to an :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` to change the behavior RLlib's default models. .. literalinclude:: ../../../rllib/models/catalog.py :language: python :start-after: __sphinx_doc_begin__ :end-before: __sphinx_doc_end__ While Catalogs have a base class :py:class:`~ray.rllib.core.models.catalog.Catalog`, you mostly interact with Algorithm-specific Catalogs. Therefore, this doc also includes examples around PPO from which you can extrapolate to other algorithms. Prerequisites for this user guide is a rough understanding of `RLModules `__. This user guide covers the following topics: - Basic usage - What are Catalogs - Inject your custom models into RLModules - Inject your custom action distributions into RLModules .. - Extend RLlib’s selection of Models and Distributions with your own .. - Write a Catalog from scratch Catalog and AlgorithmConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since Catalogs effectively control what ``models`` and ``distributions`` RLlib uses under the hood, they are also part of RLlib’s configurations. As the primary entry point for configuring RLlib, :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` is the place where you can configure the Catalogs of the RLModules that are created. You set the ``catalog class`` by going through the :py:class:`~ray.rllib.core.rl_module.rl_module.SingleAgentRLModuleSpec` or :py:class:`~ray.rllib.core.rl_module.marl_module.MultiAgentRLModuleSpec` of an AlgorithmConfig. For example, in heterogeneous multi-agent cases, you modify the MultiAgentRLModuleSpec. .. image:: images/catalog/catalog_rlmspecs_diagram.svg :align: center The following example shows how to configure the Catalog of an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` created by PPO. .. literalinclude:: doc_code/catalog_guide.py :language: python :start-after: __sphinx_doc_algo_configs_begin__ :end-before: __sphinx_doc_algo_configs_end__ Basic usage ~~~~~~~~~~~ The following three examples illustrate three basic usage patterns of Catalogs. The first example showcases the general API for interacting with Catalogs. .. literalinclude:: doc_code/catalog_guide.py :language: python :start-after: __sphinx_doc_basic_interaction_begin__ :end-before: __sphinx_doc_basic_interaction_end__ The second example showcases how to use the :py:class:`~ray.rllib.algorithms.ppo.ppo_catalog.PPOCatalog` to create a ``model`` and an ``action distribution``. This is more similar to what RLlib does internally. .. dropdown:: **Use catalog-generated models** :animate: fade-in-slide-down .. literalinclude:: doc_code/catalog_guide.py :language: python :start-after: __sphinx_doc_ppo_models_begin__ :end-before: __sphinx_doc_ppo_models_end__ The third example showcases how to use the base :py:class:`~ray.rllib.core.models.catalog.Catalog` to create an ``encoder`` and an ``action distribution``. Besides these, we create a ``head network`` that fits these two by hand to show how you can combine RLlib's :py:class:`~ray.rllib.core.models.base.ModelConfig` API and Catalog. Extending Catalog to also build this head is how :py:class:`~ray.rllib.core.models.catalog.Catalog` is meant to be extended, which we cover later in this guide. .. dropdown:: **Customize a policy head** :animate: fade-in-slide-down .. literalinclude:: doc_code/catalog_guide.py :language: python :start-after: __sphinx_doc_modelsworkflow_begin__ :end-before: __sphinx_doc_modelsworkflow_end__ What are Catalogs ~~~~~~~~~~~~~~~~~ Catalogs have two primary roles: Choosing the right :py:class:`~ray.rllib.core.models.Model` and choosing the right :py:class:`~ray.rllib.models.distributions.Distribution`. By default, all catalogs implement decision trees that decide model architecture based on a combination of input configurations. These mainly include the ``observation space`` and ``action space`` of the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, the ``model config dict`` and the ``deep learning framework backend``. The following diagram shows the break down of the information flow towards ``models`` and ``distributions`` within an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`. An :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` creates an instance of the Catalog class they receive as part of their constructor. It then create its internal ``models`` and ``distributions`` with the help of this Catalog. .. note:: You can also modify :py:class:`~ray.rllib.core.models.Model` or :py:class:`~ray.rllib.models.distributions.Distribution` in an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` directly by overriding the RLModule's constructor! .. image:: images/catalog/catalog_and_rlm_diagram.svg :align: center The following diagram shows a concrete case in more detail. .. dropdown:: **Example of catalog in a PPORLModule** :animate: fade-in-slide-down The :py:class:`~ray.rllib.algorithms.ppo.ppo_catalog.PPOCatalog` is fed an ``observation space``, ``action space``, a ``model config dict`` and the ``view requirements`` of the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`. The model config dicts and the view requirements are only of interest in special cases, such as recurrent networks or attention networks. A PPORLModule has four components that are created by the :py:class:`~ray.rllib.algorithms.ppo.ppo_catalog.PPOCatalog`: ``Encoder``, ``value function head``, ``policy head``, and ``action distribution``. .. image:: images/catalog/ppo_catalog_and_rlm_diagram.svg :align: center Inject your custom model or action distributions into RLModules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can make a :py:class:`~ray.rllib.core.models.catalog.Catalog` build custom ``models`` by overriding the Catalog’s methods used by RLModules to build ``models``. Have a look at these lines from the constructor of the :py:class:`~ray.rllib.algorithms.ppo.ppo_torch_rl_module.PPOTorchRLModule` to see how Catalogs are being used by an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`: .. literalinclude:: ../../../rllib/algorithms/ppo/ppo_rl_module.py :language: python :start-after: __sphinx_doc_begin__ :end-before: __sphinx_doc_end__ Consequently, in order to build a custom :py:class:`~ray.rllib.core.models.Model` compatible with a PPORLModule, you can override methods by inheriting from :py:class:`~ray.rllib.algorithms.ppo.ppo_catalog.PPOCatalog` or write a :py:class:`~ray.rllib.core.models.catalog.Catalog` that implements them from scratch. The following showcases such modifications. This example shows two modifications: - How to write a custom :py:class:`~ray.rllib.models.distributions.Distribution` - How to inject a custom action distribution into a :py:class:`~ray.rllib.core.models.catalog.Catalog` .. literalinclude:: ../../../rllib/examples/catalog/custom_action_distribution.py :language: python :start-after: __sphinx_doc_begin__ :end-before: __sphinx_doc_end__ Notable TODOs ------------- - Add cross references to Model and Distribution API docs - Add example that shows how to inject own model - Add more instructions on how to write a catalog from scratch - Add section "Extend RLlib’s selection of Models and Distributions with your own" - Add section "Write a Catalog from scratch"