Curious Hierarchical Actor-Critic Reinforcement Learning

Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity. However, there is a lack of approaches that combine these paradigms, and it is currently unknown whether curiosity also helps to perform the hierarchical abstraction. As a novelty and scientific contribution, we tackle this issue and develop a method that combines hierarchical reinforcement learning with curiosity. Herein, we extend a contemporary hierarchical actor-critic approach with a forward model to develop a hierarchical notion of curiosity. We demonstrate in several continuous-space environments that curiosity can more than double the learning performance and success rates for most of the investigated benchmarking problems. We also provide our <a href="https://github.com/knowledgetechnologyuhh/goal_conditioned_RL_baselines" target="_blank">source code</a> and a supplementary <a href="https://www2.informatik.uni-hamburg.de/wtm/videos/chac_icann_roeder_2020.mp4" target="_blank">video</a>.

@InProceedings{RENW20a, 
 	 author =  {Röder, Frank and Eppe, Manfred and Nguyen, D.H. Phuong and Wermter, Stefan},  
 	 title = {Curious Hierarchical Actor-Critic Reinforcement Learning}, 
 	 booktitle = {Artificial Neural Networks and Machine Learning – ICANN 2020},
 	 editors = {Igor Farkaš, Paolo Masulli, Stefan Wermter},
 	 number = {},
 	 volume = {},
 	 pages = {408--419},
 	 year = {2020},
 	 month = {May},
 	 publisher = {Springer},
 	 doi = {10.1007/978-3-030-61616-8_33}, 
 }