Imitation-Initialized Reinforcement Learning for Strategic Decision-Making In Settlers of  Catan

Authors

  • Sorin-Nicolae TURCULET Department of Computer Science,, Faculty of Mathematics and Computer Science,, Babes¸-Bolyai University, Kog˘alniceanu no. 1, Cluj-Napoca, Romania

DOI:

https://doi.org/10.24193/subbi.2026.04

Keywords:

reinforcement learning, behavioral cloning, imitation learning, Proximal Policy Optimization, Settlers of Catan, game AI

Abstract

This paper studies how to train a reinforcement learning agent for Settlers of Catan, a board game that is difficult to learn because of dice randomness, hidden opponent cards, a large action space, and sparse win/loss rewards at the end of a long game. We apply Proximal Policy Optimization (PPO) and compare three ways of training the same agent: Pure-PPO (learning from scratch with no external help), Expert- Guided PPO (EG-PPO)  adding an online reward bonus whenever the agent picks a move favored by a heuristic expert), and BC-PPO (first copying an expert via Behavioral Cloning and then fine-tuning freely with reinforcement learning). Pure-PPO never exceeds a 10% win rate against a strong heuristic baseline regardless of architecture or reward shaping. EGPPO lifts this ceiling to 31.2%, and BC-PPO reaches 69.6%, more than doubling the best reward-shaping result. We also report a counter-intuitive teacher-quality paradox: cloning a stronger multi-ply Alpha-Beta search expert yields a worse final agent than cloning a simpler 1-ply heuristic, because the reactive network cannot observe  the search tree that motivated the stronger teacher’s moves.

References

[1] Collazo, B. Catanatron: A Settlers of Catan bot and Gymnasium environment for reinforcement learning. https://github.com/bcollazo/catanatron, 2023. Accessed: 2026-03-05.

[2] Collazo, B. Making Catanatron stronger – Catanatron documentation. https://docs.catanatron.com/advanced/making-catanatron-stronger, 2023. Accessed: 2026-03-05.

[3] Cullback. Canopy: A Catan bot. https://github.com/cullback/canopy, 2024. Accessed: 2026-05-01.

[4] Farama Foundation. Gymnasium: A standard API for reinforcement learning environments. https://gymnasium.farama.org, 2023. Accessed: 2026-06-11.

[5] Kaelbling, L. P., Littman, M. L., and Moore, A. W. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237–285.

[6] McGovern, A., Fagg, A. G., Pintar, N. J., and Sutton, R. S. Policy gradient vs. value function approximation: A reinforcement learning shootout. Tech. rep., University of Massachusetts Amherst, 2006.

[7] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (2019), pp. 8024–8035.

[8] Pfeiffer, M. Reinforcement learning of strategies for Settlers of Catan. In Proceedings of the International Conference on Computer Games: Artificial Intelligence, Design and Education (2004), pp. 384–388.

[9] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-Baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8.

[10] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR abs/1707.06347 (2017), 1–12.

[11] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484–489.

[12] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144.

[13] Sutton, R. S., and Barto, A. G. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.

[14] Szita, I., Chaslot, G., and Spronck, P. Monte-Carlo tree search in Settlers of Catan. In Advances in Computer Games. Springer Berlin Heidelberg, 2010, pp. 21–32.

[15] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems 30 (2017), pp. 5998–6008.

[16] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018), pp. 2920 – 2931.

[17] Whelan, M., Maddi, N., and Burre, S. Reinforcement learning in Catan. https://sidhardhburre.com/CS6501RL/final.pdf, 2023. Course report, University of Virginia CS 6501.

Downloads

Published

2026-11-12

Issue

Section

Articles