漫画讲解Advantage-Actor-Critic（A2C）

旧城等待， 2022-05-10 22:16 106阅读 0赞

编者按：强化学习并不是什么新鲜的内容，时至今日，关于RL的优秀教程很多，尽管大家都在尽力把内容说得浅显易懂一些，但复杂的数学公式和大量图表仍让初学者望而却步。近日，数据科学家Rudy Gilman受此启发，决定制作一些漫画讲解强化学习的基本思路，以下是论智编译的内容和重制的图像。

大多数情况下，优秀的强化学习的教程离不开数学方程和抽象概念图，但它们实在不是讲解内容的最佳方法。作为一个数据科学家，从不同角度思考问题是一个必备优秀品质，既然RL本身是从动物学习中获得启发的，那我们为什么不能从自然现象入手，用讲故事的方法谈谈RL呢？

本文是一个关于A2C模型的漫画故事。A2C是策略梯度模型（Policy Gradient model）中一个比较流行的形式，如果你掌握了A2C，你就对深度强化学习有了一些了解。

![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 1]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 2]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 3]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 4]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 5]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 6]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 7]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 8]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 9]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 10]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 11]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 12]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 13]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 14]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 15]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 16]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 17]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 18]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 19]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 20]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 21]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 22]![【原创】直观的强化学习：漫画讲解Advantage-Actor-Critic（A2C）][Advantage-Actor-Critic_A2C 23]

如果看完上述漫画后，你希望对A2C建立更深入的认识，以下是一些推荐资料：

*  一个简单的A2C代码实现：[github.com/rgilman33/simple-A2C/blob/master/3\_A2C-nstep-TUTORIAL.ipynb][github.com_rgilman33_simple-A2C_blob_master_3_A2C-nstep-TUTORIAL.ipynb]
 *  OpenAI的开源工具Baselines：[github.com/openai/baselines][github.com_openai_baselines]，及相关PyTorch实现：[github.com/rgilman33/baselines-A2C][github.com_rgilman33_baselines-A2C]
 *  UCL的强化学习课程：[www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html][www0.cs.ucl.ac.uk_staff_d.silver_web_Teaching.html]
 *  论文：Deep Reinforcement Le[ar][]ning: An Overview——[arxiv.org/abs/1701.07274][arxiv.org_abs_1701.07274]
 *  用TensorFlow实现强化学习系列文章：[medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2][medium.com_emergent-future_simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2]

注：漫画重制时间匆忙，如有错译错字，欢迎指出，小编将不甚感谢。

原文地址：[medium.com/rudygilman/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752][medium.com_rudygilman_intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752]

[Advantage-Actor-Critic_A2C]: https://www.jqr.com/editor/103/185/1031859733-5a5d784dddcc8
[Advantage-Actor-Critic_A2C 1]: https://www.jqr.com/editor/318/212/3182123126-5a5d7bda2783f
[Advantage-Actor-Critic_A2C 2]: https://www.jqr.com/editor/304/376/3043762307-5a5d930ce74e2
[Advantage-Actor-Critic_A2C 3]: https://www.jqr.com/editor/379/157/3791574117-5a5d9685655f5
[Advantage-Actor-Critic_A2C 4]: https://www.jqr.com/editor/135/165/135165518-5a5d99fe1003d
[Advantage-Actor-Critic_A2C 5]: https://www.jqr.com/editor/215/486/2154863469-5a5d9da097beb
[Advantage-Actor-Critic_A2C 6]: https://www.jqr.com/editor/168/286/1682861322-5a5d9eeb1cb8d
[Advantage-Actor-Critic_A2C 7]: https://www.jqr.com/editor/262/785/2627852454-5a5da33c95c14
[Advantage-Actor-Critic_A2C 8]: https://www.jqr.com/editor/367/488/3674881885-5a5dab149025d
[Advantage-Actor-Critic_A2C 9]: https://www.jqr.com/editor/607/856/607856320-5a5dad9659484
[Advantage-Actor-Critic_A2C 10]: https://www.jqr.com/editor/320/771/3207714571-5a5db93b59ada
[Advantage-Actor-Critic_A2C 11]: https://www.jqr.com/editor/109/790/1097909684-5a5dbba7167a5
[Advantage-Actor-Critic_A2C 12]: https://www.jqr.com/editor/299/274/2992745981-5a5dc06a8d87a
[Advantage-Actor-Critic_A2C 13]: https://www.jqr.com/editor/100/687/1006877507-5a5dc546c2190
[Advantage-Actor-Critic_A2C 14]: https://www.jqr.com/editor/346/003/3460039915-5a5dc6290eddf
[Advantage-Actor-Critic_A2C 15]: https://www.jqr.com/editor/722/183/722183539-5a5dc9fdbe99a
[Advantage-Actor-Critic_A2C 16]: https://www.jqr.com/editor/101/764/1017644734-5a5dcc39bbdcf
[Advantage-Actor-Critic_A2C 17]: https://www.jqr.com/editor/425/839/4258390678-5a5dce74b0f0d
[Advantage-Actor-Critic_A2C 18]: https://www.jqr.com/editor/569/350/56935017-5a5dd0d7d551f
[Advantage-Actor-Critic_A2C 19]: https://www.jqr.com/editor/354/375/3543751828-5a5dd2ac45809
[Advantage-Actor-Critic_A2C 20]: https://www.jqr.com/editor/391/696/3916968985-5a5dd636b6098
[Advantage-Actor-Critic_A2C 21]: https://www.jqr.com/editor/474/675/474675316-5a5de16d8d3f2
[Advantage-Actor-Critic_A2C 22]: https://www.jqr.com/editor/240/756/2407567499-5a5de75af381f
[Advantage-Actor-Critic_A2C 23]: https://www.jqr.com/editor/386/670/386670207-5a5de7b454f59
[github.com_rgilman33_simple-A2C_blob_master_3_A2C-nstep-TUTORIAL.ipynb]: https://github.com/rgilman33/simple-A2C/blob/master/3_A2C-nstep-TUTORIAL.ipynb
[github.com_openai_baselines]: https://github.com/openai/baselines
[github.com_rgilman33_baselines-A2C]: https://github.com/rgilman33/baselines-A2C
[www0.cs.ucl.ac.uk_staff_d.silver_web_Teaching.html]: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html
[ar]: https://www.jqr.com/service/company?business=17
[arxiv.org_abs_1701.07274]: https://arxiv.org/abs/1701.07274
[medium.com_emergent-future_simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2]: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2
[medium.com_rudygilman_intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752]: https://medium.com/rudygilman/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752