Image for post
Image for post

I recently read a new pre-print titled Symbolic Behaviour in Artificial Intelligence, by Adam Santoro, Andrew Lampinen, and collaborators at DeepMind, and was so taken by it that I thought I would write up some brief thoughts. Unlike much of the research published by DeepMind, which is largely empirical in nature, this paper presents a philosophical perspective on the classic symbolic vs connectionist debate. Rather than making the now-typical proposal to combine symbolic reasoning and neural network approaches into a hybrid system where deep learning is used for “low level” processing, and a classical AI system is used at the…


Image for post
Image for post

Sapiens by Yuval Noah Harari was released in English in 2014, but it was not until last month that I finally read it. As such, the thoughts presented here are likely not new. Though, I hope they are at least of interest.

Before reading Sapiens, my only experience with Harari was watching about fifteen minutes of an interview he gave at Google for his more recent book Homo Deus, which was published in English in 2016. I don’t perfectly remember the conversation but it was far enough into gene editing and cognitive enhancement that I felt I had heard enough…


A few years ago wrote a series of articles on the basics of Reinforcement Learning (RL). In those, I walked through a number of the fundamental algorithms and ideas of RL, providing code along the way. This is a kind of continuation of that series, with the difference being that this article (and possibly future ones) will dive into some of the more eclectic and experimental elements of the RL world. I want to begin with a topic close to my heart and recent research interests, something called the successor representation (SR). …


Image for post
Image for post

As those who follow this blog are probably aware, I spend a lot of time thinking about Reinforcement Learning (RL). These thoughts naturally extend into my everyday life, and the ways in which the formalisms provided by RL can be applied to the world beyond artificial agents. For those unfamiliar, Reinforcement Learning is an area of Machine Learning dedicated to optimizing behaviors in the context of external rewards. If you’d like an introduction, I wrote a series of introductory articles on the topic a couple years ago. …


Looking beyond the hype of recent Deep RL successes

Image for post
Image for post

In recent weeks DeepMind and OpenAI have each shared that they developed agents which can learn to complete the first level of the Atari 2600 game Montezuma’s Revenge. These claims are important because Montezuma’s Revenge is important. Unlike the vast majority of the games in the Arcade Learning Environment (ALE), which are now easily solved at superhuman level by learned agents, Montezuma’s Revenge has been hitherto unsolved by Deep Reinforcement Learning methods and was thought by some to be unsolvable for years to come.


What goes into a stable, accurate reinforcement signal?

Image for post
Image for post

(This post assumes some familiarity with machine learning, and reinforcement learning in particular. If you are new to RL, I’d recommend checking out a series of blog posts I wrote in 2016 on the topic as a primer)

Introduction

Since the launch of the ML-Agents platform a few months ago, I have been surprised and delighted to find that thanks to it and other tools like OpenAI Gym, a new, wider audience of individuals are building Reinforcement Learning (RL) environments, and using them to train state-of-the-art models. The ability to work with these algorithms, previously something reserved for ML PhDs, is…


Image for post
Image for post
A bear learning to hunt for fish with it’s parent.

A few weeks ago OpenAI made a splash in the Deep Learning community with the release of their paper “Evolution Strategies as a Scalable Alternative to Reinforcement Learning.” The work contains impressive results suggesting that looking elsewhere than Reinforcement Learning (RL) methods may be worthwhile when training complex neural networks. It sparked a debate around the importance of Reinforcement Learning, and perhaps it’s less than necessary status as the go-to technique for learning to solve tasks. What I want to argue here is that instead of being seen as two competing strategies, one of which being necessarily better than the…


Image for post
Image for post

Reinforcement Learning provides a framework for training agents to solve problems in the world. One of the limitations of these agents however is their inflexibility once trained. They are able to learn a policy to solve a specific problem (formalized as an MDP), but that learned policy is often useless in new problems, even relatively similar ones.

Imagine the simplest possible agent: one trained to solve a two-armed bandit task in which one arm always provides a positive reward, and the other arm always provides no reward. Using any RL algorithm such as Q-Learning or Policy Gradient, the agent can…


Image for post
Image for post

Happy New Year! In this first post of 2017 I wanted to do something fun and a little different, and momentarily switch gears away from RL to generative networks. I have been working with a Generative Adversarial Network called Pix2Pix for the past few days, and want to share the fruits of the project. This framework comes from the paper “Image-to-Image Translation with Conditional Adversarial Networks” recently out of Berkeley. Unlike vanilla GANs, which take noise inputs and produce images, Pix2Pix learns to take an image and translate it into another image using an adversarial framework. Examples of this include…


Image for post
Image for post

In this article I want to provide a tutorial on implementing the Asynchronous Advantage Actor-Critic (A3C) algorithm in Tensorflow. We will use it to solve a simple challenge in a 3D Doom environment! With the holidays right around the corner, this will be my final post for the year, and I hope it will serve as a culmination of all the previous topics in the series. If you haven’t yet, or are new to Deep Learning and Reinforcement Learning, I suggest checking out the earlier entries in the series before going through this post in order to understand all the…

Arthur Juliani

PhD. Interests include Deep (Reinforcement) Learning, Computational Neuroscience, and Phenomenology.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store