Discovering Hierarchy in Reinforcement Learning

We are relying more and more on machines to performtasks that were previously the sole domain ofhumans. There is a need to make machines more self-adaptable and for them to set their own sub-goals.Designing machines that can make sense of the worldthey inhabit is still an open research problem.Fortunately many complex environments exhibitstructure that can be modelled as an inter-relatedset of subsystems. Subsystems are often repetitivein time and space and reoccur many times ascomponents of different tasks. A machine may be ableto learn how to tackle larger problems if it cansuccessfully find and exploit this repetition.Evidence suggests that a bottom up approach, thatrecursively finds building-blocks at one level ofabstraction and uses them at the next level, makeslearning in many complex environments tractable.This book describes a machine learning algorithmcalled HEXQ that automatically discovershierarchical structure in its environment purelythrough sense-act interactions, setting its own sub-goals and solving decision problems usingreinforcement learning.