{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Open AI gym tutorial" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Open AI gym is a toolkit for building and comparing different reinforcement learning algorithms. It is an open source interface reinforcement learning algorithms. Gym provides numerous environments to run reinforcement agents on." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can read the original paper from [here](https://arxiv.org/abs/1606.01540)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installing Gym \n", "You need pyhton 3.5+ installed. You can use the Python package manager pip to install it using\n", "\n", "`pip install gym`\n", "\n", "If you would like to build the library from source you can do so using\n", "\n", "`git clone https://github.com/openai/gym\n", "cd gym\n", "pip install -e .`\n", "\n", "On colab you can run the following commands\n", "\n", "`!pip install gym\n", "!apt-get install python-opengl -y\n", "!apt install xvfb -y` \n", "\n", "These install the requirements and the gym library. To render and display the environments you need the some libraries which can be installed by running\n", "\n", "`!pip install pyvirtualdisplay\n", "!pip install piglet`\n", "\n", "For further information on Colab specific rendering and information you can read through [here](https://colab.research.google.com/drive/18LdlDDT87eb8cCTHZsXyS9ksQPzL3i6H#scrollTo=7wY4qZhPXotR)\n", "\n", "To install all environments you can run\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Importing libraries " ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:10.281715Z", "start_time": "2020-03-09T15:30:09.777131Z" } }, "outputs": [], "source": [ "from matplotlib import animation\n", "import matplotlib.pyplot as plt\n", "from IPython.display import display\n", "import time\n", "import gym\n", "from gym import wrappers\n", "import io\n", "import base64\n", "from IPython.display import HTML" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-03-08T05:35:13.774277Z", "start_time": "2020-03-08T05:35:13.769595Z" } }, "source": [ "## You can see the available gym environments [here](https://github.com/openai/gym/blob/master/docs/environments.md)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:10.308157Z", "start_time": "2020-03-09T15:30:10.284951Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EnvSpec(Copy-v0)\n", "EnvSpec(RepeatCopy-v0)\n", "EnvSpec(ReversedAddition-v0)\n", "EnvSpec(ReversedAddition3-v0)\n", "EnvSpec(DuplicatedInput-v0)\n", "EnvSpec(Reverse-v0)\n", "EnvSpec(CartPole-v0)\n", "EnvSpec(CartPole-v1)\n", "EnvSpec(MountainCar-v0)\n", "EnvSpec(MountainCarContinuous-v0)\n", "EnvSpec(Pendulum-v0)\n", "EnvSpec(Acrobot-v1)\n", "EnvSpec(LunarLander-v2)\n", "EnvSpec(LunarLanderContinuous-v2)\n", "EnvSpec(BipedalWalker-v2)\n", "EnvSpec(BipedalWalkerHardcore-v2)\n", "EnvSpec(CarRacing-v0)\n", "EnvSpec(Blackjack-v0)\n", "EnvSpec(KellyCoinflip-v0)\n", "EnvSpec(KellyCoinflipGeneralized-v0)\n", "EnvSpec(FrozenLake-v0)\n", "EnvSpec(FrozenLake8x8-v0)\n", "EnvSpec(CliffWalking-v0)\n", "EnvSpec(NChain-v0)\n", "EnvSpec(Roulette-v0)\n", "EnvSpec(Taxi-v3)\n", "EnvSpec(GuessingGame-v0)\n", "EnvSpec(HotterColder-v0)\n", "EnvSpec(Reacher-v2)\n", "EnvSpec(Pusher-v2)\n", "EnvSpec(Thrower-v2)\n", "EnvSpec(Striker-v2)\n", "EnvSpec(InvertedPendulum-v2)\n", "EnvSpec(InvertedDoublePendulum-v2)\n", "EnvSpec(HalfCheetah-v2)\n", "EnvSpec(HalfCheetah-v3)\n", "EnvSpec(Hopper-v2)\n", "EnvSpec(Hopper-v3)\n", "EnvSpec(Swimmer-v2)\n", "EnvSpec(Swimmer-v3)\n", "EnvSpec(Walker2d-v2)\n", "EnvSpec(Walker2d-v3)\n", "EnvSpec(Ant-v2)\n", "EnvSpec(Ant-v3)\n", "EnvSpec(Humanoid-v2)\n", "EnvSpec(Humanoid-v3)\n", "EnvSpec(HumanoidStandup-v2)\n", "EnvSpec(FetchSlide-v1)\n", "EnvSpec(FetchPickAndPlace-v1)\n", "EnvSpec(FetchReach-v1)\n", "EnvSpec(FetchPush-v1)\n", "EnvSpec(HandReach-v0)\n", "EnvSpec(HandManipulateBlockRotateZ-v0)\n", "EnvSpec(HandManipulateBlockRotateZTouchSensors-v0)\n", "EnvSpec(HandManipulateBlockRotateZTouchSensors-v1)\n", "EnvSpec(HandManipulateBlockRotateParallel-v0)\n", "EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v0)\n", "EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v1)\n", "EnvSpec(HandManipulateBlockRotateXYZ-v0)\n", "EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v0)\n", "EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v1)\n", "EnvSpec(HandManipulateBlockFull-v0)\n", "EnvSpec(HandManipulateBlock-v0)\n", "EnvSpec(HandManipulateBlockTouchSensors-v0)\n", "EnvSpec(HandManipulateBlockTouchSensors-v1)\n", "EnvSpec(HandManipulateEggRotate-v0)\n", "EnvSpec(HandManipulateEggRotateTouchSensors-v0)\n", "EnvSpec(HandManipulateEggRotateTouchSensors-v1)\n", "EnvSpec(HandManipulateEggFull-v0)\n", "EnvSpec(HandManipulateEgg-v0)\n", "EnvSpec(HandManipulateEggTouchSensors-v0)\n", "EnvSpec(HandManipulateEggTouchSensors-v1)\n", "EnvSpec(HandManipulatePenRotate-v0)\n", "EnvSpec(HandManipulatePenRotateTouchSensors-v0)\n", "EnvSpec(HandManipulatePenRotateTouchSensors-v1)\n", "EnvSpec(HandManipulatePenFull-v0)\n", "EnvSpec(HandManipulatePen-v0)\n", "EnvSpec(HandManipulatePenTouchSensors-v0)\n", "EnvSpec(HandManipulatePenTouchSensors-v1)\n", "EnvSpec(FetchSlideDense-v1)\n", "EnvSpec(FetchPickAndPlaceDense-v1)\n", "EnvSpec(FetchReachDense-v1)\n", "EnvSpec(FetchPushDense-v1)\n", "EnvSpec(HandReachDense-v0)\n", "EnvSpec(HandManipulateBlockRotateZDense-v0)\n", "EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v1)\n", "EnvSpec(HandManipulateBlockRotateParallelDense-v0)\n", "EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v1)\n", "EnvSpec(HandManipulateBlockRotateXYZDense-v0)\n", "EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v1)\n", "EnvSpec(HandManipulateBlockFullDense-v0)\n", "EnvSpec(HandManipulateBlockDense-v0)\n", "EnvSpec(HandManipulateBlockTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateBlockTouchSensorsDense-v1)\n", "EnvSpec(HandManipulateEggRotateDense-v0)\n", "EnvSpec(HandManipulateEggRotateTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateEggRotateTouchSensorsDense-v1)\n", "EnvSpec(HandManipulateEggFullDense-v0)\n", "EnvSpec(HandManipulateEggDense-v0)\n", "EnvSpec(HandManipulateEggTouchSensorsDense-v0)\n", "EnvSpec(HandManipulateEggTouchSensorsDense-v1)\n", "EnvSpec(HandManipulatePenRotateDense-v0)\n", "EnvSpec(HandManipulatePenRotateTouchSensorsDense-v0)\n", "EnvSpec(HandManipulatePenRotateTouchSensorsDense-v1)\n", "EnvSpec(HandManipulatePenFullDense-v0)\n", "EnvSpec(HandManipulatePenDense-v0)\n", "EnvSpec(HandManipulatePenTouchSensorsDense-v0)\n", "EnvSpec(HandManipulatePenTouchSensorsDense-v1)\n", "EnvSpec(Adventure-v0)\n", "EnvSpec(Adventure-v4)\n", "EnvSpec(AdventureDeterministic-v0)\n", "EnvSpec(AdventureDeterministic-v4)\n", "EnvSpec(AdventureNoFrameskip-v0)\n", "EnvSpec(AdventureNoFrameskip-v4)\n", "EnvSpec(Adventure-ram-v0)\n", "EnvSpec(Adventure-ram-v4)\n", "EnvSpec(Adventure-ramDeterministic-v0)\n", "EnvSpec(Adventure-ramDeterministic-v4)\n", "EnvSpec(Adventure-ramNoFrameskip-v0)\n", "EnvSpec(Adventure-ramNoFrameskip-v4)\n", "EnvSpec(AirRaid-v0)\n", "EnvSpec(AirRaid-v4)\n", "EnvSpec(AirRaidDeterministic-v0)\n", "EnvSpec(AirRaidDeterministic-v4)\n", "EnvSpec(AirRaidNoFrameskip-v0)\n", "EnvSpec(AirRaidNoFrameskip-v4)\n", "EnvSpec(AirRaid-ram-v0)\n", "EnvSpec(AirRaid-ram-v4)\n", "EnvSpec(AirRaid-ramDeterministic-v0)\n", "EnvSpec(AirRaid-ramDeterministic-v4)\n", "EnvSpec(AirRaid-ramNoFrameskip-v0)\n", "EnvSpec(AirRaid-ramNoFrameskip-v4)\n", "EnvSpec(Alien-v0)\n", "EnvSpec(Alien-v4)\n", "EnvSpec(AlienDeterministic-v0)\n", "EnvSpec(AlienDeterministic-v4)\n", "EnvSpec(AlienNoFrameskip-v0)\n", "EnvSpec(AlienNoFrameskip-v4)\n", "EnvSpec(Alien-ram-v0)\n", "EnvSpec(Alien-ram-v4)\n", "EnvSpec(Alien-ramDeterministic-v0)\n", "EnvSpec(Alien-ramDeterministic-v4)\n", "EnvSpec(Alien-ramNoFrameskip-v0)\n", "EnvSpec(Alien-ramNoFrameskip-v4)\n", "EnvSpec(Amidar-v0)\n", "EnvSpec(Amidar-v4)\n", "EnvSpec(AmidarDeterministic-v0)\n", "EnvSpec(AmidarDeterministic-v4)\n", "EnvSpec(AmidarNoFrameskip-v0)\n", "EnvSpec(AmidarNoFrameskip-v4)\n", "EnvSpec(Amidar-ram-v0)\n", "EnvSpec(Amidar-ram-v4)\n", "EnvSpec(Amidar-ramDeterministic-v0)\n", "EnvSpec(Amidar-ramDeterministic-v4)\n", "EnvSpec(Amidar-ramNoFrameskip-v0)\n", "EnvSpec(Amidar-ramNoFrameskip-v4)\n", "EnvSpec(Assault-v0)\n", "EnvSpec(Assault-v4)\n", "EnvSpec(AssaultDeterministic-v0)\n", "EnvSpec(AssaultDeterministic-v4)\n", "EnvSpec(AssaultNoFrameskip-v0)\n", "EnvSpec(AssaultNoFrameskip-v4)\n", "EnvSpec(Assault-ram-v0)\n", "EnvSpec(Assault-ram-v4)\n", "EnvSpec(Assault-ramDeterministic-v0)\n", "EnvSpec(Assault-ramDeterministic-v4)\n", "EnvSpec(Assault-ramNoFrameskip-v0)\n", "EnvSpec(Assault-ramNoFrameskip-v4)\n", "EnvSpec(Asterix-v0)\n", "EnvSpec(Asterix-v4)\n", "EnvSpec(AsterixDeterministic-v0)\n", "EnvSpec(AsterixDeterministic-v4)\n", "EnvSpec(AsterixNoFrameskip-v0)\n", "EnvSpec(AsterixNoFrameskip-v4)\n", "EnvSpec(Asterix-ram-v0)\n", "EnvSpec(Asterix-ram-v4)\n", "EnvSpec(Asterix-ramDeterministic-v0)\n", "EnvSpec(Asterix-ramDeterministic-v4)\n", "EnvSpec(Asterix-ramNoFrameskip-v0)\n", "EnvSpec(Asterix-ramNoFrameskip-v4)\n", "EnvSpec(Asteroids-v0)\n", "EnvSpec(Asteroids-v4)\n", "EnvSpec(AsteroidsDeterministic-v0)\n", "EnvSpec(AsteroidsDeterministic-v4)\n", "EnvSpec(AsteroidsNoFrameskip-v0)\n", "EnvSpec(AsteroidsNoFrameskip-v4)\n", "EnvSpec(Asteroids-ram-v0)\n", "EnvSpec(Asteroids-ram-v4)\n", "EnvSpec(Asteroids-ramDeterministic-v0)\n", "EnvSpec(Asteroids-ramDeterministic-v4)\n", "EnvSpec(Asteroids-ramNoFrameskip-v0)\n", "EnvSpec(Asteroids-ramNoFrameskip-v4)\n", "EnvSpec(Atlantis-v0)\n", "EnvSpec(Atlantis-v4)\n", "EnvSpec(AtlantisDeterministic-v0)\n", "EnvSpec(AtlantisDeterministic-v4)\n", "EnvSpec(AtlantisNoFrameskip-v0)\n", "EnvSpec(AtlantisNoFrameskip-v4)\n", "EnvSpec(Atlantis-ram-v0)\n", "EnvSpec(Atlantis-ram-v4)\n", "EnvSpec(Atlantis-ramDeterministic-v0)\n", "EnvSpec(Atlantis-ramDeterministic-v4)\n", "EnvSpec(Atlantis-ramNoFrameskip-v0)\n", "EnvSpec(Atlantis-ramNoFrameskip-v4)\n", "EnvSpec(BankHeist-v0)\n", "EnvSpec(BankHeist-v4)\n", "EnvSpec(BankHeistDeterministic-v0)\n", "EnvSpec(BankHeistDeterministic-v4)\n", "EnvSpec(BankHeistNoFrameskip-v0)\n", "EnvSpec(BankHeistNoFrameskip-v4)\n", "EnvSpec(BankHeist-ram-v0)\n", "EnvSpec(BankHeist-ram-v4)\n", "EnvSpec(BankHeist-ramDeterministic-v0)\n", "EnvSpec(BankHeist-ramDeterministic-v4)\n", "EnvSpec(BankHeist-ramNoFrameskip-v0)\n", "EnvSpec(BankHeist-ramNoFrameskip-v4)\n", "EnvSpec(BattleZone-v0)\n", "EnvSpec(BattleZone-v4)\n", "EnvSpec(BattleZoneDeterministic-v0)\n", "EnvSpec(BattleZoneDeterministic-v4)\n", "EnvSpec(BattleZoneNoFrameskip-v0)\n", "EnvSpec(BattleZoneNoFrameskip-v4)\n", "EnvSpec(BattleZone-ram-v0)\n", "EnvSpec(BattleZone-ram-v4)\n", "EnvSpec(BattleZone-ramDeterministic-v0)\n", "EnvSpec(BattleZone-ramDeterministic-v4)\n", "EnvSpec(BattleZone-ramNoFrameskip-v0)\n", "EnvSpec(BattleZone-ramNoFrameskip-v4)\n", "EnvSpec(BeamRider-v0)\n", "EnvSpec(BeamRider-v4)\n", "EnvSpec(BeamRiderDeterministic-v0)\n", "EnvSpec(BeamRiderDeterministic-v4)\n", "EnvSpec(BeamRiderNoFrameskip-v0)\n", "EnvSpec(BeamRiderNoFrameskip-v4)\n", "EnvSpec(BeamRider-ram-v0)\n", "EnvSpec(BeamRider-ram-v4)\n", "EnvSpec(BeamRider-ramDeterministic-v0)\n", "EnvSpec(BeamRider-ramDeterministic-v4)\n", "EnvSpec(BeamRider-ramNoFrameskip-v0)\n", "EnvSpec(BeamRider-ramNoFrameskip-v4)\n", "EnvSpec(Berzerk-v0)\n", "EnvSpec(Berzerk-v4)\n", "EnvSpec(BerzerkDeterministic-v0)\n", "EnvSpec(BerzerkDeterministic-v4)\n", "EnvSpec(BerzerkNoFrameskip-v0)\n", "EnvSpec(BerzerkNoFrameskip-v4)\n", "EnvSpec(Berzerk-ram-v0)\n", "EnvSpec(Berzerk-ram-v4)\n", "EnvSpec(Berzerk-ramDeterministic-v0)\n", "EnvSpec(Berzerk-ramDeterministic-v4)\n", "EnvSpec(Berzerk-ramNoFrameskip-v0)\n", "EnvSpec(Berzerk-ramNoFrameskip-v4)\n", "EnvSpec(Bowling-v0)\n", "EnvSpec(Bowling-v4)\n", "EnvSpec(BowlingDeterministic-v0)\n", "EnvSpec(BowlingDeterministic-v4)\n", "EnvSpec(BowlingNoFrameskip-v0)\n", "EnvSpec(BowlingNoFrameskip-v4)\n", "EnvSpec(Bowling-ram-v0)\n", "EnvSpec(Bowling-ram-v4)\n", "EnvSpec(Bowling-ramDeterministic-v0)\n", "EnvSpec(Bowling-ramDeterministic-v4)\n", "EnvSpec(Bowling-ramNoFrameskip-v0)\n", "EnvSpec(Bowling-ramNoFrameskip-v4)\n", "EnvSpec(Boxing-v0)\n", "EnvSpec(Boxing-v4)\n", "EnvSpec(BoxingDeterministic-v0)\n", "EnvSpec(BoxingDeterministic-v4)\n", "EnvSpec(BoxingNoFrameskip-v0)\n", "EnvSpec(BoxingNoFrameskip-v4)\n", "EnvSpec(Boxing-ram-v0)\n", "EnvSpec(Boxing-ram-v4)\n", "EnvSpec(Boxing-ramDeterministic-v0)\n", "EnvSpec(Boxing-ramDeterministic-v4)\n", "EnvSpec(Boxing-ramNoFrameskip-v0)\n", "EnvSpec(Boxing-ramNoFrameskip-v4)\n", "EnvSpec(Breakout-v0)\n", "EnvSpec(Breakout-v4)\n", "EnvSpec(BreakoutDeterministic-v0)\n", "EnvSpec(BreakoutDeterministic-v4)\n", "EnvSpec(BreakoutNoFrameskip-v0)\n", "EnvSpec(BreakoutNoFrameskip-v4)\n", "EnvSpec(Breakout-ram-v0)\n", "EnvSpec(Breakout-ram-v4)\n", "EnvSpec(Breakout-ramDeterministic-v0)\n", "EnvSpec(Breakout-ramDeterministic-v4)\n", "EnvSpec(Breakout-ramNoFrameskip-v0)\n", "EnvSpec(Breakout-ramNoFrameskip-v4)\n", "EnvSpec(Carnival-v0)\n", "EnvSpec(Carnival-v4)\n", "EnvSpec(CarnivalDeterministic-v0)\n", "EnvSpec(CarnivalDeterministic-v4)\n", "EnvSpec(CarnivalNoFrameskip-v0)\n", "EnvSpec(CarnivalNoFrameskip-v4)\n", "EnvSpec(Carnival-ram-v0)\n", "EnvSpec(Carnival-ram-v4)\n", "EnvSpec(Carnival-ramDeterministic-v0)\n", "EnvSpec(Carnival-ramDeterministic-v4)\n", "EnvSpec(Carnival-ramNoFrameskip-v0)\n", "EnvSpec(Carnival-ramNoFrameskip-v4)\n", "EnvSpec(Centipede-v0)\n", "EnvSpec(Centipede-v4)\n", "EnvSpec(CentipedeDeterministic-v0)\n", "EnvSpec(CentipedeDeterministic-v4)\n", "EnvSpec(CentipedeNoFrameskip-v0)\n", "EnvSpec(CentipedeNoFrameskip-v4)\n", "EnvSpec(Centipede-ram-v0)\n", "EnvSpec(Centipede-ram-v4)\n", "EnvSpec(Centipede-ramDeterministic-v0)\n", "EnvSpec(Centipede-ramDeterministic-v4)\n", "EnvSpec(Centipede-ramNoFrameskip-v0)\n", "EnvSpec(Centipede-ramNoFrameskip-v4)\n", "EnvSpec(ChopperCommand-v0)\n", "EnvSpec(ChopperCommand-v4)\n", "EnvSpec(ChopperCommandDeterministic-v0)\n", "EnvSpec(ChopperCommandDeterministic-v4)\n", "EnvSpec(ChopperCommandNoFrameskip-v0)\n", "EnvSpec(ChopperCommandNoFrameskip-v4)\n", "EnvSpec(ChopperCommand-ram-v0)\n", "EnvSpec(ChopperCommand-ram-v4)\n", "EnvSpec(ChopperCommand-ramDeterministic-v0)\n", "EnvSpec(ChopperCommand-ramDeterministic-v4)\n", "EnvSpec(ChopperCommand-ramNoFrameskip-v0)\n", "EnvSpec(ChopperCommand-ramNoFrameskip-v4)\n", "EnvSpec(CrazyClimber-v0)\n", "EnvSpec(CrazyClimber-v4)\n", "EnvSpec(CrazyClimberDeterministic-v0)\n", "EnvSpec(CrazyClimberDeterministic-v4)\n", "EnvSpec(CrazyClimberNoFrameskip-v0)\n", "EnvSpec(CrazyClimberNoFrameskip-v4)\n", "EnvSpec(CrazyClimber-ram-v0)\n", "EnvSpec(CrazyClimber-ram-v4)\n", "EnvSpec(CrazyClimber-ramDeterministic-v0)\n", "EnvSpec(CrazyClimber-ramDeterministic-v4)\n", "EnvSpec(CrazyClimber-ramNoFrameskip-v0)\n", "EnvSpec(CrazyClimber-ramNoFrameskip-v4)\n", "EnvSpec(Defender-v0)\n", "EnvSpec(Defender-v4)\n", "EnvSpec(DefenderDeterministic-v0)\n", "EnvSpec(DefenderDeterministic-v4)\n", "EnvSpec(DefenderNoFrameskip-v0)\n", "EnvSpec(DefenderNoFrameskip-v4)\n", "EnvSpec(Defender-ram-v0)\n", "EnvSpec(Defender-ram-v4)\n", "EnvSpec(Defender-ramDeterministic-v0)\n", "EnvSpec(Defender-ramDeterministic-v4)\n", "EnvSpec(Defender-ramNoFrameskip-v0)\n", "EnvSpec(Defender-ramNoFrameskip-v4)\n", "EnvSpec(DemonAttack-v0)\n", "EnvSpec(DemonAttack-v4)\n", "EnvSpec(DemonAttackDeterministic-v0)\n", "EnvSpec(DemonAttackDeterministic-v4)\n", "EnvSpec(DemonAttackNoFrameskip-v0)\n", "EnvSpec(DemonAttackNoFrameskip-v4)\n", "EnvSpec(DemonAttack-ram-v0)\n", "EnvSpec(DemonAttack-ram-v4)\n", "EnvSpec(DemonAttack-ramDeterministic-v0)\n", "EnvSpec(DemonAttack-ramDeterministic-v4)\n", "EnvSpec(DemonAttack-ramNoFrameskip-v0)\n", "EnvSpec(DemonAttack-ramNoFrameskip-v4)\n", "EnvSpec(DoubleDunk-v0)\n", "EnvSpec(DoubleDunk-v4)\n", "EnvSpec(DoubleDunkDeterministic-v0)\n", "EnvSpec(DoubleDunkDeterministic-v4)\n", "EnvSpec(DoubleDunkNoFrameskip-v0)\n", "EnvSpec(DoubleDunkNoFrameskip-v4)\n", "EnvSpec(DoubleDunk-ram-v0)\n", "EnvSpec(DoubleDunk-ram-v4)\n", "EnvSpec(DoubleDunk-ramDeterministic-v0)\n", "EnvSpec(DoubleDunk-ramDeterministic-v4)\n", "EnvSpec(DoubleDunk-ramNoFrameskip-v0)\n", "EnvSpec(DoubleDunk-ramNoFrameskip-v4)\n", "EnvSpec(ElevatorAction-v0)\n", "EnvSpec(ElevatorAction-v4)\n", "EnvSpec(ElevatorActionDeterministic-v0)\n", "EnvSpec(ElevatorActionDeterministic-v4)\n", "EnvSpec(ElevatorActionNoFrameskip-v0)\n", "EnvSpec(ElevatorActionNoFrameskip-v4)\n", "EnvSpec(ElevatorAction-ram-v0)\n", "EnvSpec(ElevatorAction-ram-v4)\n", "EnvSpec(ElevatorAction-ramDeterministic-v0)\n", "EnvSpec(ElevatorAction-ramDeterministic-v4)\n", "EnvSpec(ElevatorAction-ramNoFrameskip-v0)\n", "EnvSpec(ElevatorAction-ramNoFrameskip-v4)\n", "EnvSpec(Enduro-v0)\n", "EnvSpec(Enduro-v4)\n", "EnvSpec(EnduroDeterministic-v0)\n", "EnvSpec(EnduroDeterministic-v4)\n", "EnvSpec(EnduroNoFrameskip-v0)\n", "EnvSpec(EnduroNoFrameskip-v4)\n", "EnvSpec(Enduro-ram-v0)\n", "EnvSpec(Enduro-ram-v4)\n", "EnvSpec(Enduro-ramDeterministic-v0)\n", "EnvSpec(Enduro-ramDeterministic-v4)\n", "EnvSpec(Enduro-ramNoFrameskip-v0)\n", "EnvSpec(Enduro-ramNoFrameskip-v4)\n", "EnvSpec(FishingDerby-v0)\n", "EnvSpec(FishingDerby-v4)\n", "EnvSpec(FishingDerbyDeterministic-v0)\n", "EnvSpec(FishingDerbyDeterministic-v4)\n", "EnvSpec(FishingDerbyNoFrameskip-v0)\n", "EnvSpec(FishingDerbyNoFrameskip-v4)\n", "EnvSpec(FishingDerby-ram-v0)\n", "EnvSpec(FishingDerby-ram-v4)\n", "EnvSpec(FishingDerby-ramDeterministic-v0)\n", "EnvSpec(FishingDerby-ramDeterministic-v4)\n", "EnvSpec(FishingDerby-ramNoFrameskip-v0)\n", "EnvSpec(FishingDerby-ramNoFrameskip-v4)\n", "EnvSpec(Freeway-v0)\n", "EnvSpec(Freeway-v4)\n", "EnvSpec(FreewayDeterministic-v0)\n", "EnvSpec(FreewayDeterministic-v4)\n", "EnvSpec(FreewayNoFrameskip-v0)\n", "EnvSpec(FreewayNoFrameskip-v4)\n", "EnvSpec(Freeway-ram-v0)\n", "EnvSpec(Freeway-ram-v4)\n", "EnvSpec(Freeway-ramDeterministic-v0)\n", "EnvSpec(Freeway-ramDeterministic-v4)\n", "EnvSpec(Freeway-ramNoFrameskip-v0)\n", "EnvSpec(Freeway-ramNoFrameskip-v4)\n", "EnvSpec(Frostbite-v0)\n", "EnvSpec(Frostbite-v4)\n", "EnvSpec(FrostbiteDeterministic-v0)\n", "EnvSpec(FrostbiteDeterministic-v4)\n", "EnvSpec(FrostbiteNoFrameskip-v0)\n", "EnvSpec(FrostbiteNoFrameskip-v4)\n", "EnvSpec(Frostbite-ram-v0)\n", "EnvSpec(Frostbite-ram-v4)\n", "EnvSpec(Frostbite-ramDeterministic-v0)\n", "EnvSpec(Frostbite-ramDeterministic-v4)\n", "EnvSpec(Frostbite-ramNoFrameskip-v0)\n", "EnvSpec(Frostbite-ramNoFrameskip-v4)\n", "EnvSpec(Gopher-v0)\n", "EnvSpec(Gopher-v4)\n", "EnvSpec(GopherDeterministic-v0)\n", "EnvSpec(GopherDeterministic-v4)\n", "EnvSpec(GopherNoFrameskip-v0)\n", "EnvSpec(GopherNoFrameskip-v4)\n", "EnvSpec(Gopher-ram-v0)\n", "EnvSpec(Gopher-ram-v4)\n", "EnvSpec(Gopher-ramDeterministic-v0)\n", "EnvSpec(Gopher-ramDeterministic-v4)\n", "EnvSpec(Gopher-ramNoFrameskip-v0)\n", "EnvSpec(Gopher-ramNoFrameskip-v4)\n", "EnvSpec(Gravitar-v0)\n", "EnvSpec(Gravitar-v4)\n", "EnvSpec(GravitarDeterministic-v0)\n", "EnvSpec(GravitarDeterministic-v4)\n", "EnvSpec(GravitarNoFrameskip-v0)\n", "EnvSpec(GravitarNoFrameskip-v4)\n", "EnvSpec(Gravitar-ram-v0)\n", "EnvSpec(Gravitar-ram-v4)\n", "EnvSpec(Gravitar-ramDeterministic-v0)\n", "EnvSpec(Gravitar-ramDeterministic-v4)\n", "EnvSpec(Gravitar-ramNoFrameskip-v0)\n", "EnvSpec(Gravitar-ramNoFrameskip-v4)\n", "EnvSpec(Hero-v0)\n", "EnvSpec(Hero-v4)\n", "EnvSpec(HeroDeterministic-v0)\n", "EnvSpec(HeroDeterministic-v4)\n", "EnvSpec(HeroNoFrameskip-v0)\n", "EnvSpec(HeroNoFrameskip-v4)\n", "EnvSpec(Hero-ram-v0)\n", "EnvSpec(Hero-ram-v4)\n", "EnvSpec(Hero-ramDeterministic-v0)\n", "EnvSpec(Hero-ramDeterministic-v4)\n", "EnvSpec(Hero-ramNoFrameskip-v0)\n", "EnvSpec(Hero-ramNoFrameskip-v4)\n", "EnvSpec(IceHockey-v0)\n", "EnvSpec(IceHockey-v4)\n", "EnvSpec(IceHockeyDeterministic-v0)\n", "EnvSpec(IceHockeyDeterministic-v4)\n", "EnvSpec(IceHockeyNoFrameskip-v0)\n", "EnvSpec(IceHockeyNoFrameskip-v4)\n", "EnvSpec(IceHockey-ram-v0)\n", "EnvSpec(IceHockey-ram-v4)\n", "EnvSpec(IceHockey-ramDeterministic-v0)\n", "EnvSpec(IceHockey-ramDeterministic-v4)\n", "EnvSpec(IceHockey-ramNoFrameskip-v0)\n", "EnvSpec(IceHockey-ramNoFrameskip-v4)\n", "EnvSpec(Jamesbond-v0)\n", "EnvSpec(Jamesbond-v4)\n", "EnvSpec(JamesbondDeterministic-v0)\n", "EnvSpec(JamesbondDeterministic-v4)\n", "EnvSpec(JamesbondNoFrameskip-v0)\n", "EnvSpec(JamesbondNoFrameskip-v4)\n", "EnvSpec(Jamesbond-ram-v0)\n", "EnvSpec(Jamesbond-ram-v4)\n", "EnvSpec(Jamesbond-ramDeterministic-v0)\n", "EnvSpec(Jamesbond-ramDeterministic-v4)\n", "EnvSpec(Jamesbond-ramNoFrameskip-v0)\n", "EnvSpec(Jamesbond-ramNoFrameskip-v4)\n", "EnvSpec(JourneyEscape-v0)\n", "EnvSpec(JourneyEscape-v4)\n", "EnvSpec(JourneyEscapeDeterministic-v0)\n", "EnvSpec(JourneyEscapeDeterministic-v4)\n", "EnvSpec(JourneyEscapeNoFrameskip-v0)\n", "EnvSpec(JourneyEscapeNoFrameskip-v4)\n", "EnvSpec(JourneyEscape-ram-v0)\n", "EnvSpec(JourneyEscape-ram-v4)\n", "EnvSpec(JourneyEscape-ramDeterministic-v0)\n", "EnvSpec(JourneyEscape-ramDeterministic-v4)\n", "EnvSpec(JourneyEscape-ramNoFrameskip-v0)\n", "EnvSpec(JourneyEscape-ramNoFrameskip-v4)\n", "EnvSpec(Kangaroo-v0)\n", "EnvSpec(Kangaroo-v4)\n", "EnvSpec(KangarooDeterministic-v0)\n", "EnvSpec(KangarooDeterministic-v4)\n", "EnvSpec(KangarooNoFrameskip-v0)\n", "EnvSpec(KangarooNoFrameskip-v4)\n", "EnvSpec(Kangaroo-ram-v0)\n", "EnvSpec(Kangaroo-ram-v4)\n", "EnvSpec(Kangaroo-ramDeterministic-v0)\n", "EnvSpec(Kangaroo-ramDeterministic-v4)\n", "EnvSpec(Kangaroo-ramNoFrameskip-v0)\n", "EnvSpec(Kangaroo-ramNoFrameskip-v4)\n", "EnvSpec(Krull-v0)\n", "EnvSpec(Krull-v4)\n", "EnvSpec(KrullDeterministic-v0)\n", "EnvSpec(KrullDeterministic-v4)\n", "EnvSpec(KrullNoFrameskip-v0)\n", "EnvSpec(KrullNoFrameskip-v4)\n", "EnvSpec(Krull-ram-v0)\n", "EnvSpec(Krull-ram-v4)\n", "EnvSpec(Krull-ramDeterministic-v0)\n", "EnvSpec(Krull-ramDeterministic-v4)\n", "EnvSpec(Krull-ramNoFrameskip-v0)\n", "EnvSpec(Krull-ramNoFrameskip-v4)\n", "EnvSpec(KungFuMaster-v0)\n", "EnvSpec(KungFuMaster-v4)\n", "EnvSpec(KungFuMasterDeterministic-v0)\n", "EnvSpec(KungFuMasterDeterministic-v4)\n", "EnvSpec(KungFuMasterNoFrameskip-v0)\n", "EnvSpec(KungFuMasterNoFrameskip-v4)\n", "EnvSpec(KungFuMaster-ram-v0)\n", "EnvSpec(KungFuMaster-ram-v4)\n", "EnvSpec(KungFuMaster-ramDeterministic-v0)\n", "EnvSpec(KungFuMaster-ramDeterministic-v4)\n", "EnvSpec(KungFuMaster-ramNoFrameskip-v0)\n", "EnvSpec(KungFuMaster-ramNoFrameskip-v4)\n", "EnvSpec(MontezumaRevenge-v0)\n", "EnvSpec(MontezumaRevenge-v4)\n", "EnvSpec(MontezumaRevengeDeterministic-v0)\n", "EnvSpec(MontezumaRevengeDeterministic-v4)\n", "EnvSpec(MontezumaRevengeNoFrameskip-v0)\n", "EnvSpec(MontezumaRevengeNoFrameskip-v4)\n", "EnvSpec(MontezumaRevenge-ram-v0)\n", "EnvSpec(MontezumaRevenge-ram-v4)\n", "EnvSpec(MontezumaRevenge-ramDeterministic-v0)\n", "EnvSpec(MontezumaRevenge-ramDeterministic-v4)\n", "EnvSpec(MontezumaRevenge-ramNoFrameskip-v0)\n", "EnvSpec(MontezumaRevenge-ramNoFrameskip-v4)\n", "EnvSpec(MsPacman-v0)\n", "EnvSpec(MsPacman-v4)\n", "EnvSpec(MsPacmanDeterministic-v0)\n", "EnvSpec(MsPacmanDeterministic-v4)\n", "EnvSpec(MsPacmanNoFrameskip-v0)\n", "EnvSpec(MsPacmanNoFrameskip-v4)\n", "EnvSpec(MsPacman-ram-v0)\n", "EnvSpec(MsPacman-ram-v4)\n", "EnvSpec(MsPacman-ramDeterministic-v0)\n", "EnvSpec(MsPacman-ramDeterministic-v4)\n", "EnvSpec(MsPacman-ramNoFrameskip-v0)\n", "EnvSpec(MsPacman-ramNoFrameskip-v4)\n", "EnvSpec(NameThisGame-v0)\n", "EnvSpec(NameThisGame-v4)\n", "EnvSpec(NameThisGameDeterministic-v0)\n", "EnvSpec(NameThisGameDeterministic-v4)\n", "EnvSpec(NameThisGameNoFrameskip-v0)\n", "EnvSpec(NameThisGameNoFrameskip-v4)\n", "EnvSpec(NameThisGame-ram-v0)\n", "EnvSpec(NameThisGame-ram-v4)\n", "EnvSpec(NameThisGame-ramDeterministic-v0)\n", "EnvSpec(NameThisGame-ramDeterministic-v4)\n", "EnvSpec(NameThisGame-ramNoFrameskip-v0)\n", "EnvSpec(NameThisGame-ramNoFrameskip-v4)\n", "EnvSpec(Phoenix-v0)\n", "EnvSpec(Phoenix-v4)\n", "EnvSpec(PhoenixDeterministic-v0)\n", "EnvSpec(PhoenixDeterministic-v4)\n", "EnvSpec(PhoenixNoFrameskip-v0)\n", "EnvSpec(PhoenixNoFrameskip-v4)\n", "EnvSpec(Phoenix-ram-v0)\n", "EnvSpec(Phoenix-ram-v4)\n", "EnvSpec(Phoenix-ramDeterministic-v0)\n", "EnvSpec(Phoenix-ramDeterministic-v4)\n", "EnvSpec(Phoenix-ramNoFrameskip-v0)\n", "EnvSpec(Phoenix-ramNoFrameskip-v4)\n", "EnvSpec(Pitfall-v0)\n", "EnvSpec(Pitfall-v4)\n", "EnvSpec(PitfallDeterministic-v0)\n", "EnvSpec(PitfallDeterministic-v4)\n", "EnvSpec(PitfallNoFrameskip-v0)\n", "EnvSpec(PitfallNoFrameskip-v4)\n", "EnvSpec(Pitfall-ram-v0)\n", "EnvSpec(Pitfall-ram-v4)\n", "EnvSpec(Pitfall-ramDeterministic-v0)\n", "EnvSpec(Pitfall-ramDeterministic-v4)\n", "EnvSpec(Pitfall-ramNoFrameskip-v0)\n", "EnvSpec(Pitfall-ramNoFrameskip-v4)\n", "EnvSpec(Pong-v0)\n", "EnvSpec(Pong-v4)\n", "EnvSpec(PongDeterministic-v0)\n", "EnvSpec(PongDeterministic-v4)\n", "EnvSpec(PongNoFrameskip-v0)\n", "EnvSpec(PongNoFrameskip-v4)\n", "EnvSpec(Pong-ram-v0)\n", "EnvSpec(Pong-ram-v4)\n", "EnvSpec(Pong-ramDeterministic-v0)\n", "EnvSpec(Pong-ramDeterministic-v4)\n", "EnvSpec(Pong-ramNoFrameskip-v0)\n", "EnvSpec(Pong-ramNoFrameskip-v4)\n", "EnvSpec(Pooyan-v0)\n", "EnvSpec(Pooyan-v4)\n", "EnvSpec(PooyanDeterministic-v0)\n", "EnvSpec(PooyanDeterministic-v4)\n", "EnvSpec(PooyanNoFrameskip-v0)\n", "EnvSpec(PooyanNoFrameskip-v4)\n", "EnvSpec(Pooyan-ram-v0)\n", "EnvSpec(Pooyan-ram-v4)\n", "EnvSpec(Pooyan-ramDeterministic-v0)\n", "EnvSpec(Pooyan-ramDeterministic-v4)\n", "EnvSpec(Pooyan-ramNoFrameskip-v0)\n", "EnvSpec(Pooyan-ramNoFrameskip-v4)\n", "EnvSpec(PrivateEye-v0)\n", "EnvSpec(PrivateEye-v4)\n", "EnvSpec(PrivateEyeDeterministic-v0)\n", "EnvSpec(PrivateEyeDeterministic-v4)\n", "EnvSpec(PrivateEyeNoFrameskip-v0)\n", "EnvSpec(PrivateEyeNoFrameskip-v4)\n", "EnvSpec(PrivateEye-ram-v0)\n", "EnvSpec(PrivateEye-ram-v4)\n", "EnvSpec(PrivateEye-ramDeterministic-v0)\n", "EnvSpec(PrivateEye-ramDeterministic-v4)\n", "EnvSpec(PrivateEye-ramNoFrameskip-v0)\n", "EnvSpec(PrivateEye-ramNoFrameskip-v4)\n", "EnvSpec(Qbert-v0)\n", "EnvSpec(Qbert-v4)\n", "EnvSpec(QbertDeterministic-v0)\n", "EnvSpec(QbertDeterministic-v4)\n", "EnvSpec(QbertNoFrameskip-v0)\n", "EnvSpec(QbertNoFrameskip-v4)\n", "EnvSpec(Qbert-ram-v0)\n", "EnvSpec(Qbert-ram-v4)\n", "EnvSpec(Qbert-ramDeterministic-v0)\n", "EnvSpec(Qbert-ramDeterministic-v4)\n", "EnvSpec(Qbert-ramNoFrameskip-v0)\n", "EnvSpec(Qbert-ramNoFrameskip-v4)\n", "EnvSpec(Riverraid-v0)\n", "EnvSpec(Riverraid-v4)\n", "EnvSpec(RiverraidDeterministic-v0)\n", "EnvSpec(RiverraidDeterministic-v4)\n", "EnvSpec(RiverraidNoFrameskip-v0)\n", "EnvSpec(RiverraidNoFrameskip-v4)\n", "EnvSpec(Riverraid-ram-v0)\n", "EnvSpec(Riverraid-ram-v4)\n", "EnvSpec(Riverraid-ramDeterministic-v0)\n", "EnvSpec(Riverraid-ramDeterministic-v4)\n", "EnvSpec(Riverraid-ramNoFrameskip-v0)\n", "EnvSpec(Riverraid-ramNoFrameskip-v4)\n", "EnvSpec(RoadRunner-v0)\n", "EnvSpec(RoadRunner-v4)\n", "EnvSpec(RoadRunnerDeterministic-v0)\n", "EnvSpec(RoadRunnerDeterministic-v4)\n", "EnvSpec(RoadRunnerNoFrameskip-v0)\n", "EnvSpec(RoadRunnerNoFrameskip-v4)\n", "EnvSpec(RoadRunner-ram-v0)\n", "EnvSpec(RoadRunner-ram-v4)\n", "EnvSpec(RoadRunner-ramDeterministic-v0)\n", "EnvSpec(RoadRunner-ramDeterministic-v4)\n", "EnvSpec(RoadRunner-ramNoFrameskip-v0)\n", "EnvSpec(RoadRunner-ramNoFrameskip-v4)\n", "EnvSpec(Robotank-v0)\n", "EnvSpec(Robotank-v4)\n", "EnvSpec(RobotankDeterministic-v0)\n", "EnvSpec(RobotankDeterministic-v4)\n", "EnvSpec(RobotankNoFrameskip-v0)\n", "EnvSpec(RobotankNoFrameskip-v4)\n", "EnvSpec(Robotank-ram-v0)\n", "EnvSpec(Robotank-ram-v4)\n", "EnvSpec(Robotank-ramDeterministic-v0)\n", "EnvSpec(Robotank-ramDeterministic-v4)\n", "EnvSpec(Robotank-ramNoFrameskip-v0)\n", "EnvSpec(Robotank-ramNoFrameskip-v4)\n", "EnvSpec(Seaquest-v0)\n", "EnvSpec(Seaquest-v4)\n", "EnvSpec(SeaquestDeterministic-v0)\n", "EnvSpec(SeaquestDeterministic-v4)\n", "EnvSpec(SeaquestNoFrameskip-v0)\n", "EnvSpec(SeaquestNoFrameskip-v4)\n", "EnvSpec(Seaquest-ram-v0)\n", "EnvSpec(Seaquest-ram-v4)\n", "EnvSpec(Seaquest-ramDeterministic-v0)\n", "EnvSpec(Seaquest-ramDeterministic-v4)\n", "EnvSpec(Seaquest-ramNoFrameskip-v0)\n", "EnvSpec(Seaquest-ramNoFrameskip-v4)\n", "EnvSpec(Skiing-v0)\n", "EnvSpec(Skiing-v4)\n", "EnvSpec(SkiingDeterministic-v0)\n", "EnvSpec(SkiingDeterministic-v4)\n", "EnvSpec(SkiingNoFrameskip-v0)\n", "EnvSpec(SkiingNoFrameskip-v4)\n", "EnvSpec(Skiing-ram-v0)\n", "EnvSpec(Skiing-ram-v4)\n", "EnvSpec(Skiing-ramDeterministic-v0)\n", "EnvSpec(Skiing-ramDeterministic-v4)\n", "EnvSpec(Skiing-ramNoFrameskip-v0)\n", "EnvSpec(Skiing-ramNoFrameskip-v4)\n", "EnvSpec(Solaris-v0)\n", "EnvSpec(Solaris-v4)\n", "EnvSpec(SolarisDeterministic-v0)\n", "EnvSpec(SolarisDeterministic-v4)\n", "EnvSpec(SolarisNoFrameskip-v0)\n", "EnvSpec(SolarisNoFrameskip-v4)\n", "EnvSpec(Solaris-ram-v0)\n", "EnvSpec(Solaris-ram-v4)\n", "EnvSpec(Solaris-ramDeterministic-v0)\n", "EnvSpec(Solaris-ramDeterministic-v4)\n", "EnvSpec(Solaris-ramNoFrameskip-v0)\n", "EnvSpec(Solaris-ramNoFrameskip-v4)\n", "EnvSpec(SpaceInvaders-v0)\n", "EnvSpec(SpaceInvaders-v4)\n", "EnvSpec(SpaceInvadersDeterministic-v0)\n", "EnvSpec(SpaceInvadersDeterministic-v4)\n", "EnvSpec(SpaceInvadersNoFrameskip-v0)\n", "EnvSpec(SpaceInvadersNoFrameskip-v4)\n", "EnvSpec(SpaceInvaders-ram-v0)\n", "EnvSpec(SpaceInvaders-ram-v4)\n", "EnvSpec(SpaceInvaders-ramDeterministic-v0)\n", "EnvSpec(SpaceInvaders-ramDeterministic-v4)\n", "EnvSpec(SpaceInvaders-ramNoFrameskip-v0)\n", "EnvSpec(SpaceInvaders-ramNoFrameskip-v4)\n", "EnvSpec(StarGunner-v0)\n", "EnvSpec(StarGunner-v4)\n", "EnvSpec(StarGunnerDeterministic-v0)\n", "EnvSpec(StarGunnerDeterministic-v4)\n", "EnvSpec(StarGunnerNoFrameskip-v0)\n", "EnvSpec(StarGunnerNoFrameskip-v4)\n", "EnvSpec(StarGunner-ram-v0)\n", "EnvSpec(StarGunner-ram-v4)\n", "EnvSpec(StarGunner-ramDeterministic-v0)\n", "EnvSpec(StarGunner-ramDeterministic-v4)\n", "EnvSpec(StarGunner-ramNoFrameskip-v0)\n", "EnvSpec(StarGunner-ramNoFrameskip-v4)\n", "EnvSpec(Tennis-v0)\n", "EnvSpec(Tennis-v4)\n", "EnvSpec(TennisDeterministic-v0)\n", "EnvSpec(TennisDeterministic-v4)\n", "EnvSpec(TennisNoFrameskip-v0)\n", "EnvSpec(TennisNoFrameskip-v4)\n", "EnvSpec(Tennis-ram-v0)\n", "EnvSpec(Tennis-ram-v4)\n", "EnvSpec(Tennis-ramDeterministic-v0)\n", "EnvSpec(Tennis-ramDeterministic-v4)\n", "EnvSpec(Tennis-ramNoFrameskip-v0)\n", "EnvSpec(Tennis-ramNoFrameskip-v4)\n", "EnvSpec(TimePilot-v0)\n", "EnvSpec(TimePilot-v4)\n", "EnvSpec(TimePilotDeterministic-v0)\n", "EnvSpec(TimePilotDeterministic-v4)\n", "EnvSpec(TimePilotNoFrameskip-v0)\n", "EnvSpec(TimePilotNoFrameskip-v4)\n", "EnvSpec(TimePilot-ram-v0)\n", "EnvSpec(TimePilot-ram-v4)\n", "EnvSpec(TimePilot-ramDeterministic-v0)\n", "EnvSpec(TimePilot-ramDeterministic-v4)\n", "EnvSpec(TimePilot-ramNoFrameskip-v0)\n", "EnvSpec(TimePilot-ramNoFrameskip-v4)\n", "EnvSpec(Tutankham-v0)\n", "EnvSpec(Tutankham-v4)\n", "EnvSpec(TutankhamDeterministic-v0)\n", "EnvSpec(TutankhamDeterministic-v4)\n", "EnvSpec(TutankhamNoFrameskip-v0)\n", "EnvSpec(TutankhamNoFrameskip-v4)\n", "EnvSpec(Tutankham-ram-v0)\n", "EnvSpec(Tutankham-ram-v4)\n", "EnvSpec(Tutankham-ramDeterministic-v0)\n", "EnvSpec(Tutankham-ramDeterministic-v4)\n", "EnvSpec(Tutankham-ramNoFrameskip-v0)\n", "EnvSpec(Tutankham-ramNoFrameskip-v4)\n", "EnvSpec(UpNDown-v0)\n", "EnvSpec(UpNDown-v4)\n", "EnvSpec(UpNDownDeterministic-v0)\n", "EnvSpec(UpNDownDeterministic-v4)\n", "EnvSpec(UpNDownNoFrameskip-v0)\n", "EnvSpec(UpNDownNoFrameskip-v4)\n", "EnvSpec(UpNDown-ram-v0)\n", "EnvSpec(UpNDown-ram-v4)\n", "EnvSpec(UpNDown-ramDeterministic-v0)\n", "EnvSpec(UpNDown-ramDeterministic-v4)\n", "EnvSpec(UpNDown-ramNoFrameskip-v0)\n", "EnvSpec(UpNDown-ramNoFrameskip-v4)\n", "EnvSpec(Venture-v0)\n", "EnvSpec(Venture-v4)\n", "EnvSpec(VentureDeterministic-v0)\n", "EnvSpec(VentureDeterministic-v4)\n", "EnvSpec(VentureNoFrameskip-v0)\n", "EnvSpec(VentureNoFrameskip-v4)\n", "EnvSpec(Venture-ram-v0)\n", "EnvSpec(Venture-ram-v4)\n", "EnvSpec(Venture-ramDeterministic-v0)\n", "EnvSpec(Venture-ramDeterministic-v4)\n", "EnvSpec(Venture-ramNoFrameskip-v0)\n", "EnvSpec(Venture-ramNoFrameskip-v4)\n", "EnvSpec(VideoPinball-v0)\n", "EnvSpec(VideoPinball-v4)\n", "EnvSpec(VideoPinballDeterministic-v0)\n", "EnvSpec(VideoPinballDeterministic-v4)\n", "EnvSpec(VideoPinballNoFrameskip-v0)\n", "EnvSpec(VideoPinballNoFrameskip-v4)\n", "EnvSpec(VideoPinball-ram-v0)\n", "EnvSpec(VideoPinball-ram-v4)\n", "EnvSpec(VideoPinball-ramDeterministic-v0)\n", "EnvSpec(VideoPinball-ramDeterministic-v4)\n", "EnvSpec(VideoPinball-ramNoFrameskip-v0)\n", "EnvSpec(VideoPinball-ramNoFrameskip-v4)\n", "EnvSpec(WizardOfWor-v0)\n", "EnvSpec(WizardOfWor-v4)\n", "EnvSpec(WizardOfWorDeterministic-v0)\n", "EnvSpec(WizardOfWorDeterministic-v4)\n", "EnvSpec(WizardOfWorNoFrameskip-v0)\n", "EnvSpec(WizardOfWorNoFrameskip-v4)\n", "EnvSpec(WizardOfWor-ram-v0)\n", "EnvSpec(WizardOfWor-ram-v4)\n", "EnvSpec(WizardOfWor-ramDeterministic-v0)\n", "EnvSpec(WizardOfWor-ramDeterministic-v4)\n", "EnvSpec(WizardOfWor-ramNoFrameskip-v0)\n", "EnvSpec(WizardOfWor-ramNoFrameskip-v4)\n", "EnvSpec(YarsRevenge-v0)\n", "EnvSpec(YarsRevenge-v4)\n", "EnvSpec(YarsRevengeDeterministic-v0)\n", "EnvSpec(YarsRevengeDeterministic-v4)\n", "EnvSpec(YarsRevengeNoFrameskip-v0)\n", "EnvSpec(YarsRevengeNoFrameskip-v4)\n", "EnvSpec(YarsRevenge-ram-v0)\n", "EnvSpec(YarsRevenge-ram-v4)\n", "EnvSpec(YarsRevenge-ramDeterministic-v0)\n", "EnvSpec(YarsRevenge-ramDeterministic-v4)\n", "EnvSpec(YarsRevenge-ramNoFrameskip-v0)\n", "EnvSpec(YarsRevenge-ramNoFrameskip-v4)\n", "EnvSpec(Zaxxon-v0)\n", "EnvSpec(Zaxxon-v4)\n", "EnvSpec(ZaxxonDeterministic-v0)\n", "EnvSpec(ZaxxonDeterministic-v4)\n", "EnvSpec(ZaxxonNoFrameskip-v0)\n", "EnvSpec(ZaxxonNoFrameskip-v4)\n", "EnvSpec(Zaxxon-ram-v0)\n", "EnvSpec(Zaxxon-ram-v4)\n", "EnvSpec(Zaxxon-ramDeterministic-v0)\n", "EnvSpec(Zaxxon-ramDeterministic-v4)\n", "EnvSpec(Zaxxon-ramNoFrameskip-v0)\n", "EnvSpec(Zaxxon-ramNoFrameskip-v4)\n", "EnvSpec(CubeCrash-v0)\n", "EnvSpec(CubeCrashSparse-v0)\n", "EnvSpec(CubeCrashScreenBecomesBlack-v0)\n", "EnvSpec(MemorizeDigits-v0)\n" ] } ], "source": [ "all_env = list(gym.envs.registry.all())\n", "for e in list(all_env):\n", " print(e)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:10.315458Z", "start_time": "2020-03-09T15:30:10.310654Z" } }, "outputs": [ { "data": { "text/plain": [ "859" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(all_env)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example environment of cartpole " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Definition of problem statement\n", "A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic functions and uses of gym environment functions\n", "**make**: make is used to create the selected environment given from the parameters\n", "\n", "**reset**: reset the environment to default state\n", "\n", "**action_space**: all possible actions for the environment\n", "\n", "**observation_space**: all possible observations for the environment\n", "\n", "**render**: rendering a visual for each environment\n", "\n", "**step**: a function where you pass the action to take for the environment\n", "\n", "**observation**: observation noted from the environment\n", "\n", "**reward**: reward obtained for taking the step in the environment\n", "\n", "**done**: single boolean values which tells if the episode has ended\n", "\n", "**info**: additional information used to debug the running of the environment\n", "\n", "**wrappers**: extend the environment defined and allow for more features and additionial build details into the environment, can ddefine observation, action and reward wrappers individually" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:23.453275Z", "start_time": "2020-03-09T15:30:10.318012Z" } }, "outputs": [], "source": [ "env = gym.make('CartPole-v0')\n", "n_episodes = 1000\n", "env_vis = []\n", "for i_episode in range(n_episodes):\n", " observation = env.reset()\n", " for t in range(100):\n", " env_vis.append(env.render(mode='rgb_array'))\n", " action = env.action_space.sample()\n", " observation, reward, done, info = env.step(action)\n", " if done:\n", " print(\"Episode finished at t{}\".format(t+1))\n", " break" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:23.459712Z", "start_time": "2020-03-09T15:30:23.455211Z" } }, "outputs": [], "source": [ "def env_render(env_vis):\n", " plt.figure()\n", " plot = plt.imshow(env_vis[0])\n", " plt.axis('off')\n", "\n", "\n", "def animate(i):\n", " plot.set_data(env_vis[i])\n", " anim = anm.FuncAnimation(plt.gcf(), animate, frames=len(env_vis), interval=24000, repeat=True, repeat_delay=2)\n", " display(display_animation(anim, default_mode='loop'))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:23.557238Z", "start_time": "2020-03-09T15:30:23.461821Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAADnCAYAAABBu67aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAADsklEQVR4nO3c0WnCUBiA0aa4ROdwjc6hM+kcXcM5Okb6VkpNVejXJNpzQDC5CP9D+LiEmGEcxycAfu956QEAHoWgAkQEFSAiqAARQQWIbK6sewQA4NwwddIOFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIpulB4CfnI77s3Pb3WGBSeA2dqgAEUEFiAgqQERQASKCChARVICIoAJEBBUgIqgAEUEFiAgqQERQASKCChARVICIoAJEBBUgIqgAEUFllabe1g9rJ6gAEUEFiAgqQERQASKCChARVICIoAJEBBUgIqgAEUEFiAgqQERQASKCChARVICIoAJEBBUgIqjcje3usPQIcJGgAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkQEldU5Hfdn57ytn3sgqAARQQWICCpARFABIoIKEBFUgIigAkQElVkMw3Dz5y9+D3MQVIDIZukBYMrb++7z++vLccFJ4HZ2qKzO15hOHcNaCSpARFABIoLK6ny/Z+oeKvdiGMfx0vrFRbjVnI8zXbmmoTB5QduhAkQEFSAiqAARQQWICCpARFABIoIKEBFUgIigAkS8vo9Z+PcS/4EdKkBEUAEiggoQEVSAiKACRAQVICKoABFBBYgIKkBEUAEiggoQEVSAiKACRAQVICKoABFBBYgIKkBEUAEiggoQEVSAiKACRAQVICKoABFBBYgIKkBEUAEiggoQEVSAiKACRAQVICKoABFBBYgIKkBEUAEiggoQEVSAiKACRAQVICKoABFBBYgIKkBEUAEiggoQEVSAiKACRDZX1odZpgB4AHaoABFBBYgIKkBEUAEiggoQEVSAyAcgGyIR8AfytQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "env_render(env_vis)" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-03-08T06:06:19.218394Z", "start_time": "2020-03-08T06:06:19.215417Z" } }, "source": [ "## Example environment of space invaders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Definition of problem statement\n", "Maximize your score in the Atari 2600 game SpaceInvaders. In this environment, the observation is an RGB image of the screen, which is an array of shape (210, 160, 3) Each action is repeatedly performed for a duration of k frames, where k is uniformly sampled from {2,3,4}." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:24.455214Z", "start_time": "2020-03-09T15:30:23.560610Z" } }, "outputs": [], "source": [ "env = gym.make('SpaceInvaders-v0')\n", "env = wrappers.Monitor(env, \"./gym-visuals\", force=True)\n", "env.reset()\n", "for _ in range(1000):\n", " action = env.action_space.sample()\n", " observation, reward, done, info = env.step(action)\n", " if done:\n", " break\n", "env.close()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2020-03-09T15:30:24.470442Z", "start_time": "2020-03-09T15:30:24.459658Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " " ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "video = io.open('./gym-visuals/openaigym.video.%s.video000000.mp4' % env.file_infix, 'r+b').read()\n", "encoded = base64.b64encode(video)\n", "HTML(data='''\n", " '''\n", ".format(encoded.decode('ascii')))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }