Codex for Minecraft NPCs

This prototype uses GPT-3 Codex to power a Non-Player Character (NPC) in Minecraft. Through the discipline of "prompt engineering" (see section below), we show the model how to use the Minecraft SimulatedPlayer API to write code to navigate the world, mine, craft and even carry on conversation.

Requirements

Please read the readme carefully to make sure you get the right bits. This repo works with very specific versions of Minecraft, Minecraft dedicated server, and Node, and requires setup commands to be called in specific order.

• An OpenAI account

  • OpenAI API Key.
  • OpenAI Organization Id. If you have multiple organizations, please update your default organization to the one that has access to codex engines before getting the organization Id.
  • OpenAI Model Id. The name of the Codex model you're using. This repository was primarily tested using code-davinci-002. (update, September 2023: code-davinci-002 does not appear available; consider using text-davinci-002 instead.) See here for checking available engines.

• You need to be an owner of Minecraft, or have PC GamePass, in order to use Minecraft or Minecraft Preview.

  • This sample uses a separate Dedicated Server and a Minecraft client app. For more information on Minecraft Bedrock Edition, Dedicated Server and how you can use scripting with them, see this article.
  • This build uses the latest Minecraft version (version 1.19.0 or higher). You will need a version of Minecraft Dedicated Server, downloadable from https://www.minecraft.net/en-us/download/server/bedrock, and Minecraft client version 1.19.0 or higher, which the below links will help you to gather.
  • Note that Minecraft has both main and "Preview" versions of Minecraft: Bedrock Edition. You can now use the main Minecraft editions to use this sample; you no longer need to use the Minecraft Preview app with this sample in particular.
  • You can access Minecraft from the MS Store here to make sure you have access: https://www.microsoft.com/store/productId/9PGW18NPBZV5
  • Download the Xbox app from Microsoft Store, and sign in with your consumer MSA account (same thing as your Xbox account if you have one) https://www.microsoft.com/store/productId/9MV0B5HZVK9Z
  • Open the Xbox App to install Minecraft (https://www.microsoft.com/store/productId/9P5X4QVLC2XR).
    • LTS 16.15 version of [Node.JS](https://nodejs.org/en/)
    • Visual Studio Code
    • OPTIONAL: Clear chat texture pack to make it easier to see the bot actions without the dim screen of chat. Download from here and follow install directions: https://mcpedl.com/clear-chat-tranparent-chat/

Setup

1. Clone the repo wherever you like: git clone https://github.com/microsoft/MinecraftCodex.git. The project is setup to work with your local Minecraft install.
2. Open the folder you placed the repo in locally in VSCode
3. You will get a recommendations to install Minecraft extensions to enable debugging, you want to accept those
4. Open Terminal in VSCode in the code directory, and run npm install to pull down the package dependencies.
5. Under the scripts folder, create a file called vars.ts with the following key value pairs:

   export let OPENAI_API_KEY = "<YOUR_KEY_HERE>";
   export let OPENAI_ORGANIZATION_ID = "<YOUR_ORG_ID_HERE>";
   export let OPENAI_ENGINE_ID = "<YOUR_CODEX_MODEL_NAME_HERE>";
   export let DEBUG = false;
   

6. Place your OpenAI API key between the quotes, and save the file.

Install Minecraft Bedrock Edition Dedicated Server

1. Download and unzip the Minecraft Bedrock Dedicated Server to a location on your hard drive.

2. In the Minecraft Codex code directory, open gulpfile.js and update the dedicatedServerPath variable to the folder where your dedicated server is located. Note: it must end with a '\' character

3. You will likely need to disable PowerShell signing requirements for the session. In the VSCode Terminal type: Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass You will need to do this every time you open a new terminal, or restart VSCode. It only works on the current process, so nowhere else should this take effect.

4. Run these three commandsin the VSCode terminal, which should be in the repo folder. gulp updateconfig gulp updateserver gulp updateworld

5. Those commands will:

   • Add additional configuration files that enable this module to make net requests on Dedicated Server.
   • Update server configuration properties (server.properties)
   • Reset the Minecraft Dedicated Server to the default world for the Codex World.

6. The first time you run Bedrock Dedicated Server, you may see a prompt within Windows to enable ports on your firewall for public and/or private networks. Within the pop up Firewall prompt in Windows that you may receive, you will want to potentially enable Bedrock Server port access on your Private networks. Alternately, ensure at least ports 19132 and 19133 are open to Bedrock Dedicated Server via your Firewall tools.

7. Ensure that local Minecraft local clients can connect to your locally-hosted server - to do this, you need to enable 'loopback' connections for Minecraft UWP.

   To enable loopback for Minecraft on Windows, run:

   npm run enablemcloopback
   

   To enable loopback for Minecraft Preview on Windows, run:

   npm run enablemcpreviewloopback
   

8. To run the game server with proper configs and defaults, you need to use gulp serve from your code directory, typically from the VSCode terminal.

9. Run the dedicated server from it's directory once to get it up and running. After it has completed loading (you will see the message "Debugger Listening" on the console), type in the following command to ensure you have full access to controls in game. op <your username>

Building and Deploying

1. To have it continuously re-build as you make changes, run gulp watch. This will stop the server and restart it with your latest code regularly.
2. The deployment step automatically moves the compiled code to a Minecraft Behavior Pack folder within your Dedicated Server.

Connecting to the Dedicated Server

1. Open your Minecraft client and click Play. Select "Servers". The first time you play, you will need to add a server:

2. Scroll all the way down on the left side to and select Add Server

3. Type a name for the server ("local") and use a Server Address of 127.0.0.1 if you are running Dedicated Server on the same machine.

4. Select the server, and select Join Server.

5. Open the chat (click 't') and type /gametest run codex:codex to spawn the Non-Player Character, along with a test structure from the BehaviorPack. When loading your world in the future, you can push the button on the command block that appears in the test structure to start the codex code running

Debugging

Use the following steps to debug:

1. Make sure you have the Minecraft Bedrock Edition Debugger installed within Visual Studio Code, or this won't work properly, make sure it is version 0.2.0 or later
2. Running gulp serve in your code folder will start the server running, and then you can run the debugger from VSCode which will automatically connect to the server.
3. You can set breakpoints in the TypeScript files directly. NOTE: The code in CodexBot is not hitting breakpoints correctly, this is a known error and being looked at.

Interacting with the NPC

Now that you've spawned the NPC into the game, you can have a multi-turn conversation with it!

1. Click 't' to open the chat and send messages to the NPC. If you interact conversationally (e.g. "What are you up to?" or "What's the meaning of life?), the NPC should respond with natural language. If given a command (e.g. "look at me", "move forward a bit", etc.) the NPC should respond with code and evaluate it.
2. The NPC is able to carry-on multi-turn conversations, carrying context across conversation turns. Example:

Player: What's your favorite Minecraft resource'
NPC: bot.chat("I love oak logs!")
Player: "Why's that?
NPC: bot.chat("Because they're so versatile!")

Note: The NPC isn't perfect, and sometimes the conversation can get stuck or begin repeating itself. Type "reset" to reset the conversational experience.

How it Works - Prompt Engineering

The Codex model being used in this sample has never seen the SimulatedPlayer API we're asking it to use (it's a new API that wasn't in its training set). In order to get the model to use the SimulatedPlayer API, we need to engineer prompts that give the model examples of the kinds of commands it will receive and the kind of code it should write. From just a few Natual Language to Code examples, we can use the model to generate new code from new commands.

We coax the correct code and natural language out of Codex through a discipline called prompt engineering. Large Language Models like GPT-3 and Codex are trained to guess the next word in a sentence, or more generally, the next token in a sequence. They can do this continuously to genereate whole sentences, paragraphs, lines of code and functions. To coax very specific code out of the models, it's best to give prompts that convey our intent and give examples of what we're looking for.

Please note: The NPC capabilities are limited to abilities in the capabilities of the SimulatedPlayer API, along with some extra capabilities we gave it. It can walk, chat with you, walk around, look at you, follow you, mine blocks, and