# Let's build our smart robot NPC demo 🤖 ## Step 0: Get the project files You can find the 👉 [complete Unity project here](https://huggingface.co/datasets/huggingface-ml-4-games-course/unity-demos/resolve/main/unit1/Jammo%20the%20Robot%20Sentis%20(v2).zip?download=true) ## Step 1: Install Unity Sentis The Sentis Documentation 👉 https://docs.unity3d.com/Packages/com.unity.sentis@latest 1. Open the Jammo project 2. Click [Sentis Pre-Release package](https://tinyurl.com/4eun48fb) or go to Window > Package Manager, click the + icon, select "Add package by name…" and type "com.unity.sentis" **Set the version to 1.3.0** 3. Press the Add button to install the package. ## Step 2: Install Sharp Transformers 💪 [Sharp Transformers](https://github.com/huggingface/sharp-transformers) is a Unity plugin of utilities to **run Transformer 🤗 models in Unity games**. We need to do it for the tokenization step. Go to "Window" > "Package Manager" to open the Package Manager. Click the "+" in the upper left-hand corner and select "Add package from git URL". Enter the URL of this repository and click "Add": [https://github.com/huggingface/sharp-transformers.git](https://github.com/huggingface/sharp-transformers.git) ## Step 3: Build the inference process 🧠 As described in [Sentis documentation](https://docs.unity3d.com/Packages/com.unity.sentis@latest), to use Sentis to run a neural network in Unity, we need to follow these steps. 1. Use the Unity.Sentis namespace. 2. **Load a neural network** model file. 3. **Create input** for the model. 4. **Create an inference engine** (a worker). 5. **Run the model** with the input to infer a result. 6. **Get the result**. In our case, we do all of this in **SentenceSimilarity.cs** file, that will be attached to our robot. In awake, we: 1. **Load our neural network.** 2. Create an **inference engine** (a worker). 3. **Create an operator**, that will allow us to perform operations with tensors. We have three functions: 1. *Encode*: takes our player input (text), **tokenizes it, and embeds it**. 2. *SentenceSimilarityScores*: **calculate the similarity scores** between the input embed (what the user typed) and the comparison embeds (the robot action list) 3. *RankSimilarityScores*: get the **most similar action** and its index given the player input ## Step 4: Build the Robot Behavior 🤖 We need to define the **behavior of our robot**. The idea is that our robot has different possible actions and the choice of the actions **will depend on most similar actions**. We need first to define the *Finite State Machine*, a simple **classical AI technique where each State defines a certain behavior**. Then, we’ll make the utility function **that will select the State hence the series of actions to perform**. ### The State Machine 🧠🤖 In a state machine, **each state represents a behavior**, for instance, moving to a column, saying hello, etc. Based on the state the agent is **it will perform a series of actions**. In our case, we have 7 states: The first thing we need to do is create an enum called State that contains each of the possible States: Because we need to constantly check the state, **we define the state machine into the Update() method using a switch system where each case is a state**. For each state case, we define the behavior of our agents, for instance in our state Hello, the robot **must move towards the player, face him correctly, and then launch its Hello animation, then go back to an Idle State**. We have now defined the behavior for each different State. The magic here will come from the fact **that’s the language model that will define what State is the closest to the Player input**. And in the utility function, we call this state. ### Let’s define the Utility Function 📈 Our action list looks like this: - The sentence is what will be embedded in the AI model. - The verb is the State - Noun (if any) is the object to interact with (Pillar, Cube, etc.) This utility function will **select the Verb and Noun associated with the sentence having the highest similarity score with the player input text.** But first, to **get rid of a lot of strange input text**, we need to have a similarity **score threshold**. For example, if I say “Look at all the rabbits”, **none of our possible actions are relevant**. Hence instead of choosing the action with the highest score, we’ll call the State Puzzled which will animate the robot with a perplexed animation. If the score is higher, then **we’ll get the verb corresponding to a State and the noun (goalObject) if any**. We set the state corresponding to the verb. That will activate the behavior corresponding to it. And that’s it, **now we’re ready to interact with our robot!** ## Step 5: Let’s interact with our Robot 🤖 In this step, you just need to click on the play button in the editor. And you **can prompt some orders and see the results**! That's nice! But the best way to learn is to try things, break things and modify the demo. ## (Optional) Improving the game: let’s add a new action Adding a new action is quite easy. Let’s take an example: - Copy the YellowPillar game object and move it - Change the name to GreenPillar - Create a new material and set it to green - Place the material on GreenPillar Now that we’ve placed the new game object, we need to add this possibility into the sentences and click on Jammo_Player. In the list of actions click on the plus button and fill in this new action item: - Add Go to the green column - GoTo - GreenColumn And that’s it! **You can easily iterate and add more actions to your game**. Again, don't forget to test the similarity threshold, **to see if you need to increase it or decrease it**.