Publish tested Next AI Lab release
Browse files
README.md
CHANGED
|
@@ -1,92 +1,51 @@
|
|
| 1 |
-
---
|
| 2 |
-
language:
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
At the marked closure, the original route becomes unusable. With the updated map, the shortest remaining journey grows from **44 to 64 steps**. Gray shows the completed portion, dashed blue shows the canceled plan, and green traces the new route from the decision point.
|
| 53 |
-
|
| 54 |
-

|
| 55 |
-
|
| 56 |
-
## Held-out evaluation
|
| 57 |
-
|
| 58 |
-
Three training seeds (73, 101 and 149) were evaluated on the same 120 held-out layouts. Every reachable floor cell served as a start. Counts are per seed, and all three seeds produced the same scores.
|
| 59 |
-
|
| 60 |
-
| Grid | Layouts | Reachable starts | Next-Reward shortest routes | Greedy arrivals |
|
| 61 |
-
|---|---:|---:|---:|---:|
|
| 62 |
-
| 5 × 5 | 40 | 569 | 569 | 474 |
|
| 63 |
-
| 8 × 8 | 40 | 1,417 | 1,417 | 927 |
|
| 64 |
-
| 12 × 12 | 40 | 3,373 | 3,373 | 1,555 |
|
| 65 |
-
| **Total** | **120** | **5,359** | **5,359** | **2,956** |
|
| 66 |
-
|
| 67 |
-
Another 1,122 floor cells had no path to the goal and were counted separately. The greedy baseline chooses the safe neighboring cell closest to the goal by Manhattan distance. BFS supplies the shortest-path reference.
|
| 68 |
-
|
| 69 |
-
## Try the checkpoint
|
| 70 |
-
|
| 71 |
-
Download `model.json` and `next_reward.py` into the same directory, install NumPy, and run:
|
| 72 |
-
|
| 73 |
-
```python
|
| 74 |
-
from next_reward import NextReward, ACTIONS
|
| 75 |
-
|
| 76 |
-
board = (".....", ".X...", ".X...", "...X.", "....G")
|
| 77 |
-
agent = NextReward.load("model.json")
|
| 78 |
-
policy = agent.policy(board)
|
| 79 |
-
print(ACTIONS[policy[0, 0]])
|
| 80 |
-
```
|
| 81 |
-
|
| 82 |
-
Coordinates use `(row, column)`. A dot denotes floor, `#` a wall, `X` a terminal pit, and `G` the goal.
|
| 83 |
-
|
| 84 |
-
## Training and reproduction
|
| 85 |
-
|
| 86 |
-
Training samples transitions on procedural 5 × 5, 6 × 6 and 7 × 7 grids. The checkpoint stores action displacement estimates for sixteen neighboring wall patterns and learned rewards for three destination types. Planning uses a discount factor of 0.98. The exported checkpoint uses seed 73.
|
| 87 |
-
|
| 88 |
-
Run `python experiment.py --output run` to train and score all three seeds. The release includes the training notebook, `evaluation.json`, and the exact held-out layouts in `test_maps.json`.
|
| 89 |
-
|
| 90 |
-
## Scope
|
| 91 |
-
|
| 92 |
-
The experiments cover fully visible simulated grids with deterministic movement. The closure experiment supplies the changed floor plan to the agent. Physical robot control, obstacle sensing, traffic between multiple agents, and optimization of the shelf visit order remain untested.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
tags:
|
| 4 |
+
- reinforcement-learning
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
# Next Reward
|
| 9 |
+
|
| 10 |
+
Next Reward learns a transition table and reward values from simulated grid moves. Given a visible map, it computes a new policy by value iteration. One JSON checkpoint runs on CPU with NumPy.
|
| 11 |
+
|
| 12 |
+
## Try the checkpoint
|
| 13 |
+
|
| 14 |
+
Download the repository files into one directory and run this Python snippet there.
|
| 15 |
+
|
| 16 |
+
~~~python
|
| 17 |
+
from next_reward import NextReward, ACTIONS
|
| 18 |
+
|
| 19 |
+
board = (".....", ".X...", ".X...", "...X.", "....G")
|
| 20 |
+
agent = NextReward.load("model.json")
|
| 21 |
+
policy = agent.policy(board)
|
| 22 |
+
print(ACTIONS[policy[0, 0]])
|
| 23 |
+
~~~
|
| 24 |
+
|
| 25 |
+
Coordinates are (row, column). Each dot is a valid starting cell. A wall (#) blocks movement. Entering a pit (X) or the goal (G) ends the episode. Policy entries for terminal cells and walls are -1.
|
| 26 |
+
|
| 27 |
+
## What it learns
|
| 28 |
+
|
| 29 |
+
Training samples random state-action transitions on 5×5, 6×6 and 7×7 procedural maps. The model estimates displacement frequencies for four actions under sixteen neighboring wall patterns and mean rewards for three destination types. Inference uses the map geometry, known terminal types, those learned estimates, and a discount of 0.98. The exported checkpoint uses the prespecified training seed 73.
|
| 30 |
+
|
| 31 |
+
## Held-out evaluation
|
| 32 |
+
|
| 33 |
+
All three training seeds (73, 101, 149) were evaluated on the same 120 new layouts. Test maps were excluded from all three training sets by content hash. Every reachable floor cell was used as a start. BFS supplied the shortest-path reference during scoring. The greedy baseline picks the safe neighboring cell closest to the goal by Manhattan distance, breaking ties by action order.
|
| 34 |
+
|
| 35 |
+
| Map size | Maps | Reachable starts | Agent reached goal on a shortest path | Greedy reached goal |
|
| 36 |
+
|---|---:|---:|---:|---:|
|
| 37 |
+
| 5×5 | 40 | 569 | 569 | 474 |
|
| 38 |
+
| 8×8 | 40 | 1,417 | 1,417 | 927 |
|
| 39 |
+
| 12×12 | 40 | 3,373 | 3,373 | 1,555 |
|
| 40 |
+
|
| 41 |
+
Counts are per training seed. All three produced the same scores. Another 1,122 floor cells had no path to the goal and were counted separately. The original demo layout was scored separately at 21/21 for both methods.
|
| 42 |
+
|
| 43 |
+
This task assumes a fully visible static grid and deterministic motion. The evaluation covers rearranged layouts under the same movement rules. Real-world navigation, changing obstacles, partial observations and stochastic actuators are outside the measured scope.
|
| 44 |
+
|
| 45 |
+
## Reproduce
|
| 46 |
+
|
| 47 |
+
Run `python experiment.py --output run` to train and score three seeds. The command writes a reloadable checkpoint, per-seed metrics and the exact test layouts. It checks that serialization preserves the exported policy on every evaluation board.
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
[Open the demo](https://huggingface.co/spaces/thelamapi/Next-Reward)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|