Lamapi commited on
Commit
fb5bc19
·
verified ·
1 Parent(s): dad2bf1

Publish tested Next AI Lab release

Browse files
Files changed (1) hide show
  1. README.md +51 -92
README.md CHANGED
@@ -1,92 +1,51 @@
1
- ---
2
- language:
3
- - en
4
- tags:
5
- - reinforcement-learning
6
- - model-based-rl
7
- - gridworld
8
- - path-planning
9
- - numpy
10
- ---
11
-
12
- ![reward](https://cdn-uploads.huggingface.co/production/uploads/67d46bc5fe6ad6f6511d6f44/z7NT6VY_eIDZXf_1h78iR.png)
13
-
14
- # Next-Reward
15
-
16
- A warehouse picker has three shelves to visit and a dock to return to. The nearest-looking aisle can end at a wall, and a closure can invalidate a route halfway through delivery.
17
-
18
- Next-Reward learns how moves change an agent’s position and what rewards follow. Given a floor plan and a goal, it uses value iteration to choose a move from each reachable cell. The checkpoint runs on CPU with NumPy.
19
-
20
- ![Three shelf visits and the return to the dock](assets/warehouse-route.png)
21
-
22
- ## One order, four shortest paths
23
-
24
- On the seed-2026 simulated warehouse, Next-Reward visited three shelves in a prescribed order and returned to the dock in **316 steps**. Every leg matched the shortest distance computed independently by breadth-first search (BFS).
25
-
26
- | Leg | Next-Reward | BFS |
27
- |---|---:|---:|
28
- | Dock → Pick 1 | 87 | 87 |
29
- | Pick 1 → Pick 2 | 58 | 58 |
30
- | Pick 2 → Pick 3 | 68 | 68 |
31
- | Pick 3 → Dock | 103 | 103 |
32
- | **Total** | **316** | **316** |
33
-
34
- ![Measured route length against the BFS reference for each leg](assets/route-efficiency.png)
35
-
36
- ## Five ways a shortcut can fail
37
-
38
- A closed loading bay forces a bypass. A dead-end aisle points straight toward the goal. A third layout requires the first move to go west even though the goal lies east. The final two layouts add an unsafe crossing and a sequence of switchbacks.
39
-
40
- Next-Reward found a shortest route on **5 of 5** layouts. The greedy baseline reached the goal on **0 of 5**. These layouts were selected to illustrate specific failure modes. The held-out evaluation below gives a broader comparison.
41
-
42
- ![Five warehouse layouts with Next-Reward and greedy trajectories](assets/five-routing-tests.png)
43
-
44
- ## Long-haul dispatch
45
-
46
- A **51 × 51** warehouse contains **1,284 reachable cells**. The green trajectory connects dispatch to the goal in **306 steps**, matching the shortest route on this layout.
47
-
48
- ![A 306-step dispatch route](assets/long-haul.png)
49
-
50
- ## An aisle closes mid-delivery
51
-
52
- At the marked closure, the original route becomes unusable. With the updated map, the shortest remaining journey grows from **44 to 64 steps**. Gray shows the completed portion, dashed blue shows the canceled plan, and green traces the new route from the decision point.
53
-
54
- ![Replanning after an aisle closure](assets/mid-delivery-closure.png)
55
-
56
- ## Held-out evaluation
57
-
58
- Three training seeds (73, 101 and 149) were evaluated on the same 120 held-out layouts. Every reachable floor cell served as a start. Counts are per seed, and all three seeds produced the same scores.
59
-
60
- | Grid | Layouts | Reachable starts | Next-Reward shortest routes | Greedy arrivals |
61
- |---|---:|---:|---:|---:|
62
- | 5 × 5 | 40 | 569 | 569 | 474 |
63
- | 8 × 8 | 40 | 1,417 | 1,417 | 927 |
64
- | 12 × 12 | 40 | 3,373 | 3,373 | 1,555 |
65
- | **Total** | **120** | **5,359** | **5,359** | **2,956** |
66
-
67
- Another 1,122 floor cells had no path to the goal and were counted separately. The greedy baseline chooses the safe neighboring cell closest to the goal by Manhattan distance. BFS supplies the shortest-path reference.
68
-
69
- ## Try the checkpoint
70
-
71
- Download `model.json` and `next_reward.py` into the same directory, install NumPy, and run:
72
-
73
- ```python
74
- from next_reward import NextReward, ACTIONS
75
-
76
- board = (".....", ".X...", ".X...", "...X.", "....G")
77
- agent = NextReward.load("model.json")
78
- policy = agent.policy(board)
79
- print(ACTIONS[policy[0, 0]])
80
- ```
81
-
82
- Coordinates use `(row, column)`. A dot denotes floor, `#` a wall, `X` a terminal pit, and `G` the goal.
83
-
84
- ## Training and reproduction
85
-
86
- Training samples transitions on procedural 5 × 5, 6 × 6 and 7 × 7 grids. The checkpoint stores action displacement estimates for sixteen neighboring wall patterns and learned rewards for three destination types. Planning uses a discount factor of 0.98. The exported checkpoint uses seed 73.
87
-
88
- Run `python experiment.py --output run` to train and score all three seeds. The release includes the training notebook, `evaluation.json`, and the exact held-out layouts in `test_maps.json`.
89
-
90
- ## Scope
91
-
92
- The experiments cover fully visible simulated grids with deterministic movement. The closure experiment supplies the changed floor plan to the agent. Physical robot control, obstacle sensing, traffic between multiple agents, and optimization of the shelf visit order remain untested.
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - reinforcement-learning
5
+ ---
6
+
7
+
8
+ # Next Reward
9
+
10
+ Next Reward learns a transition table and reward values from simulated grid moves. Given a visible map, it computes a new policy by value iteration. One JSON checkpoint runs on CPU with NumPy.
11
+
12
+ ## Try the checkpoint
13
+
14
+ Download the repository files into one directory and run this Python snippet there.
15
+
16
+ ~~~python
17
+ from next_reward import NextReward, ACTIONS
18
+
19
+ board = (".....", ".X...", ".X...", "...X.", "....G")
20
+ agent = NextReward.load("model.json")
21
+ policy = agent.policy(board)
22
+ print(ACTIONS[policy[0, 0]])
23
+ ~~~
24
+
25
+ Coordinates are (row, column). Each dot is a valid starting cell. A wall (#) blocks movement. Entering a pit (X) or the goal (G) ends the episode. Policy entries for terminal cells and walls are -1.
26
+
27
+ ## What it learns
28
+
29
+ Training samples random state-action transitions on 5×5, 6×6 and 7×7 procedural maps. The model estimates displacement frequencies for four actions under sixteen neighboring wall patterns and mean rewards for three destination types. Inference uses the map geometry, known terminal types, those learned estimates, and a discount of 0.98. The exported checkpoint uses the prespecified training seed 73.
30
+
31
+ ## Held-out evaluation
32
+
33
+ All three training seeds (73, 101, 149) were evaluated on the same 120 new layouts. Test maps were excluded from all three training sets by content hash. Every reachable floor cell was used as a start. BFS supplied the shortest-path reference during scoring. The greedy baseline picks the safe neighboring cell closest to the goal by Manhattan distance, breaking ties by action order.
34
+
35
+ | Map size | Maps | Reachable starts | Agent reached goal on a shortest path | Greedy reached goal |
36
+ |---|---:|---:|---:|---:|
37
+ | 5×5 | 40 | 569 | 569 | 474 |
38
+ | 8×8 | 40 | 1,417 | 1,417 | 927 |
39
+ | 12×12 | 40 | 3,373 | 3,373 | 1,555 |
40
+
41
+ Counts are per training seed. All three produced the same scores. Another 1,122 floor cells had no path to the goal and were counted separately. The original demo layout was scored separately at 21/21 for both methods.
42
+
43
+ This task assumes a fully visible static grid and deterministic motion. The evaluation covers rearranged layouts under the same movement rules. Real-world navigation, changing obstacles, partial observations and stochastic actuators are outside the measured scope.
44
+
45
+ ## Reproduce
46
+
47
+ Run `python experiment.py --output run` to train and score three seeds. The command writes a reloadable checkpoint, per-seed metrics and the exact test layouts. It checks that serialization preserves the exported policy on every evaluation board.
48
+
49
+
50
+
51
+ [Open the demo](https://huggingface.co/spaces/thelamapi/Next-Reward)