Spaces:
Sleeping
Sleeping
File size: 182,443 Bytes
8294cde | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 | # ShopStack Exploration Map
**Project:** ShopStack / GharStock
**Purpose:** A living exploration map for agents and collaborators.
**Rule:** Keep adding to this file whenever a model, product angle, dataset, architecture pattern, evaluation method, market idea, or risk deserves future investigation.
---
## 0. Product Summary
ShopStack is a small-model shopping copilot and household commerce memory layer.
It helps a household:
- know what is already at home,
- build shopping lists,
- scan shelves, markets, receipts, packets, fridges, and pantry spaces,
- ask questions by voice,
- decide buy / skip / compare / replace,
- add purchases into inventory,
- track freshness, expiry, price, quantity, and location,
- remember where items are stored,
- learn which stores are cheaper/better,
- understand travel, weather, timing, and convenience,
- create anonymized traces and field notes for the Build Small Hackathon.
The product is intentionally broad long-term, but every exploration should connect back to household shopping, inventory, market intelligence, or daily-use memory.
---
## 1. Capability Map
### 1.1 Language + Reasoning
Current / possible capabilities:
- NER for item, brand, unit, quantity, store, price, location, date.
- Intent classification.
- Command parsing.
- Tool-call planning.
- Natural-language inventory query.
- Shopping decision explanation.
- Multilingual / Hinglish / Indian household phrasing.
- Synonym and alias normalization.
- Structured JSON extraction.
- RAG over household memory.
- Safety disclaimers for nutrition/medicine/price uncertainty.
Exploration questions:
- Which small model is best at Indian household command parsing?
- Can a tiny fine-tuned parser outperform a larger general model on our exact commands?
- How much can rules + schema validation reduce model errors?
- Which model produces the cleanest JSON tool calls locally?
- Should we separate command parser and answer generator?
Candidate model families:
- MiniCPM / OpenBMB
- Qwen
- LFM / LiquidAI
- Gemma
- GPT-OSS
- Granite
- GLM
- Mistral / Voxtral for speech-heavy flows
- Llama / Phi GGUF models for llama.cpp badge
---
### 1.2 NER / Entity Extraction
Entities to extract:
- item name
- canonical item name
- item category
- brand
- local alias
- quantity
- unit
- price
- normalized unit price
- expiry date
- manufacturing date
- store
- location at home
- storage instruction
- nutrition facts
- household member preference
- purchase timestamp
- travel context
- weather context
Exploration questions:
- Should NER be rule-based, model-based, or hybrid?
- Can a fine-tuned small model map Hinglish utterances to canonical items?
- How do we handle ambiguous items like “Surf,” “Vim,” “Maggi,” “bread,” “pav,” “dahi,” “curd,” “yogurt”?
- Can we build a household-specific lexicon that improves over time?
- Should item names be canonicalized using embeddings?
Dataset ideas:
- Indian household item alias dataset.
- Grocery command dataset.
- Receipt-line-to-canonical-item dataset.
- Hinglish quantity normalization dataset.
- Expiry and storage instruction dataset.
---
### 1.3 Time Series
Time-series capabilities:
- consumption rate estimation,
- days-until-stockout,
- restock interval prediction,
- price trend by item,
- price trend by store,
- day-of-week price patterns,
- seasonality,
- freshness decay,
- expiry forecasting,
- travel effort over time,
- store quality over time,
- household category spend.
Exploration questions:
- Which items have predictable restock cycles?
- How much history is needed for useful next-buy predictions?
- Can we estimate stockout without exact daily consumption?
- What is the right confidence language: “likely low,” “probably enough,” “uncertain”?
- How do we handle multiple lots of the same item?
- Should predictions be rule-based initially and learned later?
Possible models / tools:
- statsmodels / Prophet-like approaches,
- DuckDB time-series queries,
- simple rolling averages,
- exponential smoothing,
- Bayesian inventory estimates,
- local notebooks / HF Jobs / Modal experiments.
---
### 1.4 Spatial Intelligence
Spatial layers:
- fridge shelf memory,
- pantry shelf memory,
- household location map,
- item last-seen memory,
- item movement events,
- room/shelf heatmap,
- “find this item” assistant,
- storage recommendation,
- household location graph,
- future AR/SLAM-like map.
Exploration questions:
- Can shelf/fridge photos become stable “location snapshots”?
- How do we identify the same item across days?
- Can we track “milk moved from shopping bag → fridge door”?
- How much user confirmation is needed?
- Should locations be user-defined first: fridge top shelf, pantry shelf 2, bathroom cabinet?
- Is a heatmap useful before AR/SLAM exists?
- Can we use image embeddings for “this looks like the same shelf”?
Possible tools:
- OpenCV,
- object detection,
- image embeddings,
- segmentation,
- H3/geospatial only for external market map,
- graph DB or SQLite graph tables,
- future WebAR / mobile capture.
---
### 1.5 Voice: STT, TTS, Audio
Voice capabilities:
- voice shopping list creation,
- voice correction,
- voice ask while shopping,
- voice answer in noisy market mode,
- voice-based item movement,
- voice-based price logging,
- Indian-language / Hinglish UX,
- audio confidence and retry UX.
STT candidates to evaluate:
- Qwen3-ASR-1.7B
- NVIDIA Parakeet / Nemotron 0.6B streaming ASR
- Mistral Voxtral Mini Realtime
- SenseVoiceSmall
- VibeVoice ASR
- Cohere Transcribe model
- Granite Speech
- Whisper large-v3-turbo baseline
TTS candidates to evaluate:
- MOSS-TTS-v1.5
- VoxCPM2
- Qwen3-TTS 0.6B / 1.7B
- Higgs Audio v3 TTS
- Kokoro-82M
- CosyVoice
- OmniVoice
- Chatterbox / XTTS-style models if useful
Exploration questions:
- Which model best handles Hinglish grocery commands?
- Can the response voice be short, warm, and market-friendly?
- Should we use browser/device TTS as fallback?
- How do we handle noisy market audio?
- Can audio be processed locally in the Space without cloud APIs?
- How do we benchmark STT/TTS quickly?
Benchmark phrases:
- “Doodh ghar pe hai kya?”
- “Tamatar aadha kilo add karo.”
- “Nahi, yeh pyaaz hai aloo nahi.”
- “Bread expiry kal ka hai, skip karo.”
- “Surf Excel already ghar pe hai kya?”
- “Isko pantry mein move karo.”
- “Kal breakfast ke liye kya hai?”
---
### 1.6 Vision Understanding
Vision capabilities:
- item detection,
- item classification,
- visual grounding,
- shelf/market scan,
- packet understanding,
- receipt understanding,
- fridge/pantry scan,
- freshness/ripeness hints,
- damaged/spoiled item detection,
- visual confirmation cards,
- annotated photos.
Candidate models / tools:
- Gemma multimodal models
- MiniCPM-V
- LocateAnything
- Qwen VL / image models
- RF-DETR
- YOLO variants
- Marlin-2B for video
- TimeLens / video grounding models
- OpenCV + OCR hybrids
- CLIP/embedding-based matching
Exploration questions:
- Which model works best on Indian market photos?
- Can open-vocabulary grounding identify “dhaniya,” “pav,” “dahi,” “atta”?
- Does a general VLM beat object detection for household goods?
- Can we use image crops + text model instead of one large VLM?
- How do we show uncertainty without frustrating users?
---
### 1.7 OCR / Extraction
OCR targets:
- receipts,
- packet labels,
- expiry dates,
- MRP,
- quantity,
- brand,
- nutrition facts,
- store name,
- receipt totals,
- bill line items,
- handwritten notes.
Candidate tools:
- PaddleOCR / PaddleOCR-VL
- NuExtract3
- Tesseract fallback
- Donut-like document models
- DocTR / LayoutLM-style pipelines
- OCR + LLM extraction hybrid
Exploration questions:
- Can receipt OCR handle local Indian store bills?
- How do we normalize quantities from OCR?
- Can we reliably detect expiry dates from packets?
- How do we distinguish MRP from sale price?
- Should packet OCR be a separate close-up mode?
---
### 1.8 Classification
Classification tasks:
- item category,
- storage location,
- food vs household vs medicine vs cleaning,
- perishable vs shelf-stable,
- urgent vs optional,
- buy / skip / compare,
- confidence class,
- freshness class,
- trace safety/redaction class,
- price anomaly class,
- user intent class.
Exploration questions:
- Which classifications can be rules?
- Which need fine-tuning?
- Should we use a lightweight classifier before LLM calls?
- Can the fine-tuned model handle both intent and item category?
- How do we evaluate classification in Field Notes?
---
### 1.9 Segmentation / Grounding
Segmentation capabilities:
- product crop cards,
- item cutouts,
- shelf zones,
- fridge zones,
- visual confirmation,
- background removal,
- annotated maps.
Candidate tools:
- RMBG
- BiRefNet
- ClipSeg
- SAM variants if feasible
- YOLO segmentation
- RF-DETR segmentation
- LocateAnything grounding
- OpenCV masks
Exploration questions:
- Is segmentation necessary for every flow, or only review cards?
- Which model runs reliably in Spaces?
- Can segmentation improve user trust?
- Can shelf-zone segmentation power household spatial memory?
- How should we handle overlapping groceries?
---
### 1.10 Image Generation / Editing
Use cases:
- annotated shopping photos,
- item cards,
- shelf maps,
- “use soon” visual cards,
- printable pantry labels,
- shopping summary posters,
- household heatmap illustrations,
- price comparison cards,
- Field Notes visuals.
Candidate models:
- Black Forest Labs FLUX.2-klein-4B
- FLUX.2-klein-9B
- Qwen Image Edit
- ControlLight
- Lightweight PIL/HTML card generation
- SVG-based generated layouts
Exploration questions:
- Is image generation useful enough for the product, or should deterministic visual cards come first?
- Can generated cards improve sharing and polish?
- How do we keep image edits from hallucinating wrong product details?
- Should all factual text be rendered by code, not generated into image pixels?
---
### 1.11 Embeddings / Retrieval
Embedding uses:
- item alias matching,
- receipt line matching,
- memory search,
- similar purchase recall,
- store note retrieval,
- trace retrieval,
- product substitute matching,
- voice phrase similarity,
- user preference retrieval,
- location snapshot matching.
Candidate embedding models:
- Qwen embeddings
- MiniLM / sentence-transformers
- multilingual E5 family
- BGE multilingual
- Jina embeddings
- small local embedding models
- CLIP / SigLIP for image similarity
Exploration questions:
- Which multilingual embedding model handles Hinglish item aliases best?
- Should text and image embeddings share one store?
- Is SQLite vector extension enough?
- Should we use FAISS, LanceDB, Chroma, or DuckDB extensions?
- Can embeddings help trace retrieval for Sharing is Caring?
---
### 1.12 Graph / Linkage Memory
Nodes:
- Item
- ItemLot
- Store
- MarketArea
- HouseholdLocation
- Shelf
- FridgeZone
- PurchaseEvent
- PriceObservation
- ShoppingTrip
- WeatherContext
- UserPreference
- Recipe
- Trace
- ModelRun
Edges:
- bought_at
- stored_in
- moved_to
- consumed_by
- expires_before
- substitutes
- usually_bought_with
- preferred_by
- cheaper_at
- best_quality_at
- seen_in_photo
- mentioned_in_voice
- derived_from_receipt
- weather_affected
- route_to
- worth_travelling_for
Exploration questions:
- Is a graph DB needed, or can SQLite edge tables do enough?
- Which queries need graph traversal?
- Can graph memory make the product feel smarter quickly?
- How do we visualize this without making it technical?
---
## 2. User Use Case Map
### 2.1 Before Shopping
Use cases:
- create list from voice,
- check what is already at home,
- predict next buys,
- suggest what to buy for meals,
- avoid buying duplicates,
- choose store based on list,
- compare travel effort,
- account for weather,
- generate shopping route,
- plan by budget.
Questions:
- “What should I buy today?”
- “What is low at home?”
- “Can we make dinner without shopping?”
- “Which store should I go to?”
- “Is the Sunday market worth it today?”
---
### 2.2 During Shopping
Use cases:
- scan shelf,
- ask if item is needed,
- compare with home inventory,
- compare price with memory,
- check expiry,
- identify unknown item,
- quantity advice,
- substitution advice,
- allergy/preference warning,
- budget warning.
Questions:
- “Do I need this?”
- “Is this price okay?”
- “Which one should I pick?”
- “Is this enough?”
- “Do we already have this?”
- “Is this near expiry?”
- “Can I skip this?”
---
### 2.3 After Shopping
Use cases:
- purchase photo ingestion,
- receipt ingestion,
- inventory update,
- expiry tracking,
- price observation logging,
- store rating,
- trip context logging,
- trace export,
- family summary.
Questions:
- “Add all this.”
- “What expires first?”
- “What did we spend?”
- “Where should this go?”
- “Did we overbuy anything?”
---
### 2.4 At Home
Use cases:
- find items,
- check stock,
- plan meals,
- use-soon reminders,
- move item location,
- consume item,
- estimate remaining quantity,
- household member asks questions,
- shelf/fridge scan,
- spatial memory.
Questions:
- “Where is the dahi?”
- “Do we have detergent?”
- “What should we use today?”
- “What is in the fridge?”
- “Move toothpaste to bathroom cabinet.”
- “How much rice is left?”
---
### 2.5 Market Intelligence
Use cases:
- price trends,
- store ranking,
- cheapest location,
- freshness/quality memory,
- travel-time decision,
- weather-aware recommendation,
- route-aware shopping,
- neighborhood price map,
- Market Intelligence Graph / Market Map: home inventory + live market cards + price memory + substitutions + freshness risk,
- snapshot-based benchmarking from real market dataset imports.
Questions:
- “Where was tomato cheapest?”
- “Is this price high?”
- “Which store is better for fruits?”
- “Should I travel to the market today?”
- “What did we learn from last month’s shopping?”
- “How much can a Swiggy price snapshot improve price memory and trip decisions?”
---
## 3. Build Small Hackathon Constraint Map
Non-negotiables:
- total loaded model parameters must be <= 32B,
- app must be built on Gradio,
- app must be hosted as a Hugging Face Space,
- short walkthrough video and social post required,
- main track should show a real person / real problem,
- Codex is a parallel track, not the product.
Bonus quests:
- Off the Grid: no cloud APIs in runtime path.
- Well-Tuned: use a fine-tuned model published on HF.
- Off-Brand: custom frontend beyond default Gradio.
- Llama Champion: run a model through llama.cpp.
- Sharing is Caring: publish anonymized agent traces.
- Field Notes: write report/blog about what was built and learned.
Exploration questions:
- Which bonus quests are product-aligned?
- How do we claim Off the Grid while using credits for build-time jobs?
- How do we make llama.cpp visible in the app/report?
- What is the smallest useful fine-tune?
- How should traces be anonymized?
---
## 4. Sponsor Alignment Map
### Hugging Face + Gradio
Explore:
- Spaces deployment,
- Space README metadata,
- dataset/model linking,
- model publishing,
- dataset publishing,
- Jobs,
- GPU Spaces,
- custom Gradio Blocks UI,
- Gradio API endpoints,
- traces as datasets.
### OpenBMB
Explore:
- MiniCPM5-1B as parser/planner,
- MiniCPM5-1B-GGUF for llama.cpp badge,
- MiniCPM-V for vision,
- VoxCPM2 for TTS,
- OpenBMB special category angle.
### OpenAI / Codex
Explore:
- Codex-attributed commits,
- AGENTS.md,
- codex build log,
- tests and docs generated/reviewed by Codex,
- public GitHub repo,
- “Built with Codex” README section,
- Codex as engineering lane only.
### NVIDIA
Explore:
- LocateAnything for grounding,
- Parakeet/Nemotron ASR,
- GPU experiments,
- accelerated vision/audio workflows,
- possible RTX 5080 relevance in final story.
### Modal
Explore:
- fine-tuning jobs,
- benchmark jobs,
- model comparison runs,
- trace generation,
- quantization,
- batch inference,
- dataset generation.
### Black Forest Labs
Explore:
- FLUX image edit / visual cards,
- use-soon cards,
- shelf maps,
- shopping summaries,
- annotated item visuals.
### Cohere
Explore:
- ASR/transcription model comparison,
- embeddings/reranking comparisons if available,
- not required for local-first runtime.
---
## 5. Dataset Exploration Map
Potential datasets to create/publish:
1. Indian household shopping utterances.
2. Hinglish grocery command parser dataset.
3. Item alias/canonicalization dataset.
4. Purchase photo annotation dataset.
5. Receipt OCR/extraction dataset.
6. Packet label/expiry extraction dataset.
7. Inventory tool-call traces.
8. Market decision traces.
9. Price observation synthetic dataset.
10. Shelf/fridge location memory dataset.
11. Voice benchmark dataset.
12. TTS pronunciation phrase set.
13. Store memory schema examples.
14. Redacted agent trace dataset for Sharing is Caring.
15. Field Notes dataset with examples and failure modes.
16. Swiggy Instamart fresh vegetables snapshot dataset for price/unit-price benchmarking and market intelligence experiments.
Dataset quality questions:
- What can be public?
- What must be synthetic?
- What must be anonymized?
- What needs user consent?
- Which datasets help Well-Tuned most?
- Which datasets help judges understand the product?
## 5.1 Decision‑First Today / ShopStack Topic Map
A decision-first Today experience should be built around the questions users actually ask in the moment, not around tabs.
- What I have: household stock and location memory for pantry, fridge, and kitchen items.
- What I should buy: buy recommendations shaped by stock, waste risk, price snapshots, and market intelligence.
- What I should skip: overbuy warnings, duplicate items, and suggestions to avoid waste.
- What I should use soon: use-soon actions, meal prompts, and practical expiry-aware reminders.
- My own list: personal shopping list, voice list edits, and familiar/repeat item memory.
- Compare prices / offers: price/unit-price benchmarking, Swiggy Instamart fresh vegetables snapshot comparisons, and store choice tradeoffs.
- Market basket: basket-level intelligence, category budget signals, and market snapshot analytics.
- What needs confirmation: uncertain OCR/image matches, missing quantities, and low-confidence items.
- What I usually buy: recurring purchase patterns, cadence, and habitual home shopping behavior.
- What I’m overbuying / wasting: waste risk, stale stock alerts, and spend leakage from items that are not being used.
This should explicitly tie the Today dashboard to exploration-level research, including the Swiggy Instamart fresh vegetables snapshot dataset and market intelligence experiments.
**2026-06-08 Update:** The decision-first navigation has been implemented. The 15 flat tabs were restructured into 5 primary tabs (Today → Basket → ShopLens → Reconcile → Memory) with sub-tabs preserving all functionality. See DR-005 in DECISION_RECORDS.md for the full rationale and migration record. The product now opens to Today (decision-first dashboard with integrated Ask) rather than a flat feature catalog. Next exploration targets:
- Wire purchase cadence and waste prevention widgets into the Today dashboard so it answers "what should I do today?" without switching tabs.
- Surface Basket-level decision classification (buy/skip/compare/use-soon) more prominently in the Today view.
- Track which sub-tabs get the most real-world usage to validate or adjust the navigation hierarchy.
---
## 6. Evaluation Map
### Product Evaluations
- Can the user create a list by voice?
- Can the app detect relevant visible items?
- Can the app correctly say buy/skip?
- Can it add confirmed purchases?
- Can it answer inventory questions?
- Can it find item locations?
- Can it explain uncertainty?
### Model Evaluations
- STT exactness and intent retention.
- TTS clarity and warmth.
- OCR field extraction accuracy.
- object detection recall.
- segmentation usability.
- tool-call JSON validity.
- parser intent accuracy.
- latency.
- memory use.
- parameter count.
- install/deploy pain.
- local-first compatibility.
### Trace Evaluations
- trace completeness,
- privacy redaction,
- reproducibility,
- educational value,
- tool-call correctness.
### Field Notes Evaluations
- real user used it,
- what worked,
- what failed,
- what changed after feedback,
- small-model fit,
- honest limitations.
---
## 7. Marketing / Positioning Exploration
Possible positioning:
- “Remember what’s at home while you shop.”
- “A shopping copilot for Indian homes.”
- “Photo + voice inventory for everyday shopping.”
- “Your fridge, pantry, market, and shopping list in one memory.”
- “Small models for small household decisions.”
- “The home commerce memory layer.”
- “Not another grocery app — a memory for what you buy, where it goes, and when to buy again.”
Potential audiences:
- Indian families,
- parents,
- students/hostels,
- shared flats,
- home cooks,
- small kirana shoppers,
- apartment households,
- elderly users,
- caregivers,
- domestic helpers managing stock,
- local sellers later as adjacent market.
Potential channels:
- hackathon demo,
- YouTube build stream,
- Twitter/X build thread,
- LinkedIn product post,
- HF Space leaderboard,
- Gradio Discord,
- Indian tech/ProductHunt style launch,
- Reddit India/frugal/mealprep communities,
- WhatsApp family-group proof-of-use story.
Exploration questions:
- Is “ShopStack” too general or perfect for scaling?
- Should the India-local layer be in the tagline, not name?
- Can demo be recorded with real household shopping?
- What short video moment makes people immediately understand?
- Which poster/screenshot is most shareable?
---
## 8. Risk / Constraint Exploration
Risks:
- too broad,
- over-reliance on imperfect vision,
- noisy market audio,
- privacy concerns,
- live price unreliability,
- too many models in one Space,
- 32B parameter accounting,
- non-commercial model licenses,
- Gradio UI becoming complex,
- agent anchoring to a reduced scope,
- overclaiming accuracy,
- app feeling like a demo instead of product direction.
Mitigations:
- confirmation-first UX,
- model registry,
- local-first mode,
- connected modes clearly separated,
- visible uncertainty,
- trace redaction,
- field notes,
- provider interfaces,
- benchmark scripts,
- privacy-first README,
- no auto-purchase,
- no medical/diet/legal claims,
- no private data in repo.
---
## 9. Open Questions
Agents should keep adding questions here.
1. Which model stack gives the best local-first performance under 32B?
2. Which model is easiest to run through llama.cpp for the parser?
3. What exact fine-tuned model should be published for Well-Tuned?
4. How much real household data can be safely used?
5. What is the first public trace dataset schema?
6. How should we count parameters when multiple models are optional but not loaded together?
7. Can Gradio handle the desired custom UI without too much friction?
8. Which STT model handles Hinglish best?
9. Which TTS model sounds warm enough for household use?
10. Should we use OCR or VLM-first for receipts?
11. How do we estimate quantity from photos without overclaiming?
12. Should price intelligence be mostly manual-memory-first?
13. How much map/heatmap functionality belongs in the app surface?
14. What is the simplest useful household map?
15. How do we benchmark live market/shelf scans?
16. What should be in the walkthrough video?
17. What should the social post emphasize?
18. What should the Field Notes title be?
19. How can Codex involvement be made obvious and authentic?
20. What is the strongest sponsor-alignment story?
---
## 10. Parking Lot
Use this for anything that may be interesting later.
- AR mode for finding items at home.
- Barcode scanning.
- Local store loyalty memory.
- Recipe planning from inventory.
- Waste tracking.
- Family member preferences.
- Domestic helper voice workflow.
- Shared household mode.
- WhatsApp integration.
- Calendar/reminder integration.
- Price community map.
- Privacy-preserving neighborhood price sharing.
- Offline mobile app.
- Browser extension for online grocery carts.
- Email receipt ingestion.
- Smart label printing.
- Shelf-life prediction from images.
- Freshness/ripeness model.
- Food waste report.
- Shopping carbon/effort score.
- Festival shopping planning.
- Monthly household budget intelligence.
- Elder-friendly voice-only mode.
- Accessibility mode for low-vision shoppers.
- Agentic shopping comparison, with user confirmation only.
- Local-language onboarding.
- Synthetic data generation pipeline.
- Human review UI for fine-tune data.
- Public leaderboard for household command parsing models.
- Sponsor-specific benchmark tables.
- Model replacement changelog.
---
## 11. Agent Contribution Protocol
When an agent adds an exploration item:
1. Add it under the relevant section.
2. Include why it matters for ShopStack.
3. Add model/tool links if known.
4. Mark whether it affects:
- product,
- model stack,
- dataset,
- evaluation,
- UI,
- privacy,
- sponsor alignment,
- bonus quest,
- marketing.
5. Do not delete old ideas unless they are unsafe or clearly obsolete.
6. Move rejected ideas to a “rejected/paused” note with reason.
7. Keep this map broad; implementation tasks belong in task docs, not here.
8. Avoid anchoring language that frames the product as temporary or small.
9. Keep hackathon constraints visible.
10. Prefer swappable interfaces over hardcoded models.
Template:
```md
### Idea / Model / Tool / Angle
**Category:** model / product / dataset / marketing / eval / privacy / sponsor / UI
**Why it matters:**
**How to test:**
**Risks:**
**Links:**
**Status:** explore / test / adopt / pause / reject
```
---
## 12. Current Strongest Directions
1. Voice-first shopping list and correction.
2. Vision-based market/shelf scan.
3. Purchase photo ingestion.
4. Inventory and freshness memory.
5. Household spatial memory.
6. Price and store memory.
7. Fine-tuned Indian household command parser.
8. llama.cpp parser/planner.
9. Anonymized trace dataset.
10. Field Notes with real household use.
11. Off-brand custom Gradio UI.
12. Model benchmarking and replacement policy.
---
_Last updated: 2026-06-05_
---
## 13. Explorer Addendum — Product and Architecture Discovery Backlog (2026-06-06)
**Source mix:** current docs, current package shape, test inventory, product hardening notes, and first-principles household-commerce analysis.
**Status:** exploration backlog; not implementation commitment.
**Update policy:** append-only unless an item is later superseded with a dated note.
### 13.1 Decision Service Extraction: make Gradio an adapter, not the brain
**Category:** architecture / product / evaluation / UI
**Why it matters:** Current docs already identify shopping service extraction as the canonical direction. The same boundary should cover Market Lens, Today dashboard, inventory recommendations, trace composition, and price intelligence so decisions are testable without UI wiring. This turns ShopStack from a demo UI into a durable household decision engine.
**How to test:** Create service-level tests that accept typed inputs and return typed decision objects; UI tests should only verify rendering/wiring.
**Risks:** Over-abstracting before the domain stabilizes. Keep services narrow and behavior-first.
**Status:** adopt incrementally.
### 13.2 Confidence-calibrated household recommendations
**Category:** product / model stack / evaluation / UI
**Why it matters:** “Buy,” “skip,” and “use soon” decisions need calibrated confidence language because household data is incomplete: unknown exact quantities, stale market prices, missing expiry dates, and approximate consumption rates. A clear uncertainty model increases trust and avoids fake precision.
**How to test:** Build fixtures for complete, partial, stale, and contradictory household states; assert both recommendation and confidence wording.
**Risks:** Too many caveats can make the product feel timid. Use concise labels: confident / likely / uncertain / needs confirmation.
**Status:** explore → adopt.
### 13.3 Household memory quality score
**Category:** product / data / UI / evaluation
**Why it matters:** The app should tell users whether its memory is good enough to trust. A “memory freshness / completeness” score can surface stale inventory, unpriced items, missing locations, uncertain quantities, and old market snapshots.
**How to test:** Score deterministic seeded households with known gaps; verify the score changes after purchases, consumption, imports, and stale data.
**Risks:** A score without actionable fixes becomes vanity UI. Pair every low score with the next best cleanup action.
**Status:** explore.
### 13.4 Receipt and invoice ingestion pipeline
**Category:** product / vision / OCR / dataset / privacy
**Why it matters:** Manual purchase entry is high friction. Receipts, app invoices, WhatsApp order summaries, and email receipts can populate inventory, price memory, store memory, and purchase cadence in one action.
**How to test:** Start with local sample receipt text/images; evaluate extraction into PurchaseEvent + PriceObservation + InventoryLot candidates with user confirmation.
**Risks:** PII leakage, messy receipt formats, and false inventory additions. Keep local-first, redacted, and confirmation-backed.
**Status:** explore.
### 13.5 Post-shopping reconciliation flow
**Category:** product / UI / data integrity
**Why it matters:** A shopping list is not complete until the household confirms what was actually bought, skipped, substituted, or already found at home. This flow would close the loop between plan → purchase → inventory → price memory.
**How to test:** Simulate a list with bought/partial/substituted/skipped items and assert inventory lots, price observations, shopping list state, and traces update together.
**Risks:** Too much bookkeeping. Use batch confirmation with sensible defaults.
**Status:** adopt candidate.
### 13.6 Substitution intelligence
**Category:** product / model stack / dataset / evaluation
**Why it matters:** Real shopping involves substitutions: tomato types, milk sizes, dal brands, detergent pack sizes, produce quality, and budget alternatives. ShopStack should recommend acceptable swaps based on household preference, price, unit price, shelf life, and recipe intent.
**How to test:** Build a substitution fixture set with item families, unacceptable swaps, dietary constraints, and unit-price comparisons.
**Risks:** Bad substitutions can break recipes or preferences. Require confirmation and learn from rejections.
**Status:** explore.
### 13.7 Waste-aware meal/use suggestions
**Category:** product / model stack / UI / marketing
**Why it matters:** “Use soon” becomes more valuable when it suggests practical actions: “use spinach tonight,” “make raita with curd,” “freeze bread,” or “move onions out of fridge.” This ties inventory memory to waste reduction.
**How to test:** Fixture expiring items and assert suggestions respect available pantry items, shelf life, household preferences, and safety disclaimers.
**Risks:** Recipe hallucination and unsafe food advice. Keep suggestions simple, local, and confidence-tagged.
**Status:** explore.
### 13.8 Store reliability and quality memory
**Category:** product / data / evaluation / UI
**Why it matters:** Cheapest is not always best. Produce quality, delivery reliability, substitutions, stale items, missing items, and refund friction should influence future recommendations.
**How to test:** Add store observations and verify recommendation ranking balances price, quality, freshness, travel/time, and confidence.
**Risks:** Sparse data can bias rankings unfairly. Show “based on N observations.”
**Status:** explore.
### 13.9 Freshness-first market data layer
**Category:** data / architecture / UI / privacy
**Why it matters:** Swiggy snapshots are useful but point-in-time. Every price/availability answer should carry source, captured-at timestamp, stale-data warning, and fallback behavior when freshness is insufficient.
**How to test:** Snapshot fixtures at fresh/stale/expired ages; assert UI and services never present old prices as live.
**Risks:** Over-warning can reduce usefulness. Use compact timestamp badges.
**Status:** adopt.
### 13.10 Local model routing policy by task risk
**Category:** model stack / architecture / evaluation / privacy
**Why it matters:** Not every task needs the same model. Deterministic parsers can handle quantity/unit extraction, embeddings can handle alias matching, and larger local planners can handle explanation. Routing reduces latency and improves reliability under the 32B cap.
**How to test:** Benchmark task families separately: command parse, receipt parse, recommendation explanation, alias match, and trace summarization.
**Risks:** Too many moving parts. Keep routing explicit and inspectable in Model Stack.
**Status:** explore.
### 13.11 Household-specific alias and preference learning
**Category:** product / dataset / model stack / privacy
**Why it matters:** Every household has names like “red dal,” “Amul wala dahi,” “kids bread,” “big Surf,” or “Friday sabzi.” A local adaptive lexicon can make voice and search feel personal without cloud training.
**How to test:** Record accepted corrections and rejected matches; assert future parsing improves while preserving auditability.
**Risks:** Wrong learned aliases can poison memory. Require reversible alias review.
**Status:** explore.
### 13.12 Inventory state reconciliation and contradiction detection
**Category:** data integrity / architecture / UI / evaluation
**Why it matters:** Household state will drift: items consumed without logging, duplicates, stale lots, impossible negative quantities, expired-but-used items, and contradictory locations. The app needs a cleanup assistant rather than silently accumulating bad memory.
**How to test:** Seed contradictory databases and assert the assistant proposes safe, confirmable fixes without destructive writes.
**Risks:** Cleanup flows can feel accusatory or tedious. Make them lightweight: “Is this still true?”
**Status:** explore.
### 13.13 Trace replay and evaluation harness
**Category:** evaluation / architecture / model stack / privacy
**Why it matters:** Existing anonymized traces can become a regression harness: replay planner inputs, compare tool calls, verify redaction, and benchmark model/provider changes before adoption.
**How to test:** Store golden trace fixtures and assert parser/tool-call outputs remain valid across model/config changes.
**Risks:** Golden traces can freeze bad behavior. Version them and review failures for intended improvements.
**Status:** adopt candidate.
### 13.14 Offline-first backup, restore, and household portability
**Category:** product / privacy / architecture / operations
**Why it matters:** Local-first users need safe backup/restore, device transfer, and human-readable exports. Portability already exists; next exploration is reliable restore semantics, conflict handling, and encrypted backups.
**How to test:** Round-trip realistic households through export/import, including duplicate items, traces, locations, prices, and market snapshots.
**Risks:** Bad imports can corrupt the source of truth. Use dry-run diff and confirmation before writes.
**Status:** explore.
### 13.15 Mobile capture path without losing local-first guarantees
**Category:** product / UI / architecture / privacy
**Why it matters:** Fridge scans, market scans, receipt photos, barcode scans, and voice commands are naturally mobile. The Gradio UI is useful for prototype velocity, but the product direction likely needs a mobile capture layer that syncs to local household memory.
**How to test:** Prototype mobile-friendly capture screens first; later evaluate a native/offline shell or local network companion.
**Risks:** Mobile sync can break the local-first trust model. Keep explicit sync boundaries and exportable data.
**Status:** explore.
### 13.16 Barcode and package knowledge base
**Category:** product / dataset / vision / UI
**Why it matters:** Barcodes can reduce ambiguity for packaged goods and enrich inventory with brand, size, category, nutrition, and repeat-purchase memory. A local cache avoids repeated lookups.
**How to test:** Build local barcode fixtures for common household goods and assert scan → candidate item → confirmation → inventory path.
**Risks:** Public barcode databases may be incomplete or noisy. Treat lookup as candidate data, not truth.
**Status:** explore.
### 13.17 Pantry economics and household budget intelligence
**Category:** product / commercial / evaluation / UI
**Why it matters:** ShopStack can become a household operating system for grocery spend: price inflation by item, category budgets, bulk-buy payoff, waste cost, and store choice tradeoffs. This is a strong retention surface beyond reminders.
**How to test:** Generate monthly fixture data and assert summaries match known spend, savings, waste, and stockout events.
**Risks:** Budget advice can feel judgmental. Use neutral, practical language.
**Status:** explore.
### 13.18 Safety boundaries for food, medicine, and sensitive household data
**Category:** safety / privacy / product / evaluation
**Why it matters:** Household inventory can include medicines, baby food, allergens, alcohol, and sensitive routines. The product needs explicit boundaries for advice, logging, redaction, and exports.
**How to test:** Add fixtures for allergens, expired medicine, baby formula, and personal data; assert conservative warnings and redaction.
**Risks:** Over-blocking benign grocery flows. Scope warnings to genuinely sensitive categories.
**Status:** explore → policy.
### 13.19 “Why this recommendation?” explanation cards
**Category:** product / UI / evaluation / trust
**Why it matters:** Users should see the concrete evidence behind decisions: at-home stock, last purchase, expected use rate, price snapshot age, store comparison, expiry/waste signal, and confidence. This improves trust and makes model behavior auditable.
**How to test:** Snapshot render decision cards for representative buy/skip/use-soon/substitute cases.
**Risks:** Cards can become verbose. Use progressive disclosure.
**Status:** adopt candidate.
### 13.20 Market Intelligence Graph / Market Map
**Category:** product / data / evaluation / UI
**Why it matters:** The next unique ShopStack wedge is not a plain price comparator. It is a living household graph that connects what is at home, what the market shows, what is stale, what is overpriced, what is already covered by inventory, what should be substituted, and what is likely to be wasted. This turns ShopCompare + ShopMemory + ShopStock into one decision surface.
**How to test:** Build fixtures with stale snapshots, sponsored cards, sold-out items, combo packs, partial inventory overlap, and substitution candidates; assert the graph produces stable buy / skip / use-soon / compare / wait / substitute lanes and truth-score labels.
**Risks:** If the graph reuses retailer language too directly, it can drift back into a shopping ad surface. Keep the default ranking household-first and explicitly label stale or sponsored signals.
**Status:** adopt candidate.
### 13.20 Launch story: from inventory app to household commerce memory
**Category:** marketing / product / positioning
**Why it matters:** “Inventory app” sounds like chores. “Household commerce memory” is more distinctive: it remembers what you have, what you paid, where it is, what spoils, and what to buy next. This framing should guide demos, docs, and feature prioritization.
**How to test:** Compare landing/demo narratives: inventory tracker vs shopping copilot vs household commerce memory; evaluate which makes the product feel inevitable and differentiated.
**Risks:** Too abstract for first-time users. Lead with concrete daily workflows, then name the deeper system.
**Status:** explore.
### 13.21 Data source expansion map
**Category:** data / product / privacy / evaluation
**Why it matters:** Swiggy fresh vegetables is a strong start, but durable market intelligence needs multiple source types: user-entered prices, receipts, quick-commerce snapshots, supermarket invoices, local kirana observations, barcode/package metadata, and seasonal produce calendars.
**How to test:** Define a normalized source contract and fixture each source type into PriceObservation/MarketSnapshot without UI assumptions.
**Risks:** Scraping or terms-of-service risk for live platforms. Prefer user-provided exports/snapshots and explicit provenance.
**Status:** explore.
### 13.22 Operator/developer diagnostics tab
**Category:** operations / UI / architecture / evaluation
**Why it matters:** Local-first apps need inspectability: DB path, provider mode, model availability, snapshot freshness, trace counts, redaction status, and recent errors. A diagnostics view makes failures understandable without cloud logs.
**How to test:** Mock provider/model/data states and assert the diagnostics view surfaces actionable status.
**Risks:** Too much internal detail in user UI. Gate as “Diagnostics” or “Local system status.”
**Status:** explore.
### 13.23 Evaluation leaderboard for small local household models
**Category:** model stack / evaluation / sponsor / marketing
**Why it matters:** The project can produce a reusable benchmark for local household agents: Hinglish command parsing, tool-call JSON validity, receipt extraction, alias matching, and recommendation explanation. This supports model choice and gives the project external credibility.
**How to test:** Build a small, versioned eval dataset from synthetic + anonymized traces; run candidate local models with latency, validity, and correction metrics.
**Risks:** Public benchmark data must not leak household details. Keep anonymization and synthetic generation strict.
**Status:** explore.
### 13.24 Human correction as the primary learning loop
**Category:** product / dataset / model stack / UI
**Why it matters:** The fastest path to a trustworthy household agent is not autonomous action; it is fast correction capture. Every “no, this is onions,” “skip this brand,” or “we call this atta” should improve future behavior locally.
**How to test:** Build correction fixtures and assert corrections update aliases/preferences/traces without mutating unrelated history.
**Risks:** Correction UI can interrupt flow. Offer one-tap corrections at decision points.
**Status:** explore.
### 13.25 Household graph model
**Category:** architecture / data / product
**Why it matters:** Items, lots, locations, stores, brands, prices, household members, preferences, recipes, and traces form a graph. SQLite can still store this, but a graph-shaped mental model helps avoid one-off tables and clarifies future recommendations.
**How to test:** Draw core relationships and map them to current tables; identify missing edges such as item↔alias, item↔substitute, store↔quality, member↔preference.
**Risks:** Premature graph infrastructure. Keep SQLite; use the graph as domain design, not necessarily a graph database.
**Status:** explore.
### 13.26 Market graph-to-basket compare actions
**Category:** product / UI / architecture / evaluation
**Why it matters:** The Market Intelligence Graph should not stay isolated. Compare lanes need direct “compare with home” and “compare with memory” actions so the graph can feed the shopping list and basket workflow, not just display intelligence.
**How to test:** Build compare-lane fixtures with partial inventory overlap and assert the compare view can expose the graph explanation, truth score, and next action.
**Risks:** Too much duplication between compare panels and the graph view. Keep the compare panel as a compact pointer into the graph rather than a second full graph UI.
**Status:** explore.
### 13.27 Market sponsored/stale de-emphasis policy
**Category:** product / safety / UI / evaluation
**Why it matters:** Sponsored and stale cards should be visually quieter and rank lower, but still remain inspectable. This is a trust and anti-manipulation policy, not a hard block.
**How to test:** Render fixtures with stale, sponsored, and upgrade-tagged items and verify they are visibly muted while their data remains accessible.
**Risks:** If the de-emphasis is too aggressive, legitimate deals get buried. Tune the signal carefully.
**Status:** explore.
### 13.28 Signal drill-down cards
**Category:** product / UI / trust / evaluation
**Why it matters:** Each market decision should explain itself with a compact breakdown of freshness, availability, sizing confidence, price confidence, memory confidence, and penalties. This makes the Market Intelligence Graph feel inspectable rather than magical.
**How to test:** Render representative buy/compare/substitute cards and assert the signal breakdown is present behind a progressive-disclosure control.
**Risks:** Too much detail can overwhelm the user. Keep the breakdown collapsed by default.
**Status:** explore.
### 13.29 Compare-to-basket bridge
**Category:** product / architecture / UI / evaluation
**Why it matters:** Compare-lane items should be able to flow into the shopping list, basket summary, or substitute confirmation with one clear step. This turns market intelligence into action rather than a dead-end display.
**How to test:** Build compare fixtures with home overlap and verify the next action can point to basket, pantry, or substitute confirmation without duplicating list state.
**Risks:** Action shortcuts can over-automate. Keep the bridge explicit and confirmable.
**Status:** explore.
### 13.30 Graph substrate projections
**Category:** architecture / product / evaluation
**Why it matters:** The Market Intelligence Graph should be the canonical substrate, with Today, Unified Shopping, Market Lens, and Ask ShopStack acting as projections over the same decision state. That keeps recommendations consistent across surfaces and makes graph reasoning reusable.
**How to test:** Build one graph fixture and project it into Today, unified shopping, market lens, and ask-context views; assert the same canonical item lands in the same lane and next action family.
**Risks:** Projection helpers can become another layer of indirection if they duplicate logic. Keep them thin wrappers over the canonical graph.
**Status:** explore.
### 13.31 Structured evidence and reason atoms
**Category:** data / trust / evaluation / product
**Why it matters:** Natural-language reasons are useful, but the graph needs structured reason atoms and evidence claims so traces, UI drill-downs, and corrections can all point at the same underlying facts.
**How to test:** Render a fixture graph and assert each decision cluster carries reason atoms, claim types, and a breakdown that can be serialized losslessly.
**Risks:** Over-structuring can produce ceremony without value. Keep the atom vocabulary small and readable.
**Status:** explore.
### 13.32 Combo solver and substitution ranker
**Category:** product / architecture / evaluation
**Why it matters:** Combos are where household intelligence beats a store page. A combo should be compared against home overlap and split purchase candidates, while substitutions should rank by cooking role, availability, freshness, and preference.
**How to test:** Use fixtures like onion/potato/tomato, herb bundles, and sold-out premium variants; assert the solver chooses compare/skip/buy-separate and the substitute ranking prefers the closest usable alternative.
**Risks:** Over-fancy ranking can confuse users if it is not explainable. Always show the reason for the ranking.
**Status:** explore.
### 13.33 Decision run memory loop
**Category:** product / data / evaluation
**Why it matters:** The graph becomes much more valuable once the app remembers what the household did after the recommendation: bought, skipped, substituted, or corrected. This turns one-off advice into learning.
**How to test:** Feed reconciliation events back into the graph and verify the next run uses the updated memory and preference signals.
**Risks:** If the feedback loop is too aggressive, noisy corrections can distort the model. Keep manual correction explicit.
**Status:** explore.
### 13.34 Frontier HF sweep and benchmark journal
**Category:** model stack / evaluation / documentation
**Why it matters:** The repo now has a fresh live Hugging Face frontier sweep in addition to the older local benchmarks. We should keep the frontier shortlist current without confusing it with the stable local runtime truth.
**How to test:** Run the reusable frontier harness in `tools/model_lab_frontier.py`, compare the saved JSONL/markdown artifacts, and keep the sweep notes synchronized with `shopstack/model_registry.py` and `Docs/MODEL_CATALOG.md`.
**Recent findings:**
- `Qwen/Qwen3.5-9B` scored 40.0% on the production-style planner prompt suite.
- `Qwen/Qwen3.6-35B-A3B` scored 20.0% but remained fast on HF inference.
- Live HF checks also surfaced `Qwen/Qwen3-ASR-0.6B`, `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`, `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign`, `openbmb/MiniCPM-V-4.6`, `deepseek-ai/DeepSeek-OCR-2`, `PaddlePaddle/PaddleOCR-VL-1.6`, and `briaai/RMBG-2.0` as current frontier candidates.
**Artifacts:** `benchmarks/model_lab/results/frontier_planner_20260613_113615.jsonl`, `benchmarks/model_lab/results/frontier_planner_20260613_113615.md`, `Docs/audits/MODEL_FRONTIER_SWEEP_2026-06-13.md`.
**Status:** adopt candidate.
### 13.35 Modal benchmark governance and GPU budgeting
**Category:** evaluation / infra / documentation
**Why it matters:** The Modal sweeps are intentionally parallel per candidate, but that only stays useful if the suite launches are budgeted together. The current plan-limit warning is a sign to sequence the sweeps, not to stop benchmarking.
**How to test:** Track the GPU count per suite, confirm the harness uses isolated per-candidate workers, and rerun one suite at a time until the artifact is written successfully. For STT, OCR, planner, vision, and TTS, record the result file path and the worker count in the run note.
**Risks:** Running planner + OCR + vision + STT sweeps concurrently can saturate the Modal workspace limit and produce stalled or wasted GPU time.
**Status:** adopt candidate.
---
## 14. Explorer Priority Cut (2026-06-06)
If only five exploration threads get near-term attention, choose these:
1. **Decision service extraction** — unlocks testability and keeps UI thin.
2. **Freshness-first market data layer** — prevents stale prices from becoming fake truth.
3. **Post-shopping reconciliation** — closes the core product loop.
4. **Trace replay/eval harness** — makes local model/provider changes safe.
5. **Human correction learning loop** — turns daily use into compounding household intelligence.
_Last updated: 2026-06-06_
---
## 15. Priority Addendum — Architecture Foundation + Visible User Loop (2026-06-06)
**Context:** Follow-up prioritization pass after reviewing the explorer backlog. This refines the earlier priority cut by separating architecture foundation from the visible user loop. The key adjustment is pulling confidence-calibrated recommendation cards earlier because they make the product’s trust model visible to users.
### Revised priority order
1. **Service / decision extraction**
- **Why:** This remains the foundation. ShopStack’s brain should live in typed services, not scattered Gradio callbacks.
- **Scope:** inventory state, market candidates, recommendation decisions, trace building, correction handling, reconciliation.
- **Reason to do first:** if freshness, reconciliation, cards, and learning are built inside UI handlers, they will need to be rewritten for mobile/API/agent surfaces.
2. **Freshness + provenance market data model**
- **Why:** Swiggy and future market snapshots are useful but time-sensitive. Recommendations must never imply stale snapshot data is live inventory.
- **Target market datum shape:**
```text
source = swiggy_instamart
captured_at = 2026-06-06
location/context = snapshot/browser/page context if known
freshness_state = fresh / aging / stale / expired
availability_state = available / sold_out / unknown
confidence = high / medium / low
```
- **Reason to do second:** bad market truth destroys user trust faster than missing features.
3. **Confidence-calibrated recommendation cards**
- **Why:** This is the first visible “this is smarter than a grocery list” surface.
- **Desired output style:**
```text
Buy Onion — ₹31/kg from Swiggy snapshot, available at capture time,
common staple, likely useful if home stock is low. Medium confidence
because current availability may have changed.
```
- **Reason to do third:** it turns backend correctness into visible product trust.
4. **Post-shopping reconciliation**
- **Why:** This closes the real household loop: planned → bought → skipped → substituted → already had → price changed.
- **Examples:**
```text
Bought: onion, tomato, potato
Skipped: cauliflower
Substituted: hybrid tomato instead of Indian tomato
Already had: carrot
Price changed: onion was ₹34 not ₹31
```
- **Reason to do fourth:** without reconciliation, inventory decays and ShopStack becomes a one-time planner instead of household memory.
5. **Human correction learning loop**
- **Why:** Corrections become local durable preferences and aliases.
- **Examples:**
```text
“atta” = wheat flour
Prefer Indian tomato over hybrid tomato
Avoid expensive premium/chemical-free variants unless requested
Don’t recommend sold-out items
Usually keep onion/potato/tomato stocked
```
- **Reason to do fifth:** this makes ShopStack personal without cloud training.
6. **Trace replay / evaluation harness**
- **Why:** Critical for model/provider/parser experimentation, but most meaningful after the first complete decision loop exists.
- **Replay contract:**
```text
inventory state + market snapshot + user intent
→ planner decision
→ recommendation cards
→ expected trace
→ expected user-facing output
```
- **Reason to do sixth:** evaluation should protect real product behavior, not incomplete scaffolding.
7. **Receipt / invoice / order ingestion**
- **Why:** High-leverage ingestion path, but it should feed an already-solid purchase/reconciliation model.
- **Output targets:** purchase_event, price_observation, inventory_delta, merchant_history, brand preference, quantity normalization.
- **Reason to do later:** ingestion without a solid reconciliation model creates messy data faster.
8. **Inventory contradiction cleanup / Household Memory Health**
- **Why:** Long-term reliability layer for stale lots, duplicate items, impossible quantities, and old produce.
- **Examples:**
```text
You marked tomatoes bought 9 days ago. Still have them?
You usually consume coriander within 2 days. Remove from inventory?
You said you already had carrots, but also bought 500g today. Merge?
```
- **Reason to do last among this cut:** it becomes valuable once enough real usage data exists.
### Build sequence
```text
services → provenance → recommendation cards → reconciliation → corrections → trace replay → ingestion → memory health
```
### Product thesis
ShopStack’s core magic should be:
> It knows what I have, knows what the market snapshot says, admits uncertainty, recommends clearly, then learns from what actually happened.
This sequence preserves clean internal architecture while quickly creating a visible user-facing reason to believe ShopStack is more than a grocery list.
_Last updated: 2026-06-06_
---
## 11. Feature Audit + Implementation Delegation Map (2026-06-07)
### Context
Pranay wants the full feature backlog implemented, but not by this agent. This section is a durable implementation map for future agents: what to build, why it matters, likely module ownership, and the local/open model + library stack that can support each feature while preserving the active constraint that simultaneously loaded models must stay at **≤32B total parameters**.
This is documentation-first. Future implementers should turn each feature cluster into focused tickets/PRs, use existing canonical modules, avoid duplicate routes/parallel systems, and keep core behavior local-first. Connected/cloud modes may exist only as optional, explicitly labeled modes.
### Research anchors checked
- Hugging Face OCR open-model guide, 2025-10-21: documents modern open OCR/VLM models and local tooling; notable ≤32B candidates include PaddleOCR-VL (~0.9B), Granite-Docling-258M, dots.ocr (~3B), DeepSeek-OCR (~3B), OlmOCR-2 (~8B), Chandra (~9B), Qwen3-VL (~9B), and Nanonets-OCR2 (~3-4B depending card/source wording). It also calls out MLX-VLM for Apple Silicon and vLLM/SGLang for batch/served OCR.
- SenseVoice repository: SenseVoice-Small is open-source, multilingual, low-latency ASR with speech understanding capabilities.
- Qwen3-32B model card: Qwen3-32B is Apache-2.0, 32.8B total / 31.2B non-embedding; this is at/just over the project’s 32B cap depending counting method, so treat it as a boundary-case candidate requiring explicit budget accounting before activation.
- Current grocery app market scan: competitive apps emphasize real-time shared lists, smart autocomplete, price estimates, receipt scanning, item-level spending analytics, plan-vs-actual comparison, pantry tracking, offline mode, and recipe-to-list flows. ShopStack’s differentiator should be local household memory + price/inventory/trace intelligence, not another checklist.
### Implementation rule for future agents
Build these through existing canonical boundaries:
```text
shopstack/services/* product logic and typed workflows
shopstack/persistence/* durable local state
shopstack/schemas/models.py domain contracts
shopstack/providers/* model/provider adapters
shopstack/tools/registry.py validated mutation/query tools
shopstack/ui/screens/* Gradio adapters only
shopstack/traces/* redaction/export primitives
shopstack/market/* market normalization/analytics/sources
```
Do **not** add a second app, second database, second trace pipeline, or route-like duplicate surface. Extend the existing service/provider/tool architecture.
---
### 11.1 P0 — make the current product feel real
#### F-001 Receipt ingestion → inventory + price memory
**Build:** Upload/photo receipt ingestion that extracts merchant/date/line items/quantities/prices/totals and proposes confirmable purchase events, inventory lots, and price observations.
**Why:** This closes the highest-leverage data loop. Receipt apps prove demand for item-level spend analytics; ShopStack can go further by updating home stock and future shopping decisions locally.
**Likely owners:** `shopstack/services/receipt.py`, `shopstack/providers/interfaces.py`, `shopstack/providers/*ocr*`, `shopstack/persistence/database.py`, `shopstack/ui/screens/market_lens.py` or a new canonical tab only if module registry is extended deliberately.
**Local/open model candidates ≤32B:**
| Candidate | Params | Fit | Notes |
|---|---:|---|---|
| `PaddlePaddle/PaddleOCR-VL` | ~0.9B | primary OCR/document parser | Multilingual, outputs Markdown/JSON/HTML; strong first test for receipts and packet labels. |
| `ibm-granite/granite-docling-258M` | 0.258B | lightweight document extraction | Very small; good CPU/Mac candidate for structured receipt/label extraction experiments. |
| `rednote-hilab/dots.ocr` | ~3B | OCR VLM | Markdown/JSON, grounding, handwriting; promising for messy receipts. |
| `deepseek-ai/DeepSeek-OCR` | ~3B | OCR + general visual understanding | Multilingual; good fallback candidate. |
| `allenai/olmOCR-2-7B-1025` | ~8B | high-quality document OCR | English-first; use for benchmark/comparison, not necessarily Indian receipt default. |
| `datalab-to/chandra` | ~9B | document OCR | Strong score, OpenRAIL; verify license before commercial claims. |
| `Qwen/Qwen3-VL-*` small variants | ~2B/4B/8-9B depending variant | general VLM + OCR | Useful if one VLM handles receipt, shelf, packet, and explanation flows. |
**Libraries/packages:** `paddleocr`, `pytesseract`/Tesseract, `surya-ocr`, `marker-pdf`, `docling`, `doctr`, `opencv-python`, `pillow`, `pydantic`, `rapidfuzz`, `dateparser`, `price-parser`, `babel`, `pandas`.
**Acceptance:** No database writes happen until the user confirms extracted rows. Every proposed write is trace-backed and redacted export-safe.
#### F-002 Plan-vs-actual shopping review
**Build:** Compare planned shopping list against actual receipt/manual purchases: bought, forgot, substituted, impulse-bought, overpaid, already-had, price changed.
**Why:** This is a competitor-visible differentiator and creates memory quality. Current grocery-list competitors highlight planned-vs-actual receipt comparison; ShopStack should connect that to inventory and future decisions.
**Likely owners:** `shopstack/services/reconciliation.py`, `shopstack/services/shopping.py`, `shopstack/persistence/database.py`, `shopstack/ui/screens/shopping.py`.
**Model/library candidates:** Rule-first with `rapidfuzz` + embeddings (`BAAI/bge-m3`, `intfloat/multilingual-e5-small/base`, `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`). Optional LLM arbitration with current 3B planner or `Qwen2.5/3 3B-8B Instruct`.
**Acceptance:** Reconciliation must preserve both user intent and actual purchase evidence; do not silently merge ambiguous items.
#### F-003 Trace service boundary
**Build:** Move trace bundle/export composition behind `shopstack/services/trace.py`, while keeping low-level redaction/export in `shopstack/traces/export.py`.
**Why:** Traces are core product trust, hackathon evidence, and evaluation substrate. They should not remain screen-local composition logic.
**Likely owners:** `shopstack/services/trace.py`, `shopstack/ui/screens/traces.py`, `shopstack/traces/export.py`.
**Model/library candidates:** No model required. Use typed Pydantic view models and deterministic JSONL export.
**Acceptance:** Existing trace UI behavior preserved; service can be unit-tested without Gradio.
#### F-004 Real local provider dogfood path
**Build:** A verified flow where at least one real local model path performs user input → perception/parser → validated tool call → trace → UI output.
**Why:** Provider interfaces exist, but product credibility requires a real, repeatable local model demo beyond mock providers.
**Likely owners:** `shopstack/providers/registry.py`, `shopstack/providers/local_provider.py`, `shopstack/planner/engine.py`, `shopstack/ui/screens/model_stack.py`.
**Local/open model candidates ≤32B:**
| Role | Candidate | Params | Notes |
|---|---|---:|---|
| Planner/parser | current `Llama-3.2-3B-Instruct-Q4_K_M.gguf` | 3B | Already downloaded/tested. |
| Planner/parser | `Qwen/Qwen2.5-7B-Instruct` or `Qwen/Qwen3-8B` | 7-8B | Strong structured output/tool-call candidate. |
| Planner/parser | `mistralai/Mistral-Small-3.2-24B-Instruct` GGUF | 24B | Strong single-model reasoning under cap if no big VLM loaded concurrently. |
| Edge parser | `HuggingFaceTB/SmolLM3-3B` | 3B | Good small parser candidate if tool JSON is reliable. |
| Boundary max | `Qwen/Qwen3-32B` | 31.2B non-embedding / 32.8B total | Use only with explicit budget policy; leaves no room for other loaded models. |
**Libraries/packages:** `llama-cpp-python`, `mlx-lm`, `mlx-vlm`, `transformers`, `accelerate`, `bitsandbytes` where supported, `outlines`, `guidance`, `instructor`, `jsonschema`, `pydantic`.
**Acceptance:** Model Stack tab shows exact backend/model/params, and traces identify model/provider used.
#### F-005 Empty-state and honesty polish
**Build:** Make empty charts/tables, mock perception, stale snapshots, and unavailable providers explicit and visually consistent.
**Why:** Trust is damaged faster by fake-looking output than by missing features.
**Model/library candidates:** No model required.
**Acceptance:** No chart/table renders as a broken blank shell when data is empty; mock/stale/preview mode is labeled wherever it affects decisions.
---
### 11.2 P1 — product intelligence features
#### F-006 Household-specific item aliases and canonicalization
**Build:** Local alias memory for “dahi/curd/yogurt,” “Surf,” “Vim,” “pav,” “atta,” etc. Add correction capture and confidence.
**Models/libraries:** `rapidfuzz`, `textdistance`, `sqlite-vec` or `sqlite-vss`, `BAAI/bge-m3` (0.6B), multilingual E5, MiniLM, `sentence-transformers`, optional local LLM parser (`Qwen3-4B/8B`, current Llama 3B).
**Acceptance:** Corrections persist and influence future shopping, receipt, and market matching.
#### F-007 Purchase cadence + stockout forecast
**Build:** Days-until-stockout and restock interval predictions using local history.
**Models/libraries:** Start deterministic: SQLite queries, `pandas`, `numpy`, `statsmodels`, simple exponential smoothing. Optional later: `sktime`, `statsforecast`, `prophet`-like alternatives.
**Acceptance:** Recommendations include confidence language: “likely low,” “uncertain,” not false precision.
#### F-008 Waste / overbuy detection
**Build:** Detect old active lots, duplicate purchases, unused perishables, and recurring waste patterns.
**Models/libraries:** Rule-first with `shopstack/decisions.py`; optional embeddings for item equivalence.
**Acceptance:** User can confirm cleanup/merge/consume; no automatic deletion.
#### F-009 Store memory
**Build:** Store entities with type, quality/freshness/price/trust/convenience scores, notes, and item-store edges.
**Models/libraries:** No model required initially. Later embeddings for store-note retrieval.
**Acceptance:** Store memory feeds “where should I buy?” and price intelligence surfaces.
#### F-010 Price fairness bands
**Build:** For scanned/manual price: compare against household memory and source snapshots; classify fair/high/low/unknown.
**Models/libraries:** `pandas`, rolling averages, robust median/IQR, unit normalization already in `shopstack/market/normalization.py`.
**Acceptance:** Always show source and freshness: household observation vs Swiggy snapshot vs manual entry.
#### F-011 Budget-aware list planning
**Build:** Estimate list total, compare against target budget, downgrade optional items, show “must buy / optional / wait” budget plan.
**Models/libraries:** Deterministic first; optional planner LLM for explanation only.
**Acceptance:** No hidden price assumptions; every estimate has source/freshness.
#### F-012 Substitutes and staples
**Build:** Item relationships: substitutes, usually-bought-with, staples, recipe-use groups.
**Models/libraries:** SQLite edge table, `networkx` for analysis if useful, embeddings for semantic substitute suggestions, optional small LLM for candidate generation.
**Acceptance:** Generated substitutes must be confirmable before becoming durable household rules.
---
### 11.3 P2 — ShopLens expansion
#### F-013 Packet label close-up mode
**Build:** OCR MRP, expiry, quantity, brand, nutrition/storage text from packet labels.
**Models:** PaddleOCR-VL, Granite-Docling, dots.ocr, DeepSeek-OCR, Qwen3-VL small, MiniCPM-V.
**Libraries:** `opencv-python` preprocessing, `pillow`, `dateparser`, `pydantic` validators.
#### F-014 Barcode catalog memory
**Build:** First scan creates barcode → item mapping; later scans resolve instantly.
**Libraries:** Existing `shopstack/scanner.py`; add/compare `pyzbar`, `zxing-cpp`, `opencv`, JS `html5-qrcode`, `quagga2`, `zxing-js/browser`.
**Acceptance:** Barcode mappings are user-correctable and local.
#### F-015 Fridge/pantry photo snapshot
**Build:** Save visual snapshots and confirmed item detections per household location.
**Models:** YOLOv8/YOLO11, RF-DETR, GroundingDINO/MM-GroundingDINO, OWL-ViT, Florence-2, MiniCPM-V, Qwen3-VL small.
**Current grounding note (2026-06-15):** the real-photo household grounding benchmark now exists with three imagegen-created scenes (fridge, cupboard, tabletop). `GroundingDINO` reached `0.2042` mean box IoU / `0.3333` success after provider fixes; `Qwen3-VL` stayed helper-only at `0.0136` mean box IoU.
**Libraries:** `ultralytics`, `rfdetr`, `transformers`, `supervision`, `opencv-python`, `faiss`/`sqlite-vec` for image-snapshot matching.
#### F-016 Shelf/location recommendations
**Build:** Recommend storage location based on category/perishability/history.
**Models/libraries:** Rule-first from category metadata; optional classifier/LLM for ambiguous items.
#### F-017 Home Scan scene understanding
**Build:** A broader ShopLens Home Scan that understands kitchen, bathroom, medicine, cleaning, bedroom, and document scenes; count visible items, read labels/expiry, accept local-language voice corrections, and propose safe inventory updates.
**Models:** Object detection + segmentation + OCR + STT first, then VLM scene summarization and multilingual correction parsing.
**Libraries:** Existing provider interfaces (`object_detection`, `segmentation`, `ocr`, `stt`, `image_edit`), `opencv-python`, `pydantic`, `dateparser`, optional VLM backends for frame captioning and cross-frame consolidation.
**Current implementation note (2026-06-14):** Shelf Scan now accepts a short video sweep, extracts multiple frames locally, merges repeated detections into one household-memory result, and uses the real OCR pipeline instead of the old mock-style compatibility fallback. The still-photo path remains intact.
**Current grounding note (2026-06-15):** generated fridge/cupboard/table photos are now saved under `benchmarks/modal/assets/household_grounding` and used in the dedicated grounding benchmark harness (`benchmarks/modal/bench_grounding_v1.py`).
**Acceptance:** Shelf scans stay confirmation-first, can be zone-aware, can merge short video sweeps, and update household memory without assuming the home is only pantry/fridge.
#### F-017 Batch confirmation UI
**Build:** Show proposed structured actions from OCR/image/list parsing and let user accept/edit/reject before writes.
**Models/libraries:** Pydantic action schemas, Gradio DataFrame/editable components, trace service.
---
### 11.4 P3 — market and context intelligence
#### F-018 Multi-source retailer adapters
**Build:** Add a common adapter interface for Blinkit, Zepto, BigBasket, local CSV/manual imports, and future connected sources.
**Libraries:** `pydantic`, `pandas`, `duckdb`, `httpx` only in connected mode, Playwright/browser automation only if explicitly allowed.
**Acceptance:** Every source record carries source, captured_at, freshness, confidence, and terms/collection notes.
#### F-019 Basket-level retailer choice
**Build:** Compare basket total across sources and split basket if savings justify it.
**Libraries:** `pandas`, `scipy`/simple optimization, existing `shopstack/market/basket.py`.
#### F-020 Travel-effort decision
**Build:** Manual-first trip effort model; optional route engines later.
**Libraries:** `osmnx`, `networkx`, OSRM/OpenRouteService/Valhalla only as optional connected mode; `geopy`, `h3` for coarse privacy cells.
#### F-021 Weather-aware shopping note
**Build:** Manual/mock weather context; optional weather lookup later.
**Libraries:** `meteostat`, Open-Meteo client in connected mode, local config/manual input in Off the Grid.
#### F-022 Private price heatmap
**Build:** Coarse local price cells with minimum observation counts.
**Libraries:** `h3`, `folium` or deterministic SVG/HTML cards; avoid exact home/route exposure in traces.
---
### 11.5 P4 — launch, evaluation, and hackathon assets
#### F-023 Walkthrough mode
**Build:** Guided demo story: ask what to buy → scan item → skip duplicate → add purchase → export trace.
**Models:** Use the smallest reliable active stack: current 3B planner + mock/real OCR path as available.
#### F-024 Public redacted trace dataset
**Build:** Curated anonymized JSONL trace examples and dataset card.
**Libraries:** existing redaction + `datasets` library for HF publishing.
#### F-025 Field Notes report generator
**Build:** Generate polished build report from traces, model stack, decisions, failures, and screenshots/outputs.
**Models:** Local LLM optional for drafting, but factual claims must come from deterministic trace/model metadata.
#### F-026 Well-Tuned narrow model
**Build:** Fine-tune a small command/entity parser for Hinglish grocery commands and publish it.
**Base candidates ≤32B:** `Qwen2.5-1.5B/3B-Instruct`, `Qwen3-1.7B/4B`, `SmolLM3-3B`, `MiniCPM5-1B`, current Llama 3B if license permits desired artifact.
**Libraries:** `trl`, `peft`, `unsloth`, `axolotl`, `datasets`, `evaluate`, `jsonschema`, `pydantic`.
**Dataset:** Indian/Hinglish household command parser dataset with tool-call JSON labels.
#### F-027 README / HF Space launch checklist
**Build:** Launch checklist covering privacy, local-first, model budget, data freshness, demo flow, trace dataset, licenses, and known limitations.
**Models:** None required.
---
## 12. Local/Open Model and Library Selection Matrix (≤32B constraint)
### 12.1 Recommended active-stack scenarios
The cap is about **simultaneously loaded active models**, not every candidate in the repo. Future agents should declare one active scenario before implementation/testing.
| Scenario | Active models | Approx params | Use when |
|---|---|---:|---|
| Minimal reliable | Llama 3.2 3B planner + BGE-M3 embeddings + Kokoro TTS optional | ~3.7B | Current local-first baseline, fast demos. |
| Receipt-first | PaddleOCR-VL 0.9B + Llama 3B planner + BGE-M3 | ~4.5B | Receipt/packet extraction with structured reconciliation. |
| Vision-heavy | MiniCPM-V 8B or Qwen3-VL 8-9B + Llama 3B + BGE-M3 | ~12-13B | Market Lens, shelf/fridge photos, OCR fallback. |
| Strong planner | Mistral Small 24B GGUF + lightweight OCR 0.9B + BGE-M3 | ~25.5B | Harder reasoning, still under cap if no large VLM. |
| Boundary single-brain | Qwen3-32B alone | 31.2B non-embedding / 32.8B total | Only if project explicitly accepts counting method and gives up simultaneous specialist models. |
| Full practical multimodal | Qwen3-VL 9B + Qwen/Llama planner 7-8B + SenseVoice 0.2B + Kokoro 0.082B + BGE-M3 0.6B + segmentation 0.3B | ~18-19B | Rich local multimodal stack under cap. |
### 12.2 Capability-by-capability candidate shortlist
| Capability | Best first candidate | Strong alternatives | Libraries/runtime |
|---|---|---|---|
| Planner/tool JSON | current Llama 3B GGUF | Qwen3 4B/8B, Qwen2.5 7B, Mistral Small 24B | `llama-cpp-python`, `mlx-lm`, `transformers`, `outlines`, `instructor` |
| Receipt OCR | PaddleOCR-VL | Granite-Docling, dots.ocr, DeepSeek-OCR, OlmOCR-2 | `paddleocr`, `mlx-vlm`, `vllm`, `sglang`, `docling`, `marker` |
| Packet label OCR | PaddleOCR-VL | dots.ocr, Qwen3-VL, MiniCPM-V | `opencv-python`, `dateparser`, OCR/VLM runtime |
| General VLM | MiniCPM-V or Qwen3-VL small | Florence-2, SmolVLM variants | `transformers`, `mlx-vlm` |
| Object detection | YOLO11/YOLOv8 | RF-DETR, GroundingDINO, OWL-ViT | `ultralytics`, `rfdetr`, `supervision`, `opencv-python` |
| Segmentation/cards | RMBG-1.4 | BiRefNet, SAM variants, ClipSeg | `transformers`, `opencv-python`, `pillow` |
| ASR | SenseVoice-Small | Parakeet 0.6B/V3, Distil-Whisper, Whisper large-v3-turbo | `funasr`, `faster-whisper`, `mlx-audio`, `transformers` |
| TTS | Kokoro-82M | Qwen3-TTS 0.6B, CosyVoice, Piper, KittenTTS | `kokoro`, `piper-tts`, `onnxruntime`, `mlx-audio` |
| Embeddings | BGE-M3 | multilingual E5, MiniLM, Jina embeddings | `sentence-transformers`, `faiss-cpu`, `sqlite-vec`, `lancedb`, `chromadb` |
| Reranking | BGE reranker small/base | Jina reranker small | `sentence-transformers`, `FlagEmbedding` |
| Time series | deterministic stats | statsmodels/sktime/statsforecast | `pandas`, `numpy`, `statsmodels`, `sktime` |
| Route/geo optional | manual + H3 | OSRM/OpenRouteService/Valhalla | `h3`, `osmnx`, `networkx`, `geopy`, `folium` |
| Barcode | existing scanner + pyzbar | zxing-cpp, html5-qrcode, Quagga2, ZXing JS | `pyzbar`, `zxing-cpp`, `opencv`, JS browser libs |
### 12.3 Specific model notes for future benchmarking
- **PaddleOCR-VL (~0.9B):** highest-priority OCR test because it is small, multilingual, and supports JSON/Markdown/HTML-style structured output. Best first model for receipts and packet labels.
- **Granite-Docling-258M:** extremely small document-understanding candidate. Best for CPU/Mac fallback and quick tests.
- **dots.ocr (~3B):** good OCR VLM candidate for Markdown/JSON and grounding. Test on messy local receipts.
- **DeepSeek-OCR (~3B):** useful for OCR + general visual understanding; test on labels and screenshots.
- **OlmOCR-2 (~8B):** strong document OCR but English-oriented; use as benchmark/reference more than default Indian household model.
- **Chandra (~9B):** strong document OCR, but verify OpenRAIL restrictions before product claims.
- **Qwen3-VL small family (~2B-9B variants):** promising single VLM for OCR, packet labels, shelf images, and explanation. Confirm exact chosen variant params/license before activation.
- **MiniCPM-V (~8B):** strong general VLM candidate and sponsor-aligned with OpenBMB angle.
- **SenseVoice-Small (~Whisper-small class):** strong ASR candidate for fast multilingual speech understanding. Benchmark Hinglish grocery phrases.
- **Parakeet 0.6B/V3:** very fast ASR candidate; confirm language coverage for Hindi/Hinglish before relying on it.
- **Kokoro-82M:** excellent tiny local TTS baseline. Use for short, warm household answers if pronunciation is acceptable.
- **Qwen3-TTS 0.6B / CosyVoice:** better voice quality candidates; benchmark latency and Hindi/Hinglish pronunciation.
- **BGE-M3 (0.6B):** already aligned with workspace embedding use; use for aliases, trace retrieval, and household memory search.
- **Qwen3-32B:** boundary case. It is useful as a “single strong brain” experiment but total params are listed as 32.8B and non-embedding 31.2B. Do not activate without updating the budget rule/counter and documenting whether total or non-embedding params are counted.
### 12.4 Non-model libraries that can implement much of the backlog
Many features should be deterministic first:
- **Parsing and validation:** `pydantic`, `jsonschema`, `dateparser`, `price-parser`, `babel`, `python-slugify`.
- **Matching/canonicalization:** `rapidfuzz`, `textdistance`, `unidecode`, `regex`, `sentence-transformers`.
- **Analytics/time series:** `pandas`, `numpy`, `duckdb`, `statsmodels`, `sktime`, `statsforecast`.
- **Vector/local memory:** `sqlite-vec`, `sqlite-vss`, `faiss-cpu`, `lancedb`, `chromadb`.
- **Vision preprocessing:** `opencv-python`, `pillow`, `scikit-image`, `imagehash`.
- **OCR baselines:** `pytesseract`, `paddleocr`, `surya-ocr`, `docling`, `marker-pdf`, `doctr`.
- **Barcode:** `pyzbar`, `zxing-cpp`, browser `html5-qrcode`, `quagga2`, `@zxing/browser`.
- **Maps/geo:** `h3`, `geopy`, `osmnx`, `networkx`, `folium` for optional map experiments.
- **Voice pipeline:** `funasr`, `faster-whisper`, `piper-tts`, `kokoro`, `onnxruntime`, `mlx-audio`, `pipecat` for a more advanced local voice loop.
- **Model serving:** `llama-cpp-python`, `mlx-lm`, `mlx-vlm`, `transformers`, `vllm`, `sglang`, `ollama` as optional local dev runtime.
- **Constrained outputs:** `outlines`, `guidance`, `instructor`, `lm-format-enforcer`.
- **Training/fine-tuning:** `datasets`, `peft`, `trl`, `unsloth`, `axolotl`, `evaluate`.
### 12.5 Implementation sequencing recommendation
Future agents should implement in this order:
```text
1. Trace service boundary
2. Receipt extraction proposal schema, no writes
3. Receipt confirmation → purchase/price/inventory writes
4. Plan-vs-actual reconciliation
5. Alias/canonicalization correction loop
6. Price fairness + budget-aware list planning
7. Packet label mode + barcode catalog memory
8. Fridge/pantry snapshot and storage recommendations
9. Store memory + basket-level retailer choice
10. Trace replay/eval harness + public redacted trace dataset
11. Fine-tuned narrow command parser
12. Optional travel/weather/heatmap intelligence
```
This order protects data integrity first, then adds intelligence, then expands perception/context. It also gives a future non-this-agent implementation team clean seams for independent work.
_Last updated: 2026-06-07_
---
## 13. Fresh HF / 2026 Model Launch Addendum (2026-06-07)
### Why this addendum exists
Pranay challenged whether the previous model scan was fresh enough. That challenge was valid. The first pass covered strong late-2025 OCR/VLM candidates, but it underweighted newer/open launches and HF ecosystem updates relevant to ShopStack, especially omni-modal, realtime speech, multimodal embeddings/rerankers, and HF Pro/Jobs workflows.
This section supersedes the older shortlist where it is more specific. Keep the older section as historical context, but future agents should check this addendum first when choosing models.
### Fresh research anchors checked
- **Qwen3-Omni-30B-A3B-Instruct / Thinking / Captioner** model cards and GitHub: Apache-2.0, omni-modal, text/image/audio/video input, text/speech output, 119 text languages, 19 speech-input languages, 10 speech-output languages, real-time streaming; 30B MoE with ~3B active parameters. Strong candidate for a single multimodal “one brain does a lot” experiment, but likely heavy operationally.
- **Mistral Voxtral Mini 4B Realtime 2602** model card: Apache-2.0, multilingual realtime ASR, 4B params, <500ms delay target, 13 languages, designed for on-device/minimal hardware compared with heavier speech stacks.
- **Hugging Face OCR open-model guide + community comments:** added/flagged models not previously emphasized enough: `opendatalab/MinerU2.5-2509-1.2B`, `lightonai/LightOnOCR-1B-1025`, `tencent/POINTS-Reader`, `Logics-MLLM/Logics-Parsing`, `SmolDocling-256M-preview`, `RolmOCR-7B`, `Typhoon-OCR-7B`, `MonkeyOCR-Recognition`.
- **Hugging Face multimodal Sentence Transformers v5.4 blog (2026-04):** new multimodal embedding/reranker support for text/images/audio/video. `Qwen/Qwen3-VL-Embedding-2B` is explicitly shown and needs ~8GB VRAM; 8B variants need ~20GB. This matters for shelf image retrieval, receipt visual retrieval, and trace/memory search.
- **HF Jobs pricing/docs:** Jobs can run batch compute on HF infrastructure when the account has positive credit balance; billed by minute while starting/running. This fits Pranay’s HF Pro/credits for benchmarking and batch OCR experiments without adding cloud dependency to the Off-the-Grid runtime.
- **vLLM-Omni speech API docs:** supports serving Qwen3-TTS, Fish Speech, Voxtral TTS, CosyVoice3, OmniVoice, VoxCPM2, MOSS-TTS-Nano, Qwen3-Omni and related omni/audio models via OpenAI-style APIs. Useful for benchmark rigs and optional connected/dev mode.
- **Aya Vision 8B/32B:** multilingual VLM family from CohereForAI. 8B is relevant for multilingual image/text tasks; 32B is boundary/too tight for simultaneous loaded stack and should be treated carefully.
- **SmolVLM / SmolVLM2 family:** tiny VLMs (256M/500M/2.2B class) useful for on-device-ish image caption/doc QA/video-lite experiments, not likely enough for robust receipt extraction alone but excellent baseline/performance fallback.
---
### 13.1 Updated “newer launch” candidates for ShopStack
#### Omni / single-model-does-a-lot candidates
| Model | Params / active | License | Why it matters for ShopStack | Caveat |
|---|---:|---|---|---|
| `Qwen/Qwen3-Omni-30B-A3B-Instruct` | 30B total, ~3B active MoE | Apache-2.0 | One model can cover text, image, audio, video, ASR-like input, and speech output. Strong candidate for a “single local/offline brain” experiment and sponsor-aligned demo. | Transformers inference can be slow; vLLM/Docker recommended. Counts as 30B loaded for project budget unless policy explicitly counts active params. Leaves little room under 32B if total params counted. |
| `Qwen/Qwen3-Omni-30B-A3B-Thinking` | 30B total, ~3B active MoE | Apache-2.0 | Better for complex multimodal reasoning over receipt/photo/audio traces. | Same budget/runtime caveat; likely slower. |
| `Qwen/Qwen3-Omni-30B-A3B-Captioner` | 30B total, ~3B active MoE | Apache-2.0 | Audio captioning and market/environment audio notes; useful for “what happened in this noisy shopping audio?” | More specialized; not a default core runtime. |
| `openbmb/MiniCPM-o-2_6` | ~8B | check model card | Any-to-any style multimodal candidate with vision/speech/language. Good OpenBMB sponsor alignment. | Must verify current license/runtime before product use. |
**Opinionated take:** Qwen3-Omni is the most exciting newer launch for the product vision, but it is not the first implementation model. Use it with HF Pro/credits for benchmarking and one “wow” demo. For reliable local-first core, keep specialist small models + deterministic services.
#### Fresh OCR / document-understanding candidates to add to benchmark set
| Model | Params | Why it matters | Suggested ShopStack test |
|---|---:|---|---|
| `opendatalab/MinerU2.5-2509-1.2B` | ~1.2B | Mentioned in HF OCR guide comments as missing from comparison; compact document parser. | Indian grocery receipt with line items + packet label close-up. |
| `lightonai/LightOnOCR-1B-1025` | ~1B | Community-flagged “punches above weight” OCR candidate. | Low-quality local receipt photo and multilingual label. |
| `tencent/POINTS-Reader` | ~3B text backbone + vision encoder | Efficient end-to-end document AI replacing larger 7B backbone with Qwen2.5-3B; Chinese/English strengths. | Receipt OCR and structured text extraction speed/accuracy. |
| `Logics-MLLM/Logics-Parsing` | verify card | Single-model document parsing for complex docs. | Use as secondary benchmark, not default. |
| `SmolDocling-256M-preview` | ~256M | Tiny document understanding candidate. | CPU/Mac fallback for basic labels/receipts. |
| `RolmOCR-7B` | ~7B | Lightweight VLM-level OCR option. | Compare against OlmOCR and PaddleOCR-VL. |
| `Typhoon-OCR-7B` | ~7B | OCR model seen in HF OCR demo collections. | Test if receipts are multilingual/noisy. |
| `MonkeyOCR-Recognition` | verify card | OCR recognition candidate in HF demo collections. | Baseline OCR recognition, not full structured extraction. |
**Updated OCR priority:**
```text
1. PaddleOCR-VL (~0.9B)
2. MinerU2.5-1.2B + LightOnOCR-1B
3. Granite/SmolDocling tiny fallbacks
4. dots.ocr / DeepSeek-OCR / POINTS-Reader
5. OlmOCR-2 / RolmOCR / Typhoon-OCR as heavier benchmarks
6. Qwen3-VL / Qwen3-Omni for general multimodal fallback
```
#### Fresh ASR / realtime voice candidates
| Model | Params | License | Why it matters | Caveat |
|---|---:|---|---|---|
| `mistralai/Voxtral-Mini-4B-Realtime-2602` | 4B | Apache-2.0 | Newer realtime ASR, 13 languages, <500ms delay target, open weights. Stronger fresh candidate than older Whisper-only baseline. | Hindi/Hinglish support must be tested; 4B is heavier than SenseVoice/Parakeet. |
| `Qwen/Qwen3-Omni-30B-A3B-Instruct` | 30B total / 3B active | Apache-2.0 | Speech input and speech output in one model; supports Urdu but not explicitly Hindi speech input in listed languages. | Heavy; use as benchmark/wow path. |
| `FunAudioLLM/SenseVoiceSmall` | ~0.2B class | likely permissive; verify exact card | Fast multilingual ASR + emotion/audio events. | Language list may not cover Hindi directly; benchmark Hinglish. |
| NVIDIA Parakeet V3 / 0.6B family | ~0.6B | check card | Very fast ASR; good low-latency candidate. | Often English-focused; verify multilingual/Hinglish. |
| Whisper large-v3-turbo | ~0.8B | MIT | Mature baseline with broad language support. | Not native realtime; use faster-whisper/whisper.cpp. |
#### Fresh TTS / speech output candidates
| Model | Params | License | Why it matters | Caveat |
|---|---:|---|---|---|
| `Qwen/Qwen3-TTS-12Hz-0.6B-*` | 0.6B | check card; Qwen family often Apache-2.0 | Small Qwen TTS; vLLM-Omni supports CustomVoice/Base. | Benchmark Hindi/Hinglish pronunciation and runtime complexity. |
| `Qwen/Qwen3-TTS-12Hz-1.7B-*` | 1.7B | check card | Better quality than 0.6B; supports CustomVoice/VoiceDesign/Base variants via vLLM-Omni. | Heavier, still fine under 32B. |
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 0.5B | verify card | vLLM-Omni supported; voice cloning style path. | More complex request requirements. |
| `mistralai/Voxtral-4B-TTS-2603` | ~4.1B | reported CC-BY-NC 4.0 in third-party scan; verify HF card | High-quality multilingual/Hindi-capable TTS candidate. | Non-commercial likely; not default for product unless license accepted. |
| `OpenMOSS-Team/MOSS-TTS-Nano` | small; verify | verify | vLLM-Omni supported; potential lightweight voice cloning. | Needs ref audio; not preset voice default. |
| Kokoro-82M | 82M | Apache-2.0 | Still best tiny default TTS baseline. | Pronunciation/voice quality may be weaker than newer TTS. |
| Piper | small | permissive | Reliable local TTS fallback. | Less “wow,” but robust. |
#### Fresh multimodal embedding / reranker candidates
| Model / stack | Params | Why it matters |
|---|---:|---|
| `Qwen/Qwen3-VL-Embedding-2B` | 2B | Directly relevant to visual receipt/shelf/image retrieval, multimodal memory search, and trace retrieval. HF Sentence Transformers v5.4 shows loading with image support. |
| Qwen3-VL Embedding 8B variants | ~8B | Higher-quality multimodal retrieval if GPU memory allows. |
| Sentence Transformers v5.4 multimodal rerankers | varies | Lets ShopStack rerank text-image/audio/video memories with one API. |
| CLIP/SigLIP/SigLIP2 family | small-medium | Good fast image/text similarity for shelf snapshots and visual duplicate matching. |
| BGE-M3 | 0.6B | Still best text-heavy multilingual local memory default. |
**New product implication:** ShopStack should not only do text embeddings. It should plan for **multimodal household memory**: receipt images, shelf photos, voice snippets/transcripts, and traces should be retrievable across modalities.
---
### 13.2 HF Pro / credits usage plan
Pranay has HF Pro and credits. Use them for research, benchmarking, and artifact creation — not as hidden runtime dependency for the Off-the-Grid path.
#### Good uses of HF Pro/credits
1. **Batch OCR bakeoff with HF Jobs**
- Create a small representative dataset: 20 receipt photos, 20 packet labels, 10 shelf/fridge photos, 10 Swiggy screenshots/product cards.
- Run PaddleOCR-VL, MinerU2.5, LightOnOCR, dots.ocr, DeepSeek-OCR, Granite/SmolDocling, POINTS-Reader via Jobs or temporary GPU Space.
- Save outputs as a versioned benchmark dataset/artifact.
2. **Temporary GPU Space for demo/model exploration**
- Spin up model-specific Spaces for Qwen3-Omni, Voxtral Realtime, Qwen3-TTS, Qwen3-VL Embedding.
- Record latency, memory, model output, failure modes.
- Tear down or downgrade hardware after experiment.
3. **Dataset publishing**
- Publish redacted/synthetic command parser dataset.
- Publish redacted trace examples.
- Publish OCR benchmark fixtures if privacy-safe.
4. **Fine-tuning / LoRA training**
- Narrow parser LoRA on Qwen/SmolLM/MiniCPM-sized base.
- Do not fine-tune a broad “do everything” model first.
5. **Model comparison reports**
- Use HF Jobs for repeatable batch inference and store metrics.
- Feed results back into `Docs/MODEL_CATALOG.md` and `shopstack/model_registry.py` only after local/runtime verification.
#### Bad uses of HF Pro/credits
- Making HF Inference Endpoint mandatory for core app behavior while claiming Off the Grid.
- Using credits to bypass implementing local confirmation, validation, and traceability.
- Benchmarking on generic OCR samples only; ShopStack needs grocery/household-specific data.
- Promoting a model to active runtime without testing memory, latency, license, and JSON/tool-call reliability.
### 13.3 Revised model strategy
The correct strategy is not “pick one best model.” Use a three-lane model strategy:
```text
Lane A — Reliable local core
deterministic services + Llama/Qwen 3B-8B planner + PaddleOCR-VL/Granite/SmolDocling + BGE-M3 + Kokoro/Piper
Lane B — Specialist excellence
Voxtral Realtime for ASR, Qwen3-TTS/CosyVoice3 for TTS, Qwen3-VL Embedding for multimodal memory, YOLO/RF-DETR for detection
Lane C — Wow / future omni demo
Qwen3-Omni-30B-A3B for audio/image/video/text/speech unified experience, benchmarked with HF credits, optional not required
```
### 13.4 Updated benchmark tasks for future agents
Create a repo-local benchmark spec before model promotion:
| Task | Inputs | Required output | Metrics |
|---|---|---|---|
| Receipt line extraction | receipt photos | merchant/date/items/qty/unit/line price/total | field F1, human correction count, latency |
| Packet label extraction | packet label closeups | brand/item/MRP/expiry/quantity/storage | exact date/price accuracy, confidence calibration |
| Hinglish voice command | short audio phrases | transcript + intent + tool JSON | WER, intent accuracy, JSON validity |
| Market photo scan | shelf/product photo | candidate item + price/freshness uncertainty | top-1/top-3 accuracy, false certainty rate |
| Alias matching | text/receipt/list variants | canonical item ID | accuracy, ambiguous deferral rate |
| Multimodal retrieval | text query over images/traces | relevant receipt/shelf/trace | recall@k, rerank quality |
| Short TTS answer | generated household answers | natural speech | latency, intelligibility, Hindi/Hinglish pronunciation |
| Omni demo | photo/audio/video prompt | useful answer + optional speech | latency, trustworthiness, failure modes |
### 13.5 Immediate doc follow-ups
Future agents should update these artifacts after each real benchmark:
- `Docs/MODEL_CATALOG.md`: add the fresh model, params, license, runtime, status, benchmark notes.
- `shopstack/model_registry.py`: add only candidates that passed basic source/license checks; mark active only after runtime verification.
- `Docs/ShopStack_Exploration_Map.md`: add findings, rejected models, and changed priorities.
- `tests/`: add fixtures for model-independent parsing/validation paths so model swaps do not break product contracts.
### 13.6 Bottom line
The previous shortlist was useful but incomplete. The biggest newer additions are:
1. **Qwen3-Omni-30B-A3B** as the ambitious single-model omni experiment.
2. **Voxtral Mini 4B Realtime** as a fresh ASR candidate.
3. **Qwen3-TTS / CosyVoice3 / vLLM-Omni speech stack** as a newer TTS evaluation lane.
4. **MinerU2.5, LightOnOCR, POINTS-Reader, SmolDocling, RolmOCR, Typhoon-OCR** as added OCR benchmark candidates.
5. **Qwen3-VL Embedding + Sentence Transformers multimodal v5.4** as the right direction for multimodal household memory.
6. **HF Jobs/Spaces with Pro credits** as the right way to benchmark without making cloud a product dependency.
_Last updated: 2026-06-07_
---
## 14. Living Model Research Intake + Leaderboard-Driven Discovery Protocol (2026-06-07)
### Why this section exists
Pranay pushed back that the model research was not thorough enough. That pushback is correct: generic “best model” lists are too shallow for ShopStack. The project needs a repeatable intake process that can absorb user-provided model names, watch new Hugging Face launches, compare leaderboards by modality and size, and still stay grounded in the product’s actual tasks.
This is a **living research queue**, not an implementation instruction. Future agents should document new findings here first, then only promote models into `Docs/MODEL_CATALOG.md` / `shopstack/model_registry.py` after license and runtime verification.
### Hard constraints for model candidates
- Prefer **open-source / open-weight / locally runnable** models.
- Prefer models with **≤32B total loaded parameters**. MoE models with ≤32B active params but much larger total params may be listed, but must be marked as **active-param-only fit** and treated as eval/API/large-rig candidates unless a local hardware plan exists.
- Keep the **Off-the-Grid** promise: HF Pro/credits may be used for research, benchmarking, fine-tuning, and batch evals, but must not become a hidden runtime dependency for core behavior.
- Rank by **ShopStack usefulness**, not generic benchmark prestige.
- Every candidate must eventually be tested on grocery/household data, not just public benchmark samples.
### Intake template for names Pranay provides
When Pranay gives a model/library name, append a row here or in a dated subtable using this schema:
| Field | Required note |
|---|---|
| Name / link | Exact HF repo, GitHub repo, paper, or package name. |
| Modality | Text, OCR/doc, VLM, ASR, TTS, embedding/rerank, detection, segmentation, time-series, agent/tooling. |
| Size | Total params, active params if MoE, expected quantized RAM/VRAM. |
| License | Exact license and commercial risk. Do not infer. |
| Runtime | `transformers`, `llama.cpp`, `mlx`, `vLLM`, `SGLang`, ONNX, WebGPU, library API, etc. |
| ShopStack fit | Receipt OCR, packet label OCR, pantry photo, Hinglish voice, TTS answers, planner/tool calls, multimodal memory, price intelligence, etc. |
| Evidence source | Official model card/paper, leaderboard, GitHub, independent eval, or user-provided hunch. |
| Verdict | `evaluate now`, `watch`, `reject`, `eval-only`, or `blocked on license/runtime`. |
| Benchmark task | The first concrete ShopStack eval this model should run. |
### Leaderboard/source watchlist by capability
Use these sources first before relying on blog roundups:
| Capability | Primary sources to check | What to extract | Caveat |
|---|---|---|---|
| General small LLM / reasoning | Hugging Face Open LLM Leaderboard org/results, Artificial Analysis with open-weights filter, Vellum Open Source LLM Leaderboard, Onyx self-hosted/open-source leaderboards | Params, open-weight status, reasoning/coding scores, context, latency/cost where available | General reasoning scores do not prove tool-call reliability or JSON correctness. |
| Small local / edge LLMs | HF model cards, small-model leaderboards, llama.cpp/MLX/Ollama community benchmarks | Quantized memory, speed, instruction following, multimodal support | Many “under 10B” lists mix official, vendor-reported, and community scores. Verify model cards. |
| Tool calling / structured output | Berkeley Function Calling Leaderboard, Gorilla/OpenFunctions/xLAM/Hammer reports, independent local LM Studio/Open WebUI tests | Tool selection, argument validity, multi-tool support, format compatibility | Chat-template/runtime mismatch can make a leaderboard winner fail locally. Test with ShopStack’s actual server. |
| VLM / multimodal | OpenCompass Open VLM Leaderboard, VLMEvalKit, HF OpenVLM records, model cards for Qwen/Gemma/MiniCPM/SmolVLM/Aya/Phi | OCR, chart/doc QA, visual reasoning, GUI grounding, video/image support | VLM leaderboard rank is not the same as receipt OCR or packet label extraction. |
| OCR / document AI | OCRBench-style reports, OmniDocBench, DocVQA/InfoVQA/ChartQA benchmarks, HF OCR guides, Docling/MinerU/PaddleOCR docs | Field extraction, layout retention, multilingual OCR, structured Markdown/JSON support | Product needs receipts and labels; academic docs may not transfer. |
| ASR / speech input | HF Open ASR Leaderboard, ASR Leaderboard paper/code, model cards for Canary/Qwen, Parakeet, Whisper, SenseVoice, Voxtral | WER, real-time factor, multilingual support, streaming support | Hindi/Hinglish and noisy market audio must be tested separately. |
| TTS / speech output | TTS Arena, HF trending TTS, model cards for Kokoro/Piper/Qwen3-TTS/CosyVoice/Chatterbox | Naturalness, latency, voice cloning, language/pronunciation, license | Subjective voice quality needs human listening tests. |
| Embeddings / rerankers | MTEB leaderboard, Sentence Transformers docs/blogs, BGE/Jina/Qwen embedding model cards | Retrieval score, multilingual score, multimodal support, rerank ability | Overall MTEB score hides retrieval-specific performance. Test household memory queries. |
| Detection/segmentation | COCO/LVIS/open-vocabulary detection reports, Ultralytics/RF-DETR/GroundingDINO/SAM docs | Object detection, grounding, segmentation, runtime | Model may detect generic objects but not branded groceries. |
| Time series / forecasting | Nixtla/statsforecast docs, sktime, NeuralForecast, Chronos/TinyTimeMixers model cards | Forecast accuracy, local CPU viability, probabilistic outputs | Use deterministic statistical baselines before neural forecasting. |
### Current leaderboard-driven candidate intake
This table captures newer candidates and source directions surfaced by the broader leaderboard scan. It is deliberately conservative: **candidate ≠ approved runtime**.
| Candidate | Modality | Size / fit | Evidence/source direction | ShopStack fit | Verdict / first eval |
|---|---|---|---|---|---|
| `Qwen3.5-4B` / `Qwen3.5-9B` small series | Planner, tool calling, possible multimodal depending exact repo | ≤10B | Small-model rankings and independent local tool-calling tests report very strong structured/tool behavior for 4B-class Qwen | Local planner, command parser, JSON/tool calls, cheap local assistant | **Evaluate now**: tool JSON harness + shopping command parser. Verify exact HF repos/licensing. |
| `NVIDIA Nemotron Nano 4B` / `Nemotron Nano 9B/12B/VL` | Tool calling, general LLM, VLM variants | ≤12B | Artificial Analysis and local tool-calling evals surface strong speed/structured behavior for Nano class | Local planner fallback, possible VLM route if VL variant is open/licensed | **Evaluate** after license/runtime check: tool call schema + receipt/photo QA. |
| `GPT-OSS-20B` | General/reasoning/tooling | 20B | Vellum/Artificial Analysis list strong open-weight cost/speed; local tool eval shows decent but not best tool calls | Larger local brain for planning/evals, not first core model | **Watch/evaluate**: compare against Qwen 4B/9B on JSON and latency. Confirm license/runtime. |
| `Gemma 3 4B / 12B / 27B` and `Gemma 3n E4B` | Text + vision, edge/local | ≤27B total; 3n effective smaller | Small-model and open-source leaderboards consistently cite strong local multimodal and edge viability | Pantry/shelf photo QA, on-device assistant, fallback VLM | **Evaluate now** for vision + local memory; test receipts but do not assume OCR excellence. |
| `Phi-4-mini` / `Phi-4 multimodal` / `Phi-4 reasoning-vision` | Small reasoning, multimodal | ~3.8B to ~15B | Small-model rankings cite math/reasoning; multimodal variants appear useful for GUI/vision | Lightweight reasoning, possible product-photo understanding | **Watch/evaluate**: tool-call reliability looked weaker in one local test; verify current variants. |
| `Mistral Nemo 12B` | Planner/tooling | 12B | Independent local tool-call eval ranked it highly under LM Studio | Structured local planner alternative | **Evaluate** if Qwen fails or license/runtime is simpler. |
| `Mistral Small 3.x / 24B` | Planner/general | 24B | Self-hosted leaderboards list it, but local tool-call tests may underperform via incompatible templates | Possible high-quality planner if native runtime works | **Blocked on runtime-template validation**. Do not assume BFCL/official score transfers. |
| `xLAM-2 8B` / Salesforce function-calling models | Tool calling | ~8B | BFCL-oriented function calling family; independent local tests warn about template incompatibility | Specialist local tool caller | **Evaluate with native template**, not generic OpenAI-compatible server only. |
| `Hammer 2.x 7B` | Tool calling | ~7B | Function-calling specialist with benchmark claims; local runtime mismatch risk | Specialist command-to-action parser | **Evaluate only after chat-template support confirmed**. |
| `Qwen3-VL-Embedding-2B` | Multimodal embedding/rerank | 2B | HF Sentence Transformers v5.4 multimodal docs/blog explicitly support multimodal embeddings | Multimodal household memory: retrieve receipt images, shelf photos, voice traces | **Evaluate now**: recall@k on text query → receipt/photo/trace. |
| Sentence Transformers v5.4 multimodal stack | Library/runtime | model-dependent | HF blog adds text/image/audio/video embedding and reranker support | One API for multimodal memory experiments | **Adopt in benchmark tooling** before app runtime. |
| `NVIDIA Canary-Qwen 2.5B` | ASR | 2.5B | HF Open ASR Leaderboard / ASR paper sources cite strong open ASR accuracy | Voice receipt notes, spoken shopping commands | **Evaluate now**: Hinglish/noisy command WER + latency. |
| `NVIDIA Parakeet TDT/CTC 0.6B–1.1B` | ASR | ≤1.1B | ASR leaderboard cites very high real-time factor for fast transcription | Realtime local voice if language coverage fits | **Evaluate speed lane**: English/Hinglish WER and streaming. |
| `IBM Granite Speech 3.3 8B` | ASR/audio LLM | ~8B/9B class | ASR leaderboard/blog sources cite strong WER and Apache license claims; verify exact card | Accurate voice command transcription | **Watch/evaluate** after license/runtime check. |
| `Voxtral Mini 4B Realtime` | ASR/audio | 4B | Mistral/HF realtime speech model candidate | Low-latency voice loop | **Evaluate**: latency + Hindi/Hinglish support; compare to Canary/Parakeet/SenseVoice. |
| `Chatterbox` | TTS | ~0.5B class | TTS roundups/trending cite strong small voice cloning/naturalness | Warm local household voice, optional personalized voice | **Evaluate**: license, voice quality, latency, pronunciation. |
| `Kokoro-82M` | TTS | 82M | Still strong tiny baseline | Default local TTS baseline | **Keep baseline**; benchmark against Qwen3-TTS/CosyVoice/Chatterbox. |
| `Qwen3-TTS 0.6B/1.7B` | TTS | ≤1.7B | vLLM-Omni supported Qwen TTS family | Higher-quality local speech output | **Evaluate**: runtime complexity + Hinglish pronunciation. |
| `PaddleOCR-VL` | OCR/doc | ~0.9B | Prior section already flagged; still top first OCR candidate | Receipt/label structured extraction | **Evaluate first** on grocery dataset. |
| `MinerU2.5-1.2B` / `LightOnOCR-1B` | OCR/doc | ~1B | HF OCR guide/community-cited newer compact OCR candidates | Receipt/label extraction, document parsing | **Evaluate now** in OCR bakeoff. |
| `SmolDocling` / `Granite-Docling` | OCR/doc | 256M-ish | Tiny document understanding candidates | CPU fallback, quick document parse | **Evaluate as fallback**, not likely top accuracy. |
| `Qwen3-Omni-30B-A3B` | Omni multimodal | 30B total / ~3B active | Fresh HF model cards; omni audio/image/video/text/speech | “One model does a lot” wow demo and benchmark oracle | **Eval-only/wow lane**: total loaded params nearly consumes cap; not reliable first core. |
| `Qwen3-VL 235B-A22B` | VLM/OCR/GUI | 235B total / 22B active | Open-source VLM leaderboards and provider writeups cite strong OCR/GUI | Reference oracle for vision/OCR quality | **Active-param-only / cloud eval**, not local Off-the-Grid default due total size. |
| `MiniMax-M2.5` | Agentic LLM | 229B total / 10B active | Open-source/self-hosted leaderboards cite strong coding/function-calling | Reference for complex planning | **API/eval-only** unless hardware plan exists. Total params exceed local cap. |
| `Kimi K2 Thinking/K2.5` | Agentic LLM | ~1T total / 32B active | Open-source leaderboards place it high for reasoning/agentic coding | High-end external judge/planner reference | **Out of local cap by total size**; can be benchmark reference with credits/API only. |
### ShopStack-specific model eval suite to create before promotion
A future implementation agent should create this as a benchmark harness before adding active runtime models:
| Eval | Dataset size | Pass criteria | Why it matters |
|---|---:|---|---|
| Receipt OCR bakeoff | 20–50 real/synthetic receipts | item/date/merchant/total field F1, correction count, latency | Core automation must not corrupt inventory/price history. |
| Packet label extraction | 20–50 packet labels | MRP/expiry/brand/size exactness, ambiguity detection | Prevent stale/expired item and wrong unit bugs. |
| Hinglish voice command | 50–100 short clips | WER + intent accuracy + valid JSON tool calls | Voice UX depends on this. |
| Tool-call schema harness | 40–100 prompts | exact tool selection, required args, no unauthorized action | Planner safety and trace correctness. |
| Multimodal memory retrieval | 30–100 image/text/audio trace entries | recall@5 and rerank quality | Enables “where did we buy this?” and “show last receipt for atta.” |
| Shelf/fridge image understanding | 30–50 photos | top-k item detection/caption confidence, false-certainty rate | Supports pantry/photo features without hallucinated writes. |
| TTS listening test | 20 generated responses | intelligibility, warmth, pronunciation, latency | Voice output should feel useful, not gimmicky. |
| Local runtime budget | all promoted models | RAM/VRAM, cold start, tokens/sec, fallback behavior | Keeps Off-the-Grid credible. |
### Research quality rules for future passes
- Do not cite a model as “best” without saying **best for what benchmark/task**.
- Always separate **official/vendor-reported** metrics from **third-party** and **our own** metrics.
- For MoE models, record both **total params** and **active params**. The project cap should default to total loaded params unless explicitly overridden for a demo/eval lane.
- A model with strong BFCL/function-calling score still needs local runtime validation because chat templates often break OpenAI-compatible serving.
- A VLM with strong MMMU/MMBench score still needs receipt/label OCR testing.
- An ASR model with low English WER still needs Hinglish and noisy-market audio tests.
- A TTS model with high community preference still needs licensing and pronunciation checks.
- Prefer deterministic libraries for core arithmetic, matching, time-series, and validation; models should propose, classify, transcribe, extract, or explain — not silently mutate state.
### Immediate next research actions
1. As Pranay provides model names, append them to the intake table with exact links and verdicts.
2. Pull exact HF model cards/licenses for the top candidates above, starting with Qwen3.5 small, Nemotron Nano, Canary-Qwen, Granite Speech, Chatterbox, Qwen3-VL-Embedding, and the OCR candidates.
3. Build the benchmark fixture list before any implementation agent wires models into runtime.
4. Update `Docs/MODEL_CATALOG.md` only after exact model card/license/runtime evidence is collected.
5. Keep a rejected/blocked list; avoiding bad models is as useful as finding good ones.
_Last updated: 2026-06-07_
---
## Design System & UI Architecture (2026-07-12)
### Context
Full design audit completed (see `Docs/audits/audit_05_full_design_audit.md`). UI surface has solid foundation but needs component library expansion, design token completion, and architectural cleanup.
### Completed
- **Split `other.py`** → `price_memory.py`, `household_map.py`, `field_notes.py`, `swiggy_market.py`
- **Moved `DEMO_SEED_INVENTORY`** → `shopstack/data/seed_demo.py`
- **CSS tokens v2**: spacing, font-size, shadow, z-index, animation scales
- **Unified `DECISION_COLORS`** with CSS variables, created `theme_tokens.py`
- **Fixed Audit 3**: removed 3 wrapper fns, private imports, extracted `_runtime_label`
- **P1 components**: `item_row`, `stat_card`, `data_table`, `confirm_dialog`, `toast`, `loading_skeleton`, `empty_state_enhanced`
### Open
- Phase 2/3 components, dark mode, keyboard shortcuts, a11y audit
- Remaining hardcoded color migration in swiggy_market, image_cards, market_lens, shopping
- Component adoption: migrate existing screens to use new components
_Last updated: 2026-07-12_
---
## Architecture Consolidation & Stash Recovery (2026-06-08)
### Context
An intentional architectural branch (stashed as WIP) performed three major
transformations: screen module extraction, design system v2, and 5-tab UX
restructure. Full architecture documented in `Docs/STASH_ARCHITECTURE_RECOVERY.md`.
### Completed Architecture Changes
- **5-tab nested UX**: 13 flat tabs → 5 nested tabs (Today, Basket, Market Lens,
Reconcile, Memory). Groups related product surfaces.
- **other.py** → 51L re-export shim. Functions extracted to canonical modules:
`price_memory.py`, `household_map.py`, `field_notes.py`, `swiggy_market.py`.
- **Design system v2**: 728-line token-based theme with Emil Kowalski animations,
WCAG AA focus-visible, reduced-motion, responsive breakpoints.
- **Lazy provider loading**: ProviderRegistry defers heavy imports (funasr, torch,
kokoro) to first `get()` call. `import app` dropped from 90s to 2.3s.
- **Local model defaults**: STT=mlx-whisper, TTS=kokoro-82M, planner=MLX/llama.cpp.
All real models by default, mock only for off-trunk providers (vision/OCR).
- **Cost tracking + tracing**: OpenTelemetry trace_call() spans in local and
OpenAI providers with model tier estimation.
- **semantic_find_item**: Tiered search (exact → prefix → semantic embedding)
registered as canonical tool with ToolSpec.
- **Decision cards**: SVG-based visual cards in dashboard for buy/skip/use-soon.
### New Capabilities Added (Aug 2026-06-08)
- **Receipt scanning**: OCR → purchase events pipeline with `services/receipt.py`
- **Multi-retailer adapters**: Blinkit, Zepto, DMart source adapters with
cross-source price comparison via `market/sources/_comparison.py`
- **Semantic inventory search**: BGE-M3 embedding provider with tiered fallback
to prefix search
- **Dashboard intelligence**: Cadence detection, waste warnings, weather context
all wired into DashboardState via service layer
- **Nutrition tracking**: 30-item reference database + service + UI tab
- **Weather/trip context**: Open-meteo API integration with caching and offline
mock fallback
- **Demo walkthrough**: `scripts/demo_walkthrough.py` — idempotent, seeds inventory,
prices, shopping list
- **Performance benchmarks**: 30 pytest-benchmark tests with results in
`benchmarks/RESULTS.md`
### Test Health
619 passed, 0 failed, zero regressions from 597 baseline
### Decision Records (DR-005 through DR-012)
- DR-005: Screen module extraction — split other.py into canonical modules
- DR-006: Design system v2 — token-based CSS replacing flat hardcoded values
- DR-007: 5-tab nested UX — 13 flat tabs collapsed to 5 primary tabs
- DR-008: Lazy provider loading — deferred heavy imports for fast app init
- DR-009: Local model defaults — real models by default, optical upgrades
- DR-010: SVG decision cards — visual card system with PNG fallback via Flux
- DR-011: semantic_find_item — tiered search registered as canonical tool
- DR-012: Cost tracking + tracing — OpenTelemetry spans in LLM providers
### Exploration Areas Opened
- Dark mode theme variant using design tokens
- Keyboard shortcut system for power users
- Accessibility audit against WCAG 2.1 AA
- Component migration: adopt primitives (ItemRow, Toast, etc.) across screens
- Model download UX: progress indicators for initial local model downloads
- Offline-first backup/restore: encrypted household snapshots
- Mobile capture path: PWA wrapper for Gradio with local-first guarantees
---
## 14. Expansion Sweep — Cross-Project Inspired Areas (2026-06-08)
This section expands the open exploration surface using patterns from sibling
Projects docs such as `family-dinner-os` exploration audits, `invoice-intelligence`
autoresearch/evaluation docs, `idea_pad` domain census, and local ShopStack docs.
All items remain documentation-only until promoted into `ROADMAP.md` or decision
records.
### 14.1 Household Operating-System Verticals
ShopStack can stay anchored in shopping while growing into adjacent household
systems:
- **Family meal operating system** — meal debates, preferences, leftovers,
nutrition, budget, and available stock converge into one dinner decision.
- **Caregiver / elder household mode** — low-vision voice workflows, medicine-like
caution labels, easy “what is at home?” answers, and remote family check-ins.
- **Shared flat / hostel mode** — roommate ownership, split costs, communal vs
personal shelves, expiry responsibility, and “who used the milk?” event trails.
- **Domestic helper / staff workflow** — voice-first Hindi/Hinglish commands,
confirmation cards, shopping handoff lists, and audit-friendly purchase logs.
- **Baby / pet / dietary inventory** — formula, diapers, pet food, allergy-safe
items, repeat cadence, and “never run out” thresholds.
- **Emergency prep / monsoon stock** — critical household reserves, power-cut
foods, water, medicines, flashlights, and weather-aware restock prompts.
- **Festival / event planning** — Diwali, Eid, weddings, birthdays, guests, bulk
shopping, perishability, and post-event leftover/waste analysis.
- **Home ops ledger** — groceries plus household consumables, repairs, cleaning,
subscriptions, appliance filters, warranty docs, and recurring errands.
Exploration questions:
- Which vertical keeps the strongest wedge while preserving “shopping memory” as
the core mental model?
- Which mode can be demoed with real household evidence in under 3 minutes?
- What user roles need separate permissions, not just separate labels?
- Which verticals create sensitive claims requiring stronger disclaimers?
### 14.2 Retail, Kirana, and Community Commerce Angles
ShopStack can horizontally extend from one household into local commerce memory:
- **Kirana companion** — compare home inventory against nearby kirana price/quality
memory, not only quick-commerce snapshots.
- **Neighborhood price commons** — privacy-preserving aggregate prices for staples,
normalized by unit, quality, and freshness.
- **Store quality memory** — subjective freshness, staff helpfulness, return issues,
packaging quality, substitution reliability, and delivery timing.
- **Local seller mini-CRM** — seller notes, WhatsApp order templates, credit ledger,
preferred brands, and recurring household baskets.
- **Market route planner** — “buy vegetables here, detergent there” based on
travel time, weather, price variance, and perishability.
- **Community-supported buying** — bulk rice/oil/dal buys across apartments,
savings split, expiry/storage planning, and consented group orders.
- **Retailer API / scraping ethics map** — what data is official, user-provided,
manually entered, public, or not safe to automate.
Exploration questions:
- What is the trust boundary between personal memory and community data?
- How do we prevent stale community prices from misleading users?
- Can quality/freshness be represented without pretending to be objective?
- Which integrations are product-critical vs demo theater?
### 14.3 Autoresearch and Evaluation Harness
Borrow the invoice-extraction pattern: fixed benchmark, fixed evaluator, one
candidate change, keep/discard logging.
Candidate benchmark tracks:
- **Command parser benchmark** — Hinglish and English shopping commands → JSON tool
calls, with exact intent, item, quantity, unit, location, and confidence scoring.
- **Receipt extraction benchmark** — receipt photos/text → purchase events with
item, quantity, price, store, date, total, tax/fee ambiguity, and review flags.
- **Packet label OCR benchmark** — expiry/MRP/nutrition extraction with false-positive
penalty for confusing MRP, sale price, manufacturing date, and best-before date.
- **Market Lens benchmark** — shelf/photo/barcode inputs → identify item, compare
with inventory, classify buy/skip/compare, and explain uncertainty.
- **Semantic alias benchmark** — pav/bread, dahi/curd/yogurt, Surf/detergent,
dhaniya/coriander, etc. → canonical item IDs.
- **Voice benchmark** — noisy-market audio → preserved intent and slot accuracy.
- **Decision benchmark** — inventory + market price + cadence + weather → expected
buy/skip/use-soon/compare/watch classification.
Immutable surfaces for each run:
- benchmark manifest,
- ground-truth labels,
- scoring formula,
- dataset split,
- validation schema,
- evaluator code.
Mutable candidate surfaces:
- prompt text,
- model/provider,
- parser/OCR/vision routing,
- validation thresholds,
- retry/fallback policy,
- normalization maps,
- confidence language templates.
Artifacts to store in-project:
- `experiments/results.tsv`,
- per-run `summary.json`,
- `error_analysis.json`,
- candidate prompt/config snapshots,
- keep/discard rationale,
- hard examples promoted to fixtures.
### 14.4 Data and Schema Expansion
Potential canonical data layers:
- household item taxonomy with Indian/local variants,
- alias and brand synonym map,
- unit normalization and pack-size grammar,
- storage-location ontology,
- shelf-life and freshness rules by category,
- price-quality observation schema,
- receipt line item schema,
- packet label schema,
- nutrition/allergen schema,
- household preference schema,
- user role/permission schema,
- consent and anonymization schema for traces,
- retailer source provenance schema,
- model-run provenance schema,
- uncertainty/explanation schema.
High-leverage datasets to publish or keep internal:
- 500–2,000 human-reviewed grocery commands,
- 100 receipt examples with synthetic/redacted variants,
- 300 packet label close-ups / synthetic crops,
- 1,000 alias-to-canonical mappings,
- 100 household shelf/fridge snapshots with zones,
- 100 market photo examples with review cards,
- 50 noisy voice snippets for benchmark phrases,
- real/synthetic price observations across Swiggy, Blinkit, Zepto, DMart, kirana,
- anonymized trace set showing tool-call planning and corrections.
Exploration questions:
- Which maps belong in SQLite tables vs versioned YAML/JSON?
- Which maps need user-editable UI because every household differs?
- Which data can be safely public for hackathon/storytelling?
- What validation catches stale or contradictory data before it reaches decisions?
### 14.5 UX, Interface, and Interaction Surfaces
Open product surfaces:
- **Today as command center** — one screen for buy, skip, use soon, compare, confirm,
and “why this recommendation?”
- **Review queue** — uncertain OCR/vision/voice events become cards to approve,
merge, split, edit, or discard.
- **Cinematic household timeline** — family-dinner-style event playback showing how
ShopStack reasoned from inventory, price memory, weather, and preferences.
- **Shelf map editor** — user-defined locations, shelf zones, item cards, last-seen
photos, and confidence badges.
- **Market mode** — large buttons, short voice responses, offline-safe fallback,
high contrast, and no dense tables.
- **Kitchen mode** — hands-free voice, recipe/use-soon prompts, timer-adjacent flow,
and “what can I cook now?”
- **Family summary** — shareable daily/weekly household digest: what to use, buy,
avoid, and what was wasted/saved.
- **Operator/debug view** — model used, fallback path, confidence, source data age,
and trace export without leaking private content.
- **Onboarding wizard** — household type, preferred stores, staple items, language,
dietary constraints, budget, and reminder preferences.
Design exploration topics:
- warm household visual language vs utilitarian inventory tool,
- mobile-first capture vs desktop-first Gradio demo,
- confidence language and iconography,
- accessibility for low vision / elder users,
- no-shame waste messaging,
- deterministic factual cards vs generated visuals,
- offline/connected mode indicators,
- empty states that teach the mental model.
### 14.6 Agent, Model, and Pipeline Architecture
Potential specialized agents/pipelines:
- **Inventory Reconciler** — merges receipt, voice, photo, and manual updates into
lots without double-counting.
- **Market Analyst** — compares unit prices, source freshness, travel cost, and
quality memory.
- **Waste Coach** — detects overbuy patterns and suggests practical use-soon actions.
- **Meal Planner** — proposes meals from current stock, preferences, and expiry.
- **Data Steward** — flags duplicates, stale prices, bad aliases, and low-confidence
observations for review.
- **Privacy Guard** — redacts traces, receipts, addresses, phone numbers, faces, and
household member names before export.
- **Benchmark Runner** — executes candidate prompts/models against fixed datasets
and writes keep/discard results.
- **Source Adapter Auditor** — validates retailer adapters for freshness, schema
drift, and unit-price correctness.
Pipeline questions:
- Where should rules beat models outright?
- Which model outputs must never mutate inventory without confirmation?
- How should fallback paths be visible to the user and trace logs?
- Can one canonical event ledger drive inventory, prices, traces, and explanations?
- Should ShopStack adopt event sourcing for all household changes?
### 14.7 Privacy, Safety, Trust, and Governance
Trust surfaces to design explicitly:
- local-first mode and clear cloud-mode disclosure,
- household data export/delete,
- encrypted backup snapshots,
- consent per household member,
- child/elder-safe voice interactions,
- face/address/phone redaction in images and receipts,
- private vs shareable trace boundaries,
- nutrition/allergen caution language,
- medicine-like item handling without medical claims,
- price freshness and source disclaimers,
- no auto-purchase without explicit confirmation,
- audit trail for who added/consumed/moved items,
- conflict handling when multiple users edit the same household.
Open questions:
- What is the minimum privacy story credible for real household photos?
- How do we make anonymized hackathon traces useful but safe?
- Which features should be disabled in public demo mode?
- What should be impossible for the agent to do automatically?
### 14.8 Business, Distribution, and Packaging Exploration
Potential packages:
- free local household inventory app,
- paid family sync / encrypted backup,
- premium market intelligence and retailer comparison,
- caregiver/elder household plan,
- apartment/community price commons,
- kirana/seller companion,
- meal-planning add-on,
- privacy-first offline appliance / NAS-style version,
- API/data layer for other home apps.
Distribution channels:
- Hugging Face Space demo,
- Product Hunt / X build thread,
- WhatsApp-family proof video,
- Indian home-cooking creators,
- Reddit/Discord frugal and meal-prep communities,
- apartment society pilots,
- local kirana pilot,
- caregiver and elder-tech communities,
- student hostel beta.
Positioning experiments:
- “Your home’s shopping memory.”
- “Know what you have before you buy again.”
- “The anti-overbuy grocery assistant.”
- “A local-first inventory copilot for Indian homes.”
- “From receipt to fridge to next shopping decision.”
- “A memory layer for household commerce.”
### 14.9 Platform and Integration Backlog
Potential integrations:
- WhatsApp list import/export,
- email receipt ingestion,
- browser extension for online grocery carts,
- mobile PWA capture,
- barcode/QR product databases,
- calendar/reminders,
- weather APIs,
- maps/travel-time APIs,
- smartwatch/voice shortcut,
- printer/label maker,
- CSV/JSON/SQLite backup,
- Home Assistant / smart fridge / camera feeds,
- supermarket loyalty emails where user-consented,
- nutrition databases,
- recipe APIs or local recipe notebooks.
Integration rules to define before building:
- canonical source of truth,
- data freshness,
- user consent,
- failure/fallback behavior,
- duplicate detection,
- manual correction path,
- export portability,
- test fixture strategy.
### 14.10 Repo, Documentation, and Quality Exploration
Process surfaces worth improving:
- keep `FEATURE_MAP.md`, `ROADMAP.md`, decision records, and this exploration map
synchronized without rewriting history,
- add exploration area IDs so roadmap items can cite their origin,
- create an `experiments/README.md` describing benchmark promotion rules,
- create a data catalog for all JSON/CSV/YAML datasets,
- track “built / partially built / only explored” status per idea,
- maintain screenshot/visual evidence under `Docs/review/assets/`, not root,
- add docs for provider fallback behavior and model parameter accounting,
- add a privacy review checklist for trace/dataset publication,
- add a demo-script checklist for hackathon video reproducibility,
- create hard-example fixtures whenever a model or OCR failure is found.
### 14.11 New Open Questions
21. Should ShopStack become an event-sourced household ledger, or stay CRUD-first
with derived event logs?
22. Which single vertical expansion is the strongest: caregiver, shared flat,
kirana/community, meal planning, or festival planning?
23. What is the smallest fixed benchmark that prevents model/prompt regressions?
24. Which decisions require hard rule gates instead of model judgment?
25. How should household member permissions work without making onboarding heavy?
26. Can price memory combine subjective quality and objective unit price cleanly?
27. What public demo data feels real without exposing private household details?
28. Should receipt ingestion be the primary acquisition wedge because it creates
instant inventory and price memory?
29. What is the right event model for correction: amend, reverse, merge, split, or
overwrite?
30. Which integrations are worth building first because they compound the memory
layer rather than distract from it?
---
## 15. HF Pipeline Model Sweep (2026-06-09)
Created a dedicated pipeline/model exploration note at
`Docs/exploration/HF_PIPELINE_MODEL_EXPLORATION_2026-06-09.md` after querying
Hugging Face Hub metadata with the local `SHOPSTACK_HF_API_KEY` from `.env`.
The key was used only for read-only discovery and was not printed or stored.
Coverage added:
- speech-to-text / market voice capture,
- text-to-speech / spoken household responses,
- planner, command parser, slot extraction, and reranking,
- OCR, receipt extraction, packet labels, document layout,
- vision, VQA, object detection, grounding, and visual similarity,
- segmentation, cropping, and background removal,
- text/image embeddings and semantic retrieval,
- image generation/editing for non-factual cards,
- privacy/redaction model surfaces,
- candidate active stacks under the 32B parameter cap,
- benchmark promotion plan for fixed evals.
Important exploration decisions:
- Do not choose models by downloads alone; require product-fit benchmarks.
- Keep factual prices, expiry dates, and inventory counts rendered by code, not
generated into image pixels.
- Keep direct inventory mutation behind validated tool calls and confirmation.
- Treat each pipeline stage as separately benchmarked: ASR, parser, OCR, vision,
embeddings, segmentation, and decision reasoning should have different evals.
- Distinguish submitted active models, optional fallbacks, and research-only
candidates in future model docs.
New model families/lanes opened:
- WhisperKit/CoreML and pyannote VAD/diarization for future mobile/voice capture.
- Indic Parler TTS, VibeVoice, OmniVoice, Chatterbox for voice quality and Indian
language support.
- PaddleOCR v5, TrOCR, Donut, LayoutLM, Table Transformer, and PP-DocLayout for
receipts and packet labels.
- BLIP/ViLT/Pix2Struct/DETR/YOLOS/RT-DETR/CLIP/SigLIP for visual understanding and
shelf/product matching.
- RMBG-2.0, BiRefNet, ClipSeg, SegFormer, Mask2Former for item cards and shelf zones.
- Qwen3-Embedding, multilingual-e5, Jina v3, MiniLM, Nomic, and BGE rerankers for
semantic memory and alias search.
- Qwen Image Edit, FLUX.2 variants, Z-Image Turbo, SDXL/SD-turbo for optional visual
assets, with deterministic overlays for factual content.
---
## 16. Travel Agency Agent Pattern Transfer (2026-06-09)
Sampled `travel_agency_agent` and `exploration_maps/travel_agency_agent` for
transferable exploration patterns. The strongest reusable ideas are not travel
features themselves; they are operating-system patterns: lifecycle pipelines,
dual-output safety boundaries, destination/context intelligence, supplier scoring,
governance, continuity, and research-status discipline.
### 16.1 Pipeline Lifecycle: Inquiry → Decision → Strategy
Travel pattern:
```text
Inquiry → Intake Normalization → Decision/Gaps → Strategy/Bundle → Safe Output
```
ShopStack equivalent:
```text
Household signal → Normalize → Gap/Decision → Action Bundle → User-safe Output
```
Candidate ShopStack lifecycle stages:
- **Signal intake** — voice command, receipt photo, packet label, fridge photo,
shopping list edit, manual inventory update, market price observation.
- **Normalization** — canonical item, unit, quantity, store/source, location,
confidence, provenance, timestamp.
- **Gap analysis** — missing quantity, ambiguous item, stale price, duplicate lot,
expiry uncertainty, source freshness, low-confidence OCR/vision.
- **Decision state** — buy, skip, use soon, compare, watch, confirm, reconcile,
split/merge, discard.
- **Action bundle** — inventory mutation proposal, price observation, reminder,
shopping list update, review card, trace event.
- **Safe output** — household-facing explanation that excludes internal-only model
prompts, raw OCR noise, private metadata, and unsupported claims.
Exploration question: should ShopStack formalize a canonical `HouseholdSignal →
DecisionBundle` contract similar to Travel Agency Agent's trip packet pipeline?
### 16.2 Dual-Output Safety Boundary
Travel pattern: internal operator bundle can contain richer decision context, while
traveler-facing output must be sanitized and leakage-checked.
ShopStack transfer:
- **Internal/debug bundle** — raw OCR text, model output, confidence vectors,
fallback path, source age, prompt/run metadata, private notes.
- **Household-safe bundle** — concise explanation, action recommendation, source
caveat, and confirmation affordance.
- **Public/demo-safe bundle** — anonymized screenshots/traces with household names,
addresses, phone numbers, receipts, faces, and payment identifiers removed.
Potential feature:
- Add a `Strict Public Demo Mode` that blocks export/share if unsafe terms or
unredacted receipt fields are detected.
### 16.3 Gap-State and Review-Queue Taxonomy
Travel Agent emphasizes blockers, confidence, next-action state, and operator
workbench review. ShopStack should mirror this for household automation.
Candidate gap states:
- `missing_quantity`
- `ambiguous_item`
- `ambiguous_unit`
- `possible_duplicate_lot`
- `stale_price_source`
- `expiry_date_conflict`
- `ocr_layout_uncertain`
- `vision_low_confidence`
- `needs_location_confirmation`
- `needs_household_member_confirmation`
- `unsafe_to_auto_apply`
- `source_adapter_schema_drift`
Review queue actions:
- approve,
- edit,
- merge with existing lot,
- split into multiple lots,
- mark as price-only observation,
- save as field note,
- discard,
- promote to benchmark fixture.
This would turn model uncertainty into a productive UI surface instead of hidden
backend ambiguity.
### 16.4 Destination Intelligence → Household Context Intelligence
Travel Agent has destination intelligence: weather, AQI, crime, disease,
political stability, crowding, cost of living, local laws, digital infra,
accessibility, emergencies.
ShopStack equivalent context layers:
- **Weather/seasonality** — rain, heat, humidity, monsoon spoilage, festival demand.
- **Neighborhood context** — local market day, store closures, delivery delays,
apartment/community bulk buys.
- **Household calendar** — guests, travel, school/work schedules, fasting days,
festivals, parties.
- **Health/diet context** — allergies, low-salt/diabetic preferences, baby/elder
household needs, without making medical claims.
- **Appliance/storage context** — fridge space, freezer availability, power cuts,
pantry capacity.
- **Mobility/accessibility context** — who is shopping, carrying capacity, elder
user constraints, delivery vs walk/drive tradeoff.
- **Budget/pay-cycle context** — monthly stocking, weekly top-up, sale timing,
price volatility.
Potential score systems:
- `buy_now_score`,
- `skip_risk_score`,
- `waste_risk_score`,
- `travel_worth_it_score`,
- `freshness_risk_score`,
- `stockout_risk_score`,
- `confidence_to_auto_apply`.
### 16.5 Supplier Reliability → Store and Source Reliability
Travel Agent explores supplier ecosystems, dark inventory, vendor reliability,
commission economics, and disruption risk. ShopStack has an analogous store/source
layer.
Store/source reliability dimensions:
- price accuracy,
- delivery punctuality,
- produce freshness,
- substitution quality,
- return/refund friction,
- stockout frequency,
- packaging quality,
- unit-size honesty,
- stale-data risk,
- adapter schema stability,
- household preference fit.
Candidate source types:
- manual household observation,
- receipt-derived observation,
- Swiggy/Blinkit/Zepto/DMart snapshot,
- kirana WhatsApp quote,
- community aggregate,
- loyalty/email receipt,
- public product database,
- synthetic/demo fixture.
Exploration question: should every price/availability recommendation carry a
`source_reliability` and `freshness_timestamp`, not just a numeric price?
### 16.6 Autonomy Gradient and Governance
Travel Agent has explicit autonomy/governance research: when AI acts, when it asks,
and how settings control autonomy.
ShopStack autonomy levels:
1. **Observe only** — save scan/receipt/voice as note, no action.
2. **Suggest** — show recommendation, no mutation.
3. **Draft action** — prefill add/consume/move/list update for approval.
4. **Auto-apply low-risk** — apply only rule-safe actions, e.g. mark a manually
checked shopping item as purchased.
5. **Never automate** — purchases, deletes, public trace export, medical/nutrition
claims, ambiguous inventory reductions.
Governance settings to explore:
- per-household autonomy level,
- per-source trust level,
- per-member permissions,
- confirmation thresholds by action type,
- “strict mode” for public demos,
- audit log visibility.
### 16.7 Operational Continuity and Disruption Handling
Travel Agent includes continuity/resilience thinking for supplier outages,
geopolitical disruption, and operational recovery. ShopStack should do the same for
household data and sources.
Continuity scenarios:
- quick-commerce source unavailable,
- OCR/vision model fails or times out,
- local model missing/download failed,
- stale price snapshot,
- corrupted SQLite/backup restore,
- duplicate household edits from two users,
- offline market mode,
- fridge/pantry photo too poor,
- receipt not readable,
- user disputes an auto-applied action.
Recovery UX:
- visible fallback path,
- “save for later review”,
- manual entry fallback,
- source freshness warning,
- undo/reverse event,
- backup/export prompt,
- hard-example capture for benchmark improvement.
### 16.8 Real-Validation Status Discipline
The Travel Agent exploration map is useful because it marks the gap between deep
theory and real validation. ShopStack should adopt that discipline.
Recommended statuses for exploration/roadmap items:
- `idea_only`
- `researched`
- `prototype_exists`
- `wired_in_app`
- `tested_with_fixtures`
- `tested_with_real_household_data`
- `dogfooded_in_market`
- `public_demo_safe`
- `production_ready`
Add validation metadata:
- evidence path,
- test command,
- data source,
- last verified date,
- known failure modes,
- privacy status,
- benchmark score if applicable.
### 16.9 Content Prism and Free-Tool Strategy
Travel Agent maps every research topic into content, SEO tools, and multiple
audiences. ShopStack can do a household-commerce version.
Three-audience prism:
- **Household user** — practical: save money, avoid waste, know what is at home.
- **Builder/judge** — small-model, local-first, multimodal, benchmarked pipeline.
- **Retail/community partner** — source reliability, price memory, local commerce.
Free-tool ideas:
- grocery unit-price calculator,
- “is this price high?” checker,
- receipt-to-spend summary,
- food waste estimator,
- pantry stockout calculator,
- festival shopping planner,
- rainy-day grocery checklist,
- bulk-buy break-even calculator,
- expiry label parser demo,
- household alias dataset explorer.
### 16.10 New Open Questions from Travel-Agent Transfer
31. Should ShopStack create a canonical `DecisionBundle` object that separates
internal/debug fields from household-safe output?
32. What is the exact autonomy gradient for add, consume, move, delete, export,
and purchase-like actions?
33. Should stale source data automatically downgrade recommendations to “watch” or
“confirm” instead of buy/skip?
34. What score formulas are explainable enough for `travel_worth_it`, `waste_risk`,
and `stockout_risk`?
35. Which review-queue actions are essential for real household correction flows?
36. Can every failed OCR/vision/parser case become a benchmark fixture with one
click?
37. What is the strict public-demo leakage guard for receipts, traces, images, and
voice?
38. Which household context layers matter first: weather, calendar, budget,
storage capacity, or mobility?
39. Should store/source reliability become a first-class table alongside price
observations?
40. How do we track exploration maturity without making docs heavy to maintain?
---
## 17. ShopStack Deep Exploration Status Map — Travel-Agent-Style Inventory (2026-06-09)
This is the deeper exploration structure that should have been added first: a
status-and-gap map like `travel_agency_agent/EXPLORATION_MAP.md`, but for
ShopStack. It is intentionally broad and detailed. It distinguishes what exists,
what is only theorized, what is missing, and what deserves priority.
### 17.0 Status Legend
| Symbol | Meaning |
|---|---|
| ✅ | Built/researched with concrete docs, code, tests, or verified evidence |
| 🟡 | Partially explored or partially built; needs validation or completion |
| ❌ | Identified gap; little/no durable research or implementation |
| 🔵 | New angle opened by this exploration pass |
| 🧪 | Needs benchmark/dogfood evidence before promotion |
Priority scale:
- **P0** — existential for credible demo/product wedge.
- **P1** — critical for launch-quality product.
- **P2** — strategic expansion / moat.
- **P3** — moonshot or later-stage platform option.
---
### A. Household Customer & User Research
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| A1. Real household interviews | ❌ | No structured interviews with Indian households, parents, students, shared flats, helpers, elder/caregiver users | **P0** |
| A2. Day-in-the-life shopping shadowing | ❌ | No observed shopping trips from list-making through purchase/reconcile/use-soon | **P0** |
| A3. Domestic helper workflow research | 🔵 | Helper-led shopping/inventory is a distinct user role with language/trust constraints | P1 |
| A4. Shared-flat / hostel behavior | 🔵 | Ownership, split-cost, “who used it,” and low-friction correction patterns unknown | P1 |
| A5. Elder/caregiver household research | 🔵 | Accessibility, low-vision, medicine-adjacent safety, and remote check-in needs unvalidated | P1 |
| A6. Willingness-to-pay | ❌ | No pricing validation: free local tool vs family sync vs market intelligence vs caregiver plan | P1 |
| A7. Non-consumption research | 🔵 | Why do households not use inventory apps? Friction, shame, setup burden, habit failure | **P0** |
| A8. Cultural/language phrase research | 🟡 | Hinglish examples exist, but no corpus of real household commands | **P0** |
### B. Core Product Wedge & Feature Validation
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| B1. Wedge product identification | 🟡 | Many features exist; not validated which one makes users come back daily | **P0** |
| B2. Receipt-to-inventory wedge | 🟡 | Pipeline discussed/built partially; needs real receipt dogfood and review UX | **P0** |
| B3. “Do I need this?” Market Lens wedge | 🟡 | Strong concept; needs real market photo/audio tests | **P0** |
| B4. Use-soon / waste prevention wedge | 🟡 | Logic exists; needs household behavior validation and no-shame UX | P1 |
| B5. Price memory wedge | 🟡 | Swiggy/source adapters exist; real user perceived value untested | P1 |
| B6. Household map / find item wedge | 🟡 | Built conceptually; needs spatial UX validation | P1 |
| B7. Meal planning from stock | 🔵 | High adjacent value but may distract unless tied to use-soon/waste | P2 |
| B8. Festival/event planning | 🔵 | Strong India-specific seasonal wedge; not researched | P2 |
| B9. Caregiver “never run out” mode | 🔵 | High trust/value vertical; requires safety and permissions | P2 |
### C. Inventory, Memory, and Data Model
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| C1. Inventory lots and CRUD | ✅ | Built/tested; needs real-world correction flow stress tests | P0 |
| C2. Event ledger / audit trail | 🟡 | Traces exist, but canonical event-sourced household ledger is not formalized | **P0** |
| C3. Lot merge/split/correction model | ❌ | Real receipts/photos will create duplicate/partial lots; correction semantics missing | **P0** |
| C4. Multi-user household model | ❌ | User columns exist but no real roles/permissions/conflict model | P1 |
| C5. Household location ontology | 🟡 | Locations seeded; user-editable shelf/zone model needs research | P1 |
| C6. Item taxonomy and aliases | 🟡 | Canonicalization exists, but no robust Indian household taxonomy/dataset | **P0** |
| C7. Unit and pack grammar | 🟡 | Unit parsing exists in market data; packet/receipt/general grammar needs hard examples | P1 |
| C8. Shelf-life/freshness rules | 🟡 | Produce metadata exists; broader item shelf-life database incomplete | P1 |
| C9. Preference memory | 🟡 | Field notes/preference ideas exist; no canonical preference schema | P2 |
| C10. Backup/restore/encryption | 🟡 | Portability exists; encrypted household snapshots need exploration | P1 |
### D. Shopping List, Basket, and Decision Logic
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| D1. Shopping list CRUD/complete | ✅ | Built/tested; needs multi-user and real purchase reconciliation | P0 |
| D2. Buy/skip/use-soon classification | ✅/🟡 | Built; needs real-world benchmark and explanation calibration | **P0** |
| D3. Basket-level recommendations | 🟡 | Item-level decisions exist; basket budget/travel/source optimization incomplete | P1 |
| D4. Confirmation thresholds | ❌ | No formal autonomy/confirmation policy by action risk | **P0** |
| D5. Review queue for uncertain decisions | ❌ | Critical missing UI/contract for OCR/vision/model ambiguity | **P0** |
| D6. Budget-aware decisions | 🔵 | No household budget/pay-cycle integration | P2 |
| D7. Substitute recommendations | 🔵 | Needs item taxonomy, preferences, price, and availability | P2 |
| D8. “Watch” state for stale/uncertain data | 🔵 | Decision taxonomy should downgrade stale-source recommendations | P1 |
### E. Receipts, OCR, Packet Labels, and Purchase Reconciliation
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| E1. Real receipt OCR | 🟡 | Tesseract works better than GLM-OCR on real photos; robust pipeline not finalized | **P0** |
| E2. Receipt-to-purchase event schema | 🟡 | Service exists; needs review, merge/split, tax/fees, store/source provenance | **P0** |
| E3. Packet expiry/MRP extraction | ❌ | Core in-store/home scan use case; no benchmark or UX | **P0** |
| E4. Indian receipt layout corpus | ❌ | Need kirana, supermarket, pharmacy, delivery, thermal, mixed-language examples | **P0** |
| E5. OCR preprocessing | 🔵 | Deskew, crop, contrast, binarize, dewarp, table regions not evaluated | P1 |
| E6. MRP vs sale price disambiguation | ❌ | High-risk; wrong price memory if confused | P1 |
| E7. Manufacturing vs expiry/best-before | ❌ | High-risk; needs hard rule/model benchmark | P1 |
| E8. Receipt privacy redaction | 🟡 | Trace redaction exists; receipt image/text redaction needs stricter rules | P1 |
| E9. Email/WhatsApp receipt ingestion | 🔵 | Strong acquisition path; not researched | P2 |
### F. Voice, Conversational UX, and Multilingual Support
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| F1. Voice add command | ✅/🟡 | Tests exist; needs real audio and Hinglish corpus | **P0** |
| F2. Market-mode voice | ❌ | No noisy outdoor benchmark or short-response UX validation | **P0** |
| F3. Hindi/Hinglish support | 🟡 | Examples exist; no real corpus, ASR eval, or fallback strategy | **P0** |
| F4. Voice correction flow | ❌ | “No, this is onion not potato” needs explicit state machine | P1 |
| F5. TTS model selection | 🟡 | HF candidates mapped; no pronunciation/listening benchmark | P1 |
| F6. Wake-word / push-to-talk | 🔵 | Hands-free kitchen mode vs privacy risk not explored | P2 |
| F7. Multi-speaker household audio | 🔵 | Speaker diarization/user attribution not explored | P3 |
### G. Vision, Market Lens, Shelf/Fridge Understanding
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| G1. Barcode + image scan surface | ✅/🟡 | Built/tested partially; real scan quality unknown | P0 |
| G2. Product photo identification | 🟡 | VLM candidates exist; no fixed household/market image benchmark | **P0** |
| G3. Fridge/pantry scan | 🔵 | Huge value; needs zone model, confidence, and correction UX | P1 |
| G4. Shelf/location snapshot memory | 🔵 | Same-shelf/same-item embeddings and last-seen memory not built | P2 |
| G5. Freshness/ripeness detection | 🔵 | Valuable for produce; high false-claim risk | P2 |
| G6. Segmentation/cropping | 🟡 | RMBG exists; ClipSeg/BiRefNet/RMBG-2.0 not benchmarked | P1 |
| G7. Visual uncertainty UX | ❌ | Need review cards and “not sure” states; avoid overconfident wrong labels | **P0** |
| G8. AR/find-item mode | 🔵 | Interesting later; depends on spatial memory maturity | P3 |
### H. Market Intelligence, Retail Sources, and Store Reliability
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| H1. Swiggy data source | ✅/🟡 | Built snapshot loader; freshness and real coverage limits need product copy | P0 |
| H2. Blinkit/Zepto/DMart adapters | 🟡 | Mentioned/built in docs/tests; need current verification and source reliability map | P1 |
| H3. Kirana/manual quote memory | 🔵 | Critical India-local angle; no schema/workflow | P1 |
| H4. Store/source reliability scoring | 🔵 | Price alone is insufficient; need freshness, substitutions, returns, punctuality | P1 |
| H5. Community price commons | 🔵 | Strong moat; privacy/staleness/trust issues unsolved | P2 |
| H6. Basket-level store choice | 🔵 | Compare total basket + travel + freshness, not item price only | P1 |
| H7. Price anomaly detection | 🟡 | Basic price memory exists; no robust anomaly/freshness scoring | P1 |
| H8. Scraping/API ethics | ❌ | Need official/user-provided/public/unsafe source classification | P1 |
| H9. Delivery vs walk/drive tradeoff | 🔵 | Travel-worth-it score needs weather/time/carrying capacity | P2 |
### I. Household Context Intelligence
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| I1. Weather/trip context | 🟡 | Schema/docs exist; needs decision integration and validation | P1 |
| I2. Calendar/event context | 🔵 | Guests, festivals, fasting, school/work schedules not modeled | P2 |
| I3. Storage capacity context | 🔵 | Fridge/freezer/pantry capacity affects bulk-buy and freshness decisions | P2 |
| I4. Health/diet/allergy context | 🔵 | Useful but safety-sensitive; no medical claims | P2 |
| I5. Budget/pay-cycle context | 🔵 | Monthly stocking and payday cycles likely matter; unresearched | P2 |
| I6. Mobility/accessibility context | 🔵 | Elder/caregiver and carrying capacity decisions unmodeled | P2 |
| I7. Seasonality/festival demand | 🔵 | India-specific product moat; no dataset | P2 |
| I8. Power-cut/monsoon resilience | 🔵 | Household ops vertical; freshness and emergency stock implications | P3 |
### J. AI Pipeline, Model Ops, and Benchmarks
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| J1. Model registry | ✅/🟡 | Exists; docs drift vs code must be continuously synced | P0 |
| J2. HF pipeline model sweep | ✅ | Added 2026-06-09 exploration doc; still needs benchmarks | P1 |
| J3. Command parser benchmark | ❌ | Must exist before changing planner defaults | **P0** |
| J4. Receipt OCR benchmark | 🟡 | Some evidence exists; needs fixed manifest and real labels | **P0** |
| J5. Vision benchmark | ❌ | Need product/photo benchmark for Market Lens/fridge scans | **P0** |
| J6. Voice benchmark | ❌ | Need audio fixtures for ASR/TTS choices | **P0** |
| J7. Alias/embedding benchmark | ❌ | Needed before semantic search promotion | P1 |
| J8. Decision benchmark | ❌ | Need expected buy/skip/use-soon cases with inventory/source context | **P0** |
| J9. Cost/latency model routing | 🟡 | Local/cloud providers exist; routing by task complexity not fully documented | P1 |
| J10. Fine-tuned parser LoRA | 🟡 | Candidate exists; dataset/training/eval not done | P1 |
| J11. Model drift/error capture | 🔵 | Corrections should feed benchmark/fine-tune data | P1 |
### K. Privacy, Safety, Trust, and Governance
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| K1. Trace redaction | ✅/🟡 | Exists; needs image/receipt/audio extension | P0 |
| K2. Public demo strict mode | 🔵 | Needed for hackathon/social sharing safety | **P0** |
| K3. Household data export/delete | 🟡 | Portability exists; privacy UX and deletion semantics need work | P1 |
| K4. Encrypted backup | 🔵 | Important for trust; not built | P1 |
| K5. Nutrition/health claim guardrails | ❌ | Nutrition fields exist; no safety language/policy | P1 |
| K6. Auto-action governance | ❌ | Need autonomy levels, confirmation thresholds, never-automate list | **P0** |
| K7. Multi-user permissions | ❌ | Needed before shared household/caregiver modes | P1 |
| K8. Receipt/image PII redaction | ❌ | Critical for public trace/dataset publication | **P0** |
| K9. Source freshness disclaimers | 🟡 | Some copy exists; should be uniform across all recommendations | P1 |
### L. UX, Design, Accessibility, and Operator Workbench
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| L1. Decision-first Today | ✅/🟡 | Implemented per docs; needs real usage validation | P0 |
| L2. Review queue/workbench | ❌ | Missing central surface for uncertain signals | **P0** |
| L3. Market-mode mobile UX | ❌ | Gradio desktop-ish UI may fail in market; needs mobile/touch flow | **P0** |
| L4. Kitchen hands-free UX | 🔵 | Useful; needs voice/privacy tradeoff research | P2 |
| L5. Accessibility / low vision | 🟡 | CSS/design notes exist; no real audit for elder/low-vision users | P1 |
| L6. Empty-state onboarding | 🔵 | Setup burden is existential; needs sample data/import wizard | P1 |
| L7. No-shame waste UX | 🔵 | Waste prevention can feel judgmental; needs tone exploration | P1 |
| L8. Debug/operator view | 🟡 | Runtime diagnostics exist; per-decision provenance view missing | P1 |
| L9. Shareable cards/reports | 🟡 | Image gen/card ideas exist; factual deterministic rendering policy needed | P2 |
### M. Integrations and Platform Surface
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| M1. WhatsApp import/export | 🔵 | High India fit; no implementation/research | P1 |
| M2. Email receipt ingestion | 🔵 | Strong passive data capture; privacy/auth complexity | P2 |
| M3. Browser extension for grocery carts | 🔵 | Strong online-shopping wedge; not researched | P2 |
| M4. Mobile/PWA capture path | 🟡 | Mentioned; needs concrete architecture and test | P1 |
| M5. Calendar/reminder integration | 🔵 | Use-soon and shopping reminders; not researched | P2 |
| M6. Label printer / pantry labels | 🔵 | Nice household ops extension | P3 |
| M7. Home Assistant/smart fridge/camera | 🔵 | Future ambient inventory path | P3 |
| M8. Public product/barcode DBs | 🟡 | Barcode decode exists; product metadata source strategy missing | P1 |
### N. Business Model, Distribution, and Growth
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| N1. Hackathon positioning | 🟡 | Product story exists; needs final wedge and proof video | **P0** |
| N2. Product naming/positioning | 🟡 | ShopStack/ShopStock architecture clarified; tagline tests missing | P1 |
| N3. Pricing/packaging | ❌ | No validation for free/local vs paid sync/market/caregiver plans | P2 |
| N4. Community launch channels | 🟡 | Channels listed; no execution plan or asset map | P1 |
| N5. Free tools / SEO | 🔵 | Many ideas; no priority or build plan | P2 |
| N6. Apartment/community pilot | 🔵 | Strong distribution wedge; needs privacy/community research | P2 |
| N7. Kirana/seller partnership | 🔵 | Potential B2B2C angle; needs validation | P3 |
| N8. Data/dataset publishing story | 🟡 | Dataset ideas exist; no public/private split finalized | P1 |
### O. Operations, QA, Docs, and Project Hygiene
| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| O1. Feature map / roadmap / decision records | ✅/🟡 | Exist; some drift visible, needs regular sync | P0 |
| O2. Exploration maturity tracking | 🔵 | Need statuses/evidence paths like Travel Agent | P1 |
| O3. Demo reproducibility | 🟡 | Demo script exists; final hackathon walkthrough checklist needed | P1 |
| O4. Data catalog | ❌ | JSON/CSV/YAML sources need canonical catalog and validation | P1 |
| O5. Benchmark results registry | 🟡 | Benchmark docs exist; model/task eval registry missing | P1 |
| O6. CI/deployment verification | 🟡 | Docs mention gaps/drift; current status needs verification | P1 |
| O7. Screenshot/artifact handling | 🟡 | Need disciplined `Docs/review/assets/` policy | P2 |
| O8. Public limitation/claims docs | 🟡 | Need clear limitations: stale prices, OCR errors, no medical guarantees | **P0** |
### P. New Research Areas Not Previously Covered Deeply
1. **Household correction semantics** 🔵 — amend/reverse/merge/split/overwrite event
model for all inventory and receipt mistakes.
2. **Strict public-demo safety system** 🔵 — block unsafe trace/image/receipt export.
3. **Source reliability and freshness scoring** 🔵 — every external price/availability
needs trust metadata.
4. **Review queue as product core** 🔵 — turn uncertain model outputs into a daily
correction workbench.
5. **Household autonomy gradient** 🔵 — action risk classification and governance.
6. **Real household command corpus** 🔵 — Hinglish, English, Hindi, corrections,
noisy-market utterances.
7. **Indian receipt/packet label corpus** 🔵 — kirana, supermarket, pharmacy,
quick-commerce, bilingual, thermal, crumpled, low-light.
8. **No-shame waste coaching** 🔵 — behavior design for reducing waste without guilt.
9. **Kirana/manual quote memory** 🔵 — bridge offline retail and quick-commerce data.
10. **Household context scoring** 🔵 — weather, calendar, storage, budget, mobility,
seasonality.
11. **Community price commons** 🔵 — privacy-preserving neighborhood price data.
12. **Caregiver/elder household mode** 🔵 — accessibility, permissions, remote family
trust, never-run-out items.
13. **Apartment/shared-flat mode** 🔵 — split costs, ownership, accountability,
shared vs personal shelves.
14. **Passive ingestion moat** 🔵 — email receipts, WhatsApp orders, browser carts,
loyalty messages.
15. **Model-error-to-benchmark loop** 🔵 — one-click promotion of failures into evals.
---
### 17.1 Research Priority Recommendations
#### Tier 1 — Existential / Must Do for Credible Demo
| # | Research Area | Why | Effort |
|---|---|---|---|
| 1 | Wedge validation: receipt-to-inventory vs Market Lens vs use-soon | Current scope is broad; need one irresistible demo path | 2-4 days |
| 2 | Real household command corpus + parser benchmark | Voice/text is core; without eval, model choices are vibes | 2-3 days |
| 3 | Real receipt + packet OCR benchmark | Receipt ingestion is likely the strongest passive wedge | 3-5 days |
| 4 | Review queue contract and UX | Every imperfect model needs correction; this is core product safety | 2-4 days |
| 5 | Autonomy/confirmation policy | Prevent unsafe mutation and overclaiming | 1-2 days |
| 6 | Public demo strict privacy mode | Required before sharing traces/screenshots/receipts | 1-2 days |
| 7 | Market Lens real-photo dogfood | Validate “Do I need this?” in actual shopping context | 2-3 days |
| 8 | Limitations/claims doc | Must be honest about stale prices, OCR, nutrition, privacy | 1 day |
#### Tier 2 — Critical for Launch Quality
| # | Research Area | Why | Effort |
|---|---|---|---|
| 9 | Item taxonomy + aliases dataset | Powers parser, search, receipts, embeddings, substitutions | 3-5 days |
| 10 | Source reliability scoring | Price intelligence needs trust/freshness, not just numbers | 2-3 days |
| 11 | Lot correction semantics | Real data will be messy; correction must be first-class | 2-3 days |
| 12 | Mobile/market-mode UX | Desktop Gradio is not enough for shopping context | 2-4 days |
| 13 | TTS/STT benchmark | Voice-first claim requires real audio evidence | 2-3 days |
| 14 | Data catalog and validation | Data/config is product architecture | 1-2 days |
| 15 | Encrypted backup/export/delete story | Trust and portability for private household data | 2-4 days |
#### Tier 3 — Strategic / Moat
| # | Research Area | Why | Effort |
|---|---|---|---|
| 16 | Kirana/manual quote memory | India-local differentiation | 1 week |
| 17 | Community price commons | Network/data moat, but trust-heavy | 1-2 weeks |
| 18 | Caregiver/elder mode | High-value vertical with trust needs | 1 week |
| 19 | Festival/event planning | Strong seasonal Indian household behavior | 3-5 days |
| 20 | Browser/WhatsApp/email passive ingestion | Creates compounding memory with less manual entry | 1-2 weeks |
| 21 | Household context scoring | Differentiates from plain inventory apps | 1 week |
| 22 | Programmatic free tools/content | Acquisition engine after product wedge is clear | ongoing |
---
### 17.2 Exploration Statistics Snapshot
Approximate status across the table above:
| Metric | Approx count |
|---|---:|
| Deep categories | 15 |
| Specific rows/status items | 120+ |
| Clearly built or strongly documented | 15-20 |
| Partial / needs validation | 35-45 |
| True gaps | 35-45 |
| New angles opened | 40+ |
| P0 existential items | 20+ |
Takeaway: ShopStack has a lot built, but the main gap is **real-world validation
and correction infrastructure**. The permanent product path is not “add more
models”; it is **household signal → normalized memory → safe decision bundle →
review/confirmation → benchmark feedback loop**.
---
## 18. Exhaustive Scenario, Persona, Edge-Case, and Benchmark Expansion (2026-06-09)
This section fills the remaining obvious exploration gaps: concrete scenarios,
personas, lifecycle maps, failure catalogs, model-by-task benchmarks, data moat,
commercial expansion, free tools, operations, and safety/regulatory surfaces.
### 18.1 Persona Matrix
| Persona | Core job | Pain | ShopStack wedge | Special constraints |
|---|---|---|---|---|
| Primary household shopper | Buy what is needed, avoid duplicates | Mental inventory burden | “Do we need this?” + list + receipt reconcile | Speed, trust, low setup |
| Secondary family member | Check/add items occasionally | Doesn’t know where things are | Ask/search/find item | Simple UI, limited permissions |
| Domestic helper | Execute shopping/restocking tasks | Language, confirmation, accountability | Voice list, photo receipt, review queue | Hindi/Hinglish, no complex forms |
| Elder user | Know stock and avoid running out | Low vision, memory, mobility | Voice-first never-run-out mode | Accessibility, caregiver permissions |
| Caregiver/remote family | Monitor essentials for someone else | Can’t see what’s at home | Shared stock dashboard + alerts | Privacy/consent, low false alarms |
| Student/hostel user | Manage cheap staples | Budget, shared fridge chaos | Budget basket + shared shelf | Low cost, mobile-first |
| Shared-flat roommate | Split shared vs personal items | Disputes and freeloading | Ownership, consume log, split cost | Social friction, audit trail |
| Parent/home cook | Plan meals from stock | Waste and repeated shopping | Use-soon meals + festival/event planning | No-shame tone, dietary preferences |
| Budget optimizer | Spend less over time | Unit-price confusion | Price memory + high-price alerts | Accurate source freshness |
| Health-conscious shopper | Track nutrition/allergens | Label overload | Packet label parser + caution flags | No medical overclaims |
| Quick-commerce power user | Compare apps quickly | App switching, dynamic prices | Basket comparison + source memory | Stale data warnings |
| Kirana-first shopper | Buy locally/offline | No digital data | Manual quote/receipt memory | Offline-first, WhatsApp-friendly |
| Apartment/community admin | Coordinate bulk buys | Demand aggregation, fairness | Community basket + split logistics | Consent, settlement, governance |
| Kirana owner | Serve repeat households | Remember preferences/orders | Seller-side household memory | Separate product, B2B trust |
| Hackathon judge/builder | Understand small-model value | Demo clarity | Transparent model/pipeline evidence | Field notes, benchmark artifacts |
### 18.2 Household Scenario Library
#### Before Shopping
- User asks “what should I buy today?” and expects low-stock, use-soon, and price-aware suggestions.
- User creates list from voice while cooking.
- User checks whether staples are enough for the week.
- User plans shopping around rain/heat/market day.
- User needs a budget basket under ₹500.
- User checks if Sunday market is worth the trip.
- User wants to avoid buying duplicate detergent/rice/oil.
- User plans groceries before guests arrive.
- User plans festival shopping with bulk sweets/oil/flour/dry fruits.
- User needs a “quick top-up” list after returning from travel.
#### During Shopping
- User scans tomatoes and asks if price is high.
- User scans a packet and asks whether it is already at home.
- User asks if the visible expiry date is too soon.
- User asks which pack size is cheaper by unit.
- User sees substitute brand and asks if household usually accepts it.
- User records a kirana quote by voice.
- User asks whether to walk to another store for cheaper produce.
- User corrects model: “no, this is coriander not spinach.”
- User cannot read small MRP/expiry text and needs OCR.
- User is in noisy market and wants one-sentence answer.
#### After Shopping
- User photographs receipt and wants inventory updated.
- User photographs all groceries on the counter.
- User marks purchased list items and adds prices.
- User splits receipt into inventory vs household expense only.
- User records store freshness rating.
- User reconciles duplicate lots after OCR error.
- User adds expiry dates from packet closeups.
- User moves items to fridge/pantry/bathroom cabinet.
- User shares weekly spend/waste summary.
- User exports redacted trace for demo/field notes.
#### At Home
- User asks where the dahi is.
- User asks what can be cooked without shopping.
- User asks what expires first.
- User consumes half a packet and updates quantity.
- User finds old item hidden behind shelf.
- User scans fridge before leaving for store.
- User wants lunchbox ideas from use-soon items.
- User asks if baby formula/pet food is enough.
- User gets reminder before milk/bread stockout.
- User wants to reduce waste without guilt.
#### Shared Household / Caregiver
- Roommate consumes shared milk; others need visibility.
- One user buys item another already bought.
- Helper buys groceries and sends receipt photo.
- Caregiver checks elder household essentials remotely.
- Parent asks kid to check fridge with photo.
- Apartment group coordinates bulk rice/oil order.
- Household member disputes an auto-applied consume event.
- User wants private personal shelf hidden from others.
- Remote family wants alerts only for critical items.
- Helper needs voice-only shopping confirmation.
### 18.3 Lifecycle Maps
#### Item Lifecycle
```text
Need detected → list candidate → bought → receipt/price observed → stored → moved
→ opened → partially consumed → use-soon → consumed / wasted / donated → learned
```
State questions:
- What event created the item?
- Which source is trusted for quantity/price/expiry?
- Is this a new lot or duplicate?
- Who confirmed it?
- What correction history exists?
- What did the system learn for next time?
#### Shopping Decision Lifecycle
```text
Signal → normalize → compare against inventory → enrich with price/source/context
→ classify → explain → confirm → apply → trace → benchmark failure if corrected
```
#### Receipt Lifecycle
```text
Image/email/WhatsApp → preprocess → OCR → line grouping → structured extraction
→ source/price provenance → duplicate detection → review queue → inventory/price events
```
#### Model Failure Lifecycle
```text
Bad output → user correction → capture raw input + expected output → redact → fixture
→ benchmark run → candidate prompt/model/config → keep/discard log
```
### 18.4 Edge-Case and Failure Catalog
| Category | Edge cases to explore |
|---|---|
| OCR/receipts | crumpled receipt, thermal fading, mixed Hindi/English, right-aligned prices, discounts, taxes, delivery fees, refund lines, unreadable totals, duplicate receipt photo |
| Packet labels | MRP vs sale price, mfg vs expiry vs best-before, date format ambiguity, batch number mistaken as date, tiny font, curved packet, reflective packaging |
| Voice | negation missed, noisy market, mixed language, brand homophones, quantity omitted, correction utterance, multi-speaker command, child voice |
| Inventory | duplicate lot, partial consume, item moved without logging, stale quantity, opened vs unopened, household member disagreement, unit conversion error |
| Market price | stale app snapshot, unavailable item, combo pack, delivery fee hidden, quality lower despite cheaper price, store substitution, region mismatch |
| Vision | similar produce, occluded shelf, transparent containers, unlabeled jars, multiple items in one frame, overconfident wrong VLM label |
| Shared household | two users edit same item, private vs shared item, helper permissions, remote caregiver override, split-cost dispute |
| Privacy | receipt phone/address, UPI/card token, face in fridge photo, child voice/audio, household names in trace, location metadata in image |
| Safety | nutrition overclaim, medicine-like item advice, allergy false negative, spoiled food missed, auto-delete or auto-purchase risk |
| Offline/runtime | model missing, HF download fails, OCR timeout, DB locked, backup restore conflict, source adapter schema drift |
### 18.5 Feature Universe Backlog
| Cluster | Feature ideas |
|---|---|
| Capture | voice command, receipt photo, packet scan, fridge photo, shelf photo, barcode, email receipt, WhatsApp receipt/list, browser cart import |
| Normalize | item aliases, brand mapping, unit grammar, pack-size parser, expiry parser, store/source canonicalization, household location ontology |
| Decide | buy/skip/use-soon/watch/compare/confirm, budget basket, stockout prediction, waste risk, travel-worth-it, source reliability, substitution |
| Reconcile | review queue, merge/split lots, undo/reverse event, duplicate receipt detection, price-only observation, correction-to-fixture |
| Remember | price history, store quality, household preferences, recurring cadence, shelf locations, last-seen photo, field notes, model traces |
| Act | shopping list, reminders, label print, calendar reminder, family summary, helper handoff, caregiver alert, community bulk-buy request |
| Explain | why buy/skip, source freshness, confidence, fallback path, data provenance, no-medical-claim caveat, stale price warning |
| Share | redacted trace export, weekly digest, spend/waste report, public demo dataset, field notes, shareable card |
| Admin | model status, source adapter health, data catalog, backup/export/delete, autonomy settings, permissions, privacy guard |
### 18.6 Model-by-Task Benchmark Matrix
| Task | Candidate models/tools | Metric | Must-have fixture set |
|---|---|---|---|
| ASR command transcription | SenseVoice, Qwen3-ASR, Parakeet, Whisper, WhisperKit | WER + slot retention + latency | 100 Hinglish/English/noisy commands |
| Intent classification | Qwen small, Gemma small, DistilBERT, rules | intent F1, confusion matrix | add/move/consume/find/price/check/correct |
| Slot extraction | parser LoRA, Qwen, token classifier, rules | exact item/qty/unit/location accuracy | household command corpus |
| Tool-call generation | Qwen3.5/4B, MiniCPM5, GGUF Llama/LFM | valid JSON, executable calls, no unsafe action | 200 command/action cases |
| Receipt OCR | Tesseract, PaddleOCR, GLM-OCR, TrOCR, Donut | field accuracy, row grouping, latency | 100 Indian receipts |
| Packet OCR | OCR + VLM | expiry/MRP/date disambiguation | 100 packet labels |
| VLM item ID | MiniCPM-V, Qwen2.5-VL, BLIP/ViLT baselines | top-1/top-3, abstain quality | 100 market/fridge photos |
| Object detection | DETR, YOLOS, RT-DETR, table/document detectors | box recall/precision | shelf/receipt/table crops |
| Segmentation | RMBG, BiRefNet, ClipSeg, SegFormer | mask IoU, visual usefulness | item card/shelf zone images |
| Embeddings | BGE-M3, Qwen3 Embedding, E5, Jina, MiniLM | top-k alias retrieval | 1k alias/canonical pairs |
| Reranking | BGE reranker | top-1 after retrieve | hard alias negatives |
| Decision classifier | rules + model explanations | expected class accuracy + explanation faithfulness | inventory+price+context scenarios |
- **Current STT note:** `Qwen3-ASR-1.7B` is now benchmarked on the 20-command Hinglish set with transliteration-aware scoring: **34.46% WER**, **55.6% slot retention**, **1.11s** latency. It outputs Devanagari script, so the quality score only makes sense after Latin transliteration.
| TTS | Kokoro, Qwen TTS, Indic Parler, CosyVoice, VibeVoice | pronunciation, MOS, latency | grocery terms + short replies |
| Privacy redaction | regex, NER, OCR boxes, face detector | leak recall, false positives | receipts/images/traces/audio transcripts |
### 18.7 Data Moat and Dataset Strategy
| Dataset | Public? | Why valuable | Risks |
|---|---|---|---|
| Household command corpus | Public synthetic + private real | Parser/fine-tune moat | Household language privacy |
| Indian item alias graph | Public | Core search/parser value | Regional bias/staleness |
| Receipt OCR corpus | Mostly private/redacted synthetic | Receipt-to-inventory benchmark | Phone/address/payment leakage |
| Packet label corpus | Public/private mix | Expiry/MRP extraction | Brand/copyright/photos |
| Price observation history | Private/aggregate public | Store/source intelligence | Stale/misleading prices |
| Store reliability observations | Private/aggregate public | Quality moat beyond price | Defamation/fairness concerns |
| Fridge/shelf images | Private only by default | Spatial memory/VLM eval | Faces/home privacy/location metadata |
| Correction traces | Redacted public subset | Model improvement and Field Notes | Raw private content leakage |
| Use-soon/waste outcomes | Private/aggregate | Waste coaching efficacy | Shame/sensitive household behavior |
| Community basket data | Consent-only aggregate | Group buying/network effects | Consent/governance/trust |
### 18.8 Commercial and Expansion Map
| Segment | Product | Why it could work | Research needed |
|---|---|---|---|
| Individual households | Free/local inventory + shopping memory | Broad wedge, privacy-friendly | Setup friction, retention |
| Families | Sync, shared permissions, reminders | Multi-user value | Permissions, pricing |
| Caregivers | Never-run-out essentials + remote alerts | High pain/trust | Safety, elder UX |
| Shared flats/hostels | Shared pantry + split costs | Clear social pain | Dispute avoidance |
| Apartment societies | Bulk-buy coordination | Network effects | Governance/payment logistics |
| Kirana stores | Repeat household memory / WhatsApp ordering | Offline commerce bridge | Seller workflows |
| Quick-commerce users | Basket comparison and price memory | Existing digital behavior | Source data access |
| Health/wellness users | Label/nutrition caution | High engagement | Claim boundaries |
| Hackathon/builders | Small-model multimodal case study | Demo/credibility | Field notes/benchmarks |
| Data/API product | Aggregated price/alias/source intelligence | Moat if consented | Privacy/legal/commercial demand |
### 18.9 Programmatic Content and Free Tools
| Tool/content | Audience | Data needed | Product tie-in |
|---|---|---|---|
| Unit-price calculator | Budget shoppers | pack sizes, units | Price memory |
| “Is this price high?” checker | Market shoppers | price history/snapshots | Market Lens |
| Receipt-to-spend summary | Families | OCR receipt | Receipt ingestion |
| Expiry label parser demo | Anyone scanning packets | OCR labels | Packet scan |
| Food waste estimator | Home cooks | inventory/use-soon | Waste coach |
| Festival shopping planner | Indian households | festival lists | Event planning |
| Rainy-day grocery checklist | Weather-aware shoppers | weather + staples | Context intelligence |
| Bulk-buy break-even calculator | Apartments/families | unit prices/storage | Community buying |
| Pantry audit checklist | New users | item taxonomy | Onboarding |
| Grocery alias explorer | Builders/judges | alias graph | Dataset story |
| “What can I cook?” mini-tool | Home cooks | inventory + recipes | Meal planning |
| Caregiver essentials checklist | Caregivers | essential categories | Elder mode |
### 18.10 Regulatory, Safety, and Claims Expansion
| Surface | Rule to research/document |
|---|---|
| Nutrition | Provide informational label extraction, not diet/medical advice. |
| Allergens | Warn only from explicit label/user data; never guarantee safety. |
| Medicines/pharmacy | Track stock/expiry only; no dosage/medical guidance. |
| Prices | Always show source and freshness; avoid “best price” guarantee if stale. |
| Community data | Consent, aggregation thresholds, removal rights, anti-abuse. |
| Receipts | Redact phone/address/payment IDs before sharing/export. |
| Household images | Strip EXIF/location; detect faces/private content before public use. |
| Child/elder data | Extra consent and conservative sharing defaults. |
| Auto-actions | No auto-purchase; no destructive delete without confirmation. |
| Public demo | Synthetic/redacted data by default; strict leakage guard. |
### 18.11 Operations and Quality System
| Operational system | Purpose |
|---|---|
| Data catalog | Every JSON/CSV/YAML/source snapshot has owner, schema, freshness, tests. |
| Benchmark registry | Every model/pipeline candidate has fixed eval, score, latency, keep/discard. |
| Source adapter monitor | Detect schema drift, stale snapshots, unit parsing failures. |
| Redaction review | Gate traces/images/receipts before public export. |
| Demo mode | Deterministic seeded data and walkthrough reset. |
| Backup/restore drill | Prove household memory can be exported/restored. |
| Failure fixture promotion | Corrections become regression tests. |
| Claims checklist | Validate UI/docs copy against actual capabilities. |
| Exploration maturity tracker | Each idea has status, evidence path, priority, and last verified date. |
| Dogfood log | Real household/market sessions with outcomes and failures. |
### 18.12 More Open Questions
41. What exact 3-minute demo proves value fastest: receipt, market scan, or use-soon?
42. Which persona should the product optimize for first: primary shopper, helper,
caregiver, or shared-flat user?
43. What is the minimum review queue that makes imperfect models useful rather than annoying?
44. Which household corrections should alter future model behavior automatically?
45. How much setup can users tolerate before seeing value?
46. What data should never leave the device, even with consent?
47. Can stale price/source uncertainty be explained in one short line?
48. Is the moat item aliases, receipt corpus, price memory, source reliability, or
correction traces?
49. Should ShopStack sell sync/backup, intelligence, or community buying first?
50. What feature would a household still use after the hackathon novelty fades?
_Last updated: 2026-06-09_
|