Spaces:

NabilHzs
/

sipena

Sleeping

sipena / scripts /rerun_bertopic.log

Redeploy full backend

77c4780 4 months ago

31.5 kB

��..\.venv\Scripts\python.exe : C:\Kuliah\Skripsi\Work\Apps\.venv\Lib
\site-packages\transformers\tokenization_utils_base.py:1601: 
FutureWarning: `clean_up_tokenization_spaces` was not set. It will 
be set to `True` by default. This behavior will be depracted in 
transformers v4.45, and will be then set to `False` by default. 
For more details check this issue: 
https://github.com/huggingface/transformers/issues/31884
At line:1 char:1
+ ..\.venv\Scripts\python.exe scripts\rerun_bertopic.py 2>&1 | 
Tee-Obje ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (C:\Kuliah\Skrip...rs/ 
   issues/31884:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
  warnings.warn(
File: C:\Kuliah\Skripsi\Work\Model\metadata.csv
Shape awal: (2073, 17)

Preprocessing summary:
{
  "filter_low_context_abstracts": true,
  "min_abstract_words": 30,
  "min_abstract_unique_words": 10,
  "min_abstract_unique_ratio": 0.25,
  "max_abstract_token_ratio": 0.3,
  "n_rows_before_preprocessing": 2073,
  "n_rows_after_preprocessing": 2069,
  "n_rows_removed_preprocessing": 4,
  "n_rows_removed_empty_text": 0,
  "n_rows_removed_low_context": 4,
  "n_rows_empty_or_placeholder_abstract": 0
}

Publikasi unik untuk model: 2,069
Relasi publikasi-dosen: 3,006
Baris duplikat publikasi yang digabung: 0

Mendeteksi bahasa...
{'id': 1080, 'en': 989}
Loading paraphrase-multilingual-MiniLM-L12-v2...
Encoding 2069 dokumen dengan paraphrase-multilingual-MiniLM-L12-v2...

Batches:   0%|          | 0/65 [00:00<?, ?it/s]
Batches:   2%|1         | 1/65 [00:02<02:59,  2.80s/it]
Batches:   3%|3         | 2/65 [00:03<01:53,  1.81s/it]
Batches:   5%|4         | 3/65 [00:05<01:33,  1.51s/it]
Batches:   6%|6         | 4/65 [00:06<01:22,  1.36s/it]
Batches:   8%|7         | 5/65 [00:07<01:16,  1.27s/it]
Batches:   9%|9         | 6/65 [00:08<01:14,  1.26s/it]
Batches:  11%|#         | 7/65 [00:09<01:10,  1.22s/it]
Batches:  12%|#2        | 8/65 [00:10<01:09,  1.21s/it]
Batches:  14%|#3        | 9/65 [00:11<01:05,  1.17s/it]
Batches:  15%|#5        | 10/65 [00:13<01:03,  1.15s/it]
Batches:  17%|#6        | 11/65 [00:14<01:01,  1.14s/it]
Batches:  18%|#8        | 12/65 [00:15<01:00,  1.14s/it]
Batches:  20%|##        | 13/65 [00:16<00:59,  1.14s/it]
Batches:  22%|##1       | 14/65 [00:17<00:57,  1.14s/it]
Batches:  23%|##3       | 15/65 [00:18<01:00,  1.22s/it]
Batches:  25%|##4       | 16/65 [00:20<01:03,  1.29s/it]
Batches:  26%|##6       | 17/65 [00:21<01:03,  1.33s/it]
Batches:  28%|##7       | 18/65 [00:23<01:06,  1.41s/it]
Batches:  29%|##9       | 19/65 [00:24<01:04,  1.41s/it]
Batches:  31%|###       | 20/65 [00:26<01:03,  1.41s/it]
Batches:  32%|###2      | 21/65 [00:27<01:01,  1.41s/it]
Batches:  34%|###3      | 22/65 [00:29<01:00,  1.41s/it]
Batches:  35%|###5      | 23/65 [00:30<00:58,  1.40s/it]
Batches:  37%|###6      | 24/65 [00:31<00:57,  1.40s/it]
Batches:  38%|###8      | 25/65 [00:33<00:56,  1.41s/it]
Batches:  40%|####      | 26/65 [00:34<00:54,  1.41s/it]
Batches:  42%|####1     | 27/65 [00:36<00:53,  1.41s/it]
Batches:  43%|####3     | 28/65 [00:37<00:53,  1.44s/it]
Batches:  45%|####4     | 29/65 [00:39<00:51,  1.43s/it]
Batches:  46%|####6     | 30/65 [00:40<00:50,  1.43s/it]
Batches:  48%|####7     | 31/65 [00:42<00:50,  1.47s/it]
Batches:  49%|####9     | 32/65 [00:43<00:48,  1.48s/it]
Batches:  51%|#####     | 33/65 [00:44<00:46,  1.47s/it]
Batches:  52%|#####2    | 34/65 [00:46<00:44,  1.45s/it]
Batches:  54%|#####3    | 35/65 [00:47<00:43,  1.44s/it]
Batches:  55%|#####5    | 36/65 [00:49<00:41,  1.43s/it]
Batches:  57%|#####6    | 37/65 [00:50<00:40,  1.43s/it]
Batches:  58%|#####8    | 38/65 [00:52<00:39,  1.46s/it]
Batches:  60%|######    | 39/65 [00:53<00:37,  1.45s/it]
Batches:  62%|######1   | 40/65 [00:55<00:35,  1.44s/it]
Batches:  63%|######3   | 41/65 [00:56<00:34,  1.45s/it]
Batches:  65%|######4   | 42/65 [00:57<00:33,  1.46s/it]
Batches:  66%|######6   | 43/65 [00:59<00:32,  1.46s/it]
Batches:  68%|######7   | 44/65 [01:00<00:30,  1.45s/it]
Batches:  69%|######9   | 45/65 [01:02<00:28,  1.45s/it]
Batches:  71%|#######   | 46/65 [01:03<00:28,  1.49s/it]
Batches:  72%|#######2  | 47/65 [01:05<00:26,  1.49s/it]
Batches:  74%|#######3  | 48/65 [01:06<00:24,  1.46s/it]
Batches:  75%|#######5  | 49/65 [01:08<00:23,  1.46s/it]
Batches:  77%|#######6  | 50/65 [01:09<00:21,  1.45s/it]
Batches:  78%|#######8  | 51/65 [01:11<00:20,  1.45s/it]
Batches:  80%|########  | 52/65 [01:12<00:18,  1.45s/it]
Batches:  82%|########1 | 53/65 [01:14<00:17,  1.45s/it]
Batches:  83%|########3 | 54/65 [01:15<00:15,  1.45s/it]
Batches:  85%|########4 | 55/65 [01:16<00:14,  1.48s/it]
Batches:  86%|########6 | 56/65 [01:18<00:13,  1.51s/it]
Batches:  88%|########7 | 57/65 [01:20<00:12,  1.52s/it]
Batches:  89%|########9 | 58/65 [01:21<00:10,  1.50s/it]
Batches:  91%|######### | 59/65 [01:23<00:09,  1.50s/it]
Batches:  92%|#########2| 60/65 [01:24<00:07,  1.49s/it]
Batches:  94%|#########3| 61/65 [01:25<00:05,  1.47s/it]
Batches:  95%|#########5| 62/65 [01:27<00:04,  1.47s/it]
Batches:  97%|#########6| 63/65 [01:28<00:02,  1.48s/it]
Batches:  98%|#########8| 64/65 [01:30<00:01,  1.48s/it]
Batches: 100%|##########| 65/65 [01:31<00:00,  1.34s/it]
Batches: 100%|##########| 65/65 [01:31<00:00,  1.41s/it]
  shape (2069, 384) (91.6s)

Grid search: 4 x 4 = 16 kombinasi
Grid 1: fit selesai (19.5s)
[ 1] topics=140 noise= 19.6% sil=0.0466 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
Grid 2: fit selesai (8.8s)
[ 2] topics=64 noise= 23.3% sil=0.0482 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
Grid 3: fit selesai (8.4s)
[ 3] topics=40 noise= 24.1% sil=0.0395 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
Grid 4: fit selesai (8.5s)
[ 4] topics=36 noise= 23.4% sil=0.0569 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
Grid 5: fit selesai (9.3s)
[ 5] topics=135 noise= 23.6% sil=0.0427 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
Grid 6: fit selesai (9.0s)
[ 6] topics=58 noise= 28.4% sil=0.0467 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
Grid 7: fit selesai (9.4s)
[ 7] topics=36 noise= 23.8% sil=0.0406 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
Grid 8: fit selesai (9.0s)
[ 8] topics=31 noise= 29.2% sil=0.0478 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
Grid 9: fit selesai (10.5s)
[ 9] topics=125 noise= 24.8% sil=0.0538 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
Grid 10: fit selesai (9.3s)
[10] topics=52 noise= 26.0% sil=0.0465 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
Grid 11: fit selesai (8.6s)
[11] topics=32 noise= 18.1% sil=0.0262 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
Grid 12: fit selesai (8.6s)
[12] topics=32 noise= 22.4% sil=0.0349 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
Grid 13: fit selesai (10.6s)
[13] topics=102 noise= 30.5% sil=0.0442 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 5, 'min_samples': 3}
Grid 14: fit selesai (8.9s)
[14] topics=48 noise= 30.9% sil=0.0468 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 10, 'min_samples': 5}
Grid 15: fit selesai (9.0s)
[15] topics=31 noise= 29.3% sil=0.0596 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 15, 'min_samples': 5}
Grid 16: fit selesai (9.3s)
[16] topics=22 noise= 27.0% sil=0.0455 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 15, 'min_samples': 10}

Best config sil=0.0596 UMAP={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} HDBSCAN={'min_cluster_size': 15, 'min_samples': 5}

Dokumen EN: 989 | ID: 1080

=== EN Clustering ===
Loading all-mpnet-base-v2...
Encoding 989 dokumen dengan all-mpnet-base-v2...

Batches:   0%|          | 0/31 [00:00<?, ?it/s]
Batches:   3%|3         | 1/31 [00:20<10:06, 20.21s/it]
Batches:   6%|6         | 2/31 [00:36<08:42, 18.02s/it]
Batches:  10%|9         | 3/31 [00:52<08:00, 17.16s/it]
Batches:  13%|#2        | 4/31 [01:09<07:36, 16.90s/it]
Batches:  16%|#6        | 5/31 [01:26<07:22, 17.01s/it]
Batches:  19%|#9        | 6/31 [01:42<06:57, 16.71s/it]
Batches:  23%|##2       | 7/31 [01:58<06:37, 16.55s/it]
Batches:  26%|##5       | 8/31 [02:13<06:05, 15.88s/it]
Batches:  29%|##9       | 9/31 [02:28<05:47, 15.79s/it]
Batches:  32%|###2      | 10/31 [02:42<05:13, 14.95s/it]
Batches:  35%|###5      | 11/31 [02:53<04:40, 14.05s/it]
Batches:  39%|###8      | 12/31 [03:05<04:10, 13.19s/it]
Batches:  42%|####1     | 13/31 [03:18<03:58, 13.24s/it]
Batches:  45%|####5     | 14/31 [03:30<03:39, 12.90s/it]
Batches:  48%|####8     | 15/31 [03:40<03:12, 12.01s/it]
Batches:  52%|#####1    | 16/31 [03:50<02:51, 11.42s/it]
Batches:  55%|#####4    | 17/31 [04:00<02:32, 10.87s/it]
Batches:  58%|#####8    | 18/31 [04:11<02:21, 10.87s/it]
Batches:  61%|######1   | 19/31 [04:23<02:15, 11.32s/it]
Batches:  65%|######4   | 20/31 [04:32<01:56, 10.58s/it]
Batches:  68%|######7   | 21/31 [04:41<01:42, 10.26s/it]
Batches:  71%|#######   | 22/31 [04:53<01:35, 10.58s/it]
Batches:  74%|#######4  | 23/31 [05:05<01:28, 11.11s/it]
Batches:  77%|#######7  | 24/31 [05:22<01:30, 12.98s/it]
Batches:  81%|########  | 25/31 [05:34<01:16, 12.69s/it]
Batches:  84%|########3 | 26/31 [05:45<01:00, 12.11s/it]
Batches:  87%|########7 | 27/31 [05:58<00:48, 12.23s/it]
Batches:  90%|######### | 28/31 [06:10<00:36, 12.18s/it]
Batches:  94%|#########3| 29/31 [06:23<00:24, 12.42s/it]
Batches:  97%|#########6| 30/31 [06:30<00:11, 11.01s/it]
Batches: 100%|##########| 31/31 [06:37<00:00,  9.68s/it]
Batches: 100%|##########| 31/31 [06:37<00:00, 12.82s/it]
  shape (989, 768) (397.6s)
S3_EN: fit selesai (3.3s)

=== ID Clustering ===
Loading firqaaa/indo-sentence-bert-base...
Encoding 1080 dokumen dengan firqaaa/indo-sentence-bert-base...

Batches:   0%|          | 0/34 [00:00<?, ?it/s]
Batches:   3%|2         | 1/34 [00:21<12:01, 21.86s/it]
Batches:   6%|5         | 2/34 [00:42<11:11, 20.99s/it]
Batches:   9%|8         | 3/34 [01:01<10:31, 20.37s/it]
Batches:  12%|#1        | 4/34 [01:21<09:57, 19.90s/it]
Batches:  15%|#4        | 5/34 [01:40<09:36, 19.87s/it]
Batches:  18%|#7        | 6/34 [02:00<09:12, 19.74s/it]
Batches:  21%|##        | 7/34 [02:19<08:50, 19.66s/it]
Batches:  24%|##3       | 8/34 [02:40<08:35, 19.84s/it]
Batches:  26%|##6       | 9/34 [02:56<07:51, 18.85s/it]
Batches:  29%|##9       | 10/34 [03:11<06:59, 17.50s/it]
Batches:  32%|###2      | 11/34 [03:24<06:09, 16.07s/it]
Batches:  35%|###5      | 12/34 [03:37<05:37, 15.33s/it]
Batches:  38%|###8      | 13/34 [03:51<05:09, 14.74s/it]
Batches:  41%|####1     | 14/34 [04:03<04:39, 13.97s/it]
Batches:  44%|####4     | 15/34 [04:14<04:07, 13.04s/it]
Batches:  47%|####7     | 16/34 [04:25<03:43, 12.43s/it]
Batches:  50%|#####     | 17/34 [04:35<03:20, 11.79s/it]
Batches:  53%|#####2    | 18/34 [04:46<03:02, 11.44s/it]
Batches:  56%|#####5    | 19/34 [04:56<02:44, 10.99s/it]
Batches:  59%|#####8    | 20/34 [05:05<02:29, 10.68s/it]
Batches:  62%|######1   | 21/34 [05:15<02:14, 10.37s/it]
Batches:  65%|######4   | 22/34 [05:24<02:00, 10.02s/it]
Batches:  68%|######7   | 23/34 [05:35<01:50, 10.07s/it]
Batches:  71%|#######   | 24/34 [05:45<01:40, 10.04s/it]
Batches:  74%|#######3  | 25/34 [05:53<01:25,  9.52s/it]
Batches:  76%|#######6  | 26/34 [06:04<01:20, 10.07s/it]
Batches:  79%|#######9  | 27/34 [06:13<01:08,  9.80s/it]
Batches:  82%|########2 | 28/34 [06:21<00:55,  9.26s/it]
Batches:  85%|########5 | 29/34 [06:30<00:45,  9.19s/it]
Batches:  88%|########8 | 30/34 [06:38<00:34,  8.73s/it]
Batches:  91%|#########1| 31/34 [06:46<00:25,  8.41s/it]
Batches:  94%|#########4| 32/34 [06:56<00:17,  8.98s/it]
Batches:  97%|#########7| 33/34 [07:02<00:08,  8.21s/it]
Batches: 100%|##########| 34/34 [07:05<00:00,  6.64s/it]
Batches: 100%|##########| 34/34 [07:05<00:00, 12.53s/it]
2026-06-16 16:43:20,297 - BERTopic - WARNING: Using a custom list 
of topic assignments may lead to errors if topic reduction 
techniques are used afterwards. Make sure that manually assigning 
topics is the last step in the pipeline.Note that topic embeddings 
will also be created through weightedc-TF-IDF embeddings instead 
of centroid embeddings.
2026-06-16 16:43:20,475 - BERTopic - WARNING: Using a custom list 
of topic assignments may lead to errors if topic reduction 
techniques are used afterwards. Make sure that manually assigning 
topics is the last step in the pipeline.Note that topic embeddings 
will also be created through weightedc-TF-IDF embeddings instead 
of centroid embeddings.
  shape (1080, 768) (425.9s)
S3_ID: fit selesai (3.6s)

Reduce outliers...

EN: 18 topik (max_id=17) | ID: 63 topik (offset=18)

Menyimpan artefak (dengan backup)...
  saved embeddings_en.npy
  saved embeddings_id.npy
  backup: embeddings.npy.bak.1781603000
  saved embeddings.npy
  backup: publication_ids.npy.bak.1781603000
  saved publication_ids.npy
  backup: bt_en.bak.1781603000
  saved bt_en
  backup: bt_id.bak.1781603000
  saved bt_id
  backup: topic_info.json.bak.1781603000
  saved topic_info.json
{
  "en": {
    "model_name": "all-mpnet-base-v2",
    "n_topics": 18,
    "max_topic_id": 17
  },
  "id": {
    "model_name": "firqaaa/indo-sentence-bert-base",
    "n_topics": 63,
    "id_topic_offset": 18
  },
  "data": {
    "unit": "unique_publication",
    "n_source_rows": 2069,
    "n_unique_publications": 2069,
    "n_publication_lecturer_edges": 3006,
    "duplicate_publication_rows_merged": 0
  },
  "preprocessing": {
    "filter_low_context_abstracts": true,
    "min_abstract_words": 30,
    "min_abstract_unique_words": 10,
    "min_abstract_unique_ratio": 0.25,
    "max_abstract_token_ratio": 0.3,
    "n_rows_before_preprocessing": 2073,
    "n_rows_after_preprocessing": 2069,
    "n_rows_removed_preprocessing": 4,
    "n_rows_removed_empty_text": 0,
    "n_rows_removed_low_context": 4,
    "n_rows_empty_or_placeholder_abstract": 0
  },
  "rejected_reason_counts": {
    "abstract_words<30": 3,
    "abstract_words<30; unique_words<10": 1
  }
}
  saved lecturer_embeddings.npy
  saved lecturer_names.npy
  backup: publications.csv.bak.1781603001
  saved publications.csv (2069 rows)
  backup: metadata.csv.bak.1781603001
  saved metadata.csv (2069 rows)
  saved publication_lecturers.csv (3006 rows)
  saved publication_topics.csv (2069 rows)
  backup: lecturer_profiles.csv.bak.1781603001
  saved lecturer_profiles.csv (101 rows)

Topic distribution:
topic_id
0     103
1      94
2     101
3      90
4      88
5      88
6      58
7      42
8      50
9      37
10     40
11     37
12     51
13     26
14     24
15     23
16     21
17     16
18     81
19     43
Name: count, dtype: int64

Selesai. embeddings rows = 2069 | publications rows = 2069

	��..\.venv\Scripts\python.exe : C:\Kuliah\Skripsi\Work\Apps\.venv\Lib
	\site-packages\transformers\tokenization_utils_base.py:1601:
	FutureWarning: `clean_up_tokenization_spaces` was not set. It will
	be set to `True` by default. This behavior will be depracted in
	transformers v4.45, and will be then set to `False` by default.
	For more details check this issue:
	https://github.com/huggingface/transformers/issues/31884
	At line:1 char:1
	+ ..\.venv\Scripts\python.exe scripts\rerun_bertopic.py 2>&1 \|
	Tee-Obje ...
	+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	+ CategoryInfo : NotSpecified: (C:\Kuliah\Skrip...rs/
	issues/31884:String) [], RemoteException
	+ FullyQualifiedErrorId : NativeCommandError

	warnings.warn(
	File: C:\Kuliah\Skripsi\Work\Model\metadata.csv
	Shape awal: (2073, 17)

	Preprocessing summary:
	{
	"filter_low_context_abstracts": true,
	"min_abstract_words": 30,
	"min_abstract_unique_words": 10,
	"min_abstract_unique_ratio": 0.25,
	"max_abstract_token_ratio": 0.3,
	"n_rows_before_preprocessing": 2073,
	"n_rows_after_preprocessing": 2069,
	"n_rows_removed_preprocessing": 4,
	"n_rows_removed_empty_text": 0,
	"n_rows_removed_low_context": 4,
	"n_rows_empty_or_placeholder_abstract": 0
	}

	Publikasi unik untuk model: 2,069
	Relasi publikasi-dosen: 3,006
	Baris duplikat publikasi yang digabung: 0

	Mendeteksi bahasa...
	{'id': 1080, 'en': 989}
	Loading paraphrase-multilingual-MiniLM-L12-v2...
	Encoding 2069 dokumen dengan paraphrase-multilingual-MiniLM-L12-v2...

	Batches: 0%\| \| 0/65 [00:00<?, ?it/s]
	Batches: 2%\|1 \| 1/65 [00:02<02:59, 2.80s/it]
	Batches: 3%\|3 \| 2/65 [00:03<01:53, 1.81s/it]
	Batches: 5%\|4 \| 3/65 [00:05<01:33, 1.51s/it]
	Batches: 6%\|6 \| 4/65 [00:06<01:22, 1.36s/it]
	Batches: 8%\|7 \| 5/65 [00:07<01:16, 1.27s/it]
	Batches: 9%\|9 \| 6/65 [00:08<01:14, 1.26s/it]
	Batches: 11%\|# \| 7/65 [00:09<01:10, 1.22s/it]
	Batches: 12%\|#2 \| 8/65 [00:10<01:09, 1.21s/it]
	Batches: 14%\|#3 \| 9/65 [00:11<01:05, 1.17s/it]
	Batches: 15%\|#5 \| 10/65 [00:13<01:03, 1.15s/it]
	Batches: 17%\|#6 \| 11/65 [00:14<01:01, 1.14s/it]
	Batches: 18%\|#8 \| 12/65 [00:15<01:00, 1.14s/it]
	Batches: 20%\|## \| 13/65 [00:16<00:59, 1.14s/it]
	Batches: 22%\|##1 \| 14/65 [00:17<00:57, 1.14s/it]
	Batches: 23%\|##3 \| 15/65 [00:18<01:00, 1.22s/it]
	Batches: 25%\|##4 \| 16/65 [00:20<01:03, 1.29s/it]
	Batches: 26%\|##6 \| 17/65 [00:21<01:03, 1.33s/it]
	Batches: 28%\|##7 \| 18/65 [00:23<01:06, 1.41s/it]
	Batches: 29%\|##9 \| 19/65 [00:24<01:04, 1.41s/it]
	Batches: 31%\|### \| 20/65 [00:26<01:03, 1.41s/it]
	Batches: 32%\|###2 \| 21/65 [00:27<01:01, 1.41s/it]
	Batches: 34%\|###3 \| 22/65 [00:29<01:00, 1.41s/it]
	Batches: 35%\|###5 \| 23/65 [00:30<00:58, 1.40s/it]
	Batches: 37%\|###6 \| 24/65 [00:31<00:57, 1.40s/it]
	Batches: 38%\|###8 \| 25/65 [00:33<00:56, 1.41s/it]
	Batches: 40%\|#### \| 26/65 [00:34<00:54, 1.41s/it]
	Batches: 42%\|####1 \| 27/65 [00:36<00:53, 1.41s/it]
	Batches: 43%\|####3 \| 28/65 [00:37<00:53, 1.44s/it]
	Batches: 45%\|####4 \| 29/65 [00:39<00:51, 1.43s/it]
	Batches: 46%\|####6 \| 30/65 [00:40<00:50, 1.43s/it]
	Batches: 48%\|####7 \| 31/65 [00:42<00:50, 1.47s/it]
	Batches: 49%\|####9 \| 32/65 [00:43<00:48, 1.48s/it]
	Batches: 51%\|##### \| 33/65 [00:44<00:46, 1.47s/it]
	Batches: 52%\|#####2 \| 34/65 [00:46<00:44, 1.45s/it]
	Batches: 54%\|#####3 \| 35/65 [00:47<00:43, 1.44s/it]
	Batches: 55%\|#####5 \| 36/65 [00:49<00:41, 1.43s/it]
	Batches: 57%\|#####6 \| 37/65 [00:50<00:40, 1.43s/it]
	Batches: 58%\|#####8 \| 38/65 [00:52<00:39, 1.46s/it]
	Batches: 60%\|###### \| 39/65 [00:53<00:37, 1.45s/it]
	Batches: 62%\|######1 \| 40/65 [00:55<00:35, 1.44s/it]
	Batches: 63%\|######3 \| 41/65 [00:56<00:34, 1.45s/it]
	Batches: 65%\|######4 \| 42/65 [00:57<00:33, 1.46s/it]
	Batches: 66%\|######6 \| 43/65 [00:59<00:32, 1.46s/it]
	Batches: 68%\|######7 \| 44/65 [01:00<00:30, 1.45s/it]
	Batches: 69%\|######9 \| 45/65 [01:02<00:28, 1.45s/it]
	Batches: 71%\|####### \| 46/65 [01:03<00:28, 1.49s/it]
	Batches: 72%\|#######2 \| 47/65 [01:05<00:26, 1.49s/it]
	Batches: 74%\|#######3 \| 48/65 [01:06<00:24, 1.46s/it]
	Batches: 75%\|#######5 \| 49/65 [01:08<00:23, 1.46s/it]
	Batches: 77%\|#######6 \| 50/65 [01:09<00:21, 1.45s/it]
	Batches: 78%\|#######8 \| 51/65 [01:11<00:20, 1.45s/it]
	Batches: 80%\|######## \| 52/65 [01:12<00:18, 1.45s/it]
	Batches: 82%\|########1 \| 53/65 [01:14<00:17, 1.45s/it]
	Batches: 83%\|########3 \| 54/65 [01:15<00:15, 1.45s/it]
	Batches: 85%\|########4 \| 55/65 [01:16<00:14, 1.48s/it]
	Batches: 86%\|########6 \| 56/65 [01:18<00:13, 1.51s/it]
	Batches: 88%\|########7 \| 57/65 [01:20<00:12, 1.52s/it]
	Batches: 89%\|########9 \| 58/65 [01:21<00:10, 1.50s/it]
	Batches: 91%\|######### \| 59/65 [01:23<00:09, 1.50s/it]
	Batches: 92%\|#########2\| 60/65 [01:24<00:07, 1.49s/it]
	Batches: 94%\|#########3\| 61/65 [01:25<00:05, 1.47s/it]
	Batches: 95%\|#########5\| 62/65 [01:27<00:04, 1.47s/it]
	Batches: 97%\|#########6\| 63/65 [01:28<00:02, 1.48s/it]
	Batches: 98%\|#########8\| 64/65 [01:30<00:01, 1.48s/it]
	Batches: 100%\|##########\| 65/65 [01:31<00:00, 1.34s/it]
	Batches: 100%\|##########\| 65/65 [01:31<00:00, 1.41s/it]
	shape (2069, 384) (91.6s)

	Grid search: 4 x 4 = 16 kombinasi
	Grid 1: fit selesai (19.5s)
	[ 1] topics=140 noise= 19.6% sil=0.0466 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
	Grid 2: fit selesai (8.8s)
	[ 2] topics=64 noise= 23.3% sil=0.0482 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
	Grid 3: fit selesai (8.4s)
	[ 3] topics=40 noise= 24.1% sil=0.0395 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
	Grid 4: fit selesai (8.5s)
	[ 4] topics=36 noise= 23.4% sil=0.0569 umap={'n_neighbors': 10, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
	Grid 5: fit selesai (9.3s)
	[ 5] topics=135 noise= 23.6% sil=0.0427 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
	Grid 6: fit selesai (9.0s)
	[ 6] topics=58 noise= 28.4% sil=0.0467 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
	Grid 7: fit selesai (9.4s)
	[ 7] topics=36 noise= 23.8% sil=0.0406 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
	Grid 8: fit selesai (9.0s)
	[ 8] topics=31 noise= 29.2% sil=0.0478 umap={'n_neighbors': 15, 'n_components': 5, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
	Grid 9: fit selesai (10.5s)
	[ 9] topics=125 noise= 24.8% sil=0.0538 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 5, 'min_samples': 3}
	Grid 10: fit selesai (9.3s)
	[10] topics=52 noise= 26.0% sil=0.0465 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 10, 'min_samples': 5}
	Grid 11: fit selesai (8.6s)
	[11] topics=32 noise= 18.1% sil=0.0262 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 5}
	Grid 12: fit selesai (8.6s)
	[12] topics=32 noise= 22.4% sil=0.0349 umap={'n_neighbors': 15, 'n_components': 10, 'min_dist': 0.0} hdb={'min_cluster_size': 15, 'min_samples': 10}
	Grid 13: fit selesai (10.6s)
	[13] topics=102 noise= 30.5% sil=0.0442 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 5, 'min_samples': 3}
	Grid 14: fit selesai (8.9s)
	[14] topics=48 noise= 30.9% sil=0.0468 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 10, 'min_samples': 5}
	Grid 15: fit selesai (9.0s)
	[15] topics=31 noise= 29.3% sil=0.0596 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 15, 'min_samples': 5}
	Grid 16: fit selesai (9.3s)
	[16] topics=22 noise= 27.0% sil=0.0455 umap={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} hdb={'min_cluster_size': 15, 'min_samples': 10}

	Best config sil=0.0596 UMAP={'n_neighbors': 25, 'n_components': 5, 'min_dist': 0.1} HDBSCAN={'min_cluster_size': 15, 'min_samples': 5}

	Dokumen EN: 989 \| ID: 1080

	=== EN Clustering ===
	Loading all-mpnet-base-v2...
	Encoding 989 dokumen dengan all-mpnet-base-v2...

	Batches: 0%\| \| 0/31 [00:00<?, ?it/s]
	Batches: 3%\|3 \| 1/31 [00:20<10:06, 20.21s/it]
	Batches: 6%\|6 \| 2/31 [00:36<08:42, 18.02s/it]
	Batches: 10%\|9 \| 3/31 [00:52<08:00, 17.16s/it]
	Batches: 13%\|#2 \| 4/31 [01:09<07:36, 16.90s/it]
	Batches: 16%\|#6 \| 5/31 [01:26<07:22, 17.01s/it]
	Batches: 19%\|#9 \| 6/31 [01:42<06:57, 16.71s/it]
	Batches: 23%\|##2 \| 7/31 [01:58<06:37, 16.55s/it]
	Batches: 26%\|##5 \| 8/31 [02:13<06:05, 15.88s/it]
	Batches: 29%\|##9 \| 9/31 [02:28<05:47, 15.79s/it]
	Batches: 32%\|###2 \| 10/31 [02:42<05:13, 14.95s/it]
	Batches: 35%\|###5 \| 11/31 [02:53<04:40, 14.05s/it]
	Batches: 39%\|###8 \| 12/31 [03:05<04:10, 13.19s/it]
	Batches: 42%\|####1 \| 13/31 [03:18<03:58, 13.24s/it]
	Batches: 45%\|####5 \| 14/31 [03:30<03:39, 12.90s/it]
	Batches: 48%\|####8 \| 15/31 [03:40<03:12, 12.01s/it]
	Batches: 52%\|#####1 \| 16/31 [03:50<02:51, 11.42s/it]
	Batches: 55%\|#####4 \| 17/31 [04:00<02:32, 10.87s/it]
	Batches: 58%\|#####8 \| 18/31 [04:11<02:21, 10.87s/it]
	Batches: 61%\|######1 \| 19/31 [04:23<02:15, 11.32s/it]
	Batches: 65%\|######4 \| 20/31 [04:32<01:56, 10.58s/it]
	Batches: 68%\|######7 \| 21/31 [04:41<01:42, 10.26s/it]
	Batches: 71%\|####### \| 22/31 [04:53<01:35, 10.58s/it]
	Batches: 74%\|#######4 \| 23/31 [05:05<01:28, 11.11s/it]
	Batches: 77%\|#######7 \| 24/31 [05:22<01:30, 12.98s/it]
	Batches: 81%\|######## \| 25/31 [05:34<01:16, 12.69s/it]
	Batches: 84%\|########3 \| 26/31 [05:45<01:00, 12.11s/it]
	Batches: 87%\|########7 \| 27/31 [05:58<00:48, 12.23s/it]
	Batches: 90%\|######### \| 28/31 [06:10<00:36, 12.18s/it]
	Batches: 94%\|#########3\| 29/31 [06:23<00:24, 12.42s/it]
	Batches: 97%\|#########6\| 30/31 [06:30<00:11, 11.01s/it]
	Batches: 100%\|##########\| 31/31 [06:37<00:00, 9.68s/it]
	Batches: 100%\|##########\| 31/31 [06:37<00:00, 12.82s/it]
	shape (989, 768) (397.6s)
	S3_EN: fit selesai (3.3s)

	=== ID Clustering ===
	Loading firqaaa/indo-sentence-bert-base...
	Encoding 1080 dokumen dengan firqaaa/indo-sentence-bert-base...

	Batches: 0%\| \| 0/34 [00:00<?, ?it/s]
	Batches: 3%\|2 \| 1/34 [00:21<12:01, 21.86s/it]
	Batches: 6%\|5 \| 2/34 [00:42<11:11, 20.99s/it]
	Batches: 9%\|8 \| 3/34 [01:01<10:31, 20.37s/it]
	Batches: 12%\|#1 \| 4/34 [01:21<09:57, 19.90s/it]
	Batches: 15%\|#4 \| 5/34 [01:40<09:36, 19.87s/it]
	Batches: 18%\|#7 \| 6/34 [02:00<09:12, 19.74s/it]
	Batches: 21%\|## \| 7/34 [02:19<08:50, 19.66s/it]
	Batches: 24%\|##3 \| 8/34 [02:40<08:35, 19.84s/it]
	Batches: 26%\|##6 \| 9/34 [02:56<07:51, 18.85s/it]
	Batches: 29%\|##9 \| 10/34 [03:11<06:59, 17.50s/it]
	Batches: 32%\|###2 \| 11/34 [03:24<06:09, 16.07s/it]
	Batches: 35%\|###5 \| 12/34 [03:37<05:37, 15.33s/it]
	Batches: 38%\|###8 \| 13/34 [03:51<05:09, 14.74s/it]
	Batches: 41%\|####1 \| 14/34 [04:03<04:39, 13.97s/it]
	Batches: 44%\|####4 \| 15/34 [04:14<04:07, 13.04s/it]
	Batches: 47%\|####7 \| 16/34 [04:25<03:43, 12.43s/it]
	Batches: 50%\|##### \| 17/34 [04:35<03:20, 11.79s/it]
	Batches: 53%\|#####2 \| 18/34 [04:46<03:02, 11.44s/it]
	Batches: 56%\|#####5 \| 19/34 [04:56<02:44, 10.99s/it]
	Batches: 59%\|#####8 \| 20/34 [05:05<02:29, 10.68s/it]
	Batches: 62%\|######1 \| 21/34 [05:15<02:14, 10.37s/it]
	Batches: 65%\|######4 \| 22/34 [05:24<02:00, 10.02s/it]
	Batches: 68%\|######7 \| 23/34 [05:35<01:50, 10.07s/it]
	Batches: 71%\|####### \| 24/34 [05:45<01:40, 10.04s/it]
	Batches: 74%\|#######3 \| 25/34 [05:53<01:25, 9.52s/it]
	Batches: 76%\|#######6 \| 26/34 [06:04<01:20, 10.07s/it]
	Batches: 79%\|#######9 \| 27/34 [06:13<01:08, 9.80s/it]
	Batches: 82%\|########2 \| 28/34 [06:21<00:55, 9.26s/it]
	Batches: 85%\|########5 \| 29/34 [06:30<00:45, 9.19s/it]
	Batches: 88%\|########8 \| 30/34 [06:38<00:34, 8.73s/it]
	Batches: 91%\|#########1\| 31/34 [06:46<00:25, 8.41s/it]
	Batches: 94%\|#########4\| 32/34 [06:56<00:17, 8.98s/it]
	Batches: 97%\|#########7\| 33/34 [07:02<00:08, 8.21s/it]
	Batches: 100%\|##########\| 34/34 [07:05<00:00, 6.64s/it]
	Batches: 100%\|##########\| 34/34 [07:05<00:00, 12.53s/it]
	2026-06-16 16:43:20,297 - BERTopic - WARNING: Using a custom list
	of topic assignments may lead to errors if topic reduction
	techniques are used afterwards. Make sure that manually assigning
	topics is the last step in the pipeline.Note that topic embeddings
	will also be created through weightedc-TF-IDF embeddings instead
	of centroid embeddings.
	2026-06-16 16:43:20,475 - BERTopic - WARNING: Using a custom list
	of topic assignments may lead to errors if topic reduction
	techniques are used afterwards. Make sure that manually assigning
	topics is the last step in the pipeline.Note that topic embeddings
	will also be created through weightedc-TF-IDF embeddings instead
	of centroid embeddings.
	shape (1080, 768) (425.9s)
	S3_ID: fit selesai (3.6s)

	Reduce outliers...

	EN: 18 topik (max_id=17) \| ID: 63 topik (offset=18)

	Menyimpan artefak (dengan backup)...
	saved embeddings_en.npy
	saved embeddings_id.npy
	backup: embeddings.npy.bak.1781603000
	saved embeddings.npy
	backup: publication_ids.npy.bak.1781603000
	saved publication_ids.npy
	backup: bt_en.bak.1781603000
	saved bt_en
	backup: bt_id.bak.1781603000
	saved bt_id
	backup: topic_info.json.bak.1781603000
	saved topic_info.json
	{
	"en": {
	"model_name": "all-mpnet-base-v2",
	"n_topics": 18,
	"max_topic_id": 17
	},
	"id": {
	"model_name": "firqaaa/indo-sentence-bert-base",
	"n_topics": 63,
	"id_topic_offset": 18
	},
	"data": {
	"unit": "unique_publication",
	"n_source_rows": 2069,
	"n_unique_publications": 2069,
	"n_publication_lecturer_edges": 3006,
	"duplicate_publication_rows_merged": 0
	},
	"preprocessing": {
	"filter_low_context_abstracts": true,
	"min_abstract_words": 30,
	"min_abstract_unique_words": 10,
	"min_abstract_unique_ratio": 0.25,
	"max_abstract_token_ratio": 0.3,
	"n_rows_before_preprocessing": 2073,
	"n_rows_after_preprocessing": 2069,
	"n_rows_removed_preprocessing": 4,
	"n_rows_removed_empty_text": 0,
	"n_rows_removed_low_context": 4,
	"n_rows_empty_or_placeholder_abstract": 0
	},
	"rejected_reason_counts": {
	"abstract_words<30": 3,
	"abstract_words<30; unique_words<10": 1
	}
	}
	saved lecturer_embeddings.npy
	saved lecturer_names.npy
	backup: publications.csv.bak.1781603001
	saved publications.csv (2069 rows)
	backup: metadata.csv.bak.1781603001
	saved metadata.csv (2069 rows)
	saved publication_lecturers.csv (3006 rows)
	saved publication_topics.csv (2069 rows)
	backup: lecturer_profiles.csv.bak.1781603001
	saved lecturer_profiles.csv (101 rows)

	Topic distribution:
	topic_id
	0 103
	1 94
	2 101
	3 90
	4 88
	5 88
	6 58
	7 42
	8 50
	9 37
	10 40
	11 37
	12 51
	13 26
	14 24
	15 23
	16 21
	17 16
	18 81
	19 43
	Name: count, dtype: int64

	Selesai. embeddings rows = 2069 \| publications rows = 2069