long sequences compatibility
Thanks for your great works!
I ve been struggling on a large genome, whose first chromosome even contains 1.4gb bases in only one chromosome, the code continuously said overflow the int32 Max. I wonder if I can only split the chromosome?
Traceback (most recent call last):
File "/f2/software/Carbon-A-1.2B/annotation_pipeline/src/model_inference.py", line 1067, in
main()
File "/f2/software/Carbon-A-1.2B/annotation_pipeline/src/model_inference.py", line 1041, in main
write_prob_parquet(
File "/f2/software/Carbon-A-1.2B/annotation_pipeline/src/model_inference.py", line 291, in write_prob_parquet
pq.write_table(table, output_path, compression="zstd", use_dictionary=False)
File "/home/yty/anaconda3/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 2055, in write_table
writer.write_table(table, row_group_size=row_group_size)
File "/home/yty/anaconda3/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 1216, in write_table
self.writer.write_table(table, row_group_size=row_group_size)
File "pyarrow/_parquet.pyx", line 2473, in pyarrow._parquet.ParquetWriter.write_table
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Uncompressed data page size overflows INT32_MAX. Size:2794939026
Command exited with non-zero status 1
Hi! Thanks for your feedback and sorry for the late reply!
We just made a quick fix, the script can now handle max chromosome length up to 2gb.
We will also make a more refined fix that completely removes the length constraint.
Thanks again for your feedback.
Qiuyi
谢谢您的更新！另有一些log呈现上的小建议：在Inference时log输出的进度是以染色体个数为单位的，不能很直观地监控目前到底完成了多少序列；我不太了解模型具体的运行逻辑，如果是沿染色体序列线性推进的话，建议可以添加一个log层级，每分钟、或者每处理10Mb或者其它合适的长度，在log中输出一次已遍历的序列长度。
另外我们这边使用的网络不太方便，如果您方便的话希望后续更新可以同步到一些能够便于访问的公开平台，非常感谢！
feel free to contact me via email :)