leary-poken/storage / upload.py
leary-poken's picture
download
raw
769 Bytes
import time
from datetime import datetime
from huggingface_hub import HfApi
REPO_ID = "leary-poken/m-total"
FOLDER_PATH = "/mnt/txt-360-v2/dataset/medium/embed"
INTERVAL = 600 # 10 minutes
api = HfApi()
def upload():
api.upload_folder(
repo_id=REPO_ID,
repo_type="model",
folder_path=FOLDER_PATH,
allow_patterns=["Medium*.npy", "Medium*.json"],
commit_message=f"Periodic upload: {datetime.now():%Y-%m-%d %H:%M:%S}",
)
print("Upload completed successfully.")
while True:
start = time.monotonic()
try:
upload()
except Exception as e:
print(e)
print("Upload failed; will retry next cycle.")
elapsed = time.monotonic() - start
time.sleep(max(0, INTERVAL - elapsed))

Xet Storage Details

Size:
769 Bytes
·
Xet hash:
a6987a9738cdc1f36eaa9d4585da1ed83d1224ebe1e29386ba405965ef442aad

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.