README.md

Cloud notebook runtime blind spots

1. The blind spot I chose

I use Kaggle for long machine learning experiments because GPU compute is limited for me. I did not initially understand the difference between running cells in Kaggle's interactive session and using Save & Run All. After several experiments stopped or became difficult to resume, I tried different approaches and asked friends for help. They pointed me to Save & Run All. Some interrupted runs had checkpoints that let me continue, but others did not. Repeating those runs cost time and limited GPU hours and sometimes forced me to reduce a project's scope.

I tested whether a model could give practical advice about where notebook code runs, what happens when a browser disconnects, what progress is visible during execution, and whether saving a notebook also saves a recoverable training state. An answer can sound plausible while confusing these distinctions. My hypothesis is that standard question answering benchmarks rarely test this combination of platform specific behavior, limited compute, and uncertain connectivity. This is a hypothesis about benchmark coverage, not a result measured here.

2. Evaluation and findings

I evaluated Qwen/Qwen3-4B-Instruct-2507 on 24 prompts. They crossed four environments, local Jupyter, Kaggle interactive, Kaggle Save & Run All, and hosted Colab, with six events: browser closure or laptop sleep, internet loss, mid-run feedback, crash recovery, notebook saving versus checkpointing, and a quick-test-then-full-run workflow. The same generation setup and seed, 42, were used throughout.

The notebook, prompts, and full responses are included here. The notebook export contains code but no embedded outputs, so the response files are the record of model output. The model weights remain at the linked Hugging Face model page rather than being duplicated in this Bucket.

In the accepted working scores, nine of 24 responses received 0 or 1 on a 0 to 3 scale. Four of six Kaggle interactive responses fell in that group, compared with one of six Kaggle background responses. These are descriptive counts from one response per prompt, not an estimate of the model's general accuracy. The review workbook includes the working scores, explanations, and documentation checks. The scoring and some review notes were developed with AI assistance; my direct observations and corrections are identified in the workbook. They should not be presented as 24 independently written human reviews.

Cases 07 and 12 show the most important error to me. Case 07 said Kaggle training ran on the user's laptop, although Kaggle uses remote compute. Case 12 treated an interactive Kaggle run as local and recommended a separate deployment for unattended work, even though Kaggle provides Save & Run All for a separate remote run. Case 18 got that core workflow right when Save & Run All was named in the prompt, but its suggestion to "test locally" was ambiguous and its alerting advice was not established by the documentation I checked. The contrast suggests that the model's advice needs to identify the execution mode, not just recognise the word Kaggle. Kaggle's notebook documentation describes the run modes. Kaggle also offers live logs for background versions, although these are not the same as interactive, cell by cell control.

3. A proposed path forward

I would combine current official documentation with contrast examples. Before giving advice, a system should identify the notebook platform and run mode, retrieve the relevant documentation, and distinguish documented behavior from uncertain or changeable policy. It should explain where code runs, what a disconnect changes, what progress can be monitored, and whether a recoverable training checkpoint was actually saved. If the run mode is missing, it should ask rather than guess.

I would also build contrast pairs that change one important fact, such as interactive Run versus Save & Run All, and test whether the answer changes appropriately. Both Kaggle modes run remotely, but their execution lifecycle and feedback differ. I would hold out new pairs for evaluation. Fine tuning on separately verified examples might be useful later if documentation retrieval still leaves these errors. I have not demonstrated that this proposal fixes the model.

Clear documentation and short tutorials could help people directly, too. Better model advice would not itself make an interrupted experiment recoverable or give a background run interactive cell by cell control. A platform feature that combines durable progress with intermediate feedback is a separate idea.

Files and limitations

File	Contents
cloud-notebook-runtime-blind-spots-evaluation.ipynb	Evaluation code exported from Kaggle, without embedded outputs
evaluation_prompts_all.csv	All 24 prompts
raw_responses_full.csv	Full responses and run metadata in CSV form
raw_responses_full.jsonl	Full responses and run metadata in JSON Lines form
Research_Adjusted_Review.xlsx	Working scores and review notes

This is one model, one response per constructed prompt, and a small evaluation. Platform behavior can change. The Kaggle notebook is private, so the exported notebook and raw response files, not the Kaggle page, are the reproducibility record. Relevant references include the Kaggle notebook documentation, Kaggle live logs, and the Colab FAQ.