AbstractPhil/qwen-symbolic-thinking-image-training

Qwen symbolic thinking: image training

Teaching a Qwen3.5 vision-language model to think about a picture in writing, the way the rows of thought-symbolic-image-captions do: a short chain of plain-language thoughts, one step each (see, count, light, mood ...), ending in a caption.

see: One man in an apron stands in front of two ovens.
place: A working kitchen; pans hang on the wall behind him.
mood: A kitchen at work reads as warm and busy.
caption: A man in an apron standing by two ovens in a kitchen.

The code lives at github.com/AbstractEyes/qwen-symbolic-thinking; this repository holds the data files, the screens and the adapters.

What is here

path	what
screens/inputs/screen_v1.json	the behaviour screen's rows: 60 picture rows (COCO 2017 images named by id, fetched from the COCO site at run time) and 40 specification rows, with two worked examples per depth
sft/sft_v1.json	the first fine-tune set: 978 picture rows outside the screen and 978 specification rows (1,956 items)
screens/<stamp>/	a screen's answers (results.jsonl: the text, the parse and the reads per answer), per-model summaries, summary.md, env.json
screens/<stamp>_throughput*/	a throughput sweep: rows an hour, seconds a row, tokens a second and peak memory per batch size
lora/<model>/r<rank>/<stamp>_<arm>/	LoRA adapters shipped during and after training (step_N/, final/), with train_summary.json
sft/pictures_v2.json, sft/drawn_deep_v2.json, synth/	the whole-row training sets and the synthetic rows (below)
runs/queue_lane*.json, runs/done/*.json	the queues the cards worked through unattended, and each run's record

The base models, screened

Qwen3.5-9B and Qwen3.5-4B, shown a picture and asked plainly (an RTX PRO 6000, greedy decoding, thinking off). Reads per answer: the form held (step: text lines and a caption), the number of thoughts against the depth asked for, the share of the row's specification said, and two habits the dataset bans (defining a word instead of looking; remarking on the task).

model	form	format held	thoughts in range	spec words said	s a row (batch 4)
Qwen3.5-9B	instructions + two examples	1.00	0.98	0.43	0.56
Qwen3.5-9B	instructions only	1.00	0.43	0.47	1.42
Qwen3.5-4B	instructions + two examples	1.00	0.62	0.37	0.91
Qwen3.5-4B	instructions only	1.00	0.12	0.48	1.22

The 9B does the behaviour when shown examples; what it lacks is the dataset's own vocabulary and the depth cue, which the fine-tune teaches. The dataset's writers say 0.93 of the specification's words.

Throughput of the 9B (one-line prompt, at most 400 new tokens, the linear-attention kernels on)

batch	rows an hour	tokens a second	peak memory
64	11,957	733	26 GB
128	13,805	848	35 GB
256	16,264	1,002	52 GB

The adapters: the whole row (10-30)

The first adapter (lora/Qwen3.5-9B/r32/20261030_063222, 1,956 items, chain-only answers) learned the form, the depth and the voice in 11.6 minutes but said less of the picture than the writers and wrote no wordings or questions. The second night taught the WHOLE ROW: the thoughts, then the wordings in order (caption, prompt, tags, critique), then one line per question, question: what -> answer [record, chain or unknown], under one-line prompts (full for a picture, spec_full for a specification as records; short and spec_short for the chain and a caption). Three levers were tried alone and together, each arm screened on the same 60 held-out pictures and 40 specifications:

arm	taught from	steps	min	row shape (pictures)	long look whole (specs)	said / backed, chain-only	said / backed, whole row	definitions (specs)	adapter
A_pictures_full	951 picture rows as whole rows (and chain-only), 3 epochs	357	16.4	1.00	0.53	0.18 / 0.24	0.18 / 0.25	0.05	lora/Qwen3.5-9B/r32/20261030_075529_A_pictures_full/final
A2_pictures_2ep	the same, 2 epochs	238	9.4	1.00	0.55	0.18 / 0.28	0.17 / 0.26	0.07	lora/Qwen3.5-9B/r32/20261030_081912_A2_pictures_2ep/final
C_pictures_drawn_deep	pictures + 500 long-look drawn rows, 3 epochs	453	29.8	1.00	1.00	0.17 / 0.26	0.18 / 0.25	0.20	lora/Qwen3.5-9B/r32/20261030_075550_C_pictures_drawn_deep/final
B_pictures_synth	pictures + 2,000 synthetic closer looks, 3 epochs	732	31.0	1.00	0.72	0.25 / 0.31	0.20 / 0.28	0.03	lora/Qwen3.5-9B/r32/20261030_080839_B_pictures_synth/final
E_all	pictures + drawn + synthetic, 3 epochs	828	45.0	1.00	1.00	0.24 / 0.27	0.19 / 0.25	0.17	lora/Qwen3.5-9B/r32/20261030_082833_E_all/final
E_all_2ep	pictures + drawn + synthetic, 2 epochs	552	30.4	1.00	1.00	0.22 / 0.27	0.21 / 0.31	0.10	lora/Qwen3.5-9B/r32/20261030_083947_E_all_2ep/final
E_all_x2	pictures + drawn + synthetic twice, 2 epochs	802	41.2	1.00	1.00	0.24 / 0.30	0.19 / 0.31	0.20	lora/Qwen3.5-9B/r32/20261030_085116_E_all_x2/final
F_all_quick	pictures + drawn + synthetic closer looks + 1,138 synthetic quick looks, 2 epochs	694	35.6	1.00	0.97	0.31 / 0.22	0.18 / 0.27	0.20	lora/Qwen3.5-9B/r32/20261030_092331_F_all_quick/final
the writers' own rows	the dataset	-	-	0.95	1.00	0.23 / 0.32	the same	0.03	-

Reads: row shape = the depth's wordings with a caption and the questions in range and well formed; long look whole = the same on the specification rows (long looks need 6-10 thoughts and 5-8 questions); said = the share of the picture's own captions' content words the chain and caption say, backed = the share of the chain's content words those captions back (the writers' own rows set the level); definitions = rows with a thought that defines a word instead of looking (a habit the dataset bans).

What each lever did: the whole-row target gives the row's shape (every arm, 1.00); the long-look drawn rows give the long look (C and the E arms), at the price of their own definitional habit on specification rows; the synthetic rows give the looking (B meets the writers' level chain-only; the E arms carry it into the whole-row form); two epochs beat three on grounding and on definitions. The synthetic rows are the base 9B's own chains, shown two worked examples, on COCO pictures no row was written about, kept only when they hold the form and the depth, define nothing, remark on nothing, and both say and are backed by the picture's captions: synth/S_synth.json (2,000 closer looks of 3,500 pictures, 13.6 minutes on the card) and synth/S_quick.json (1,138 quick looks of 1,500). Training: plain Adam 2e-4 cosine, rank 32, alpha 64, batch 8 x 2, 2.4-3.9 s a step on an RTX PRO 6000; a trained model answers a picture in 0.4-0.8 s at batch 4.

Throughput of the whole-row form (the E_all_2ep adapter, 256 picture rows, at most 400 new tokens)

batch	rows an hour	tokens a second	peak memory
32	13,228	334.8	22.0 GB
64	17,407	439.8	26.5 GB
128	19,877	499.9	35.4 GB
256	23,856	599.9	53.3 GB

Using an adapter

from peft import PeftModel
from transformers import AutoModelForMultimodalLM, AutoProcessor
base = "Qwen/Qwen3.5-9B"
processor = AutoProcessor.from_pretrained(base)
model = AutoModelForMultimodalLM.from_pretrained(base, dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(model, "AbstractPhil/qwen-symbolic-thinking-image-training",
                                  subfolder="lora/Qwen3.5-9B/r32/20261030_083947_E_all_2ep/final").merge_and_unload()
# the whole row:  "Look at this picture and think it through in writing, a closer look; then write a caption and a prompt
#                  or tags, and the questions a careful looker would still ask, each as `question: what -> answer [record, chain or unknown]`."
# the chain only: "Look at this picture and think it through in writing, a closer look, ending with a caption."
# depths: "a quick look" (1-2 thoughts), "a closer look" (3-5), "a long look" (6-10); thinking off (enable_thinking=False)

The package's qwen_symbolic_thinking.prompts writes these prompts and qwen_symbolic_thinking.chain.parse reads the answers back into thoughts, wordings and questions. The data files behind the arms: sft/pictures_v2.json (951 picture rows with their wordings and questions), sft/drawn_deep_v2.json (500 long-look drawn rows), synth/seeds_v1.json and synth/seeds_simple_v1.json (the pictures the synthetic rows were written about), runs/queue_lane*.json and runs/done/*.json (the queues the cards worked through and each run's record).

License

Apache-2.0, like the Qwen3.5 models. The dataset's rows carry their own licences per row; COCO images are not redistributed.