TianRW

/

CEAEval-Model

like 0

Thanks for releasing the code, model and dataset together — having all three in one place is genuinely useful.

One thing needed to compare against Table 2 is not in the release: the test/train split.

§4.1 states:

16.1 hours of speech, split into 14.65 hours for training and 1.45 hours for testing, with strict story-level separation and no overlap in narratives, characters, or scenes.

In TianRW/CEAEval-Data, manual_annotation/story_infos.json records the following per segment:

start, end, chunk_idx, chunk_start, group, filename, speaker, content,
score, intonation, rhythm, emotion, effect, bgm, difficulty, gender,
age, origin_len, audio_path

None of these marks the split, and there is no separate split file in the dataset repo (manual_annotation/ contains audio_segments/, novel_match.json, story_infos.json, whole_novel/). The dataset README describes the annotation schema and the reliability statistics but does not mention a split either.

Since the separation is story-level, a plain text file listing the test-set story_id values would be enough (there are 82 distinct story_id values in the released subset). Any number computed on the released data currently mixes training and evaluation material, so results cannot be placed on the same footing as the paper's.

An alternative that would serve the same purpose

If publishing the split is not straightforward, per-utterance predictions on the test set — a CSV of segment identifier and predicted score — would also work, and arguably better: it lets anyone recompute the reported metrics directly and check whether a discrepancy originates in inference or in scoring, without needing to run the model at all.

Three smaller points, if you are updating the release anyway

Neither of these blocks anything; both are things a reader has to guess at.

1. Which plan wording the judge expects. The same six rhythm categories appear with three different spellings across the release:

Intonation has the same three-way split, and emotion is free-form Chinese in the dataset but free-form English in the released planner prompt, with no mapping between them. §5.1 says the best-performing prompt language is reported per model, but Table 2 does not mark which language each row used. One verbatim ideal dict as fed to the judge would settle all of this.

2. The judge prompt in the repo and the one in Appendix D are different prompts. Appendix D asks the judge to analyse "emotion, rhythm, intonation, paralinguistic vocalizations, recording condition, and overall naturalness" — six aspects — and names the fourth plan field recording_condition. The prompt built by build_scorer_messages (data_processor.py:222) asks for four:

Rate this speech (0.0–5.0) by emotion, rhythm, intonation, and effect match.
...
Ideal Performance: emotion={...}, rhythm={...}, intonation={...}, effect={...}

Appendix D also shows commented w/ CoT and w/o CoT variants, but the repo emits CoT unconditionally, so the w/o CoT column of Table 2 cannot be produced with the released code as-is. Could the system prompt actually used for each column be included?

3. CEAEVAL_FORCE_M4A defaults to on. In data_processor.py:126:

_FORCE_M4A_DEFAULT = os.environ.get("CEAEVAL_FORCE_M4A", "1").lower() not in ("0", "false", "no", "")

Under this default, load_audio_ffmpeg re-encodes any non-.m4a input to 64 kbps AAC before decoding. Every segment in manual_annotation is .wav, so out of the box each clip goes through a lossy round-trip. If that is not what the reported numbers used, it may be worth flipping the default or noting it in the README.

Thanks again for the release.