1 day ago
The gap looks huge, but what does it actually feel like in practice? Writing novels, or still serving niche needs (dating an AI)?
Try testing it with your two or three most common scenarios for a few rounds each before deciding whether to switch models.
What really sets text capabilities apart often isn't how impressive a single output is, but how few errors and how little drift there is over long tasks. In the short term, everything may seem about the same, but the time cost of repeated revisions and rework adds up—and the longer the task and the more it depends on context, the more obvious that gap becomes.
Corpora and styles vary quite a bit across languages, so performance on Chinese and English writing may not be in sync. Is what you're noticing concentrated in a particular type of writing, or is it about the same across all kinds of tasks?
Looked at from another angle, might niche demands actually be a better test of text capability?
I'm a bit hesitant about the claim that the gap is "huge in practice." With everyday short conversations and simple rewrites, several models may perform quite similarly; what really separates them is long-text structure and style control, and those are exactly the things that aren't easy to judge from a few quick impressions. Perhaps we could first break "text capability" down into a few observable dimensions, and then discuss where the gaps actually lie.
What you mentioned about writing novels and character dialogue actually puts more emphasis on contextual coherence and character consistency—benchmark scores alone may not reflect those. If you only use it occasionally, the difference may not be that obvious; but for long-form writing over time, stability will quickly become apparent.