🌎 GenAI G11n Quality Assessment β€” fr-FR Evaluation Results v1.0

#11
by AndresCas - opened
Dilato Infotech Limited org

French AI Quality Assessment Banner

French (fr-FR) β€” G11n Quality Assessment

We are sharing the results of our Globalization (G11n) Quality Assessment for French as used in France (fr-FR), evaluating how four AI model configurations handle localized language, cultural context, instructions, and multimodal scenarios.

The evaluation was conducted using the DilatoMX G11n GenAI Assessment Model, with a focus on identifying overall quality performance and specific globalization issues that can affect the experience of French users.


πŸ“‹ Assessment Execution

The evaluation was conducted by two QA evaluators using the fr-FR benchmark and defined Success Criteria. Standard prompts were generally executed three times per evaluator to observe response quality and consistency across repeated runs.

For supported capabilities, Image Generation and Multiple Generation scenarios were evaluated using one run per evaluator. However, not all capabilities were evaluated for every model configuration. When a category or subcategory was not evaluated, it was excluded from the applicable execution totals rather than counted as a failure.

After execution, valid issues were reviewed, assigned a severity, and attributed to the quality area they actually affected. This made it possible to distinguish between issues observed during a prompt execution and the quality dimension ultimately impacted.

The analysis distinguishes between unique issues and failed instances across runs. If a single reported issue occurs in three failed runs, it contributes one unique issue and three failed instances.

The final G11n Quality Score is based on the Success Criteria (MET) results. Execution outcomes, consistency patterns, and issue attribution provide additional context for interpreting the findings.

πŸ€– AI Models Evaluated

The assessment covered the following model configurations:

AI Model Model / Configuration Evaluated
ChatGPT GPT-5.6 Luna
Gemini 3.5 Flash Lite
Copilot GPT-5 / Smart Mode
DeepSeek V4.1

The results correspond to these specific configurations during the evaluation period. They should not automatically be interpreted as results for later model versions or different configurations.


πŸ’¬ ChatGPT β€” GPT-5.6 Luna

ChatGPT fr-FR

ChatGPT achieved a G11n Quality Score of 97.97, classified as Excellent Globalization. Language & Grammar scored 98.23, Instruction & Response Coherence 97.21, Cultural Adaptation 97.67, and Multimodal Consistency 98.78.

The main quality areas highlighted in the issue analysis were Audio Generation (30.00%), Legal-Regulatory Compliance (20.00%), and Localized Formats (18.33%).

The evaluation included 700 executions, with 617 passes (88.14%) and 83 failures (11.86%). Across 220 prompt-evaluator combinations, 187 produced consistent-pass results across all three runs, 22 were consistent failures, and 11 had variable results.

Issue analysis: The evaluation recorded 37 unique issues, corresponding to 83 failed instances across runs. Issue attribution classified 17 as Primary (45.95%) and 20 as Secondary (54.05%).

Capability coverage: Image Generation and Multiple Generation were included with one run per evaluator where supported.


✨ Gemini β€” 3.5 Flash Lite

Gemini fr-FR

Gemini achieved a G11n Quality Score of 97.82, classified as Excellent Globalization. Multimodal Consistency was its strongest dimension (99.05), while Cultural Adaptation recorded the lowest score (96.00).

The main areas where quality issues were identified were Audio Generation (33.33%), Legal-Regulatory Compliance (20.00%), and Spelling & Punctuation (15.00%). Output Accuracy also recorded an 11.67% failure rate.

The evaluation included 700 executions, with 625 passes (89.29%) and 75 failures (10.71%). Across 220 prompt-evaluator combinations with repeated runs, 186 produced consistent-pass results, 17 were consistent failures, and 17 had variable results.

Issue analysis: The evaluation recorded 39 unique issues, corresponding to 75 failed instances across runs. Issue attribution classified 26 as Primary (66.67%) and 13 as Secondary (33.33%).

Capability coverage: Image Generation and Multiple Generation used one run per evaluator where supported.


πŸ‹ DeepSeek β€” V4.1

DeepSeek fr-FR

DeepSeek achieved a G11n Quality Score of 97.93, classified as Excellent Globalization. Cultural Adaptation was its strongest dimension (99.50), while Multimodal Consistency recorded the lowest score (96.00).

The largest concentrations of quality issues were associated with Spelling & Punctuation (30.00%), Localized Formats (23.33%), and Output Accuracy (21.67%). Code Generation and Semantic Accuracy also recorded 10.00% failure rates.

The evaluation included 600 executions, with 534 passes (89.00%) and 66 failures (11.00%). Across 200 prompt-evaluator combinations, 172 produced consistent-pass results, 17 were consistent failures, and 11 had variable results.

Issue analysis: The evaluation recorded 30 unique issues, corresponding to 68 failed instances across runs. Issue attribution classified 17 as Primary (56.67%) and 13 as Secondary (43.33%).

Capability coverage: Audio Generation, Image Generation, and Multiple Generation were not evaluated for this model and were excluded from the execution totals. These exclusions should not be interpreted as failed tests.


πŸͺŸ Copilot β€” GPT-5 / Smart Mode

Copilot fr-FR

Copilot achieved a G11n Quality Score of 98.35, classified as Excellent Globalization. Language & Grammar was its strongest dimension (99.89), while Instruction & Response Coherence recorded the lowest score (97.33).

The primary areas where quality issues were identified were Image Generation (25.00%), Spelling & Punctuation (13.33%), and Localized Formats (13.33%). Semantic Accuracy and Code Generation also recorded 11.67% failure rates.

The evaluation included 620 executions, with 573 passes (92.42%) and 47 failures (7.58%). Across 200 prompt-evaluator combinations with repeated runs, 178 produced consistent-pass results, 9 were consistent failures, and 13 had variable results.

Issue analysis: The evaluation recorded 27 unique issues, corresponding to 47 failed instances across runs. Issue attribution classified 11 as Primary (40.74%) and 16 as Secondary (59.26%).

Capability coverage: Image Generation used one run per evaluator. Audio Generation and Multiple Generation were not evaluated and were excluded from the execution totals.


πŸ”Ž Cross-Model Observations

Although all four model configurations achieved an Excellent Globalization classification, the issue analysis revealed meaningful differences in the types of quality challenges encountered.

1. All four configurations achieved high G11n Quality Scores

The G11n Quality Scores ranged from 97.82 to 98.35. These scores are based on Success Criteria results and should be interpreted separately from execution pass rates.

2. The leading issue areas varied by model

Model Main Quality Areas Identified Impact Rate
ChatGPT Audio Generation 30.00%
ChatGPT Legal-Regulatory Compliance 20.00%
ChatGPT Localized Formats 18.33%
Gemini Audio Generation 33.33%
Gemini Legal-Regulatory Compliance 20.00%
Gemini Spelling & Punctuation 15.00%
DeepSeek Spelling & Punctuation 30.00%
DeepSeek Localized Formats 23.33%
DeepSeek Output Accuracy 21.67%
Copilot Image Generation 25.00%
Copilot Spelling & Punctuation 13.33%
Copilot Localized Formats 13.33%

These findings illustrate that globalization quality extends beyond linguistic fluency. Depending on the model, the most visible challenges may concern spelling, localized formats, output accuracy, regulatory context, or multimodal output.

3. Repeated-run consistency provides additional context

Consistent-pass results ranged from 84.55% to 89.00% among the reported prompt-evaluator combinations with repeated runs.

Consistent failures identify scenarios where a quality issue recurred across all runs. Variable results indicate that the same prompt-evaluator combination produced different outcomes across runs, highlighting response stability as another dimension of quality.

4. Capability support affects evaluation coverage

Not every model configuration supported or underwent evaluation for every benchmark scenario. DeepSeek's Audio Generation, Image Generation, and Multiple Generation scenarios were not evaluated. Copilot's Audio Generation and Multiple Generation scenarios were also not evaluated.

These scenarios were excluded from the applicable execution totals and should not be treated as failed tests.


πŸ“Š Final Results

Model G11n Quality Score Pass Fail Execution Pass Rate Unique Issues Failed Instances
ChatGPT GPT-5.6 Luna 97.97 617 83 88.14% 37 83
Gemini 3.5 Flash Lite 97.82 625 75 89.29% 39 75
DeepSeek V4.1 97.93 534 66 89.00% 30 68
Copilot GPT-5 / Smart Mode 98.35 573 47 92.42% 27 47

The G11n Quality Score and execution pass rate represent different measurements: the former is based on Success Criteria (MET) results, while the latter summarizes pass/fail outcomes across applicable executions. They should not be interpreted as interchangeable metrics.

Unique issue counts and failed-instance counts are also separate measures. Failed-instance totals are taken from each model's Analysis sheet; one issue may account for multiple failed instances across runs.


πŸ“š Explore the Assessment

The benchmark, evaluation model, Success Criteria, and supporting documentation are available in the DilatoMX G11n GenAI Assessment Model repository.

We welcome feedback and discussion from the G11n, localization, QA, and AI evaluation communities as we continue exploring the challenges of delivering high-quality AI experiences across locales.


Dilato MX β€” Globalization & Quality Engineering

Sign up or log in to comment