Evaluating an AI translation provider means reviewing the entire workflow: sources, technology, terminology, specialists, quality controls, security and the ability to learn from errors. Quality cannot be judged from the model name or a demonstration containing a few simple sentences.

A useful assessment begins with your real content, markets and consequences of failure.

1. Start with the problem, not the tool

Define volume, languages, formats, frequency, channels and deadlines. Separate repetitive content, marketing, technical documentation, conversion pages and regulated material. Each group may need a different workflow.

A credible provider will ask about intended use and risk before proposing a technology or level of automation.

2. Request a source map

The provider should explain which information the AI may use and how conflicts or gaps will be handled. Sources may include translation memories, glossaries, brand guides, catalogues, documentation and approved content.

  • Which source takes priority?
  • What may be inferred?
  • How are unauthorised claims blocked?
  • How do source updates reach every language?

These decisions belong inside an AI content system, not inside an undocumented prompt.

3. Test terminology and brand voice

Ask how glossaries, memories and style rules are created and maintained. Brand voice should become verifiable instructions and examples for each language and channel.

The sample should include difficult terms, ambiguity and market variants. Simple sentences do not demonstrate operational capability.

4. Require risk-based review

Not every asset needs full human revision, but none should be automated without criteria. The provider should classify risk by visibility, regulation, commercial impact, sensitivity, permanence and reversibility.

Human oversight should specify who reviews, what they validate and when an issue is escalated.

5. Examine the QA system

Ask which automated and human checks happen before delivery or publication. Controls may cover figures, omissions, terminology, variables, tags, format and deviation from the source.

The AI content evaluation checklist provides a fuller framework. A provider should also show how issues are recorded and turned into improvements.

6. Evaluate the specialists

Find out who designs the workflow, who reviews each language and whether they have relevant subject experience. “Human in the loop” is not sufficient without clear responsibility, scope and acceptance criteria.

7. Review security and data handling

Request details about tools, data location and retention, access, model training and incident procedures. The appropriate environment depends on the sensitivity of the material.

8. Check integration and traceability

The provider should explain how content is received, identifiers are preserved, versions are managed and results return to the CMS, PIM, TMS or repository. Ask how every version can be traced to its source, model, glossary and review.

9. Agree metrics and acceptance criteria

Define “approved” before the pilot. Useful metrics include first-pass approval, errors by type, rework, terminology consistency and cycle time. Automated scores can help classify outputs, but they do not replace every human decision.

10. Use a representative pilot

Ask each provider to work on the same representative sample, using the same approved sources, languages, required fields and review depth. Include straightforward and difficult content: ambiguous wording, numbers, critical terms, formatting and missing source information. Compare results separately where a language or content type needs different controls.

Agree what counts as accepted before work starts. Name the person who approves each language and the specialist who resolves product or other sensitive claims. Missing required information, altered figures, unsupported claims and broken variables should hold the affected asset until the issue is resolved. Record the reason for rejection, the correction and who approved it.

Compare the full pilot cost per accepted asset. Include source preparation, setup, generation or translation, review, rework, exception handling and delivery checks. Keep rejected assets in the cost total; count only assets that meet the agreed acceptance criteria in the denominator. Record setup and recurring costs separately so you can assess the next batch as well as the pilot. If no assets are accepted, record that outcome; a unit cost cannot yet be calculated.

Before signing off, have a named owner compare the approved version with the output in its destination: the correct fields, target language, figures, variables and links. A file that passes language review may still be incomplete or misplaced in the CMS. Record the approved version and the destination checked.

Our published Chachifarma case study describes an initial batch of 30 products, together with a glossary, prompts, product rules and QA instructions. It also identifies composition data as requiring specific human verification. The criterion I would take from that process is to test both the content and its controls before increasing volume. The case does not establish what another catalogue will cost or promise the same outcome.

Review how the provider raises source questions, documents decisions and applies feedback. Then test whether the next comparable batch meets the same acceptance criteria with fewer defects or less rework. A faster draft or a higher automated score does not establish that improvement on its own.

Provider evaluation table

Area Evidence to request Warning sign
Sources Map and priorities AI fills gaps without control
Terminology Glossary and maintenance A list without context or ownership
Risk Classification and review level The same workflow for everything
QA Checks and issue records Spellchecking only
Specialists Roles and experience Undefined human review
Security Processing and retention Generic answers
Metrics Thresholds and actions One score presented as a guarantee

Questions to ask before selecting a provider

  • Which content would you advise us not to automate?
  • What happens when sources conflict?
  • Which errors always block publication?
  • Who approves each language?
  • How does feedback change future batches?
  • What do we receive in addition to final copy?

The proposal should explain the system

A sound proposal describes scope, sources, technology, specialists, controls, deliverables, responsibilities and limitations. Speed or savings alone cannot demonstrate that content will be ready to publish.

To compare these criteria with a governed workflow, explore our AI-powered translation service.