Rethinking translation QA

After completing two translation tests for a prospective customer, I was given some feedback. It was not what I wanted to hear. From ‘translator is not quite familiar with the industry terminology’ to ‘needs supervision’, the comments were stinging. Why would I feel bothered by an anonymous critique, you’d say? For the same reason a stranger tells you that you don’t know how to run your business.

I wrote back to my prospective customer and expressed my frustration at the inconsistency between the level of the criticism and the kind of “errors” found in my translation tests. The main point I tried to make was that many of the “errors” were preferences of the translators or editors who checked my translations. Weeks later, I received an email expressing concern, approval of my vendor status and an offer to do better at communicating. I replied my prospective customer by saying that, apparently, she takes translation test results only as one of many factors to decide who to hire as a freelancer.  The message reads as follows:

I definitely  do not just use the errors in the sample to determine the approval of a translator.  I take into account many different things.  I even take into account the tone and wording of the e-mails and telephone conversations in general.  That tells me  a lot about a person. A sample of 350 words is hardly enough to base my entire judgment on.

What weight does this have on translation quality control? That error counting does nothing to tell you what you need to know about a freelance translator. I’ve been thinking about the whole business case for implementing translation quality standards, and I think that some in the industry are so focused on finding errors that they fail to see the tree from the woods.

For QA to work in any field, it has to offer practical, cost-effective instruments to measure things. But you need to find measurable things. Languages are not like math or geography or archeology. How do you measure a language? How do you even measure if a document is well written? By counting the typos or syntax errors? Then how do you measure style?

I posit that none of this can be measured in any meaningful way. I propose a different way to ‘measure’ translation quality: effectiveness.

Now you’ll tell me ‘But effectiveness cannot be measured!’ And you might be right…to a point. Let’s imagine marketing campaigns. An effective marketing campaign is the one that increases sales, name recognition, gets people to talk about your company and your product. A similar strategy can be employed for translation effectiveness. That the focus is on business results is the beauty of it.

This is an ongoing analysis and it is a work in progress. I am not claiming to have found the ultimate solution to measuring translation, but my experience strongly suggests in my mind that we are going about it the wrong way. Go ahead, measure words and errors all you want. You will end up empty-handed.

Advertisement

Who tests the translators?

Translation tests can be a welcome sign of interest from a prospective client for a translator…or an annoyance that elicits loud groans among some of the more seasoned linguists. I recently took part on a group discussion on LinkedIn regarding a translator’s reaction to a client requesting a translation test. The topic is so hot and thorny in the translation community that the discussion collected more than 125 postings in its thread.

I am not going to engage in endless arguments about the pros and cons of translation testing. Having said that, I strongly believe that a business (the translation agency or LSP) has every right to vet the assets it hires to perform linguistic services, especially because ours is an unregulated profession (at least, in America).

What I am going to propose to you, gentle readers (sorry about pilfering your phrase, Miss Manners), is this: let’s face the issue in a productive manner. Let’s talk about standardizing the translation test, maybe get the American Translators Association involved in the discussion. Some might say ‘But we already request ATA certification.’ Yes, but the ATA certification is no proof of performance.

What I propose is bringing all interested parties under one roof to discuss the pros and cons of translation tests, the current situation (maybe run some surveys among LSPs?) and suggest ways to address the weak points. Translation agencies need a tool to vet the translators, translators need feedback but also need feedback from peers, not bilingual staff. Also, this feedback accompanying the translation test results needs to be fairly and professionally delivered.

I am in favor of an ethical translation test, conducted by translation companies or other organizations that hire translators. The review of the tests must be in the hands of peers having no affiliation with the translation service provider administering the test; otherwise, the tested translator might think –and with reason– that the results are tainted by bias.

In addition, a standardized, ethical translation test can complement the ATA certification and benefit both translation companies and translators: everybody knows the rules, plays by the rules and abide by evidence-based decisions. Such a test can also become as valuable as a QA standard in many spaces in the language services marketplace.

What are your thoughts on the subject? If you administer translation tests, is it a chore, a necessary evil, an effective tool to hire linguists? If you are a translator, do you have examples of fair tests? Please share with us.