SDL Trados Studio
SDL Trados GroupShare
SDL Trados Business Manager
SDL Trados Live
SDL Speech to Text
SDL Managed Translation - Enterprise
Translation Management Connectors
SDL LiveContent S1000D
SDL Contenta S1000D
SDL Tridion Docs
SDL Tridion Sites
SDL Content Assistant
SDL Machine Translation Cloud
SDL Machine Translation Connectors
SDL Machine Translation Edge
Tridion Docs Developers
SDL User Experience
Language Products - GCS Internal Community
SDL Community Internal Group
SDL Access Customer Portal
SDL Professional Services
SDL Training & Certification
Language Technology Partner Group
SDL Academic Partners
SDL Enterprise Technology Partners
ETUG (European Trados User Group) Public Information
Machine Translation User Group
Nordic SDL Tridion Docs User Group
SDL Tridion UK Meetup
SDL Tridion User Group New England
SDL Tridion West Coast User Group
SDL WorldServer User Group
Tridion Docs Europe & APAC User Group
Tridion User Group Benelux
Tridion User Group Ohio Valley
SDL MultiTerm Ideas
SDL Passolo Ideas
SDL Trados GroupShare Ideas
SDL Trados Studio Ideas
SDL Machine Translation Cloud Ideas
SDL Machine Translation Edge Ideas
SDL Language Cloud TMS Ideas
SDL Language Cloud Terminology Ideas
SDL Language Cloud Online Editor Ideas
SDL Managed Translation - Enterprise Ideas
SDL TMS Ideas
SDL WorldServer Ideas
SDL Tridion Docs Ideas
SDL Tridion Sites Ideas
SDL LiveContent S1000D Ideas
SDL Contenta S1000D
SDL XPP Ideas
Events & Webinars
To SDL Documentation
To SDL Support
What's New in SDL
Detecting language please wait for.......
Hello developer team,
we are preparing the migration of our TMs on Group Share 2020. We want to create TMs that are exactly set up like on the old server. For this, we export the TM properties and we import them on the new server.
In the list of properties of a TM, we have 2 similar objects: TokenizerFlags and Recognizers. In the user interface, we can see the recognizers (date, numbers, alphanumerics...) but we can't set any tokenizer flags. Can you explain what's the difference between tokenizerflags and recognizers is and how the tokenizerflags are set when a TM is created manually?
The TokinezerFlags are impacting the word count flags. The correct value that you need to set on the language resource template of the TM is DefaultFlags. Recognizer, tokenizer…
As far as I know not really the flags are more the responsibility of the underlying components but this defaults changed over the time so it might be that.
Hi, I've gone a little deeper and have seen that tokenizer is more like the word count and it is certainly to tokenize the segment to check the similarity with existing translation units.
But the question remains the same: how can we set these flags. The function does not seem deprecated and on our server, we have 3 different sets: noFlags, DefaultTags and AllFlags. How is this possible?
Can you help? Romulus Crisan?
Can you please let me know which API you are using?
Hi Romulus,It's in the translation memory API
here the link to the flag enumeration.http://producthelp.sdl.com/SDK/TranslationMemoryApi/4.0/html/d8cc81a4-8731-9013-55b2-bf83f1f8b000.htm
The TokinezerFlags are impacting the word count flags. The correct value that you need to set on the language resource template of the TM is DefaultFlags. Recognizer, tokenizer and word count flags need to be set on the template of the TM. In your case you can either create a template and use that across the TM's or create the TM save it in the server that will create a private TM template associated with the TM and set the flags. You can set the flags on the TM : ServerBasedTranslationMemory has all the flags that will update the private language resource template.
I hope that shared some light on your questions.
thank you for your answer. Yes that's mainly what I wanted to know. I will set the tokenizer on "defaultFlags" for every TM container.
I guess the noFlags and allFlags we found for some containers are due to old TMs in older version of Studio where the property did not exist yet. Or is there any possiblity to set other Tokenizer Flags without using the API?