WebHelp Responsive search: How do "Stop Words" work?

[B-E-N]

Hello,

first of all thank you for the new search capabilities in Oxygen 19.

We are currently translating the new search strings in our various languages. Two strings mention the "stop words", such as "of", "the", and "by".

How does this work in other languages? I can see that you have translated them into Spanish, for example. How should we proceed if we would like to add Russian, for example? Is there a way to add or remove stop words?

Thanks,

Benjamin

[B-E-N]

Correction: I've just realized that the Spanish translation was done by a colleague of mine and not by you. So the more general question: How should we deal with translating the strings in the WebHelp search?

[radu_pisoi]

Hi,

The procedure for localizing the WebHelp output is described in our user manual in the Localizing the Interface of WebHelp Output (for DITA Map Transformations) topic.

We are currently translating the new search strings in our various languages. Two strings mention the "stop words", such as "of", "the", and "by".

Do you need the context where these strings are used? If yes, could you tell us which are the strings you need additional information?

[B-E-N]

Thanks for your answer.

We know how to localize the WebHelp output, I mean something different here.

The search considers some words as so called "stop words". This means, they are not considered when searching for terms. There are two strings that mention stop words:

Code: Select all

No results were found because the search query only contains <span>stop words</span> that are excluded by the search engine.

Code: Select all

Stop words are very common words or adjectives that hinder search efforts. Words such as: 'of', 'the', 'by', etc.

We must translate those strings into our target languages (Spanish, French, Japanese, Russian, etc.).

The question now is: What do we do with the stop words (of, the, by,...)? Just because we translate them, doesn't mean that the search actually ignores them in other languages.

How does the search know, which words are stop words? And can we add stop words for other languages as well?

[radu_pisoi]

Hi,

The stop words are computed dynamically depending on the language you have chosen when you publish your documentation. They are computed by the search indexer and written in the out/webhelp-responsive/oxygen-webhelp/search/index-1.js file:

Code: Select all

stopWords = new Array();

stopWords[0]= "but";

stopWords[1]= "be";

stopWords[2]= "with";

stopWords[3]= "such";

....

So, if you want to be sure which are the stop words for a certain language, you need to inspect the index-1.js file.

There is no parameter to control the stop words.

[B-E-N]

Thank you, that helps a lot.

[Gertone]

Hi,

I do realise I am reviving a pretty old thread...

In v23 I looked for the index-1.js
but realized that the array construction has moved to
...\oxygen-webhelp\app\search\index\stopwords.js

Code: Select all

define(function() {
// Auto generated list of analyzer stop words that must be ignored by search.
return ["but","be","with","such","then","for","no","will","not","are","and","their","if","this","on","into","a","or","there","in","that","they","was","is","it","an","the","as","at","these","by","to","of"];
});

Does that imply that we can now influence the stop words?

I guess I could swap that file with a project/language dependent function, either manually or through a plugin change,
but doing it from the configuration of the customization would be my preferred path.

Thanks for other suggestions

Geert Bormans

[radu_pisoi]

Hi,

Starting with version 23, you can customize the stop words list by using the following two parameters: webhelp.search.stop.words.exclude and webhelp.search.stop.words.include. They allow you to exclude/include custom stop words.

Please see the WebHelp Responsive Transformation Parameters topic in WebHelp documentation for more details.

[Gertone]

Hi Radu,

Thanks for pointing me to right place in the manual
(and thank you Oxygen for adding that functionality)

I assume this can not be made language dependent other than add all languages in one parameter?
Anyhow, the functionality is extremely useful as it is already

Thanks,

Geert

[radu_pisoi]

Hi,

I assume this can not be made language dependent other than add all languages in one parameter?

No, you should update exclude/include stop words parameters depending on the current language.