Large and heterogeneous data stocks are usually difficult to keep track of and could not be structured effectively with automatic classification until now. With the help of manual annotation (especially complex keywording), documents had to be provided with additional information. The search for this data required the creation of synonym lists, since text-based searches can only be based on content contained in the files. This problem was partially circumvented by using synonym lists (e.g. CEO, CEO = Managing Director) or thesauri. If the search terms are not directly contained in the files, synonymous search terms increase the chance of finding these contents.
As an integral part of INTERGATOR SMART SEARCH, an automatic keywording of contents enables the automatic extraction of essential terms and their context. The progress in the field of artificial intelligence (and machine learning in particular) opens up new possibilities to make the curated lists just mentioned obsolete. Instead, a trained model recognizes terms and their meaning and puts content into a context. The previous strict (or lexical) search for contained keywords is extended by a semantic-associative search, which finds documents that do not necessarily contain the search term, but are thematically similar to the information searched for.
The trained machine learning model autonomously groups documents into clusters on the basis of contained conceptual relationships and relates them to the overall inventory. For example, if a document contains terms such as “wheel”, “pedal” and “saddle”, it can be assumed that it is a “bicycle”. This automatic clustering is supported by the use of modern artificial neural networks, in whose learning result contents are mathematically transferred into a vector space. This results in spatial relationships that can be used for similarity searches.
With INTERGATOR’s KI-based SMART SEARCH even more problems can be solved elegantly. Human language, unlike mathematics, is not very exact. Ambiguities, homonyms or polysemes make understanding difficult and until now it always required human expertise to solve resulting “misunderstandings”. Humans deduce the meaning of a word from the context of its use in a text. Until now, machines lacked this ability. A “bank” can be grasped by a human being with different meanings from the context. For computers, this term is the same in terms of its sequence of characters, regardless of whether it refers to the seat or the financial institution.
Already since the 1950s, the linguistic foundations for the analysis of the meaning of natural language have been known. Since then, the methods of machine learning for computer-aided NLP have been continuously developed. But only since a few years a goal-oriented implementation is technically possible and is applied in INTERGATOR SMART SEARCH. If terms such as “credit”, “money” or “interest” appear in the texts, SMART SEARCH automatically classifies them in the context of finance.
INTERGATOR SMART SEARCH visualizes the recognized content-related connections by means of an interactive context map, in which terms and documents are combined to clusters. The desired results can be better narrowed down with just a few clicks by means of the term contexts visualized in this way. In the “Bank” search example outlined above, a click on the cluster with financial topics will only return results from this area. All other documents on “bank” that are not financially relevant are left out. In this way, users receive better, because more relevant, results that do not require maintained keywords, lists or similar.