New Trends in Natural Language Processing – How NLP Becomes Suitable for the Mass-Market
NLP (Natural Language Processing) generally describes the computer-aided processing of human language. This includes both written and spoken language. The goals pursued with NLP can be classified into two superordinate categories: Understanding language and generating language. The technical challenge for both purposes is to transfer unstructured information in the form of text into a format that can be processed by a machine. In concrete terms, this means that text must be represented in a number format that the computer can understand. Just a few years ago, this was only possible for a few tech companies. These companies had three decisive advantages:
- Access to huge amounts of unstructured text data
- Experts who can develop cutting-edge technologies
- Computing capacity to process the amount of unstructured data
In this article, we show different NLP case studies and explain how, within only a few years, the barriers to market entry have been lowered to such an extent that today every company can use NLP to solve their business problems.
In which areas can NLP be used?
Computer-aided speech processing is a very abstract term. The term becomes more understandable when it is broken down into its areas of application. For each subarea, a different, specialized model for speech processing is applied. It should be noted that tasks that are relatively easy for a human being, such as recognizing emotions, also tend to be easier for a machine in the NLP field. On the other hand, more complicated tasks, such as translating texts, tend to be more difficult for the computer to solve. The six most important NLP applications, and the business problems that can be solved with it, are discussed below.Sequence ClassificationThe classic example of NLP is sequence classification. The goal is to assign text sequences to one of several previously defined classes. An example of these classes are emotions (joyful, angry, exhilarated, etc.). A text is presented to the computer and it must decide independently which emotion the author wanted to express in this text. Other examples are assigning a text to a known author or classifying document types. When classifying sequences, it should be noted that a sequence can consist of a text of any length. A sequence can consist of a single word (sequence of letters), a sentence, a paragraph, or a complete document. An example of a shorter sequence would be a vacation rating.
Case Study 1: A travel portal wants to specifically target customers with a negative vacation experience in a marketing campaign. For this purpose, existing customer reviews are divided into three classes – positive, neutral, and negative. Each review is automatically assigned to one of these classes.
Longer documents could be, e.g., mailings of any kind.
Case Study 2: The logistics department of an international company processes different document types in different teams. For this purpose, postal items are currently selected and sorted manually. In the future, they will be automatically assigned to a category. Incoming invoices, delivery bills, and other inquiries could be defined as categories.
Question-Answer Models
For question-answer problems, the computer is provided with as many text corpora as possible. The goal is to give a content-correct answer to questions written by a person based on information from the text corpora. The difficulty of this task varies depending on how specific the required information is in the text. The simplest solution is the complete extraction of existing text passages. Based on this, the extracted information can be packaged into a grammatically correct answer. Most complex to implement are logical conclusions based on the existing information. For example, a given text could describe the structure of a company’s premises. Buildings A, B, and C are mentioned. A possible question could be, “How many buildings are there on the site? A logical conclusion of the computer would be that the site consists of 3 buildings without mentioning the number itself.
Case Study 3: A medium-sized company has been observing a continuous increase in customer inquiries for a long time. Many of these inquiries are requests for information. The company decides to develop a chatbot. Based on internal support documents, customers can now independently and automatically ask questions to the chatbot and have them answered.
Generating Texts
Based on a given text, the next matching word should be predicted as accurately as possible. The resulting text can be used to predict another word. This process can be repeated as often as desired to create texts of any length. Thereby it is possible to generate texts with any language subtleties. Thus, a particular accent or dialect can be modeled, but also a simple or more complex language can be used, depending on the target group. The greatest challenge is that text should be free of errors in terms of both content and language.
Case Study 4: A manufacturer of a document management system wants to make it easier to find documents. In the built-in search mask, the search term is automatically supplemented with additional matching words.
Recognition of Phrases
When recognizing phrases, also called Named Entity Recognition (NER), the goal is to assign one or more words in a sentence to a class. Grammatical units can be defined as possible phrases. A sentence is divided into subject, verb, object, etc. User-defined entities are often chosen instead of grammatical units. Usually, the entities are, e.g., places, natural persons, companies, or times. When a person has to decide in which category a phrase belongs, we automatically use rules (such as grammar rules). In NER, the computer should learn to use similar decision rules. However, these rules are not explicitly given; instead, the computer has to learn them independently.
Case Study 5: A hedge fund automatically analyzes the quarterly reports submitted to the SEC. The goal is to generate summaries of the company’s business activity automatically. Thus, the list of identities that have to be extracted consists of business type, division, director, etc.
Summaries
The task of creating a summary of a text is exactly like the task in a school lesson. The goal is to create a summary with all relevant content that looks as real as possible. Applicable spelling and grammar rules must be adhered to. The challenge here is to teach the computer to separate important and relevant content from unimportant content.
Case Study 6: By analyzing their website’s usage behavior, an online news agency has found out that fewer people read the articles entirely through to the end. To make it easier for readers to extract relevant information, they want to create automated summaries for existing and new articles. The length and language complexity of the summary should depend on the user profile.
Translations
In a text translation, the text is transferred from one language to another, in compliance with the appropriate spelling and grammar rules and without changing the content. The computer has similar problems with translating texts as a human being. The balance between content and grammatical correctness must always be maintained without moving away from the original text.
Case Study 7: A nationally acting supplier wants to expand its sales market internationally. For this purpose, all existing technical specifications must be translated into the language of the target markets. A particular challenge is the translation of technical, industry-specific vocabulary.
How have the NLP Models developed?
The history of NLP models can be divided into three eras: Naive models, static vector models, and dynamic vector models. In the early days of NLP models, attempts were made to determine the meaning of texts by counting words or pairs of words. This required a very intensive preparation of the texts. The actual calculation of the models, based on the counts, can be done very quickly (with today’s computers). However, any context is lost when counting the words. The next development step were static vector models. The idea behind these models is that each word is represented by a vector, i.e., a series of numbers. These number series are usually calculated by using deep learning models. They then serve as input for another model, e.g., another Deep Learning Model, which uses the number series to solve the actual task, e.g., the classification of the texts. By using the vectors, it was possible to understand the context of words better. This means that other words surrounding this word are also considered when calculating a vector for a word. However, the vectors for a word written the same way are still identical, independent of the actual meaning. In the example shown below, the vector for ‘park’ would be the same.
I don’t know how to parallel park. (park = from the verb “to park”) I’m taking my dog for a walk at the park. (park = open green area)
The calculation of the vectors, as well as the model, is very time and computationally intensive. However, due to the missing context of the vectors, the prediction is still very efficient. The latest generation of NLP models is similar to the second generation, but now vectors are calculated with respect to the word’s context. Thus, in the example above, a different vector would be calculated for the bench than for the bank. This makes both the calculation of the model and the prediction very computationally intensive.
Why has NLP become so relevant?
Google kicked off the “New-Area of NLP” with the so-called BERT model at the end of 2018 (click here LINK: https://github.com/google-research/bertto get to the official GitHub repository). Since then, monthly adaptations and further developments of the model have been published by universities, and companies such as Facebook and, of course, Google itself. The majority of these models are available to the broad masses free of charge –the use for commercial purposes is almost always permitted. The performance of this latest generation of NLP models is in many areas on par with, or already above, the results that can be achieved by humans. Research has developed data sets for various tasks and sub-areas of speech processing. These tasks were first solved by humans to create a reference value to be beaten by computers. Meanwhile, NLP models can provide almost human results in nearly all areas. It should be noted that the data sets are very generalistic. Each of these benchmark datasets tries to achieve as much coverage as possible in its subarea to make the best possible general statement about the performance. Business problems, on the other hand, are usually much more concrete. For example, a model may be very well able to capture the general mood of all kinds of texts and thus obtains a good, high rating in this area. A business problem could be to evaluate the mood of customer contributions in social networks or of incoming emails of customer complaints. From a human point of view, both tasks are very similar. For a machine, it can make a big difference, whether it is short, informal texts, such as posts from social media, or longer, formal texts, such as emails. An evaluation of the models for the business problem is essential.
How did NLP become so easy to use?
Until a few years ago, there were three fundamental problems in the development of artificial intelligence, and especially for the subarea of NLP, which made the development and adaptation of these models difficult. The problems were related to the allocation of resources in the three areas data, computing power, and human capital. The pre-training of models significantly mitigated all three issues. The large, relevant companies in NLP model development invest in these resources and provide these pre-trained models afterward, mostly free of charge. The models are already outstanding in the general understanding of texts but usually leave room for improvement for specific problems. However, the biggest part of resources is required for the first part, the generalistic representation of text. These pre-trained models can now be fine-tuned to specific business problems with relatively little effort. By fine-tuning, excellent results can be achieved with minimal effort and at a low cost.
Entry Barrier: Data Availability
As the complexity of the models’ increases, the need for data needed for training grows exponentially. The performance of these models is achieved by looking at the context of a word. Consequently, a model must see as many words in as many combinations as possible. Through the Internet, there is access to extensive text collections. Thus, the BERT model mentioned above was trained on various books with about 800 million words and the complete English platform Wikipedia, with about 2.5 billion words.
Entry Barrier: Computing Power
The increasing demand for data and the growing complexity of the model result in a higher need for computing power. Some relevant points can be observed here. The power of computers increases massively every year, and it is doubling about every two years. At the same time, computing power is becoming cheaper. Since the triumph of cloud providers, access to computing power has been democratized. Extremely high-performant and specialized computer clusters are now available not only to large companies but to everyone, with billing to the minute.
Entry Barrier: Talent Acquisition
In the early days of AI, it was necessary to either build up a competitive development team within the own organization or to purchase the complete development from specialized companies. As a result, it was necessary to invest a great deal of money in advance to put a finished AI product into operation after a development period that often lasted several years. Often such projects failed or did not add enough value. Financial investments with such a risk profile were usually only possible for large multinational companies. Most of the newly developed NLP models are available for free today, even for commercial purposes. Therefore, it is possible to get a proof of concept within weeks instead of months or years. The introduction time of a complete product has been reduced from years to months. Iterations in product development are now possible very quickly, with a low initial investment.
What challenges remain?
Many of the original problems have been defused or completely solved. These developments have, above all, immensely shortened the time needed to complete a feasibility study.
Model
Currently, pre-trained models are available from a variety of companies and providers. These are continually being developed, and often a newer, improved version is released after a few months. Also, several versions of the same model are released at the same time. These often differ in complexity or language. It is crucial to explore and evaluate models in the initial phase of an NLP project. The performance can be divided into two dimensions: The quality of the results and the execution speed. To evaluate the quality of model results is often challenging, depending on the task. When classifying emotions, it is usually possible to determine whether the model was right or wrong. Assessing summaries is much more difficult. It is vital in the initial phase of a project to determine a measure of quality that is technically feasible but also reflects the business problem. The second dimension of model performance is the speed of execution. This includes both the time required for training and prediction. It is very important to coordinate the model requirements with all project partners at an early stage. For example, a model that has to answer questions live within milliseconds has different properties than a model that calculates results once a day overnight.
Data
The topic of data is generally a double-edged sword in AI and especially in NLP. On the one hand, data is typically available, and computer systems can process it. By pre-training models, a large part of the work with data is taken off our hands. On the other hand, pre-trained models are always designed to work as good as possible in various tasks. Often the pre-trained models deliver good results without fine-tuning, but not outstanding results. The fine-tuning is usually done in two dimensions. The model must first be adapted to language peculiarities and subtleties – this can be a unique vocabulary, slang, or dialect. Thus, there is a big difference between articles from social media and instructions for production processes. The second dimension relates to the actual task at hand. To achieve outstanding performance, models must always be tailored to the business problem. A model that can translate differs significantly from a model that can classify emotions. For this fine-tuning, texts/data are needed to adapt to the target language and the target problem. The texts must be prepared and fed into the model. Depending on the complexity and quality of the data, this can still be a laborious process.
Computing power
The fact that computers are getting better and better and computing power is getting cheaper and cheaper is one of the main reasons for the adaptation of AI. As already mentioned several times, pre-trained models make it unnecessary to provide the lion’s share of the computing power by oneself. Computing power is only needed for data processing and fine-tuning the models. This is a fraction of the computing power needed for complete training from start to finish. Nevertheless, it is usually more than a standard computer could manage in a reasonable time. Therefore, cloud computing is generally used for fine-tuning. Cloud resources are usually billed by the minute and are therefore very cost-effective. However, the process of training using cloud computing differs significantly from training in a standard data center, which is why knowledge in this area must either be built up within the own organization or purchased from external service providers.
What can we expect from NLP in the Future?
Of the entire field of artificial intelligence, NLP is currently the most actively researched area. In the next months and years, some more interesting developments can be expected, and currently, two developments with very interesting practical implications are emerging. In the short to medium term, we expect the practical application of so-called zero-shot models. These models are trained for a particular task area like sequence classification. The novelty is that these models provide excellent results without having seen domain-specific data. Thus these models develop a kind of “general” intelligence. This makes the fine-tuning of models much easier or completely unnecessary. The next step to be expected are the so-called general purpose models. These models can solve any task on unseen data, eliminating the need for complete fine-tuning. The first experiments with these models seem to give outstanding results, but the models are extremely large and require very high computing power. Therefore, the commissioning of these models is currently extremely difficult and expensive. There are still almost no practical applications. We expect significant leaps in practicability and performance in the next few years.
Summary
The latest developments in the field of speech processing are both impressive and fast. Google gave the starting signal of the newest developments with the publication of the BERT model scarcely two years ago. Since then, in a weekly rhythm, new models are published by companies and universities worldwide. These models often improve the results of existing problems or enable existing resources to be used more efficiently. Problems which two years ago were considered unsolvable are now often very well solvable and affordable in terms of resources and development time. The time required to prepare a feasibility study has been extremely shortened.
