Knowledge Management with NLP: How to easily process emails with AI

In a fast-paced and data-driven world, the management of information and knowledge is essential. Businesses in particular rely on making knowledge accessible internally as quickly, clearly, and concisely as possible. Knowledge management is the process of creating, extracting, and utilizing knowledge to improve business performance. It includes methods that help organizations identify and extract knowledge, distribute it, and use it to better achieve their goals. However, this can be a complex and challenging task, especially in large companies.

Natural Language Processing (NLP) promises to provide a solution. This technology has the potential to revolutionize the knowledge strategy of companies. NLP is a branch of artificial intelligence that deals with the interaction between computers and human language. By using NLP, companies can gain insights from large amounts of unstructured text data and convert them into actionable knowledge.

In this blog post, we examine how NLP can improve knowledge management and how companies can use NLP to perform complex processes quickly, safely, and automatically. We explore the benefits of using NLP in knowledge management, the various NLP techniques used, and how companies can use NLP to achieve their goals better with artificial intelligence.

Case Study for effective knowledge management

Using the example of email correspondence in a construction project, we illustrate the application and added value of natural language processing. We use two emails as specific examples that were exchanged during the construction project: an order confirmation for ordered items and a complaint about their quality.

For a new building, the builder requested quotes for products from a variety of suppliers, including thermal insulation. Eventually, they were ordered from a supplier. In an email, the supplier clarifies the ordered items, their properties and costs, and confirms the delivery on a specified date. Later, the builder discovers that the quality of the delivered products does not meet the expected standards. The builder informs the supplier of this in a written complaint, also via email. The text of these emails contains a wealth of information that can be extracted, processed, and further processed using NLP methods to improve understanding. Due to the large number of different offers and interactions, manual processing is very time-consuming, and programmatic evaluation of the communication provides a remedy.

Next, we introduce a knowledge management pipeline that gradually checks these two emails for their content and provides users with the maximum benefit through text processing. Click on the interactive boxes to see how the Knowledge Management Pipeline works!

Summary (Task: Summarization)

In the first step, the content of each text can be summarized and brought to the point in a few sentences. This reduces the text to important information and knowledge, removes irrelevant information such as platitudes and repetitions, and greatly reduces the amount of text to be read.

Especially with long emails, the added value of summary alone is enormous: listing the important content as bullet points saves time, prevents misunderstandings, and avoids overlooking important details.

General summaries are already helpful, but with the latest language models, NLP can do much more. In a general summary, the text length is reduced as much as possible while maintaining the same information density. Large language models can not only produce a general summary but also customize this process to specific needs of employees. For example, facts can be highlighted, or technical jargon can be simplified. In particular, summaries can be performed for a specific audience, such as a specific department within the company.

Different departments and roles require different types of information. This is why summaries are particularly useful when tailored to the interests of a specific department or role. For example, the two emails in our case study contain information that is relevant to the legal, operations, or finance department in different ways. Therefore, the next step is to create a separate summary for each department:

This makes it even easier for users to identify and understand the information that is relevant to them, while also drawing the right conclusions for their work.

Generative NLP models not only allow texts to be condensed to the essential, but also provide explanations for ambiguities and details. An example of this is the explanation of a regulation mentioned only by an acronym in the confirmation of an order, whose details the user may not be familiar with. This eliminates the need for a tedious online search for a suitable explanation.

Knowledge Extraction (Task: NER, Sentiment Analysis, Classification)

The next step is to systematically categorize the emails and their contents. This allows incoming emails to be automatically assigned to the correct mailboxes, annotated with metadata, and collected in a structured way.

For example, emails received on a customer service account can be automatically classified into defined categories (complaints, inquiries, suggestions, etc.). This eliminates the manual categorization of emails, which reduces the likelihood of incorrect categorizations and ensures more robust processes.

Within these categories, the contents of emails can be further divided using semantic content analysis, for example, to determine the urgency of a request. More on that later.

Once the emails are correctly classified, metadata can be extracted and created from each text using “Named Entity Recognition (NER).”

NER allows entities in texts to be identified and named. Entities can be people, places, organizations, dates, or other named objects. Regarding email inboxes and their contents, NER can be useful in extracting important information and connections within the texts. By identifying and categorizing entities, relevant information can be quickly found and classified.

In the case of complaints, NER can be used to identify the names of the product, the customer, and the seller. This information can then be used to solve the problem or make changes to the product to avoid future complaints.

NER can also help automatically highlight relevant facts and connections in emails after they are classified. For example, if an order is received as an email from a customer, NER can extract the relevant information, enrich the email with metadata, and automatically forward it to the appropriate salesperson.

Similarity (Task: Semantic Similarity)

Successful knowledge management first requires identifying and gathering relevant data, facts, and documents in a targeted manner. This has been a particularly challenging task with unstructured text data such as emails, which are also stored in information silos (i.e. in mailboxes). To better capture the content of incoming emails and their overlaps, methods for semantic analysis of text can be employed. “Semantic Similarity Analysis” is a technology used to understand the meaning of texts and measure the similarities between different texts.

In the context of knowledge management, semantic analysis can help group emails and identify those that relate to the same topic or contain similar requests. This can increase the productivity of customer support teams by allowing them to focus on important tasks, rather than spending a lot of time manually sorting or searching through emails.

In addition, semantic analysis can help identify trends and patterns in incoming emails that may indicate problems or opportunities for improvement in the company. These insights can then be used to proactively address customer needs or improve processes and products.

Answer Generation (Task: Text Generation)

Finally, emails need to be answered. Those who have already experimented with text suggestions in email programs know that this task is not yet ready for automation. However, generative models can help answer emails faster and more accurately. A generative language model can quickly and reliably generate response templates based on incoming emails, which then only need to be supplemented, completed and checked by the person processing them. It is important to carefully check each response before sending it, as generative models are known to hallucinate results, i.e. generate convincing answers that contain errors upon closer examination. Here too, AI systems can at least partially remedy the situation by using a “control model” to verify the facts and statements of these “response models” for accuracy.

Conclusion

Natural Language Processing (NLP) offers companies numerous opportunities to improve their knowledge management strategies. NLP enables us to extract precise information from unstructured text and optimize the processing and provision of knowledge for employees.

By applying NLP methods to emails, documents, and other text sources, companies can automatically categorize, summarize, and reduce content to the most important information. This allows employees to quickly and easily access important information without having to wade through long pages of text. This saves time, reduces error-proneness, and contributes to making better business decisions.

At the example of a construction project, we demonstrated how NLP can be used in practice to process emails more efficiently and improve knowledge management. The application of NLP techniques, such as summarizing and specifying information for specific departments, can help companies better achieve their goals and improve their performance.

The application of NLP in knowledge management offers great advantages for companies. It can help automate processes, improve collaboration, increase efficiency, and optimize decision-making quality. Companies that integrate NLP into their knowledge management strategy can gain valuable insights that enable them to better navigate an increasingly complex business environment.

Image source: AdobeStock 459537717 Oliver Guggenbühl, Jonas Braun