Natural Language Processing

Natural Language Processing (NLP) is a field of computer science interacting with artificial intelligence focused on computer language and human language. It deals with modeling and predictions related to text data sources, including text extraction, text summarization, text analysis, and semantic extractions for applications of NLP. NLP is also used by search engines, social media monitoring, and target advertising and sentiment analysis.

Problem

Preprocess books, articles, text contents with a combination of automated and manual NLP preprocessing approaches to clean the data and prepare them for training via Generative Pretrained Transformers. The objective of this is to produce descriptive text paragraphs related to a given topic following the trained contents of a book, article, or any document related to a particular scope.

Solution

Our team has carried out the removal of numbers, URLs, table of contents, references, symbols, and unnecessary syntax and the conversion of the first-person narrative view of sentences into a collective one while removing odd proper nouns and product names by developing an automated script. To ensure the elimination of incomplete and inconsistent sentences that may be not possible to resolve using the automated process were manually proofread.