> For the complete documentation index, see [llms.txt](https://smartpredict.gitbook.io/smartpredict-ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://smartpredict.gitbook.io/smartpredict-ai/module-reference/core-modules/11.-natural-language-processing/text-cleaner.md).

# Text cleaner

This module belongs to the category " Natural Language Processing" .

## Description

While **text cleansing** might be a time-consuming NLP operation, it is also a well-known fact that it is at the core of *sentiment analysis, ontology feeding* and *knowledge base construction.*        

In fact,  **parsing operations, lemmatization** and **stemming** are definitely indispensable for harvesting the useful data out of the huge amount of words after *web scraping,* providing the right idiom *translation* and designing *chatbots* that can express themselves in meaningful sentences  etc.

These are the pinpointed issues that SmartPredict intends to solve by this unique module.

The **Text cleaner module** is used to clean input text for NLP purposes , in **English** or **French** languages or in **mixed** language. Select the **language used by the doc** in the parameters for applying **stop words removal.**

![As an input, the text cleaner receives several kinds of data .](https://1833277725-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lyc1OXsKqB2S62LxsOR%2F-M1VUk1UNgOHjDZQiAX7%2F-M1V_4bAmv0qjnlBKLVK%2Fimage.png?alt=media\&token=c54d3d1f-fd7e-49fc-ab1f-2c50ca56029d)

## Parameters

The text cleaner is a comprehensive little module which includes all typical cleansing operations on its own. We can select to apply these cleansing operations to a part of the sentence or as an option, decide to completely eliminate the occurrences of certain words. 

#### Splitting

As options, we can split the text according to key words. The Input to clean and split could be a:.

* **dataframe**
* **series**
* **list of text**
* **string**

###

![The text cleaner module is a comprehensive handy NLP toolset.](https://1833277725-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lyc1OXsKqB2S62LxsOR%2F-M1VbpCESafXT2rQZOPs%2F-M1Vc7No57w6yiIkc4Y7%2FSmartPredict%20-Text%20cleaner.gif?alt=media\&token=162f94bb-b7d3-4991-8e0c-17698f85690e)

###

### Cleansing operations 

Among other text cleansing operations,  we may choose to remove:

* **text between parentheses**
* **email addresses** 
* **HTML** 
* **tags** 
* **text inside brackets** 
* **retain alphabetic words only** 
* **remove stop words**
* **remove web URLS**

![The text cleaner module offers all the classic functions for text cleansing.](https://1833277725-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lyc1OXsKqB2S62LxsOR%2F-M1VbpCESafXT2rQZOPs%2F-M1Vcnjo8ddwtirNDSBi%2Fimage.png?alt=media\&token=6ee812cf-790a-4045-9cba-cf5b03db8aee)

###