Photo by editor
# Introduction
Feature Engineering Data science and machine learning are an integral process in workflows as well as in any AI system as a whole. It involves constructing meaningful explanatory variables from raw – and often messy – data. The process behind feature engineering can be extremely simple or overly complex, depending on the size, structure, and richness of the dataset(s), as well as the objectives of the machine learning modeling. While the most popular Python libraries for data manipulation and modeling Pandas And Learn to skateenable somewhat basic and moderately extensible feature engineering, there are specialized libraries that go the extra mile in dealing with large datasets and automating complex transformations, yet they are largely unknown to many.
This article lists 7 under-the-radar Python libraries that push the boundaries of the feature engineering process at scale.
# 1. Acceleration with nvtabular
First, we have Nvidia-Merlin nvtabular: a library designed to apply pre-processing and feature engineering to datasets – yes, you guessed it! – Tabular. Its distinctive feature is its GPU-accelerated approach designed to easily handle the large-scale datasets required to train large-scale deep learning models. The library is specifically designed to support pipelines for advanced recommender system engines based on Deep Neural Networks (DNNs).
# 2. Automated with feature tools
Feature Toolsdesigned by Altrex, focuses on leveraging automation in the feature engineering process. This library applies deep feature synthesis (DFS), an algorithm that generates new, deep “features” by mathematically analyzing relationships. The library can be used on both relational and time series data, making it possible to obtain complex feature output with minimal coding burden in both.
This code snippet shows an example of what is used with DFS featuretools The library looks like this, on the users dataset:
customers_df = pd.DataFrame({'customer_id': (101, 102)})
es = es.add_dataframe(
dataframe_name="customers",
dataframe=customers_df,
index="customer_id"
)
es = es.add_relationship(
parent_dataframe_name="customers",
parent_column_name="customer_id",
child_dataframe_name="transactions",
child_column_name="customer_id"
)# 3. Parallel with the disk
Disk Parallels is growing in popularity as a library to make Python computations faster and easier. The master recipe behind Dask is to scale feature transformations with traditional pandas and scikit-learn using cluster-based computations, thereby facilitating faster and cheaper feature engineering pipelines on large datasets that would otherwise consume memory.
This article shows a practical DASK walkthrough for performing data preprocessing.
# 4. Correcting with polarities
Competing with Dask in terms of growing popularity, and vying for a place on Python’s data science podium with Pandas, we have Polar: A Rust-based dataframe library that uses a slow expression API and slow computation to drive efficient, scalable feature engineering and transformations on very large datasets. Considered by many as the high-performance counterpart to Pandas, Polar is very easy to learn and become familiar with if you are familiar enough with Pandas.
Interested in learning more about Polar? This article presents several practical polar one-liners for common data science tasks, including feature engineering.
# 5. Stocking with feast
invitation is an open-source library envisioned as a feature store, helping to provide sample data sources at scale to production-level or production-ready AI applications, especially those based on large language models (LLM), for both model training and individuation tasks. One of its attractive features consists in ensuring consistency between the two phases: training in production and its evaluation. Its use as a feature store is also closely tied to the engineering process, i.e. by using it in conjunction with other open source frameworks, for example, Deformed.
# 6. Extract with TSFRESH
Turning our attention to larger time series datasets, we have tsfresh library, with a package that specializes in extensible feature extraction. From statistics to vernacular features, this library is capable of computing hundreds of meaningful features in large time series as well as applying correlation filtering, which, as its name suggests, filters features by relevance in the machine learning modeling process.
This example code snippet takes one DataFrame Contains a time series dataset that has previously been wrapped in Windows, and applied to tsfresh Feature on it:
features_rolled = extract_features(
rolled_df,
column_id='id',
column_sort="time",
default_fc_parameters=settings,
n_jobs=0
)# 7. Paving along the river
Let’s stop dipping our feet in the river (pun intended). river The library is designed to streamline online machine learning workflows. As part of its utility, it has the ability to enable online or streaming feature transformation and feature learning techniques. This can help efficiently deal with issues such as redundant data and concept growth in production. River Beech is built to robustly handle rare problems in machine learning systems, such as the appearance and disappearance of data features over time.
# wrap up
This article lists 7 notable Python libraries that can help make the feature engineering process more scalable. Some of these are directly focused on providing a specific feature engineering approach, while others can be used in conjunction with other frameworks, to further support certain engineering tasks in certain scenarios.
Ivan Palomares Carrascosa Is a leader, author, speaker, and consultant in AI, Machine Learning, Deep Learning, and LLMS. He trains and guides others in real-world applications of AI.