5 points to build the best embrace facial transformer pipelines

Photo by Editor | Chat GPT

. Introduction

The hugs face Many AI developers and data have become a standard for scientists as it rapidly reduce the barrier to working with modern AI. Instead of working with AI models from the beginning, developers can access a wide range of different models without any hassle. Customers can also adapt and deploy these models with customs data.

The throat is one of the facial framework API rapers Transformer pipelinesA series of packages, including a pre -manufactured model, its toterzer, pre -and post processing, and relevant components to work for the use of AI. These pipelines summarize complex code and provide a simple, smooth API.

However, working with transformer pipelines may be dirty and may not get the maximum pipeline. This is why we will find five different ways you can improve your transformer pipelines.

Let’s enter it.

. 1. Batch Inconing Requests

Often, when using transformer pipelines, we do not fully use the graphics processing unit (GPU). Batch processing of multiple inputs can significantly promote the use of GPUs and increase diagnostic performance.

Instead of taking action on a sample at a time, you can use the pipeline batch_size Pass the parameter or inputs so that the model can take several inputs in a forward pass. Here is a code example:

from transformers import pipeline

pipe = pipeline(
    task="text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
    device_map="auto"
)

texts = (
    "Great product and fast delivery!",
    "The UI is confusing and slow.",
    "Support resolved my issue quickly.",
    "Not worth the price."
)

results = pipe(texts, batch_size=16, truncation=True, padding=True)
for r in results:
    print(r)

Through batching requests, you can get high thruppets with only the minimum effect on the delay.

. 2. Use low precision and quantization

Many pre -produced models fail in the interference because there is no memory in the development and production environment. Helps reduce the use of low -digit precision memory and speeds up its speed without sacrificing more accuracy.

For example, how to use half precision on GPU here in the transformer pipeline is that:

import torch
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)

Similarly, quantization techniques can significantly compress the model without harassment performance:

# Requires bitsandbytes for 8-bit quantization
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    device_map="auto"
)

The use of low precision and quantization in production usually accelerates pipelines and reduces the use of memory without significantly affecting the model’s accuracy.

. 3. Choose effective model architecture

In many applications, you do not need the biggest model to solve the work. The choice of mild transformer architecture, such as an east model, often receives better delays and input with acceptable accuracy trade.

Compact models or analized version, such as DistilbertKeep most of the original model’s accuracy but with very few parameters, resulting in a rapid estimate.

Choose a model that has improved Optim to improve architecture and be in accordance with the accuracy requirements of your work.

. 4.

Many systems are wasted by repeating expensive work. Catching can significantly increase the performance by reusing expensive counting results.

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=False,
        use_cache=True
    )

Effective catching computation reduces time and improves reaction times, which delays the production system.

. 5. Use a sharp run time through maximum (ONNX run time)

Many pipelines run in one Piturich The maximum mode, which adds overhead and additional memory copies. Using Maximum Open Neural Network Exchange (ONNX) with Run Time – Via ONNX Run Time – The model converts to static graphs and fuse operations, so the run time can use a faster kernel on a central processing unit (CPU) or GPU with low head. The result is usually a faster gesture, especially on the CPU or mixed hardware, without any change on how you call the pipeline.

Install with desired packages:

pip install -U transformers optimum(onnxruntime) onnxruntime

Then, replace the model with such code:

from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    from_transformers=True
)

By maximizing the pipeline to the ONX run time, you can maintain your existing pipeline code during a less delay and more efficient diagnosis.

. Wrap

Transformer pipelines have an API reaper in the throat facial framework that facilitates the AI ​​application development by thickering the complex code in an easy interface. In this article, we have discovered five points to improve the transformer pipelines of the throat facial transformer pipelines, from the selection of batch enclosures, from the selection of effective model architectures, the catching and beyond.

I hope it has helped!

Cornelius Yodha Vijaya Data Science is Assistant Manager and Data Writer. Elijan, working in Indonesia for a full time, likes to share indicators of data and data through social media and written media. CorneLius writes on various types of AI and machine learning titles.