Picture by the writer
. Introduction
Although in modern data science you will mainly find Gapter Notebook, panda and graphical dashboards, they do not always give you the level of control you may need. On the other hand, the command line tools may not be so intuitive, but they are powerful, lightweight, and very fast to perform the specific jobs for which they are designed.
In this article, I have tried to balance the utility, maturity and strength. You will find some classic that are closely inevitable, as well as more modern additions that fill the gap or improve performance. Even you can call it A 2025 version of the CLI Tolls list. For those who are not familiar with CLI tools but want to learn, I have already added a bonus section to you before adding these tools to my workflow, so scroll before you add these tools.
. 1. Curl
curl Do I have to go to make HTTP requests like get, post, or PUT? Downloading files; And sending/receiving data to a protocol like HTTP or FTP. It is ideal for retrieving data from APIS or downloading datases, and you can easily connect it with data injection pipelines to pull it JSON, CSV, or other payloads. The best thing about Karl is that it is mostly installed on the UNIX system, so you can start using it right now. However, its syntax (especially the headers, physical pay loads, and the verification) may be a verbal and error. When you are interacting with more complicated APIS, you can prefer easy wrapper or Azigar library, but knowing the curl is still an essential plus for immediate testing and debugging.
. 2.
Ji Qi There is a lightweight JSON processor that helps you queries, filters, changes and beautiful print JSON data. With a dominant format for JSON APIS, Logs, and Data Interchange, JQ JSON is essential to pulling and changing the pipelines. It acts like “pandas for JSON in the shell”. The biggest advantage is that it provides a comprehensive language to cope with complex JSON, but it may take time to learn the syntax, and JSON files may need extra care with memory management.
. 3. Csvkit
csvkit CSV-Central Command is a suit of CSV-Central Command line utility to change, filter, collect, add and find CSV files. You can combine columns, subset rows, multiple files, convert from one format to another, and even run SQL -like questions against CSV data. CSVKIT understands CSV words referring to the cement and header, and for this format is more safer than the use of ordinary text processing. Being based on the fact that performance can be lagging behind on very large datases, and some complex questions in pandas or SQLs can be easier. If you prefer the use of speed and efficient memory, consider this csvtk Toll cut.
. 4. QWK / SED
Link (SED): https://www.gnu.org/software/sed/manual/sed.html
Classic Unique Tolls like awk And CED Be irreparable for manipulation in the text. The AWK pattern is powerful for scanning, field -based changes, and quick deposit, while CED texts take over alternatives, deletion and changes. These tools are sharp and lightweight, making them excellent at work for the pipeline. However, their syntax can be non -intuitive. As the logic increases, you have the ability to read, and you can move into the scripting language. Also, these tools are limited to these tools, such as nesting or rating data (eg, nesting json).
. 5. Parallel
Gnu parallel Parallel accelerates workflows by running a number of processes. Many data works are “mapable” in data parts. We say that you have to follow the same change on hundreds of files – parallel CPU can spread work in the core, accelerate processing, and manage to control the job. However, you must keep in mind the I/O obstacles and system burden, and it can be difficult to refer to/escape in complex pipelines. Consider Resource, Resource Schedulers (eg, spark, dask, cobrints) of cluster scale or distributed work loads.
. 6. RIPGREP (RG)
Ripgrep For,,,,,,,,,, for,, for,,,, for,,,, for,,, for,,,, for,,,, for,,,, for,,, for,,, for,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,, for,,, for,,, for,,, for,,,,, for,,,, for,,,, for,,,, for,, for,.rg) A high -speed repetition search tool is designed for speed and efficiency. It respects .gitignore Ignores default and invisible or binary files, which makes it significantly faster than traditional grep. This code base, log directory, or files are the best search for the perfect search. Since it is default to ignore some of the paths, you may need to adjust the FLA flags looking for everything, and it is not always available as default on every platform.
. 7. Data mesh
Data mesh Directly in the shell directly provides numerical, texture and data operations (money, mean, median, group bay, etc.) stdin Or files. It is lightweight and is useful for collecting quickly, such as launching heavy tools like Azgar or R, which makes it ideal for shell -based ETL or research analysis. But it is not designed for very large datases or complex analytics, where special tools perform better. Also, a lot of cardinals may need memory to group.
. 8. Htop
htop An interactive system is a monitor and process viewer that provides direct insight into CPU, memory, and I/O use per process. When you run heavy pipelines or model training, HTOP is extremely useful for resource consumption tracking and indicating obstacles. This is more user -friendly than traditional topBut being interactive means that it does not fit well in the automatic script. It may also disappear on the minimum server setup, and it does not replace special performance tools (profilers, matrix dashboards).
. 9. Gut
Got A distributed version control system is essential for tracking codes, scripts, and changes in small data assets. For reproductive capacity, cooperation, branching experiences, and rollback, gut quality. It is connected with deployment pipelines, CI/CD tools, and notebooks. The drawback is that it is not intended to see large binary data, for which GITLFS, DVC, or special systems are good. Branching and merger workflow also comes with curves to learn.
. 10. Tmux / screen
Terminal Multi Plexar likes tmux And Screen Let you run several terminal sessions in the same window, resume work after a separate and retreat session, and SSH disconnected. If you need to run long experiences or pipelines, they are important. Although TMUX is recommended because of its active development and flexibility, its formation and binding can be difficult for newcomers, and in the least environment it is not set to be installed as a pre -determined.
. Wrap
If you are starting, I will recommend mastering in the “Core Four”: curl, JQ, surprise/seed, and gut. They are used everywhere. Over time, you will have to discover specific CLIS related to SQL clients such as domain Dick DBCOr Datasate Slot in your workflow. To read more, see the following resources:
- Data Science in Command Line via Jeroin Johnsis
- The art of command line on Gut Hub
- Mark Pearl’s Bash Chat Sheet
- Like the communities Unix & & & & & & & The command line Subdes often highlight the level of useful tricks and new tools that increase your toolbox over time.
Kanwal seals A machine is a learning engineer and is a technical author that has a deep passion for data science and has AI intersection with medicine. He authored EBook with “Maximum Production Capacity with Chat GPT”. As a Google Generation Scholar 2022 for the APAC, the Champions Diversity and the Educational Virtue. He is also recognized as a tech scholar, Mitacs Global Research Scholar, and a Taradata diversity in the Harvard Wacked Scholar. Kanwal is a passionate lawyer for change, who has laid the foundation of a Fame Code to empower women in stem fields.