Photo by Author | Chat GPT
. Introduction
Getting real -world data for your data science projects is often the most difficult part. Toy datasis is easy to find, but for high quality or real -time data you usually need to use APIS or create customs scraping pipelines to extract information from the web.
In this article, I share 10 of my favorite free APIS – which I use daily data collection, data integration, and building AI agents. These APIs have been organized in five types, spreading reliable data reservoirs, web scraping, and web search, so you can immediately choose the right tool and move the insight faster by data.
. The basic data storage
One of the basic data storage is a community -based platform where various organizations and open source supporters share their datases with a broader world. With a simple command, you can access these datases for your project.
!! 1. Kagal API
Cogl Datases are extremely popular when working on data science projects. Instead of downloading them manually, you can create a data pipeline that will automatically download the data, zip it, and load it into your workplace.
These datases are shared for the use of everyone by the open source community. To get started, create an API key from your Kagal account and set it as an environmental variable. After that, you can run the following commands in your terminal. Kagal also provides a SD SD, which allows you to have easy integration with your code.
kaggle datasets download -d kingabzpro/world-vaccine-progress -p data --unzip!! 2. The throat face CLI
Like Kiegel, The hugs face There is also a data science and machine learning community where people distribute datases, models and demo. You can easily install the embraced face CLI and integrate it into your workflose by using one of the CLI commands or the codes. Both options allow you to download datases without any API key.
An API key is only necessary when the data is gathed.
hf download kingabzpro/dermatology-qa-firecrawl-dataset. Web and crawling apis
There are different types of data in the web. If you can’t find your need on the above platforms, you may need to fix your data by scratching the web or using a web search API.
!! 3. The firecrackers
Cracker Provides an API to extract content from the websites and convert it into a mark down format for easy AI integration. It also comes with an abrasive and extraction API that connects with LLM (large language model) for advanced web scraping options.
This API is mandatory. I use it every day to create data and integrate it into my AI projects.
curl -s -X POST " \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "
"formats": ("markdown", "html")
}'!! 4. Tavily
tavily There is a sharp web search API that provides 1,000 search requests every month for free. They are both correct and fast. You can use it to make it datasis, integrate it into your AI projects, or use it as a simple search API for your development needs.
curl --request POST \
--url \
--header "Authorization: Bearer " \
--header "Content-Type: application/json" \
--data '{
"query": "who is Leo Messi?",
"auto_parameters": false,
"topic": "general",
"search_depth": "basic",
"chunks_per_source": 3,
"max_results": 1,
"days": 7,
"include_answer": true,
"include_raw_content": true,
"include_images": false,
"include_image_descriptions": false,
"include_favicon": false,
"include_domains": (),
"exclude_domains": (),
"country": null
}' . APIS of geographical and weather
If you are looking for weather and geographical datases, you will know that things change. That is why you need real -time access to these datases through API.
!! 5. Open Vitheramp
Open Vitheramp There is a service that provides global climatic data through APIS, including current conditions, forecasts, nine cast, historical records, and even minutes of hyperloqual rain.
curl "!! 6. Open Street Map
Open Street Map Provides global map data, and Over Pass API The only reader is a web database that serves selected parts of OSM and can be questioned with the Overseas QL. The example below takes the cafe’s nodes to a small London -bound box.
curl -G " \
--data-urlencode 'data=(out:json);node("amenity"="cafe")(51.50,-0.15,51.52,-0.10);out;'
. Financial Market Data APIS
Financial market data is recommended if you are working on a financial project and requires real -time data on stock, crypto, and other finance information and news.
!! 7. Alpha Ventage
Alpha Ventage There is a financial data platform that offers free APIS for real -time and historical market data in stock, forex, crypto currencies, commodities and options, which are outputs in JSON or CSV. It also provides a chart for intravenous, daily, weekly, and monthly intervals, and also provides more than 50 technical indicators for analysis.
curl "!! 8. Yahoo Finance
Many early and practitioners use yfinance API to enter stock prices, historical time series data, profit and spillets as well as basic metadata. This allows them to develop a data frame ready for quick prototypes and classroom projects.
Yahoo Finance Free stock pricing, news, portfolio tools, and international markets offer coverage, which enables consumers to find a wide range of market data without any price.
import yfinance as yf
print(yf.download("AAPL", period="1y").head()). Social and community data apis
If you are working on a project to analyze the text and community conversation with high social media platforms, these APIS provides easy access to real social media data.
!! 9. Reddit
reddit Offers a full, community -driven data source, and Azgar Reddate API Raper (Prao) In Azar, access to the official reddate API for tasks such as posts, comments, and subdidated metadata.
Prah Reddit’s API works by sending applications under the hood and is commonly used in teaching to collect debates for analysis.
import praw
r = praw.Reddit(
client_id="ID",
client_secret="SECRET",
user_agent="myapp:ds-project:v1 (by u/yourname)"
)
print((s.title for s in r.subreddit("Python").hot(limit=5)))
!! 10. X
X – Access usually requires verification, adherence to the limits and policy rates, and your volume and use of the issue of use is to choose the appropriate access level.
curl -H "Authorization: Bearer YOUR_BEARER_TOKEN" \
". The final views
These APIS provide free access to data that is often difficult to achieve. They greatly enhance your web data collection or improve your web scraping efforts, allowing you to create custom datases.
When you need high quality, real -time data from the web, I highly recommend bookmarking to re -view this article. By taking advantage of these APIs, you can unlock valuable insights that will help you research and analyze.
Abid Ali Owan For,,,,,,,,,, for,, for,,,, for,,,, for,,, for,,,, for,,,, for,,,, for,,, for,,, for,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,,, for,,, for,,, for,,, for,,, for,,,,, for,,,, for,,,, for,,,, for,, for,.@1abidaliawan) A certified data scientist is a professional who loves to create a machine learning model. Currently, he is focusing on creating content and writing technical blogs on machine learning and data science technologies. Abid has a master’s degree in technology management and a bachelor’s degree in telecommunications engineering. Its vision is to create AI products using a graph neural network for students with mental illness.