FireDucks is a high-performance compiler-accelerated dataframe library for python. It is being developed with speed and pandas compatibility in mind.
License
FireDucks is released on pypi.org under the 3-Clause BSD License (the Modified BSD License).
Let's experience the power of FireDucks!🦆🦆🦆
Anyone can experience the ease of use and high-speed performance of FireDucks for free.
If you have a Google account, you can try it out right now on Google colab. Please select 'v2-8 TPU' from the runtime settings, because it has high number of cpu cores and runtime memory.
| notebook contents | sample notebook links(open in Google colab) |
|---|---|
| A short presentation of some frequently used data analysis using pandas. | pandas_nyc_demo.ipynb |
| A short presentation of some frequently used data analysis using FireDucks. | fireducks_pandas_nyc_demo.ipynb |
| Demonstrating true strength of FireDucks in Query planning and optimization. | tpch-query3-pandas-fireducks-cudf.ipynb |
| NYC Taxi trips (Jan-2015) analysis using FireDucks, pandas and polars. | FireDucks_vs_Pandas_vs_Polars.ipynb |
Install
FireDucks is currently available for Linux (manylinux) on the x86_64 architecture (we will support it for other platforms based on user requests)
and it can be simply installed using pip as follows:
Please make sure you are using Python >3.8, <=3.13, otherwise you may encounter error saying: "No matching distribution found for fireducks".
⚠ Since FireDucks 1.1.0, we have upgraded dependent pyarrow to 18.0.0. Therefore, python 3.8 is no longer supported.
Usage
FireDucks provides two types of usage: Import Hook and Explicit Import.
Import Hook
FireDucks provides an import hook utility. This utility runs the given Python
script by automatically replacing import pandas statement with FireDucks. So
this utility enables you to use FireDucks for your existing program as it is.
The import hook can be activated by Python interpreter's -m option as follows.
python3 -m fireducks.pandas your_script.py
For IPython/Jupyter Notebook, a magic command is available as follows.
%load_ext fireducks.pandas import pandas as pd
This is very useful when your program includes many Python scripts which import
pandas internally. You don't need to replace individual import statements from
those scripts manually.
If you use any external libraries (like matplotlib etc.) which may internally use pandas DataFrame, please make sure to enable the hook to avoid any unintended executional issues.
📢 The import hook feature has changed its name to fireducks.pandas from
FireDucks 0.11.0.
An old module name fireducks.imhook is also available as an alias.
Explicit Import
FireDucks has pandas-like module fireducks.pandas which can be imported instead
of pandas. If you want to use FireDucks for an existing pandas program, replace
the import statement as follows.
# import pandas as pd import fireducks.pandas as pd
💻 Web site: https://fireducks-dev.github.io
This repository is used for communication such as issue report and Q&A. Please open new issue.
✉ Contact: mailto:contact@fireducks.jp.nec.com
🤝Slack: https://join.slack.com/t/fireducks/shared_invite/zt-34qpdgr6q-_iWdIoZW4l_hGhljKS0pyg