[minor][spark] Minor bugfixes by Dharin-shah · Pull Request #180 · duckdb/duckdb-python
Navigation Menu
{{ message }}
duckdb / duckdb-python Public
- Notifications You must be signed in to change notification settings
- Fork 76
Merged
evertlammerts merged 12 commits intoduckdb:v1.4-andiumfrom
Dec 8, 2025Merged
[minor][spark] Minor bugfixes#180
evertlammerts merged 12 commits intoduckdb:v1.4-andiumfrom
[minor][spark] Minor bugfixes#180
evertlammerts merged 12 commits intoduckdb:v1.4-andiumfrom
Conversation
Copy link Copy Markdown
Contributor
This PR fixes 3 small bugs:
-
When you call asin(2.0) or acos(5.0) (values outside the valid range), PySpark is supposed to return NaN. DuckDB
was returning NULL instead, which breaks code that expects PySpark behavior. Fixed by using a SQL workaround since
the internal C++ layer converts NaN to NULL by design. -
The spark.read.json() method had all the code written, but a raise NotImplementedError at the end blocked it
from ever returning. One line removal fixed it. -
Instead of hitting a generic NotImplementedError when encountering DuckDB union types (which PySpark doesn't
support), it now throws a clear ContributionsAcceptedError explaining the limitation.
Dharin-shah
changed the title
Minor bugfixes
[minor][spark] Minor bugfixes
Dharin-shah
marked this pull request as ready for review
Dharin-shah added 11 commits
December 8, 2025 08:53
evertlammerts
force-pushed
the
minor-bugfixes
branch
from
524c852 to
b8c499c
Compare
evertlammerts
changed the base branch from
main
to
v1.4-andium
Copy link Copy Markdown
Collaborator
evertlammerts
commented
Dec 8, 2025
evertlammerts commented
Dec 8, 2025Thanks! I rebased onto v1.4-andium. When the tests pass I'll do a review.
Dharin-shah
deleted the
minor-bugfixes
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment