[minor][spark] Minor bugfixes by Dharin-shah · Pull Request #180 · duckdb/duckdb-python

Skip to content

Navigation Menu

Sign in

Appearance settings

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

Appearance settings

Conversation

@Dharin-shah

Copy link Copy Markdown

Contributor

@Dharin-shah Dharin-shah commented

Nov 15, 2025

edited

Loading

This PR fixes 3 small bugs:

  1. When you call asin(2.0) or acos(5.0) (values outside the valid range), PySpark is supposed to return NaN. DuckDB
    was returning NULL instead, which breaks code that expects PySpark behavior. Fixed by using a SQL workaround since
    the internal C++ layer converts NaN to NULL by design.

  2. The spark.read.json() method had all the code written, but a raise NotImplementedError at the end blocked it
    from ever returning. One line removal fixed it.

  3. Instead of hitting a generic NotImplementedError when encountering DuckDB union types (which PySpark doesn't
    support), it now throws a clear ContributionsAcceptedError explaining the limitation.

@Dharin-shah Dharin-shah changed the title Minor bugfixes [minor][spark] Minor bugfixes

Nov 15, 2025

@Dharin-shah Dharin-shah marked this pull request as ready for review

December 2, 2025 22:20

@evertlammerts evertlammerts changed the base branch from main to v1.4-andium

December 8, 2025 07:55

@evertlammerts

Copy link Copy Markdown

Collaborator

Thanks! I rebased onto v1.4-andium. When the tests pass I'll do a review.

@evertlammerts evertlammerts merged commit 030a76a into duckdb:v1.4-andium

Dec 8, 2025

14 of 15 checks passed

@Dharin-shah Dharin-shah deleted the minor-bugfixes branch

December 8, 2025 21:08

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Dharin-shah @evertlammerts