[1/3] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] by mohamedawnallah · Pull Request #35216 · apache/beam
added 5 commits
June 9, 2025 17:22
Co-authored-by: Danny McCormick <dannymccormick@google.com>
mohamedawnallah
changed the title
[1/2] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid]
[1/3] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid]
changliiu pushed a commit to changliiu/beam that referenced this pull request
Jul 1, 2025…ybrid] (apache#35216) * sdks/python: add pymilvus dependency * sdks/python: add `MilvusSearchEnrichmentHandler` * sdks/python: test `MilvusSearchEnrichmentHandler` * sdks/python: itest `MilvusSearchEnrichmentHandler` * examples: add `MilvusSearchEnrichmentHandler` * sdks/python: combine milvus search strategies in one Co-authored-by: Danny McCormick <dannymccormick@google.com> * sdks/python/container: update image requirements * sdks/python: add license for `milvus_search.py` * sdks/python: add docstrings for `milvus_search.py` * sdks/python: unit test milvus search handler * sdks/python: update docstrings for milvus_search * sdks/python: fix linting for `milvus_search.py` * sdks/python: add more unit tests for milvus search * sdks/python: combine test classes in one * sdks/python: add `setuptools` as dependency * sdks/python: update container image requirements * sdks/python: update definition for `ANNS` field * .github/workflows: upgrade pip & setuptools * sdks/python: fix linting issue for `milvus_search` * sdks/python: group I/O types together milvus_search * .github/workflows: upgrade pip & setuptools * .github: unify upgrading setuptools & pip * sdks/python: fix linting for `milvus_search.py` * sdks/python: update grpcio for py<=3.12 * sdks/python: update image requirements * sdks/python: add `milvus-lite` manual license * sdks/python: fix `milvus_search_it_test` failed cases * sdks/python: unify access to sparse/dense embeddings * sdks/python: invoke `unittest.main` on milvus search * sdks/pyhon: make `MilvusSearchDBContainerInfo` optional for linting * sdks/python+website: update docs * sdks/python: fix linting issues for `milvus_search` component * sdks/python: fix linting issues for milvus search component * website: add missing doc for milvus search * sdks/python: add itests for milvus search * sdks/python: complete itests for milvus search * sdks/python: fix linting * sdks/python: address (can be) flaky test cases * website: update relase version for `enrichment-milvus.md` * sdks/python: fix failed unit tests for milvus search * sdks/python: fix linting for milvus search itests * website: update docs html to ref milvus enrichment handler * sdks/python: avoid port collision for milvus container * sdks/python: remove free port allocation for milvus search * sdks/python: fix formatting issues for milvus search * sdks/python: fix linting for milvus_search_it_test * sdks/python: handle port collisions for milvus search itest * sdks/python: increase timeout for milvus container * sdks/python: experiment being explicit about the port solve the CI issue * sdks+.github: experiment running ml deps CI test onubuntu solve issue * .github/workflwos: revert python precommit ml changes * sdks/python: fix CI issues for itests * sdks/python: fix linting for milvus search itests * examples/notebook: update milvus enrichment transform * website: update milvus enrichment transform * CHANGES.md: add note for milvus enrichment handler * sdks/python: update itests for milvus search * sdks/python: fix linting issues * multi: update * multi: update * updatet * update * update * sdks/python: fix linting issues * sdks/python: see what CI workflows would fail * .github: run beam_PreCommit_Python_ML only on ubuntu-20.04 runner * .github: test workflow * .github: revert changes * .github: add milvus-integration-tests.yml * .github: update milvus it workflow * update * .github: update milvus-tests workflow * .github: try to use ubuntu version `ubuntu-20.04` * .github+sdks/python: update itests * .github: update gh runner for milvus itests * .github: update milvus itests workflow * .github+sdks/python: update itests * .github: remove `milvus-integration-tests.yml` for the PR review * sdks/python: skip itests properly if milvus db container failed to start * skds/python: restructure the code order in the example * sdks/python: reduce number of retries to avoid test timeout * sdks/python: set internal testcontainer env variable for max retries * sdks/python: update tc max retries * sdks/python: update * sdks/python: use dynamic milvus service and healthcheck ports * sdks/python: fix linting issues for milvus search itest * sdks/python: fixing linting issues for milvus search itests * .github+sdks/python: reconfigure dependencies * sdks/python: address Danny's feedback (2) * examples/notebooks: update `milvus_enrichment_transform` * website+examples: remove non-functional docs/examples * website: revert updated `enrichment.md` * sdks/python: remove duplicated `HybridSearchParameters` * sdks/python: fix linting for milvus search * sdks/python: remove examples from this PR * .github/workflows: remove unnecesssary changes * CHANGES.md: undo the feature template * sdks/python: remove `pip` and `setuptools` as explicit dependency --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
jrmccluskey pushed a commit to jrmccluskey/beam that referenced this pull request
Jul 1, 2025…ybrid] (apache#35216) * sdks/python: add pymilvus dependency * sdks/python: add `MilvusSearchEnrichmentHandler` * sdks/python: test `MilvusSearchEnrichmentHandler` * sdks/python: itest `MilvusSearchEnrichmentHandler` * examples: add `MilvusSearchEnrichmentHandler` * sdks/python: combine milvus search strategies in one Co-authored-by: Danny McCormick <dannymccormick@google.com> * sdks/python/container: update image requirements * sdks/python: add license for `milvus_search.py` * sdks/python: add docstrings for `milvus_search.py` * sdks/python: unit test milvus search handler * sdks/python: update docstrings for milvus_search * sdks/python: fix linting for `milvus_search.py` * sdks/python: add more unit tests for milvus search * sdks/python: combine test classes in one * sdks/python: add `setuptools` as dependency * sdks/python: update container image requirements * sdks/python: update definition for `ANNS` field * .github/workflows: upgrade pip & setuptools * sdks/python: fix linting issue for `milvus_search` * sdks/python: group I/O types together milvus_search * .github/workflows: upgrade pip & setuptools * .github: unify upgrading setuptools & pip * sdks/python: fix linting for `milvus_search.py` * sdks/python: update grpcio for py<=3.12 * sdks/python: update image requirements * sdks/python: add `milvus-lite` manual license * sdks/python: fix `milvus_search_it_test` failed cases * sdks/python: unify access to sparse/dense embeddings * sdks/python: invoke `unittest.main` on milvus search * sdks/pyhon: make `MilvusSearchDBContainerInfo` optional for linting * sdks/python+website: update docs * sdks/python: fix linting issues for `milvus_search` component * sdks/python: fix linting issues for milvus search component * website: add missing doc for milvus search * sdks/python: add itests for milvus search * sdks/python: complete itests for milvus search * sdks/python: fix linting * sdks/python: address (can be) flaky test cases * website: update relase version for `enrichment-milvus.md` * sdks/python: fix failed unit tests for milvus search * sdks/python: fix linting for milvus search itests * website: update docs html to ref milvus enrichment handler * sdks/python: avoid port collision for milvus container * sdks/python: remove free port allocation for milvus search * sdks/python: fix formatting issues for milvus search * sdks/python: fix linting for milvus_search_it_test * sdks/python: handle port collisions for milvus search itest * sdks/python: increase timeout for milvus container * sdks/python: experiment being explicit about the port solve the CI issue * sdks+.github: experiment running ml deps CI test onubuntu solve issue * .github/workflwos: revert python precommit ml changes * sdks/python: fix CI issues for itests * sdks/python: fix linting for milvus search itests * examples/notebook: update milvus enrichment transform * website: update milvus enrichment transform * CHANGES.md: add note for milvus enrichment handler * sdks/python: update itests for milvus search * sdks/python: fix linting issues * multi: update * multi: update * updatet * update * update * sdks/python: fix linting issues * sdks/python: see what CI workflows would fail * .github: run beam_PreCommit_Python_ML only on ubuntu-20.04 runner * .github: test workflow * .github: revert changes * .github: add milvus-integration-tests.yml * .github: update milvus it workflow * update * .github: update milvus-tests workflow * .github: try to use ubuntu version `ubuntu-20.04` * .github+sdks/python: update itests * .github: update gh runner for milvus itests * .github: update milvus itests workflow * .github+sdks/python: update itests * .github: remove `milvus-integration-tests.yml` for the PR review * sdks/python: skip itests properly if milvus db container failed to start * skds/python: restructure the code order in the example * sdks/python: reduce number of retries to avoid test timeout * sdks/python: set internal testcontainer env variable for max retries * sdks/python: update tc max retries * sdks/python: update * sdks/python: use dynamic milvus service and healthcheck ports * sdks/python: fix linting issues for milvus search itest * sdks/python: fixing linting issues for milvus search itests * .github+sdks/python: reconfigure dependencies * sdks/python: address Danny's feedback (2) * examples/notebooks: update `milvus_enrichment_transform` * website+examples: remove non-functional docs/examples * website: revert updated `enrichment.md` * sdks/python: remove duplicated `HybridSearchParameters` * sdks/python: fix linting for milvus search * sdks/python: remove examples from this PR * .github/workflows: remove unnecesssary changes * CHANGES.md: undo the feature template * sdks/python: remove `pip` and `setuptools` as explicit dependency --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
This was referenced
Jul 13, 2025This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters