Automated: Migrate {target_path} from gsutil to gcloud storage

This CL is part of the on going effort to migrate from the legacy
`gsutil` tool to the new and improved `gcloud storage` command-line
interface.
`gcloud storage` is the recommended and modern tool for interacting with
Google Cloud Storage, offering better performance, unified
authentication, and a more consistent command structure with other
`gcloud` components. 🚀

### Automation Details

This change was **generated automatically** by an agent that targets
users of `gsutil`.
The transformations applied are based on the [gsutil to gcloud storage
migration guide](http://go/gsutil-gcloud-storage-migration-guide).

### ⚠️ Action Required: Please Review and Test Carefully

While we have based the automation on the migration guide, every use
case is unique.
**It is crucial that you thoroughly test these changes in environments
appropriate to your use-case before merging.**
Be aware of potential differences between `gsutil` and `gcloud storage`
that could impact your workflows.
For instance, the structure of command output may have changed,
requiring updates to any scripts that parse it. Similarly, command
behavior can differ subtly; the `gcloud storage rsync` command has a
different file deletion logic than `gsutil rsync`, which could lead to
unintended file deletions.

Our migration guides can help guide you through a list of mappings and
some notable differences between the two tools.

Standard presubmit tests are run as part of this CL's workflow. **If you
need to target an additional test workflow or require assistance with
testing, please let us know.**

Please verify that all your Cloud Storage operations continue to work as
expected to avoid any potential disruptions in production.

### Support and Collaboration

The `GCS CLI` team is here to help! If you encounter any issues, have a
complex use case that this automated change doesn't cover, or face any
other blockers, please don't hesitate to reach out.
We are happy to work with you to test and adjust these changes as
needed.

**Contact:** `gcs-cli-hyd@google.com`

We appreciate your partnership in this important migration effort!

#gsutil-migration

Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com>

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

fix: init mp pool & grpc client once, use os.sched_setaffinity

- mp pool is initalize once per benchmark run, otherwise time to init
pool is affecting throughput calculations.
- grpc client should be intialized once per process in an event loop,
otherwise processes get stuck .
- os.sched_setaffinity helps in pinning the process to a set of cores to
avoid hard irq from nic.

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

feat: Add system test for cross-region buckets
    
- Adds a new system test, test_basic_wrd_x_region, to verify
functionality with cross-region GCS buckets.
    
- Also updates the Cloud Build configuration to pass the necessary
_CROSS_REGION_BUCKET environment variable to the test environment



This PR is a cherry-picked from
#1760 because that PR
is blocked by b/489420625

Browse the repository at this point in the history

feat: Triger bidi conformance tests

Changes / fixes

- When (read/write) stream breaks, don't do `.close` on it. (this leads
to reading the same error message again and again until retry loop times
out)
- Bidi Conf test were ran directly as a module , change that so that
they get triggered in Kokoro (done that by adding `test_` so that pytest
pickup the tests)
- Provide a way to create anonymous connection in grpc client. 
- Unit tests and other minor changes.



~blocked because of b/489420625~

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Browse the repository at this point in the history

…th` (#1768)

fix(transfer_manager): Prevent path traversal in download_many_to_path

This PR addresses a security vulnerability where `download_many_to_path`
could be exploited to write files outside the intended destination
directory.

The fix ensures that the resolved path for each blob download remains
within the bounds of the user-provided `destination_directory`. If a
blob name would result in a path outside this directory (e.g., by using
`../`), a warning is issued, and that specific blob download is skipped.
This prevents directory traversal attacks.

Absolute paths in blob names (e.g., `/etc/passwd`) are now treated as
relative to the `destination_directory`, so `/etc/passwd` will be
downloaded to `destination_directory/etc/passwd`.

See b/449616593 for more details.

BREAKING CHANGE: Blobs that would resolve to a path outside the
`destination_directory` are no longer downloaded. While this is a
security fix, users relying on the previous behavior to write files
outside the target directory will see a change.

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

Implemented bucket encryption enforcement configuration support.
Added `EncryptionEnforcementConfig` and `BucketEncryption` classes.
Added `encryption` property to `Bucket`.
Added unit tests.

---
*PR created automatically by Jules for task
[10151670798613550332](https://jules.google.com/task/10151670798613550332)
started by @rajeevpodar*

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: rajeevpodar <3637722+rajeevpodar@users.noreply.github.com>
Co-authored-by: Nidhi Nandwani <nidhiii@google.com>

4 people authored

Browse the repository at this point in the history

The proto-plus library provides protocol buffer message classes that
behave like native Python types. While convenient, my profiling confirms
this abstraction comes with a significant performance penalty in our hot
paths. Every time we access a field on a proto-plus object, the library
triggers dynamic lookups and wrapper instantiation. In our hot data
ingestion loop, this overhead accumulates rapidly over multiple chunks.

My benchmarking shows that accessing the underlying C++ Protobuf
directly is ~2x faster than going through the proto-plus wrapper
(measured over 30,000 iterations). While 30,000 operations might sound
high, it corresponds to downloading just 60GB of data (assuming 2MB
chunks) for accessing just one attribute. For high-performance workloads
(e.g., downloading at 1GB/s) so it only takes 60s to complete the
process. Additionally this wrapper overhead introduces measurable
latency not just for data access, but for every metadata check and state
update that repeats per chunk.

This PR bypass proto-plus in our critical IO loops and interact directly
with the underlying C++ Protobuf structures. This eliminates the
"Wrapper tax" without changing the external behavior of the application.

Browse the repository at this point in the history

…load_many_to_path (#1773)

This PR updates `download_many_to_path` return type in
transfer_manager.py

It now returns `List[None|Exception|UserWarning]` 
- `None` for successful download
- `UserWarning` for 
- file skipped because `skip_if_exists=True` was provided and file
exists.
- Resolve path in either invalid or skips destination_directory (because
of `..`)
- `Exception` if error occurred during download.

Browse the repository at this point in the history

Browse the repository at this point in the history

Browse the repository at this point in the history

PR created by the Librarian CLI to initialize a release. Merging this PR
will auto trigger a release.

Librarian Version: v1.0.2-0.20251119154421-36c3e21ad3ac
Language Image:
us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91
<details><summary>google-cloud-storage: 3.10.0</summary>

##
[3.10.0](v3.9.0...v3.10.0)
(2026-03-18)

### Features

* [Bucket Encryption Enforcement] add support for bucket encryption
enforcement config (#1742)
([2a6e8b0](2a6e8b0))

### Perf Improvments

* [Rapid Buckets Reads] Use raw proto access for read resumption
strategy (#1764)
([14cfd61](14cfd61))
* [Rapid Buckets Benchmarks] init mp pool & grpc client once, use
os.sched_setaffinity (#1751)
([a9eb82c](a9eb82c))
* [Rapid Buckets Writes] don't flush at every append, results in bad
perf (#1746)
([ab62d72](ab62d72))


### Bug Fixes

* [Windows] skip downloading blobs whose name contain `":" ` eg: `C:`
`D:` etc when application runs in Windows. (#1774)
([5581988](5581988))
* [Path Traversal] Prevent path traversal in `download_many_to_path`
(#1768)
([700fec3](700fec3))
* [Rapid Buckets] pass token correctly, '&' instead of ',' (#1756)
([d8dd1e0](d8dd1e0))


</details>

Browse the repository at this point in the history