Plugin and theme commands break on custom folder structures

followup to #243

WP-CLI: 2.5.0-alpha-02486ab
Command: wp plugin (e.g. wp plugin list)
PHP 7.4.7

The wp plugin command does not find any plugins if we set WP_PLUGIN_DIR using define( 'WP_PLUGIN_DIR', __DIR__ . '/world/' ); in my wp-config.
in 2.4.0 I get:
/var/www/vhosts/hello/world/

in 2.5.0 I get:
phar:///usr/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/world/

which is why suddenly the "plugin" command is broken in 2.5.0 (since my plugins are obviously not located in the phar directory)
So this definitely is a major bug, as wp cli completely changes the working directory and is therefore not running WP in the given path