Fix fnamemodify()'s handling of :r after :e by bobrippling · Pull Request #5024 · vim/vim

@bobrippling

During `fnamemodify()`, for `":r"` we will iterate up the filename.
Ensuring that we don't go before the filename's first directory
separator (the tail) is insufficient in cases where we've already
handled a `":e"` modifier, for example:

```
"path/to/this.file.ext" :e:e:r:r
         ^    ^-------- *fnamep
         +------------- tail
```

This means for a `":r"`, we'll go before `*fnamep`, and outside the
bounds of the filename. This is both incorrect and causes vim to attempt
to allocate a lot of memory, which will either fails and we'll continue
with a null string, or we'll get a runtime allocation error.

The large memory allocation comes from calculating `s - *fnamep`. Since
`s` is before `*fnamep`, we caluclate a negative length, which ends up
being interpreted as an amount to allocate, causing the above problem.

We must instead ensure we don't go before `*fnamep` nor `tail`. The
check for `tail` is still relevant, for example, here we don't want to
go before it:

```
"path/to/this.file.ext" :r:r:r
 ^       ^------------- tail
 +--------------------- *fnamep
```