module: cache stat() results more aggressively by bnoordhuis · Pull Request #4575 · nodejs/node

added 3 commits

January 7, 2016 21:59
fs.statSync() creates and returns a heavy-weight fs.Stat object whereas
fs.accessSync() simply returns nothing.  The return value is ignored,
the call is for its side effect of throwing an ELOOP error in case of
cyclic symbolic links.
Reduce the number of stat() system calls that require() makes by caching
the results more aggressively.

To avoid unbounded growth without implementing a LRU cache, scope the
cache to the lifetime of the first call to require().  Recursive calls
(i.e. require() calls in the included code) transparently profit from
the cache.

The benchmarked application is the loopback-sample-app[0] and it sees
the number of stat calls at start-up go down by 40%, from 4736 to 2810.

[0] https://github.com/strongloop/loopback-sample-app
Avoid an unneeded ArgumentsAdaptorTrampoline stack frame by passing the
the right number of arguments to Module._load() in Module.require().
Shortens the following stack trace with one frame:

    LazyCompile:~Module.load module.js:345
    LazyCompile:Module._load module.js:282
    Builtin:ArgumentsAdaptorTrampoline
    LazyCompile:*Module.require module.js:361
    LazyCompile:*require internal/module.js:11

@bnoordhuis

Use internalModuleReadFile() to read the file from disk to avoid the
fs.fstatSync() call that fs.readFileSync() makes.

It does so to know the file size in advance so it doesn't have to
allocate O(n) buffers when reading the file from disk.

internalModuleReadFile() is plenty efficient though, even more so
because we want a string and not a buffer.  This way we also don't
allocate a buffer that immediately gets thrown away again.

This commit reduces the number of fstat() system calls in a benchmark
application[0] from 549 to 29, all made by the application itself.

[0] https://github.com/strongloop/loopback-sample-app

jasnell pushed a commit that referenced this pull request

Jan 12, 2016
fs.statSync() creates and returns a heavy-weight fs.Stat object whereas
fs.accessSync() simply returns nothing.  The return value is ignored,
the call is for its side effect of throwing an ELOOP error in case of
cyclic symbolic links.

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

jasnell pushed a commit that referenced this pull request

Jan 12, 2016
Reduce the number of stat() system calls that require() makes by caching
the results more aggressively.

To avoid unbounded growth without implementing a LRU cache, scope the
cache to the lifetime of the first call to require().  Recursive calls
(i.e. require() calls in the included code) transparently profit from
the cache.

The benchmarked application is the loopback-sample-app[0] and it sees
the number of stat calls at start-up go down by 40%, from 4736 to 2810.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

jasnell pushed a commit that referenced this pull request

Jan 12, 2016
Avoid an unneeded ArgumentsAdaptorTrampoline stack frame by passing the
the right number of arguments to Module._load() in Module.require().
Shortens the following stack trace with one frame:

    LazyCompile:~Module.load module.js:345
    LazyCompile:Module._load module.js:282
    Builtin:ArgumentsAdaptorTrampoline
    LazyCompile:*Module.require module.js:361
    LazyCompile:*require internal/module.js:11

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

jasnell pushed a commit that referenced this pull request

Jan 12, 2016
Use internalModuleReadFile() to read the file from disk to avoid the
fs.fstatSync() call that fs.readFileSync() makes.

It does so to know the file size in advance so it doesn't have to
allocate O(n) buffers when reading the file from disk.

internalModuleReadFile() is plenty efficient though, even more so
because we want a string and not a buffer.  This way we also don't
allocate a buffer that immediately gets thrown away again.

This commit reduces the number of fstat() system calls in a benchmark
application[0] from 549 to 29, all made by the application itself.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

@Trott Trott mentioned this pull request

Jan 13, 2016

@Trott Trott mentioned this pull request

Jan 13, 2016

rvagg pushed a commit that referenced this pull request

Jan 14, 2016
Avoid an unneeded ArgumentsAdaptorTrampoline stack frame by passing the
the right number of arguments to Module._load() in Module.require().
Shortens the following stack trace with one frame:

    LazyCompile:~Module.load module.js:345
    LazyCompile:Module._load module.js:282
    Builtin:ArgumentsAdaptorTrampoline
    LazyCompile:*Module.require module.js:361
    LazyCompile:*require internal/module.js:11

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

Trott added a commit to Trott/io.js that referenced this pull request

Jan 15, 2016
This reverts commit 809bf5e
("change statSync to accessSync in realpathSync").

It was causing tests to fail on Windows CI.

Ref: nodejs#4575
PR-URL: nodejs#4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Trott added a commit to Trott/io.js that referenced this pull request

Jan 15, 2016
This reverts commit 7c60328.

It is causing CI failures on Windows.

Ref: nodejs#4575
PR-URL: nodejs#4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Trott added a commit that referenced this pull request

Jan 16, 2016
This reverts commit 809bf5e
("change statSync to accessSync in realpathSync").

It was causing tests to fail on Windows CI.

Ref: #4575
PR-URL: #4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Trott added a commit that referenced this pull request

Jan 16, 2016
This reverts commit 7c60328.

It is causing CI failures on Windows.

Ref: #4575
PR-URL: #4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

@rvagg rvagg mentioned this pull request

Jan 19, 2016

evanlucas pushed a commit that referenced this pull request

Jan 19, 2016
Reduce the number of stat() system calls that require() makes by caching
the results more aggressively.

To avoid unbounded growth without implementing a LRU cache, scope the
cache to the lifetime of the first call to require().  Recursive calls
(i.e. require() calls in the included code) transparently profit from
the cache.

The benchmarked application is the loopback-sample-app[0] and it sees
the number of stat calls at start-up go down by 40%, from 4736 to 2810.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

evanlucas pushed a commit that referenced this pull request

Jan 20, 2016
Reduce the number of stat() system calls that require() makes by caching
the results more aggressively.

To avoid unbounded growth without implementing a LRU cache, scope the
cache to the lifetime of the first call to require().  Recursive calls
(i.e. require() calls in the included code) transparently profit from
the cache.

The benchmarked application is the loopback-sample-app[0] and it sees
the number of stat calls at start-up go down by 40%, from 4736 to 2810.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: #4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

evanlucas added a commit that referenced this pull request

Jan 20, 2016
Notable changes:

* events: make sure console functions exist (Dave) #4479
* fs: add autoClose option to fs.createWriteStream (Saquib) #3679
* http: improves expect header handling (Daniel Sellers) #4501
* node: allow preload modules with -i (Evan Lucas) #4696
* v8,src: expose statistics about heap spaces (`v8.getHeapSpaceStatistics()`) (Ben Ripkens) #4463
* Minor performance improvements:
  - lib: Use arrow functions instead of bind where possible (Minwoo Jung) #3622
  - module: cache stat() results more aggressively (Ben Noordhuis) #4575
  - querystring: improve parse() performance (Brian White) #4675

PR-URL: #4742

evanlucas added a commit that referenced this pull request

Jan 21, 2016
Notable changes:

* events: make sure console functions exist (Dave) #4479
* fs: add autoClose option to fs.createWriteStream (Saquib) #3679
* http: improves expect header handling (Daniel Sellers) #4501
* node: allow preload modules with -i (Evan Lucas) #4696
* v8,src: expose statistics about heap spaces (`v8.getHeapSpaceStatistics()`) (Ben Ripkens) #4463
* Minor performance improvements:
  - lib: Use arrow functions instead of bind where possible (Minwoo Jung) #3622
  - module: cache stat() results more aggressively (Ben Noordhuis) #4575
  - querystring: improve parse() performance (Brian White) #4675

PR-URL: #4742

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
fs.statSync() creates and returns a heavy-weight fs.Stat object whereas
fs.accessSync() simply returns nothing.  The return value is ignored,
the call is for its side effect of throwing an ELOOP error in case of
cyclic symbolic links.

PR-URL: nodejs#4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
Reduce the number of stat() system calls that require() makes by caching
the results more aggressively.

To avoid unbounded growth without implementing a LRU cache, scope the
cache to the lifetime of the first call to require().  Recursive calls
(i.e. require() calls in the included code) transparently profit from
the cache.

The benchmarked application is the loopback-sample-app[0] and it sees
the number of stat calls at start-up go down by 40%, from 4736 to 2810.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: nodejs#4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
Avoid an unneeded ArgumentsAdaptorTrampoline stack frame by passing the
the right number of arguments to Module._load() in Module.require().
Shortens the following stack trace with one frame:

    LazyCompile:~Module.load module.js:345
    LazyCompile:Module._load module.js:282
    Builtin:ArgumentsAdaptorTrampoline
    LazyCompile:*Module.require module.js:361
    LazyCompile:*require internal/module.js:11

PR-URL: nodejs#4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
Use internalModuleReadFile() to read the file from disk to avoid the
fs.fstatSync() call that fs.readFileSync() makes.

It does so to know the file size in advance so it doesn't have to
allocate O(n) buffers when reading the file from disk.

internalModuleReadFile() is plenty efficient though, even more so
because we want a string and not a buffer.  This way we also don't
allocate a buffer that immediately gets thrown away again.

This commit reduces the number of fstat() system calls in a benchmark
application[0] from 549 to 29, all made by the application itself.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: nodejs#4575
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
This reverts commit 809bf5e
("change statSync to accessSync in realpathSync").

It was causing tests to fail on Windows CI.

Ref: nodejs#4575
PR-URL: nodejs#4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
This reverts commit 7c60328.

It is causing CI failures on Windows.

Ref: nodejs#4575
PR-URL: nodejs#4679
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

scovetta pushed a commit to scovetta/node that referenced this pull request

Apr 2, 2016
Notable changes:

* events: make sure console functions exist (Dave) nodejs#4479
* fs: add autoClose option to fs.createWriteStream (Saquib) nodejs#3679
* http: improves expect header handling (Daniel Sellers) nodejs#4501
* node: allow preload modules with -i (Evan Lucas) nodejs#4696
* v8,src: expose statistics about heap spaces (`v8.getHeapSpaceStatistics()`) (Ben Ripkens) nodejs#4463
* Minor performance improvements:
  - lib: Use arrow functions instead of bind where possible (Minwoo Jung) nodejs#3622
  - module: cache stat() results more aggressively (Ben Noordhuis) nodejs#4575
  - querystring: improve parse() performance (Brian White) nodejs#4675

PR-URL: nodejs#4742