python.d.plugin: use separate process for initial module checking by ilyam8 · Pull Request #5552

python.d.plugin: use separate process for initial module checking by ilyam8 · Pull Request #5552 · netdata/netdata

added 11 commits

March 5, 2019 01:28

ilyam8 changed the title ~~[wip] python.d.plugin: use separate process for initial module checking~~ python.d.plugin: use separate process for initial module checking

Mar 5, 2019

ilyam8 deleted the pythond_plugin_refactor branch

March 12, 2019 13:48

jackyhuang85 pushed a commit to jackyhuang85/netdata that referenced this pull request

Jan 1, 2020

…tdata#5552)

##### Summary

This PR adds (major) changes only to `python.d.plugin` file.

Fixes: netdata#5525

`pyhton.d.plugin` imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.

___
Memory utilization comparing before/after the PR (one job `example` module, py3.7.2):

> 21.1 => 8.8 MiB

![screenshot_20190305_111837](https://user-images.githubusercontent.com/22274335/53791147-c27a6e00-3f39-11e9-8eaf-8ac3809a3b6e.png)


##### Component Name
[`collectors/python.d.plugin`](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/python.d.plugin.in)

##### Additional Information

This PR adds separate process for initial module checking. 

Logic:
 - main process spawns checker process
 - checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
 - checker process returns list of modules and jobs.
 - main process loads only active modules, etc.