Suddenly PM2 Stopped working - Caught Error : "AssertionError: Resource leak detected"

Last night, We had an outage in one of our nodejs micro services. All the services were stopped working suddenly. Nodejs services are  managed by PM2 in cluster mode.

I ran the following command to see any error in PM2 logs process  itself

 sudo pm2 logs PM2 --lines 2000

Below is the extract of the specific event happened  :

2018-04-23 00:00:00: --- PM2 global error caught ---------------------------------------------------
PM2        | 2018-04-23 00:00:00: Time                 : Mon Apr 23 2018 00:00:00 GMT+0000 (UTC)
PM2        | 2018-04-23 00:00:00: Resource leak detected.
PM2        | 2018-04-23 00:00:00: AssertionError: Resource leak detected.
PM2        |     at removeWorker (cluster.js:320:7)
PM2        |     at cluster.js:377:28
PM2        |     at ChildProcess.<anonymous> (/usr/lib/node_modules/pm2/lib/God.js:136:9)
PM2        |     at ChildProcess.g (events.js:260:16)
PM2        |     at emitTwo (events.js:87:13)
PM2        |     at ChildProcess.emit (events.js:172:7)
PM2        |     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)

 This Micro services were running and serving 3000 requests /sec for last few years wihtout any issues. Strange is n't it ?

 After this event, PM2 stopped all applications and  never started again. 

further analysis, This error is in nodejs cluster mode itslef . Not in PM2. There are PR for this issue and it seems it never pushed to nodejs release  :(

https://github.com/cnpm/cnpmjs.org/issues/536 

 Solution could be to migrate to latest nodejs ^5.0.0. Have to test it first. 

Nginx 502 Bad Gateway

This summary is not available. Please click here to view the post.