Suddenly PM2 Stopped working - Caught Error : "AssertionError: Resource leak detected"

Last night, We had an outage in one of our nodejs micro services. All the services were stopped working suddenly. Nodejs services are  managed by PM2 in cluster mode.

I ran the following command to see any error in PM2 logs process  itself

 sudo pm2 logs PM2 --lines 2000

Below is the extract of the specific event happened  :

2018-04-23 00:00:00: --- PM2 global error caught ---------------------------------------------------
PM2        | 2018-04-23 00:00:00: Time                 : Mon Apr 23 2018 00:00:00 GMT+0000 (UTC)
PM2        | 2018-04-23 00:00:00: Resource leak detected.
PM2        | 2018-04-23 00:00:00: AssertionError: Resource leak detected.
PM2        |     at removeWorker (cluster.js:320:7)
PM2        |     at cluster.js:377:28
PM2        |     at ChildProcess.<anonymous> (/usr/lib/node_modules/pm2/lib/God.js:136:9)
PM2        |     at ChildProcess.g (events.js:260:16)
PM2        |     at emitTwo (events.js:87:13)
PM2        |     at ChildProcess.emit (events.js:172:7)
PM2        |     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)

 This Micro services were running and serving 3000 requests /sec for last few years wihtout any issues. Strange is n't it ?

 After this event, PM2 stopped all applications and  never started again. 

further analysis, This error is in nodejs cluster mode itslef . Not in PM2. There are PR for this issue and it seems it never pushed to nodejs release  :(

https://github.com/cnpm/cnpmjs.org/issues/536 

 Solution could be to migrate to latest nodejs ^5.0.0. Have to test it first. 

Nginx 502 Bad Gateway

This summary is not available. Please click here to view the post.

Useful Linux Commands - Memory consumed by Processes - in ascending order

Use the below command to display nice view of all processes sort by consumed memory in descending order

Linux / Unix Shell Command :

ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }'

Apache Solr 5.1 Installation on Ubuntu 14

• Install Java RunTime 

• sudo apt-get -y install openjdk-7-jdk

  Download & Install Solr from dist directly. 

  cd ~

  mkdir solr

  sudo wget http://archive.apache.org/dist/lucene/solr/5.1.0/solr-5.1.0.tgz
  sudo tar -xvf solr-5.1.0.tgz
  sudo cp -R solr-5.1.0/example /opt/solr
   
  java -jar server/start.jar 

IIS CPU Processor affinity Calculation Table

accepted

http://www.iis.net/configreference/system.applicationhost/applicationpools/add/cpu smpProcessorAffinityMask Optional uint attribute. Specifies the hexadecimal processor mask for multi-processor computers, which indicates to which CPU the worker processes in an application pool should be bound. Before this property takes effect, the smpAffinitized attribute must be set to true for the application pool. Note: On 64-bit computers, the smpProcessorAffinityMask attribute contains the low-order DWORD for the processor mask, and the smpProcessorAffinityMask2 attribute contains the high-order DWORD for the processor mask. On 32-bit computers, the smpProcessorAffinityMask2 attribute has no effect. If you set the value to 1 (which corresponds to 00000000000000001 in binary), the worker processes in an application pool run on only the first processor. If you set the value to 2 (which corresponds to 0000000000000010 in binary), the worker processes run on only the second processor. If you set the value to 3 (which corresponds to 0000000000000011 in binary) the worker processes run on both the first and second processors. Note: Do not set this property to 0. Doing so disables symmetric multiprocessing (SMP) affinity and creates an error condition. This means that processes running on one CPU will not remain affiliated with that CPU throughout their lifetime. The default value is 4294967295. First, set smpAffinitized to true. Processor identification goes from right to left. Starts at processor 0. To set the affinity to the last 12 processors: 0000 0000 1111 1111 1111 0000 0000 0000 Then you convert that to hex: 0x1FFE000

SFTP Server and Client

Terminologies

Good resource on key terminalogies has been given  in below link

https://www.bitvise.com/public-keys-in-ssh

https://www.bitvise.com/configuring-ssh-server-for-sftp

Points to remember:

1. client will have two keys (Private and public)  . Public key need to be shared to server and configured in server under user.

2. Server will have two keys (Private and public) . Both keys will not be shared to anybody. Server will show/send Server host key to client.
 
3.  While Authenticating Client, Server will show the SErver host key.. this need to be verified manually.

4. Filezilla server or client does not support SFTP.

5. Use bitvise for Server and Winscp for client.

6. SFTP  ( SSH FTP ) and  FTPS are completely different

Amazon EFS ( Elastic File System) vs S3 vs Glacier

     

EFS

• EFS launched as Network storage device for EC2 Instances with unlimited storage and Pay for Use pricing model.

• Its Available as Preview http://aws.amazon.com/efs/pricing/
• Current pricing is  $0.30/GB-Month
• Network filesystem- it can be attached to any number of EC2 Instances and you can expect slight latency
• Expensive than EBS Storage. But you only pay for use.. Let say if you use 5 GB , you just pay for 5 GB. Not the full volume size as in the EBS.
• Amazon claims fully scalable unlimited disk size.

S3 (Simple Storage File System)

• Can be consider as an Backup storage location..
• Cheaper than  EFS or EBS Volume. 10% cost of EFS.
•  $0.03/GB/Month

Glacier

• Offline backup

• Cheapest Storage
• Need to wait atleast 4 hours if you want your old glacier backup to be available to read it back
• Starts from $0.01/GB/Month
• Good for Long term backup.

For more information about EFS