Running Multiple Instances Per Host To Improve Performance
You may find that Rocket.Chat slows down once you have a lot of concurrent users. When this sluggishness begins, you will likely see Rocket.Chat node process approaching 100% CPU (even if the host CPU load is low). This is due to the single-threaded nature of Node.js applications; they can't take advantage of multiple cores natively.
While it's possible to scale out by adding more servers (and this is recommended for HA purposes), you can achieve better utilization of your existing hardware by running multiple instances of the Rocket.Chat application (Node.js/Meteor app) on your current host(s). Of course, you only want to do this if you're already running on a multi-core machine. A reasonable rule-of-thumb may be to run N-1 Rocket.Chat instances, where N=num_cores.
Running multiple instances of Rocket.Chat on a single host requires a reverse proxy in front of your application. This tutorial assumes that you've already followed the tutorial for Running behind a Nginx SSL Reverse Proxy.
There's essentially just three steps:
  1. 1.
    Enable ReplicaSet on your MongoDB installation (https://docs.mongodb.com/manual/tutorial/deploy-replica-set/)
  2. 2.
    Start multiple instances of Rocket.Chat bound to different ports
  3. 3.
    Update your proxy to point at all local Rocket.Chat instances
We'll be working with Nginx in our examples, but it should be possible with other reverse proxies as well.

Run multiple instances of Rocket.Chat

We'll assume that you've configured Rocket.Chat to run as a systemd service. Since we want to run multiple instances simultaneously, we need to run at least two services. The only difference is the service name and port. If you don't have a service yet, the easiest way to do this for Rocket.Chat is to create a file in /usr/lib/systemd/system/ and call it rocketchat.service
1
[Unit]
2
Description=Rocket.Chat Server
3
After=syslog.target
4
After=network.target
5
6
[Service]
7
Type=simple
8
Restart=always
9
StandardOutput=syslog
10
SyslogIdentifier=RocketChat
11
User=rocketchat
12
Group=rocketchat
13
Environment=MONGO_URL=mongodb://your_mongodb:27017/your_database?replicaSet=your_replica_set_name
14
Environment=MONGO_OPLOG_URL=mongodb://your_mongodb1:27017/local?replicaSet=your_replica_set_name
15
Environment=ROOT_URL=https://your_rocketchat_domain.com
16
Environment=PORT=3000
17
WorkingDirectory=/path.to.rocketchat/rocket.chat
18
ExecStart=/usr/local/bin/node /path.to.rocketchat/rocket.chat/bundle/main.js
19
20
[Install]
21
WantedBy=multi-user.target
Copied!
Make sure the User and Group exist and both have read/write/execute Permissions for the rocketchat. Now you can run start, stop, restart, and status your rocketchat service.
If you want multiple Services create another file in /usr/lib/systemd/system and call it [email protected] with the following content:
1
[Unit]
2
Description=Rocket.Chat Server
3
After=syslog.target
4
After=network.target
5
6
[Service]
7
Type=simple
8
Restart=always
9
StandardOutput=syslog
10
SyslogIdentifier=RocketChat
11
User=rocketchat
12
Group=rocketchat
13
Environment=MONGO_URL=mongodb://your_mongodb:27017/your_database?replicaSet=your_replica_set_name
14
Environment=MONGO_OPLOG_URL=mongodb://your_mongodb1:27017/local?replicaSet=your_replica_set_name
15
Environment=ROOT_URL=https://your_rocketchat_domain.com
16
Environment=PORT=%I
17
WorkingDirectory=/path.to.rocketchat/rocket.chat
18
ExecStart=/usr/local/bin/node /path.to.rocketchat/rocket.chat/bundle/main.js
19
20
[Install]
21
WantedBy=rocketchat.service
Copied!
Start the other Rocket.Chat Services with
systemctl start [email protected] (or any other desired port after the @)
If you want to run rocketchat at boot just enable the services with
systemctl enable rocketchat
The other Services will be enable since they are "WantedBy"=RocketChat.service

Ensure nodes can communicate

If you run Rocket.Chat instances on multiple physical nodes. Or even in multiple containers make sure they can communicate with each other.
Rocket.Chat makes use of a peer to peer connection to inform each other of events. Let's say you type a message and tag a friend or coworker that is connected to another instance.
Two different events are fired: 1. The user (you) is typing 2. Notify user (friend)
Each Rocket.Chat instance registers in your database the ip address it detected for its self. Other instances then use this list to discover and establish connections with each other.
If you find instances unable to talk to each other you can try setting the INSTANCE_IP environment variable to the ip the other instances can use to talk to it.

Update your Nginx proxy config

Edit /etc/nginx/sites-enabled/default or if you use nginx from docker /etc/nginx/conf.d/default.conf and be sure to use your actual hostname in lieu of the sample hostname "your_hostname.com" below.
You just need to setup a backend if one doesn't already exist. Add all local Rocket.Chat instances to it. Then swap out the host listed in the proxy section for the backend you defined with no port.
Continuing the example, we'll update our Nginx config to point to the two Rocket.Chat instances that we started running on ports 3001 and 3002.
1
# Upstreams
2
upstream backend {
3
server 127.0.0.1:3000;
4
server 127.0.0.1:3001;
5
#server 127.0.0.1:3002;
6
#server 127.0.0.1:3003;
7
.
8
.
9
.
10
}
Copied!
Now just replace proxy_pass http://IP:3000/; with proxy_pass http://backend;. Updating the sample Nginx configuration would result in a config like this:
1
# HTTPS Server
2
server {
3
listen 443;
4
server_name your_hostname.com;
5
6
error_log /var/log/nginx/rocketchat.access.log;
7
8
ssl on;
9
ssl_certificate /etc/nginx/certificate.crt;
10
ssl_certificate_key /etc/nginx/certificate.key;
11
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don’t use SSLv3 ref: POODLE
12
13
location / {
14
proxy_pass http://backend;
15
proxy_http_version 1.1;
16
proxy_set_header Upgrade $http_upgrade;
17
proxy_set_header Connection "upgrade";
18
proxy_set_header Host $http_host;
19
20
proxy_set_header X-Real-IP $remote_addr;
21
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
22
proxy_set_header X-Forwarded-Proto https;
23
proxy_set_header X-Nginx-Proxy true;
24
25
proxy_redirect off;
26
}
27
}
Copied!
Now restart Nginx: service nginx restart

Update your Apache proxy config

Run this as root (to enable the necessary modules to use proxy balancer):
1
a2enmod proxy_html
2
a2enmod proxy_balancer
3
a2enmod headers
4
a2enmod session
5
a2enmod session_cookie
Copied!
Edit /etc/apache2/sites-enabled/rocketchat.conf and be sure to use your actual hostname in lieu of the sample hostname "your_hostname.com" below.
1
<VirtualHost *:443>
2
ServerAdmin [email protected]
3
ServerName chat.domain.com
4
5
LogLevel info
6
ErrorLog /var/log/chat.domain.com_error.log
7
TransferLog /var/log/chat.domain.com_access.log
8
9
SSLEngine On
10
SSLCertificateFile /etc/ssl/certs/chat.domain.com.crt
11
SSLCertificateKeyFile /etc/ssl/private/chat.domain.com.key
12
13
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
14
ProxyPreserveHost On
15
16
<Proxy balancer://http>
17
BalancerMember http://localhost:3000 route=1
18
BalancerMember http://localhost:3001 route=2
19
...
20
ProxySet stickysession=ROUTEID
21
</Proxy>
22
23
<Proxy balancer://ws>
24
BalancerMember ws://localhost:3000 route=1
25
BalancerMember ws://localhost:3001 route=2
26
...
27
ProxySet stickysession=ROUTEID
28
</Proxy>
29
30
<Location />
31
Require all granted
32
</Location>
33
34
RewriteEngine On
35
RewriteCond %{HTTP:Upgrade} =websocket [NC]
36
RewriteRule /(.*) balancer://ws/$1 [P,L]
37
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
38
RewriteRule /(.*) balancer://http/$1 [P,L]
39
40
ProxyPassReverse / http://localhost/
41
</VirtualHost>
Copied!
Now restart Apache: systemctl restart apache2.service
Visit https://your_hostname.com just as before the update. Ooh, so fast!
To confirm you're actually using both services like you'd expect, you can stop one rocketchat service at a time and confirm that chat still works. Restart that service and stop the other. Still work? Yep, you're using both services!

Check your database

Another very important part is your database. As mentioned above, you will need to make sure you are running a replicaset.
This is important for a couple of reasons: 1. Database reliability. You will want to make sure that your data is replicated, and you have another node if something happens to your primary. 2. Rocket.Chat does what's called oplog tailing. The oplog is turned on when you setup a replicaset. Mongo makes use of this to publish events so the other nodes in the replicaset can make sure its data is up to date. Rocket.Chat makes use of this to watch for database events. If someone sends a message on Instance 1 and you are connected to Instance 2. Instance 2 watches for message insert events and then is able to show you a new message has arrived.

Database engine

Another thing to keep in mind is the storage engine you are using. By default mongo uses Wiredtiger. Wiredtiger under some loads can be very CPU and Memory intensive. Under small single instance setups we don't typically see issues. But when you run multiple instances of Rocket.Chat it can sometimes get a bit unruly.
It's because of this we recommend in multiple instance situations that you switch the mongo storage engine to mmapv1.
Last modified 1yr ago