March 06, 2018
Let’s find out what this error really means and how to fix it. Your WebSocket-enabled Node.js backend suddenly logged out this error:
Error: EMFILE, Too many open files
Here’s what I learned when I ran into this error hosting a Node.js app using WebSockets on Ubuntu.
The word ‘files’ is misleading. The problem really is that there are too many ‘file descriptors’ open. A file descriptor can be a lot of things, including a handle to an input or output resource, such as a network connection. If you are serving HTTP requests with HTTP responses, these file descriptors open and close quickly. They are open from the time the server receives the request until it finishes responding and closes the connection to the client. You might never reach the limit and see this error. However, if you open and maintain a WebSocket connection for each user who stays on the site, you’ll find the count of file descriptors going up and up when visitors stay on your site. The WebSocket connection stays open for much longer than the HTTP request/response connection, so the network connections (and open file descriptors) accumulate.
You can get the total count of open file descriptors with:
$ lsof | wc -l 5268
lsof lists open files,
wc -l counts the lines)
If you want to narrow this down to just 1 Node.js process’ open file descriptors, add a filter. Find your Node.js process’ PID
$ ps aux | grep node adam 14980 0.8 0.8 126212 24944 pts/1 Sl+ 12:29 0:02 node server.js
ps lists processes,
grep filters them)
I know the second column is my PID. I can use that to filter
$ lsof -p 14980 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME node 14980 adam cwd DIR 8,6 4096 4597208 /home/adam/Documents/socketio-demos/chat_after node 14980 adam rtd DIR 8,6 4096 2 / node 14980 adam txt REG 8,6 31604723 4855676 /home/adam/.nvm/versions/node/v7.0.0/bin/node node 14980 adam mem REG 8,6 1742312 4587818 /lib/i386-linux-gnu/libc-2.15.so node 14980 adam mem REG 8,6 124663 4588543 /lib/i386-linux-gnu/libpthread-2.15.so node 14980 adam mem REG 8,6 116232 4588449 /lib/i386-linux-gnu/libgcc_s.so.1 node 14980 adam mem REG 8,6 173576 4587822 /lib/i386-linux-gnu/libm-2.15.so node 14980 adam mem REG 8,6 905712 7345110 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 node 14980 adam mem REG 8,6 30684 4587807 /lib/i386-linux-gnu/librt-2.15.so node 14980 adam mem REG 8,6 13940 4587815 /lib/i386-linux-gnu/libdl-2.15.so node 14980 adam mem REG 8,6 134344 4588554 /lib/i386-linux-gnu/ld-2.15.so node 14980 adam 0u CHR 136,1 0t0 4 /dev/pts/1 node 14980 adam 1u CHR 136,1 0t0 4 /dev/pts/1 node 14980 adam 2u CHR 136,1 0t0 4 /dev/pts/1 node 14980 adam 3r FIFO 0,8 0t0 117628 pipe node 14980 adam 4w FIFO 0,8 0t0 117628 pipe node 14980 adam 5u 0000 0,9 0 6531 anon_inode node 14980 adam 6r FIFO 0,8 0t0 117629 pipe node 14980 adam 7w FIFO 0,8 0t0 117629 pipe node 14980 adam 8u 0000 0,9 0 6531 anon_inode node 14980 adam 9u CHR 136,1 0t0 4 /dev/pts/1 node 14980 adam 10r CHR 1,3 0t0 1056 /dev/null node 14980 adam 11u IPv6 119080 0t0 TCP *:http-alt (LISTEN) node 14980 adam 12u CHR 136,1 0t0 4 /dev/pts/1 node 14980 adam 15u IPv6 122405 0t0 TCP adam-m1330.local:http-alt->Adams-MacBook-Pro.local:64982 (ESTABLISHED) node 14980 adam 23u IPv6 119894 0t0 TCP localhost:http-alt->localhost:38218 (ESTABLISHED) node 14980 adam 24u IPv6 119436 0t0 TCP localhost:http-alt->localhost:38219 (ESTABLISHED)
These are all the open file descriptors for my Node.js process (the chat after demo from this socketio-demos repo). See the last 3 lines with a
IPv6? Those are file descriptors for 3 open web socket connections from 3 web browsers connected to a Node.js chatroom running on my M1330 laptop.
adam-m1330.local:http-alt->Adams-MacBook.local:64982is connecting from a MacBook in the same network
localhost:http-alt->localhost:38218and line after are two browser tabs on the same laptop as the server
And like all things that can be counted, there are limits in place to prevent systems from overloading themselves and crashing.
Linux puts limits on the amount of files a user (like the one who executes the Node.js process) can have open at once. Low limits can be good to keep one user from hogging too many resources on a server shared by many users. However, if you are deploying servers dedicated to running Node.js processes, you can safely raise this limit and give Node.js access to more resources.
You can see the limit with the command
$ ulimit -n 1024
To edit this limit, add new lines to the file /etc/security/limits.conf.
(if you want to all other user specific limits you can configure in this file, run
soft nofile 10000 hard nofile 10000
(the difference between hard and soft limits isn’t important in the context of a Node.js server, but I like this explanation if you are curious about it.)
The exact number is up to you. Consider:
Restart the machine, and run
ulimit -n to see if the value has stuck.
Ubuntu has an operating system wide limit on file descriptors as well. You can view it with the command
sysctl -a | grep fs.file-max:
$ sysctl -a | grep fs.file-max fs.file-max = 308115
If you need to raise this as well, edit the file /etc/sysctl.conf and add a new line with the new value, like this:
fs.file-max = 40000
Again, consider how high a limit you actually need, and what your server can handle.
To apply changes, run:
$ sysctl -p
-p will load the settings from /etc/sysctl.conf into memory)
Also, I wrote about open file descriptor counts in 2014.