Not able to connect more devices to ThingWorx via proxy and connection server respectively

[VVarshney]

Hi all,

I am facing the issue of after a certain no. of devices I am not able to connect more devices to ThingWorx. My application structure is as below
Devices -> proxy server(nginx) -> connection server -> ThingWorx.
Each device will have 872 properties. We were able to connect around 3800 devices but after this count, devices will not get reflected on Thingworx. After checking logs found the below warning at the connection server.
[vert.x-acceptor-thread-0] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107)
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:145)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Thanks in advance for your help.

[PaiChung]

My initial thought would be that your server doesn't have enough CPU/Processing power to keep up with what you are trying to do.

Also you can play with the 'thread' settings to perhaps increase those. You can check in Monitoring/subsystems and see how the system is holding up.

You don't seem to be anywhere near the actual connection limit, but if you have available, you can try to set up a load balancer and connection servers to lower the active connection count into the server.

Also check the logs on the ThingWorx side

And as maybe a final possibility, do you have any limitation based on license?

[VVarshney]

Twx server has 16 CPU core and 64Gib memory while proxy server and connection server both have 4CPU core and 8Gib memory.

There is no log found on tomcat server.

We tried to change the below settings but found the same result we are not able to connect more than 3800 devices.

Tomcat side:

Updated values for following parameter as mentioned for tomcat server.
-Xms48g
-Xmx48g

updated value for following param in tomcat server.xml
-maxThreads 10000 ⇒ 16384
-maxConnections 10000 ⇒ 16384

Proxy settings:

updated the value of proxy_connect_timeout from 75 to 300 in /etc/nginx/conf.d/server.conf
updated the value of below param in /etc/nginx/nginx.conf"
-worker_connections 8132 ⇒ 16384
-keepalive_requests 8132 ⇒ 16384

Connection server
updated value for following param in cxserver.conf
-authentication.timeout-ms 20000 ⇒ 60000
-request.default_timeout-ms 20000 ⇒ 60000
-connections.timeout-ms 20000 ⇒ 60000

is there any other setting we need to change?

[slangley]

Hi @vvarshney.

Take a look at this article.  What is the ulimit currently set to?  Also, it's not clear where you're experiencing the problem. Is the log snippet you provided from the connection server?

Regards.

--Sharon

[VVarshney]

u limit I have increased from 1024 to 819200. but still, this warning is coming and more devices are not able to connect.

[VladimirRosu]

@VVarshney the error you posted above is well known if you search also in Google the first line - java.io.IOException: Too many open files. It is related to the default number of file descriptors which is too low.

My colleague slangley posted what I believe is the correct article to solve this - note that that article makes reference to the ThingWorx Platform, but you need to apply that also to the Connection Server level.

Separate question: does this question belong to one of your colleagues in the same project? It is very similar to your question. https://community.ptc.com/t5/ThingWorx-Developers/nginx/m-p/758449#M55369

[slangley]

Hi @vvarshney.

At this point, it might be best to open a case.  I'll be happy to open one on your behalf if you agree.

Regards.

--Sharon

[VVarshney]

Yes @slangley if you open the ticket it will be a great help. Thank you

[slangley]

Hi @vvarshney.

Please provide the version of ThingWorx you are running.  It's needed for the case.

Regards.

--Sharon

[VVarshney]

ThingWorx version is 8.5.4.

[slangley]

Hi @VVarshney.

Do you still need a case opened for this?  The previous communication was several months ago.  Please confirm, and I'll be happy to open one on your behalf.

Regards.

--Sharon