1-Visitor

Question

Not able to connect more devices to ThingWorx via proxy and connection server respectively

Forum|Forum|4 years ago
November 16, 2021
4 replies
3006 views

Hi all,

I am facing the issue of after a certain no. of devices I am not able to connect more devices to ThingWorx. My application structure is as below
Devices -> proxy server(nginx) -> connection server -> ThingWorx.
Each device will have 872 properties. We were able to connect around 3800 devices but after this count, devices will not get reflected on Thingworx. After checking logs found the below warning at the connection server.
[vert.x-acceptor-thread-0] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107)
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:145)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Thanks in advance for your help.

Troubleshooting

P

PaiChung

22-Sapphire I

My initial thought would be that your server doesn't have enough CPU/Processing power to keep up with what you are trying to do.

Also you can play with the 'thread' settings to perhaps increase those. You can check in Monitoring/subsystems and see how the system is holding up.

You don't seem to be anywhere near the actual connection limit, but if you have available, you can try to set up a load balancer and connection servers to lower the active connection count into the server.

Also check the logs on the ThingWorx side

And as maybe a final possibility, do you have any limitation based on license?

V

VVarshney

Author

1-Visitor

Twx server has 16 CPU core and 64Gib memory while proxy server and connection server both have 4CPU core and 8Gib memory.

There is no log found on tomcat server.

We tried to change the below settings but found the same result we are not able to connect more than 3800 devices.

Tomcat side:

Updated values for following parameter as mentioned for tomcat server.
-Xms48g
-Xmx48g

updated value for following param in tomcat server.xml
-maxThreads 10000 ⇒ 16384
-maxConnections 10000 ⇒ 16384

Proxy settings:

updated the value of proxy_connect_timeout from 75 to 300 in /etc/nginx/conf.d/server.conf
updated the value of below param in /etc/nginx/nginx.conf"
-worker_connections 8132 ⇒ 16384
-keepalive_requests 8132 ⇒ 16384

Connection server
updated value for following param in cxserver.conf
-authentication.timeout-ms 20000 ⇒ 60000
-request.default_timeout-ms 20000 ⇒ 60000
-connections.timeout-ms 20000 ⇒ 60000

is there any other setting we need to change?

S

slangley

Community Manager

Hi @vvarshney.

Take a look at this article. What is the ulimit currently set to? Also, it's not clear where you're experiencing the problem. Is the log snippet you provided from the connection server?

Regards.

--Sharon

V

VVarshney

Author

1-Visitor

u limit I have increased from 1024 to 819200. but still, this warning is coming and more devices are not able to connect.

V

VladimirRosu_116627

19-Tanzanite

@VVarshney the error you posted above is well known if you search also in Google the first line - java.io.IOException: Too many open files. It is related to the default number of file descriptors which is too low.

My colleague slangley posted what I believe is the correct article to solve this - note that that article makes reference to the ThingWorx Platform, but you need to apply that also to the Connection Server level.

Separate question: does this question belong to one of your colleagues in the same project? It is very similar to your question. https://community.ptc.com/t5/ThingWorx-Developers/nginx/m-p/758449#M55369

S

slangley

Community Manager

Hi @vvarshney.

At this point, it might be best to open a case. I'll be happy to open one on your behalf if you agree.

Regards.

--Sharon