cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Your Friends List is a way to easily have access to the community members that you interact with the most! X

Not able to connect more devices to ThingWorx via proxy and connection server respectively

VVarshney
4-Participant

Not able to connect more devices to ThingWorx via proxy and connection server respectively

Hi all,

I am facing the issue of after a certain no. of devices I am not able to connect more devices to ThingWorx. My application structure is as below
Devices -> proxy server(nginx) -> connection server -> ThingWorx.
Each device will have 872 properties. We were able to connect around 3800 devices but after this count, devices will not get reflected on Thingworx. After checking logs found the below warning at the connection server.
[vert.x-acceptor-thread-0] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110)
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107)
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:145)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Thanks in advance for your help.

10 REPLIES 10
PaiChung
22-Sapphire I
(To:VVarshney)

My initial thought would be that your server doesn't have enough CPU/Processing power to keep up with what you are trying to do.

Also you can play with the 'thread' settings to perhaps increase those. You can check in Monitoring/subsystems and see how the system is holding up.

You don't seem to be anywhere near the actual connection limit, but if you have available, you can try to set up a load balancer and connection servers to lower the active connection count into the server.

Also check the logs on the ThingWorx side

And as maybe a final possibility, do you have any limitation based on license?

VVarshney
4-Participant
(To:PaiChung)

Twx server has 16 CPU core and 64Gib memory while proxy server and connection server both have 4CPU core and 8Gib memory.

There is no log found on tomcat server.

We tried to change the below settings but found the same result we are not able to connect more than 3800 devices.

Tomcat side:

Updated values for following parameter as mentioned for tomcat server.
-Xms48g
-Xmx48g

updated value for following param in tomcat server.xml
-maxThreads 10000 ⇒ 16384
-maxConnections 10000 ⇒ 16384

 

Proxy settings:

updated the value of proxy_connect_timeout from 75 to 300 in /etc/nginx/conf.d/server.conf
updated the value of below param in /etc/nginx/nginx.conf"
-worker_connections 8132 ⇒ 16384
-keepalive_requests 8132 ⇒ 16384

 

Connection server
updated value for following param in cxserver.conf
-authentication.timeout-ms 20000 ⇒ 60000
-request.default_timeout-ms 20000 ⇒ 60000
-connections.timeout-ms 20000 ⇒ 60000

 

is there any other setting we need to change?

slangley
23-Emerald II
(To:VVarshney)

Hi @vvarshney.

 

Take a look at this article.  What is the ulimit currently set to?  Also, it's not clear where you're experiencing the problem. Is the log snippet you provided from the connection server?

 

Regards.

 

--Sharon

VVarshney
4-Participant
(To:slangley)

u limit I have increased from 1024 to 819200. but still, this warning is coming and more devices are not able to connect.

@VVarshney the error you posted above is well known if you search also in Google the first line - java.io.IOException: Too many open files. It is related to the default number of file descriptors which is too low.

My colleague slangley posted what I believe is the correct article to solve this - note that that article makes reference to the ThingWorx Platform, but you need to apply that also to the Connection Server level.

 

Separate question: does this question belong to one of your colleagues in the same project? It is very similar to your question. https://community.ptc.com/t5/ThingWorx-Developers/nginx/m-p/758449#M55369

 

slangley
23-Emerald II
(To:VVarshney)

Hi @vvarshney.

 

At this point, it might be best to open a case.  I'll be happy to open one on your behalf if you agree.

 

Regards.

 

--Sharon

VVarshney
4-Participant
(To:slangley)

Yes @slangley if you open the ticket it will be a great help. Thank you

slangley
23-Emerald II
(To:VVarshney)

Hi @vvarshney.

 

Please provide the version of ThingWorx you are running.  It's needed for the case.

 

Regards.

 

--Sharon

VVarshney
4-Participant
(To:slangley)

ThingWorx version is 8.5.4.

slangley
23-Emerald II
(To:VVarshney)

Hi @VVarshney.

 

Do you still need a case opened for this?  The previous communication was several months ago.  Please confirm, and I'll be happy to open one on your behalf.

 

Regards.

 

--Sharon

 

Top Tags