(504) Gateway Timeout

[htran-21]

Hi,

I got a WSEMS Error: The remote server returned an error: (504) Gateway Timeout after 10 seconds waiting a response from server. I've looked into the config.json, but i don't see any value related to 10 seconds, is it content_read_timeout?

"http_server": {
	"host": "localhost", // Defaults to localhost - be careful, this likely means IPV6 localhost, not 127.0.0.1.
	"port": 8000, // Number. Defaults to 8000.
	"ssl": true, // Boolean. Enable SSL. Defaults to false.
	"certificate": "/path/to/certificate/file", // String. Default is empty.
	"private_key": "/path/to/private/key", //String. Default is empty.
	"passphrase": "password", //String. Default is empty.
	"authenticate": true, // Boolean. Enable Authentication. Defaults to false.
	"user": "johnsmith", // User for HTTP server.
	"password": "AES:EncryptedPassword", // Password for user. Can be encrypted.
	"content_read_timeout": 20000, // Setting to timeout when trying to read content from a POST or PUT.
	"ports_to_try": 10, // Number. How many additional ports to try, incrementing by one, if the server cannot bind to the configured port. Defaults to 10.
	"max_clients": 15, // Number. The maximum amount of concurrently connected clients. Defaults to 15.
	"enable_csrf_tokens" : true, // Boolean.  Flag to enable or disable the use of CSRF tokens on the server.
	"csrf_token_rotation_period" : 10 // Number. Use to set how often a CSRF token will rotate for a given session (in minutes).  Defaults to 10.
}

Any suggestions would be greatly appreciated!

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

Is "(504) Gateway Timeout" the full error?  Please provide the edge logs and the config.json file for further review.  You can mask any sensitive information in the file before forwarding.

There are some tags under ws_connection that default to 10 seconds.  You could change this by explicitly adding the tag in your config and changing the value.

Can you also clarify when this issue occurs?  During connect when executing a service, etc.?  Does it consistently fail?

Regards.

--Sharon

[htran-21]

Hi Sharon,

Yes, it is constantly failed whenever a service takes longer than 10 seconds on the server end. Edge code could not get a response because a 504 Gateway Timeout has occurred. I don't think that's related to ws_connection settings because there is no any Edge Thing defined in config.json yet at that moment.

Here is the WSEMS logs at that moment.

{"level":"DEBUG","timestamp":"2019-10-31T14:09:16,966","origin":"[-0500] HTTP Server","content":"0.0.0.0 - SeqId 52 - \"GET \/Thingworx\/Things\/SystemRepository\/Properties\/name HTTP\/1.1\" 200 146 \"-\"\n"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:16,967","origin":"TlsStream::doclose","content":"Disconeccting socket"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:16,970","origin":"SDK","content":"twTlsClient_Close: Disconnecting from server"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:16,977","origin":"SDK","content":"twTlsClient_Close: Disconnecting from server"}
{"level":"WARN","timestamp":"2019-10-31T14:09:24,865","origin":"SDK","content":"api:sendMessageBlocking: Message 15 timed out"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:24,867","origin":"SDK","content":"twOfflineMessageStore_Write: Stored message in offline msg file. RequestId 15"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:24,882","origin":"SDK","content":"twMessage_Delete:  Deleting REQUEST Message: 15"}
{"level":"INFO","timestamp":"2019-10-31T14:09:24,891","origin":"SDK","content":"twMessageHandler_CleanupOldMessages: Message 15 timed out"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:24,913","origin":"[-0500] HTTP Server","content":"0.0.0.0 - SeqId 36 - \"POST \/Thingworx\/Things\/DeviceIdentifierEdgeApp\/Services\/InvokeService HTTP\/1.1\" 504 0 \"-\"\n"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:27,23","origin":"SDK","content":"twTlsClient_Create: Initializing TLS Client"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:27,25","origin":"TlsStream:: doopen","content":"Setting SO_RCVTIMEO"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:27,35","origin":"RestThingworx::callback","content":"Handling request"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:27,41","origin":"RestThingworx","content":"Executing request isConnected"}
{"level":"DEBUG","timestamp":"2019-10-31T14:09:27,42","origin":"[-0500] HTTP Server","content":"0.0.0.0 - SeqId 54 - \"GET \/Thingworx\/Things\/LocalEms\/Properties\/isConnected HTTP\/1.1\" 200 154 \"-\"\n"}

{"level":"DEBUG","timestamp":"2019-10-31T14:09:24,913","origin":"[-0500] HTTP Server","content":"0.0.0.0 - SeqId 36 - \"POST \/Thingworx\/Things\/DeviceIdentifierEdgeApp\/Services\/InvokeService HTTP\/1.1\" 504 0 \"-\"\n"}

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

Can you provide config.json?

Regards.

--Sharon

[htran-21]

Hi Sharon,

config.json

{
	"ws_servers":	[{
			"host":	"blablabala",
			"port":	443
		}],
	"appKey":	"blablabala",
	"logger":	{
		"level":	"DEBUG",
		"audit_target":	"file://",
    "publish_directory": "C:/Thingworx/_tw_logs/",
		"publish_level":	"DEBUG",
		"max_file_storage":	"20000000",
		"auto_flush":	true
	},
	"http_server":	{
		"host":	"localhost",
		"port":	8000,
    "ssl": false
	},
	"offline_message_store":	{
		"enabled":	false,
    "directory": "C:/Thingworx/MicroServer/offlineStorage",
		"max_size":	65535
	},
	"certificates":	{
		"validate":	false,
		"allow_self_signed":	false
	},
  "auto_bind": [],
	"tunnel":	{
		"buffer_size":	8192,
		"read_timeout":	10,
		"idle_timeout":	300000,
		"max_concurrent":	4
	},
	"file":	{
		"buffer_size":	8192,
		"max_file_size":	81920000,
		"virtual_dirs":	[]
	}
}

Please let me know if you need anything else.

My Bests,

Hung Tran

[slangley]

Hi @htran-21.

If you have a support agreement, it might be easier to troubleshoot this over a WebEx session.  I will be happy to open a case for that if you will private message me with your company/contact info.

If not, please provide the full log as an attachment.  From the config.json, the only questionable thing we found is that you indicated offline message storage as false, but the log indicates otherwise.

Regards.

--Sharon

[slangley]

Hi @htran-21.

Thank you for providing your contact information via private email.  The case has been opened in your name.  Once the issue is resolved, we will update this post for others in the community who may be encountering the same issue.

Regards.

--Sharon

[slangley · Accepted solution]

Hi @htran-21.

From the case, it appears that the issue was resolved by setting msg_timeout to 30000 (default is 10000).  If you are in agreement that this was the fix for the issue, please mark this post as the Accepted Solution for the benefit of others on the community.

Regards.

--Sharon