Question
Creating Profile aborted - Spark maxResultSize
- July 31, 2018
- 0 replies
- 705 views
A 'CreateProfile' job is aborted, throwing this exception:
|
SparkException: Job aborted due to stage failure: Total size of serialized results of 69 tasks (1026.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
|
Obviously, thingworx analytics is using apache spark for analyzing datasets. I need the maxResultSize to be configurable in order to be able to get the profiles of a large dataset.
Is are any possibility to change the maxResultSize in the micro service?