cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Underscore characters in output from base64EncodeBytes

DmitryTsarev
17-Peridot

Underscore characters in output from base64EncodeBytes

I'm writing a small Thingworx service to push some data from repository in Thingworx to an external service (Elasticsearch) over REST and need to send some data in Base64 format

 

I'm using LoadBinary and base64EncodeBytes to load and convert the data, but the output result of base64EncodeBytes snippet contains "_" characters which are illegal base64 characters.

I'm using the following code:

var binaryBlob = Things["Tsar_Repository"].LoadBinary({path: path});
var binaryBlobArray = [];
for (var i = 0; i < binaryBlob.length; i++) binaryBlobArray.push(binaryBlob[i]);
base64content = base64EncodeBytes({ array: binaryBlobArray });

 

(I then send it using PostJSON and it works for small .txt files

var content = {
"name": name,
"title": downloadLink,
"data": base64content
};
var params = {
url: "http://192.168.1.217:9200/someindex/_doc/?pipeline=attachment",
content: content,
};
var tt = Resources["ContentLoaderFunctions"].PostJSON(params);

 

For example, the string "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua." gets encoded as 

"TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg_c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWdu_YSBhbGlxdWEu"

 

The result of the base64EncodeBytes seem to be of String type, and I tried to get rid of the "_" characters using string operations (like .replace() / .substring()) but they don't work for a reason on the results from base64EncodeBytes snippet.

 

Am I overlooking something / are there any workarounds (or parameters I overlooked) to get results in "normal" base64 format?

1 ACCEPTED SOLUTION

Accepted Solutions

Here is the promised solution / overeview of the subject matter.

I'm marking the thread as "Resolved", since there is a way to get the desired result, but I still believe the behaviour is incorrect / broken. 

@c_lowy , if you have any comments / thoughts regarding this, or my conclusions / steps are somehow flawed, I'd really appreciate some feedback / tips

View solution in original post

11 REPLIES 11
c_lowy
13-Aquamarine
(To:DmitryTsarev)

Hi @DmitryTsarev,

 

If you output the value of base64content before calling PostJSON, do you see underscores at that stages ? Or rather the expected spaces ?

@c_lowy Yes, I can see underscores right in the output of the base64EncodeBytes. BTW, I don't expect spaces, I expect nothing instead of underscores. Just in case, I have attached a screen with some more details.

 

I also checked the content of intermediate blob Array (tmpBlobArray in the example below) and it does not seem to have any extra / unexpected characters, so underscores seem to be generated by base64EncodeBytes.

I'm fine with it, and would be ok to just manually remove them using something like str.replace(), but it doesn't work with the result of  base64EncodeBytes for a reason.

I'm not a seasoned JS developer so might be missing something obvious, but I couldn't neither find a cause nor find a workaround.

 

Here is a short and quick way to reproduce the behaviour:

 

var tmpBlob = Things["Tsar_Repository"].LoadBinary({path: "/ConfigurationLog.txt"});
var tmpBlobArray = [];
for (var i = 0; i < tmpBlob.length; i++) tmpBlobArray.push(tmpBlob[i]);
tmp_base64content = base64EncodeBytes({ array: tmpBlobArray });
logger.warn("tmp_base64: "+tmp_base64content);

 

Here is what is in the logs: 

2021-02-12 20:42:43.859+0300 [L: WARN] [O: S.c.t.d.e.DSLScript] [I: ] [U: Administrator] [S: ] [P: ] [T: https-openssl-nio-8443-exec-5] tmp_base64: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg_c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWdu_YSBhbGlxdWEu

 

The contents of the ConfigurationLog.txt is just the phrase from the post above.

c_lowy
13-Aquamarine
(To:DmitryTsarev)

@DmitryTsarev, after doing some tests using the information and script you provided I suspect that the issue might acutally be with how the value of tmp_base64content is displayed in the ScriptLog. The actual value of this variable seems to be correct : TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEu.

You can see that if you set a String property value to tmp_base64content in the same service for example : me.StringProp = tmp_base64content;

 

 

Or if you output the value of tmp_base64content as the result of your service (in this case it will show some spaces, but the result of a Decode64 on that string would still be the same, see attachment).

 

So it looks like base64EncodeBytes is actually returning the expected value.

 

Did you see an unexpected result when doing PostJSON afterwards ?

@c_lowy  Thank you for taking time to look at my issue.

 

Can't comment on the results of your investigation yet, but it sounds interesting - will definitely check it. I will try to send the content of the variable as a part of my REST request.

 

The issue which made me create the topic is that I was getting the 'illegal base64 character 5f' error after sending PostJSON to Elasticsearch.

 

From there I started debugging and discovered this _ character (U+005F) when outputting the value of the base64EncodeBytes variable to the logs.

 

So I observe the 'incorrect' behavior in two places.

 

While pinpointing the cause and possible solutions for this approach (using base64EncodeBytes), can you suggest any workarounds / other ways to encode file to base64 from inside Thingworx service?

@c_lowy Well... Seems I managed to resolve it and the solution is, as it often happens, is "obvious" (basically, str.replace worked, thought earlier I have written that it doesn't - yet to double-check).

Hindsight is always 20/20 they say, though :), and a few things are still unclear / not tested / not work as expected - will write back the end result a tad later.

 

If you have any thoughts until then or independently came to a different solution - don't let it go waste, I'd love to hear.

c_lowy
13-Aquamarine
(To:DmitryTsarev)

@DmitryTsarev thanks for your detailed feedback.

 

To answer one of your questions, the use of base64EncodeBytes is indeed the recommended practice to encode a file to base64. 

 

Like you I am seeing the underscores in the ScriptLog, but not when outputting the value of tmp_base64content as the result of the service, sending it to a String property or saving it to a text file in a repository. In those cases I am seeing spaces instead of the underscores. I need to check if those three actions somehow modify the content of tmp_base64content (for display purposes or similar).

 

I understand of course that using str.replace is more of a workaround here and that you expect the value of tmp_base64content to not contain any underscores. Are you calling PostJSON inside the same service ?

Thanks in advance !

@c_lowy  Yes, I'm calling PostJSON from the same service (you can see it on the video attached to the previous post, starting from 2:42).

 

I also showed on the video that there are some "extra" characters (this time "\n") added when saving the variable to JSON format from the service (~0:56).

 

Now, with a workaround available, I became genuinely interested in what could cause such a different behavior (underscores, spaces, "\n", "normal" string) in a kinda simple function which just returns a string. So looking forward to hearing some feedback from you

 

BTW, thank you once again, I really appreciate the time you spent investigating the case with me

c_lowy
13-Aquamarine
(To:DmitryTsarev)

@DmitryTsarev, thanks a lot for the thorough demonstration.

I will need to perform some additional tests and research to figure out the root cause for this behavior and whether anything needs fixing.

I will post an update here when I have one !

c_lowy
13-Aquamarine
(To:DmitryTsarev)

Hello @DmitryTsarev,

 

Thanks to the information you provided I was able to easily reproduce the issue you faced. I have reported this behavior to our R&D department to confirm whether this would be considered a bug. I will update the following article once I receive a feedback : When passing result of base64EncodeBytes into JSON blank characters are replaced with \n in ThingWor....

 

Thanks for reporting this issue !

Here is the promised solution / overeview of the subject matter.

I'm marking the thread as "Resolved", since there is a way to get the desired result, but I still believe the behaviour is incorrect / broken. 

@c_lowy , if you have any comments / thoughts regarding this, or my conclusions / steps are somehow flawed, I'd really appreciate some feedback / tips

View solution in original post

c_lowy
13-Aquamarine
(To:DmitryTsarev)

After discussing with R&D, no change is planed to be implemented to base64EncodeBytes which outputs \n char after every 76 index characters. As of today for content of type text, following services can be used : LoadText and then base64EncodeString and will not output \n.
Previously mentioned 

Announcements