Skip to main content
D
DmitryTsarev17-Peridot
17-Peridot
February 12, 2021
Solved

Underscore characters in output from base64EncodeBytes

  • February 12, 2021
  • 2 replies
  • 7674 views

I'm writing a small Thingworx service to push some data from repository in Thingworx to an external service (Elasticsearch) over REST and need to send some data in Base64 format

 

I'm using LoadBinary and base64EncodeBytes to load and convert the data, but the output result of base64EncodeBytes snippet contains "_" characters which are illegal base64 characters.

I'm using the following code:

var binaryBlob = Things["Tsar_Repository"].LoadBinary({path: path});
var binaryBlobArray = [];
for (var i = 0; i < binaryBlob.length; i++) binaryBlobArray.push(binaryBlob[i]);
base64content = base64EncodeBytes({ array: binaryBlobArray });

 

(I then send it using PostJSON and it works for small .txt files

var content = {
"name": name,
"title": downloadLink,
"data": base64content
};
var params = {
url: "http://192.168.1.217:9200/someindex/_doc/?pipeline=attachment",
content: content,
};
var tt = Resources["ContentLoaderFunctions"].PostJSON(params);

 

For example, the string "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua." gets encoded as 

"TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg_c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWdu_YSBhbGlxdWEu"

 

The result of the base64EncodeBytes seem to be of String type, and I tried to get rid of the "_" characters using string operations (like .replace() / .substring()) but they don't work for a reason on the results from base64EncodeBytes snippet.

 

Am I overlooking something / are there any workarounds (or parameters I overlooked) to get results in "normal" base64 format?

Best answer by DmitryTsarev

Here is the promised solution / overeview of the subject matter.

I'm marking the thread as "Resolved", since there is a way to get the desired result, but I still believe the behaviour is incorrect / broken. 

@c_lowy , if you have any comments / thoughts regarding this, or my conclusions / steps are somehow flawed, I'd really appreciate some feedback / tips 🙂

2 replies

C
c_lowy16-Pearl
16-Pearl
February 12, 2021

Hi @DmitryTsarev,

 

If you output the value of base64content before calling PostJSON, do you see underscores at that stages ? Or rather the expected spaces ?

D
DmitryTsarev17-PeridotAuthor
17-Peridot
February 12, 2021

@c_lowy Yes, I can see underscores right in the output of the base64EncodeBytes. BTW, I don't expect spaces, I expect nothing instead of underscores. Just in case, I have attached a screen with some more details.

 

I also checked the content of intermediate blob Array (tmpBlobArray in the example below) and it does not seem to have any extra / unexpected characters, so underscores seem to be generated by base64EncodeBytes.

I'm fine with it, and would be ok to just manually remove them using something like str.replace(), but it doesn't work with the result of  base64EncodeBytes for a reason.

I'm not a seasoned JS developer so might be missing something obvious, but I couldn't neither find a cause nor find a workaround.

 

Here is a short and quick way to reproduce the behaviour:

 

var tmpBlob = Things["Tsar_Repository"].LoadBinary({path: "/ConfigurationLog.txt"});
var tmpBlobArray = [];
for (var i = 0; i < tmpBlob.length; i++) tmpBlobArray.push(tmpBlob[i]);
tmp_base64content = base64EncodeBytes({ array: tmpBlobArray });
logger.warn("tmp_base64: "+tmp_base64content);

 

Here is what is in the logs: 

2021-02-12 20:42:43.859+0300 [L: WARN] [O: S.c.t.d.e.DSLScript] [I: ] [U: Administrator] [S: ] [P: ] [T: https-openssl-nio-8443-exec-5] tmp_base64: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg_c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWdu_YSBhbGlxdWEu

 

The contents of the ConfigurationLog.txt is just the phrase from the post above.

C
c_lowy16-Pearl
16-Pearl
February 15, 2021

@DmitryTsarev, after doing some tests using the information and script you provided I suspect that the issue might acutally be with how the value of tmp_base64content is displayed in the ScriptLog. The actual value of this variable seems to be correct : TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEu.

You can see that if you set a String property value to tmp_base64content in the same service for example : me.StringProp = tmp_base64content;

 

 

Or if you output the value of tmp_base64content as the result of your service (in this case it will show some spaces, but the result of a Decode64 on that string would still be the same, see attachment).

 

So it looks like base64EncodeBytes is actually returning the expected value.

 

Did you see an unexpected result when doing PostJSON afterwards ?

    D
    DmitryTsarev17-PeridotAuthorAnswer
    17-Peridot
    February 16, 2021

    Here is the promised solution / overeview of the subject matter.

    I'm marking the thread as "Resolved", since there is a way to get the desired result, but I still believe the behaviour is incorrect / broken. 

    @c_lowy , if you have any comments / thoughts regarding this, or my conclusions / steps are somehow flawed, I'd really appreciate some feedback / tips 🙂

    C
    c_lowy16-Pearl
    16-Pearl
    April 28, 2021

    After discussing with R&D, no change is planed to be implemented to base64EncodeBytes which outputs \n char after every 76 index characters. As of today for content of type text, following services can be used : LoadText and then base64EncodeString and will not output \n.
    Previously mentioned 

      Terms & ConditionsCookie settingsAccessibility statement