Emoji support 🎉 and other updates

[michael-gillett]

Features

• Fixed toByteArray and fromByteArray to have more general support for Unicode including emojis
• Added MODULARIZE flag to Makefile
  • This gives the ability to pass functions to the module constructor such as locateFile for Webpack support
• Cleaned up pre.js and post.js to work with MODULARIZE
• Upgrade JQ submodule to lastest version of master

API changes

• Changed jq(<object>, <filter>) to jq.json(<object>, <filter>)
• Changed jq.promised(<object>, <filter>) to jq.promised.json(<object>, <filter>)

Tests

• Added emoji example to tests
• Added jq.promised.* tests

README

• Added a section on getting .wasm to load properly in Webpack
• Expanded section on building
• Added a section on testing
• Updated API references

[fiatjaf]

Great changes, thank you!

I'll do a quick test tomorrow and then merge this.

[fiatjaf]

The MODULARIZE stuff isn't working properly. The global is called Module, is it possible that it is called jq? Also, if the module initialization code is moved to inside a function the API is broken in a very hard way.

Why do you need that? Is it related to the way the wasm file is loaded?

[michael-gillett]

I can change the module name back to jq, found a flag for it in emscripten. Can you explain more about how the API breaks when the module is initialized inside of a function and/ or provide a snippet?

It would be great to get this all working with the MODULARIZE flag since it makes it much easier to get the wasm file loaded in webpack. It also allows users to override various emscripten module functions by passing them into the constructor, including locateFile

Also is there any way I can help with your additional testing? I don't want to waste any of your time if I still have more work to do on this. Thus far I've only been using test.js to determine If I had gotten this working

[fiatjaf]

Currently the API doesn't require a constructor, there's a global / exported module that has the functions jq(), jq.raw(), jq.promised.raw() etc. and these functions are synchronous. There is a "promised" version just to take into account the loading of the wasm file or the asm.js mem file, and a onInitialized listener for that same reason (when you want to use synchronous functions instead of promised, but still want to make sure these asynchronous files are loaded).

I'm not sure about how it is supposed to work now. My test is just loading jq.wasm.js in a script tag and trying to run some functions, but it didn't work at all (first I was looking for a global named jq, but even when I tried with Module I couldn't do anything, I don't know how the asynchronous initialization should work).

If you want to help with tests perhaps you could implement a better way of doing tests in the browser, it could be with a headless browser or those test runners that run whenever you open the test.html -- I think tape supports that somehow.

But first we must do something about this API, I didn't find good documentation on how MODULARIZE works, but maybe there's a way to specify arguments to emscripten's constructor by setting some globals before importing jq or something like that?

[michael-gillett]

Awesome, thanks for the detailed response. I’ll get back with fixes in the next couple days. I would definitely like to get tape working with browser tests so that moving forward the test suite represents all the functionality we want.

I agree that it’s annoying that the module now requires setting the constructor so I’m going to get it back to the way it behaved before and then figure out passing arguments to emscripten

[michael-gillett]

Just pushed an update!

• Changed the module name from Module to jq
• No longer need to call the constructor on the module (like it was before)
  • I haven't tried to find a work around for passing Emscripten methods, but I did find another way to get it working with webpack by using copy-webpack-plugin and making sure the wasm ends up in the same directory as the webpack bundle.js
• Added web_test.js to make sure it works in the browser
  • run ./node_modules/live-server/live-server.js --open="web_test.html"
  • This just creates a basic webserver that opens the html file in your browser, I couldn't figure out a way to test the <script> import through tape

I think this addresses all of the points you made in your previous comment. Let me know if there is anything else that needs to be fixed :)

[fiatjaf]

Should we remove the locateFile snippet from the README?

[fiatjaf]

Do you understand why the jq.js.mem file isn't being generated anymore?

[fiatjaf]

I was thinking about using something like the Module.memoryInitializerRequest for loading the wasm file in cases where you couldn't place it in the same folder, but I don't know enough about it.

[michael-gillett]

We should remove the locateFile snippet now, I must have missed it. I may have accidentally removed a flag from the Makefile that generated the jq.js.mem, I'll take a look.

Module.memoryInitializerRequest could be another option, but I believe it still needs to get passed into the module to be used by the prescript. I can take another look at being able to pass functions without needing to call a constructor

I'll put up another PR later today with fixes

[fiatjaf]

I think it will work if you define Module as global before loading the actual Emscripten module.

i've removed the locateFile thing from the README.

I've also checked if you had removed code from the Makefile, but it seems you didn't. The problem is that everything seems to be working without the memfile so I'm confused.

[michael-gillett]

The .mem file is for asm.js so as long as your browser supports webassembly it won't mind that its missing. It looks like the version of Emscripten I used changed from defaulting to asm.js to wasm so we probably need to add a flag to the Makefile to get the mem file back.

It might make sense to change jq.wasm.js to jq.asm.js and have the default jq.js be wasm (I think it already is). While on the topic, what is the purpose of jq.bundle.js? It seems like it is built almost identially to jq.js

[fiatjaf]

It used to be:

jq.js -> asm.js build that needs the memfile
jq.bundle.js -> asm.js build with the memfile bundled in, bigger file and worse performance, but doesn't need the hassle of a memfile
jq.wasm.js -> build that will load the wasm compiled, much better and faster than the others

[fiatjaf]

Oh, you've found the explanation!

I don't know if we should change the defaults here, defaults are confusing (as we've just discovered), I think I'll make everything explicit, jq.asm.js, jq.wasm.js, jq.asm.bundle.js.