[WIP] WebAssembly support

[magik6k]

Part of #5694

Putting this in a PR for visibility

Currently this code is POC quality, mostly to test whether it may be actually viable to run go-ipfs in the javascript land (mostly browsers, I don't think running wasm go-ipfs in node makes much sense)

If we actually want to turn this into something usable, there will be a fair bit of cleaning up to do, but it's at lest a staring point.

How to run this thing:

• go get github.com/libp2p/go-reuseport, cd into it, git checkout feat/wasm, cd back into go-ipfs
• gx-go link QmXD921xzL9EDRpD6gRm1cb7Khm8VEpZ3NT3nPK7uTX6Fq (current go-reuseport hash)
• make misc/wasm/serve
• Go to http://127.0.0.1:8000, wait a bit for wasm blob to load, press 'Run', wait a bit more
• Type something into text box and press enter
• If you adjust bootstrap list to have your node in it, you should be able to fetch the resulting hash

[sriharrsha]

Well, 39MB is super huge. for Binary WASM of Go-IPFS. Is there any alternative to improve upon decreasing the size.

[magik6k]

There are some ways to compress the code ~10x, I'm not sure how portable to wasm though

[whyrusleeping]

@magik6k this is really awesome. Do you intend to work on it more? or maybe try to get it merged in some way?

[magik6k]

Do you intend to work on it more?

I currently have a rather long list of things to do, but I'd like to get this at least to the stage where we could see how well it works/could work (say, by comparing vs js-ipfs)

[whyrusleeping]

I managed to get the bundle down to 27mb by removing some of the larger dependencies we don't use here, like quic and btcec, and a bunch of other things. There's a lot more that can be removed, like some hash functions, the entire commands lib and ipfs commands, and anything related to a disk backed datastore (badger, leveldb, etc).

This is fun :D

[whyrusleeping]

spent some more time on this, its really fun.

This tool is really useful: https://github.com/jondot/goweight

I've gotten the uncompressed wasm blob now down to 15.5MB, 3.1MB gzipped. I did this by using hectors ipfs-lite repo, and trimming a bunch of things, notably eradicating any and all imports of net/httpfrom the dependency tree. This was difficult, but i did it by cutting out the go-libp2p-nat package (not needed in the browser) and removing all opentracing/opencensus stuff from go-log. Badger also got removed in favor of the in-memory datastore (should try and write something that uses javascript native indexedDB stuff). Additionally, bitswap was having a dependency on both "testing" and the libp2p mock net, i removed this dependency by just deleting the 'testutils.go' file in bitswap. We should make sure that only gets compiled for test builds.

A few things that stick out now:

• gogo protobuf contributes over 3MB to the uncompressed blob size. This seems absurd, protobuf marshaling and unmarshaling really shouldnt take up that much space. Maybe we can swap this out with a more 'trim' protobuf lib? (perhaps just write one?)
• The go runtime is 2.4MB. Eh, what can we do about it? I guess just bug the go team.
• the 'net' package, which shouldnt actually get used in wasm, contributes one whole MB. I think we just depend on some types from here. For wasm compiles, we should try and avoid any and all net function usage.
• the json package is contributing half a MB. Seems like something we could use javascript native stuff for? also why is this a 'core' ipfs dependency?
• my cbor fork is still being used apparently, we should switch over to refmt entirely, will save half a MB
• archive/tar is getting pulled in somehow. This is around 400kb.

In any case, i don't intend for the hack i'm playing with to get used, but what this points out is that a reasonably sized go-ipfs wasm blob is very practical. I think that with some more work, and improvements from the go team, we can get things to be very very small!

[whyrusleeping]

One very short term thing that I think would be really cool:

We can compile the go yamux package to wasm, and use it in js to make communication with go nodes easier! (cc @daviddias :) )

[Stebalien]

gogo protobuf contributes

Golang protobuf is 1.8MiB.

cbor

libp2p/go-libp2p-pnet#26

[bmiller59]

@whyrusleeping @magik6k @Kubuxu Is there any update on this? I see last activity was about a year ago. What is blocking this getting merged in and what do you see as the path forward? Thanks!

[Stebalien]

The @0xProject team has added wasm support to the websocket transport (https://github.com/libp2p/go-ws-transport/). Other than that, this is just waiting for somebody to pick it up.

[unicomp21]

Amazing idea, would be cool for both browser and nodejs, especially if typescript bindings could be generated.

[Thaina]

Was this idea retracted or having any roadblock?