refactor: Use inmemory Kubo API if the local node is used.

[philwinder]

This does 2 main things:
First, change the connection to the local IPFS node to directly wire the local CoreAPI object. This will make stack traces more usefull and skip an extra HTTP step, helping to debug.

Secondly, more correct type enforcement around the ipfs.Client object, there were many places with code like this:

result, err := doStuffFunc(params, client.APIAddress())

// ...
func doStuffFunc(params paramsT, api string) (result, error) {
  cl, err := ipfs.NewClient(api)
  if err != nil {
    return nil, err
  }

  // ...
}

All of thoses has been rewriten to:

result, err := doStuffFunc(params, client)

// ...
func doStuffFunc(params paramsT, cl ipfs.Client) (result, error) {
  // ...
}

Passing typed clients object is much more expressfull than strings.

Fixes #1523

[Jorropo]

@philwinder just so you know, this is an improvement but I don't think this is solving the exact case we intended to solve (meaningfull stack traces when debugging) because AFAIT the internal node is not used when doing bacalhau serve:

• https://github.com/filecoin-project/bacalhau/pull/1615/files#diff-74403c26da79c2fce414d91b010bd07077a62bdee24dfd61cb2ab7786581cb5dR305

I think a second PR that add support for inprocess kubo node will be needed.
Currently this will be only used for devstack:

• https://github.com/filecoin-project/bacalhau/pull/1615/files#diff-a472acc5a0cf5687333961449244d00376057828e2082755734fd89ef4585450R158

[philwinder]

Hi @Jorropo no I think this PR is what we want by default: serve uses HTTP, because providers have external IPFS clusters. Everything else uses internal.

Also, the specific problem I was having was with the get functionality, i.e. communicating with an embedded ipfs node to download results.

I've just finished my load tests with this version and the gets work 1000/1000 times. Previously it would fail about 1 in 10.

So, fingers crossed, I think this fixes that issue.

However, some of the tests are broken, so I need to fix them before merging.

But you're right, in the future, there could be an option of using a direct client with the serve command.

[Jorropo]

I've just finished my load tests with this version and the gets work 1000/1000 times. Previously it would fail about 1 in 10.

Honnestly that very surprising, there are possibilities that Kubo's http api client or server is at fault but except than this, this should be the same codepaths.
Anyway ot gonna complain if we get some fixes for free.

[philwinder]

Some issues with respect to how Bacalhau currently uses timeouts. Working through them.

Thread: https://filecoinproject.slack.com/archives/CRLC5H4J0/p1673347336829899
Issue: ipfs/kubo#9532

[Jorropo]

@philwinder this happen because your tests is using the same node to add and get the files and flatfs does not use context (see ipfs/kubo#9532 (comment)).

I think your easiest path here is to use two different IPFS nodes and have them bitswap each other content (which is the real thing we care about).