Dowload of a large file (~4GB) is hanging

[ninkisa]

Version information:

go-ipfs version: 0.4.23-
Repo version: 7
System version: amd64/linux
Golang version: go1.13.7

Description:

Hello,

Somethimes download of large files (~4GB) is hanging.
I haven't noticed such problem with smaller files.
The files are added to ipfs with the nocopy option
I made a stacktrace and noticed a lot of hanging goroutines

ipfs-profile-7b6c73771327-2020-04-08T151813+0000.tar.gz

It is not happening with all files and it is hard to reproduce. Can not workaround it either.
Removing the block and running the Garbage Collector doesn't help
In this particular case:
checking the stats of the hash shows that the size is "CumulativeSize: 3881905526", but the download stops on 1.21 GiB

$ ipfs get QmeqTUSyqjxQqR7EEkp4665Raj6ghVSKb3Xjh3pMdjGjtz -o /root/export/cumulocity/sync-data/10.6/nightly-builds/EDGE-server-10.6.0.0-1486-HyperV.zip.ipart
Saving file(s) to /root/export/cumulocity/sync-data/10.6/nightly-builds/EDGE-server-10.6.0.0-1486-HyperV.zip.ipart
1.21 GiB / 3.61 GiB 33.37%^C0m01s22s
$
$
$ ipfs object stat QmeqTUSyqjxQqR7EEkp4665Raj6ghVSKb3Xjh3pMdjGjtz
NumLinks: 86
BlockSize: 4311
LinksSize: 3873
DataSize: 438
CumulativeSize: 3881905526

The ipfs nodes are part of a private network and in the case above only one peer is providing the object
Can you help with some suggestion how to resolve this issue?

[hsanjuan]

Hi, can you test with go-ipfs 0.5.0-rc1 ? https://dist.ipfs.io/go-ipfs/v0.5.0-rc1

[ninkisa]

Hi @hsanjuan ,

I tried updating but the repo migration failed with

bash-4.4# ipfs daemon --migrate
Initializing daemon...
go-ipfs version: 0.5.0-rc1
Repo version: 9
System version: 386/linux
Golang version: go1.13.8
Found outdated fs-repo, migrations need to be run.
=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
=> Failed to download fs-repo-migrations.
The migrations of fs-repo failed:
failed to download latest fs-repo-migrations: GET https://ipfs.io/ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_linux-musl-386.tar.gz error: 404 Not Found: ipfs resolve -r /ipfs/QmUgfXycSjF9R8F4Tyauaz6LZ4bj5nbksg54G9GdF4fit6/fs-repo-migrations/v1.5.1/fs-repo-migrations_v1.5.1_linux-musl-386.tar.gz: no link named "fs-repo-migrations_v1.5.1_linux-musl-386.tar.gz" under QmWgjNoXVSfZXJAG5LaovdDHMn7nT8hJGzXVeVHMq6fxy4

[hsanjuan]

are you building go-ipfs manually? We don't build 32bit musl targets. You may need to build your own fs-repo-migrations from source.

[hsanjuan]

@ninkisa did you delete a previous comment? can we close the issue?

[ninkisa]

Hello @hsanjuan ,
I'm building a custom docked image with ipfs. I changed the base image and the migartion ran fine, but now the node is not able to connect to any other peer. Should all the peers have the same ipfs version?
I'm getting
$ ipfs swarm connect /ip4/PeerIp/udp/4002/quic/ipfs/PeerID
Error: connect PeerID failure: failed to dial PeerID: all dials failed

• [/ip4/PeerIp/udp/4002/quic] context deadline exceeded

After reading the release notes, I tried to change the connection address to /ip4/PeerIp/udp/4002/quic/p2p/PeerID
but the error is the same.

But the host on which I observed the problem is a production host and I really hesitate to upgrade to a RC version. Even if I successfully update a host to version 0.5.0-rc1 on a test machine, I'm not sure I can reproduce the same situation. Are you planning to make some fixes to the current stable version, or we should wait until 0.5 become stable?

[hsanjuan]

Are you running a private network? QUIC does not work with private networks anymore. Can you try with regular tcp?

we should wait until 0.5 become stable?

You should wait for this.

[Stebalien]

RC2 and RC1 use an incompatible QUIC version. If you're not listening on TCP, they won't be able to talk to each other.

[RubenKelevra]

@ninkisa I think I've seen this bug before. An ipfs ping helps in most cases to continue the transfer.

To me it looks like the connection gets terminated, while the process transferring doesn't notice that and waits - if a new connection is established, the transfer continues.

[ninkisa]

@RubenKelevra ,
thank you for the hint. I will add 'ipfs ping' as a workaround.
Currently I'm not able to reproduce the same behavior with IPFS 5.
So I guess we will wait for the offitial release and upgrade
I will close this issue for now and reopen if I notice the same problem with version 5.

[RubenKelevra]

I would like to make sure this is a known limitation.

Do you use IPFS 0.5.0-rc3 on both sides?

Is the CID version 1 or version 0?

[RubenKelevra]

@ninkisa please reopen this, until we tracked down the issue. :)

[Stebalien]

This was likely libp2p/go-yamux#15.

[ninkisa]

Hello @RubenKelevra ,
I will reopen the issue as requested, but I'm not able to reproduce it with version 0.5
Yes, I upgraded IPFS on both sides.
Will add the ping workaround in case the issue apears in the peers that are still on 0.4.23

[RubenKelevra]

Hello @RubenKelevra ,
I will reopen the issue as requested, but I'm not able to reproduce it with version 0.5
Yes, I upgraded IPFS on both sides.
Will add the ping workaround in case the issue apears in the peers that are still on 0.4.23

Let's keep an eye on it for a while. I've had the issue disappear, too. But it was quite intermittently, so I'm not sure it's completely fixed.

[RubenKelevra]

@ninkisa I've tested extensively but couldn't reproduce the issue with 0.5.0. Had you any issues so far with 0.5.0? :)