panics on darwin arm #803

mangas · 2021-04-05T16:47:14Z

Summary of Bug

Following the docs on mac arm (apple m1) causes a panic. The same instructions work on amd64.

I'm happy to help look into it if anyone is available to pair on it, I'll have a look in meantime but some guidance would appreciated.

starting ABCI with Tendermint
unexpected fault address 0x5f6c61697469a2bd
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x5f6c61697469a2bd pc=0x1036a6eb0]

Version

v4.2.0

Steps to Reproduce

git clone -b v4.2.0 https://github.com/cosmos/gaia
cd gaia
make install
gaiad init chooseanicehandle
wget https://github.com/cosmos/mainnet/raw/master/genesis.cosmoshub-4.json.gz
gzip -d genesis.cosmoshub-4.json.gz
mv genesis.cosmoshub-4.json ~/.gaia/config/genesis.json
gaiad start --p2p.seeds bf8328b66dceb4987e5cd94430af66045e59899f@public-seed.cosmos.vitwit.com:26656,[email protected]:26656,[email protected]:26656,ba3bacc714817218562f743178228f23678b2873@public-seed-node.cosmoshub.certus.one:26656,[email protected]:26656 --x-crisis-skip-assert-invariants

For Admin Use

Not duplicate issue
Appropriate labels applied
Appropriate contributors tagged
Contributor assigned/self-assigned

The text was updated successfully, but these errors were encountered:

gsora · 2021-04-05T17:42:23Z

I wonder if Tendermint makes some running architecture assumptions...

What Go version are you compiling with?

Could you post the complete stack trace?

faddat · 2021-04-06T05:29:06Z

I do not have an M1 yet but have run a bunch of chains on arm64 Linux.

Arm64 Linux works quite well and so I bet that it is an issue with macos.

If you sent me a DM on twitter, I would be super happy to do a google meet + screenshare.

@mangas

I've never seen anything like that segfault before. Interesting.

tac0turtle · 2021-04-06T07:09:21Z

I have a m1, I run Tendermint all the time on my machine for testing. Will test gaia shortly

tac0turtle · 2021-04-06T07:18:31Z

I used two separate config files. One that was previously generated and one that was newly generated the old one works and the new comes with the same errors. @shahankhatch was there any sort of change that could cause this?

mangas · 2021-04-06T22:24:03Z

@marbar3778 could you post the files? I'm specifically interested in the database backend and possibly the indexer.

mangas · 2021-04-06T22:34:31Z

It works when I compile with the command below and set the database to badgerdb:

This is the default command generated by make build and I added badgerdb to the tags

go build -mod=readonly -tags "netgo ledger badgerdb" -ldflags '-X github.com/cosmos/cosmos-sdk/version.Name=gaia -X github.com/cosmos/cosmos-sdk/version.AppName=gaiad -X github.com/cosmos/cosmos-sdk/version.Version=v4.2.0 -X github.com/cosmos/cosmos-sdk/version.Commit=535be14a8bdbfeb0d950914b5baa2dc72c6b081c -X "github.com/cosmos/cosmos-sdk/version.BuildTags=netgo,ledger" -X github.com/tendermint/tendermint/version.TMCoreSemVer=v0.34.8 -w -s' -trimpath -o /Users/fa/git/gaia/build/ ./...

This is off tag v4.2.0.

shahankhatch · 2021-04-07T03:36:35Z

@marbar3778 Aside from sdk/tm version bumps, and some Makefile, doc and Docker updates, Gaia itself hasn't had changes that seem to impact building on an M1.

faddat · 2021-05-11T03:52:51Z

@shahankhatch correct.

It builds just fine, but well, I wanted to confirm and I love toys so I picked up one of these M1 MBP's.

Can fully confirm @mangas issue.

10:48AM INF Starting EventBus service impl=EventBus module=events
10:48AM INF Starting PubSub service impl=PubSub module=pubsub
unexpected fault address 0x5f6c61697469a2bd
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x5f6c61697469a2bd pc=0x104923010]

goroutine 26 [running]:
runtime.throw(0x10523fd01, 0x5)
        runtime/panic.go:1117 +0x54 fp=0x1400054d5a0 sp=0x1400054d570 pc=0x104237a74

faddat · 2021-05-11T03:59:15Z

@mangas I don't know the "why" of this, but build it with the badgerdb flag and it should run fine.

PS:

Actually it runs fine for about 5 minutes, and then it crashes in a different way.

cyborgshead · 2021-05-12T06:02:22Z

@faddat I have the same issue on M1, also on Nvidia Jetson AGX Xavier and Jetson Nano. Trace

This happens when trying to sync from other nodes in the network that already launched (e.g not local setup). Also tried with badgerdb - same issue

faddat · 2021-05-12T09:14:20Z

Thank you, it's helpful to know that we now have three confirmed.

Rpi works great... So now I'm very curious why the Jetson does not work.

faddat · 2021-05-13T22:24:19Z

@litvintech Do you have a raspberry pie model 4?

I think you would find that works on the RPI but not M1 ...which is weird...

cyborgshead · 2021-05-14T01:34:38Z

@faddat I think that RPi is too slow compared to others and that's why the chain is not halted. On M1 chain stop sync in less than 30 seconds, on Xavier it stops in around 1 min and a couple of minutes on Nano. (approx time as I remembered as tested 2 months ago)

Anyway, I cannot understand is it a general problem cause there is not enough reports. Does nobody run gaia and their chains on ARMs?

faddat · 2021-05-14T09:46:18Z

I have successfully run the latest gaia on ARM (raspi) but I moved my office and don't have one up on the latest version. Trying on my AWS graviton node now.

faddat · 2021-05-14T10:56:10Z

It doesn't work on my graviton either. As soon as I have a raspberry pie up I will test that but I believe that we are dealing with an ecosystem wide regression that prevents us from running on ARM CPUs

faddat · 2021-05-14T11:01:42Z

https://gist.github.com/faddat/c799093f7444899b7fd2508285c0d8d0

faddat · 2021-05-14T11:17:24Z

ubuntu@ip-172-31-63-80:~/gaia/cmd/gaiad$ ./gaiad start --p2p.seeds bf8328b66dceb4987e5cd94430af66045e59899f@public-seed.cosmos.vitwit.com:26656,[email protected]:26656,[email protected]:26656,ba3bacc714817218562f743178228f23678b2873@public-seed-node.cosmoshub.certus.one:26656,[email protected]:26656,[email protected]:26656 --x-crisis-skip-assert-invariants
11:09AM INF starting ABCI with Tendermint
11:09AM INF Starting multiAppConn service impl=multiAppConn module=proxy
11:09AM INF Starting localClient service connection=query impl=localClient module=abci-client
11:09AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
11:09AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
11:09AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
11:09AM INF Starting EventBus service impl=EventBus module=events
11:09AM INF Starting PubSub service impl=PubSub module=pubsub
11:09AM INF Starting IndexerService service impl=IndexerService module=txindex
11:09AM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=
11:09AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0

4.1.0 may resolve it, I know I've run the most recent gaia on pi

✔️ 4.1.0 works on arm64, bet it also works on M1.

@litvintech random thought: move your libs back to the cosmos-sdk version in 4.1.0, bet that works.

cyborgshead · 2021-05-14T16:19:05Z

@faddat I tested with the current dev branch of cyber that was upgraded to cosmos-sdk v42 (same as gaia 4.1.0). Will check again soon as will finish the upgrade to cosmos-sdk v43

faddat · 2021-05-14T18:11:13Z

That's interesting, thank you. I'll track what other libraries and stuff changed as I do so enjoy running my chains on arm.

tac0turtle · 2021-05-17T09:51:01Z

fixed the issue here cosmos/cosmos-sdk#9345

tac0turtle · 2021-05-20T09:23:39Z

this is fixed with https://github.com/cosmos/cosmos-sdk/releases/tag/v0.42.5. Gaia needs a release to fix it

faddat · 2021-07-30T10:31:54Z

Resolved and safe to close.

faddat · 2021-12-11T20:38:19Z

bump re: safe to close.

faddat mentioned this issue Apr 12, 2021

Add Darwin-arm64 support for the nodetime pkg ignite/cli#873

Closed

faddat mentioned this issue May 11, 2021

Generalized bloat #704

Closed

faddat mentioned this issue May 15, 2021

ARM Support is broken cosmos/cosmos-sdk#9343

Closed

4 tasks

shahankhatch mentioned this issue May 25, 2021

Release a Gaia 4.2.2 #862

Closed

6 tasks

nooomski closed this as completed Dec 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panics on darwin arm #803

panics on darwin arm #803

mangas commented Apr 5, 2021 •

edited

Loading

gsora commented Apr 5, 2021 •

edited

Loading

faddat commented Apr 6, 2021 •

edited

Loading

tac0turtle commented Apr 6, 2021

tac0turtle commented Apr 6, 2021 •

edited

Loading

mangas commented Apr 6, 2021

mangas commented Apr 6, 2021 •

edited

Loading

shahankhatch commented Apr 7, 2021

faddat commented May 11, 2021

faddat commented May 11, 2021 •

edited

Loading

cyborgshead commented May 12, 2021

faddat commented May 12, 2021

faddat commented May 13, 2021

cyborgshead commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021 •

edited

Loading

cyborgshead commented May 14, 2021 •

edited

Loading

faddat commented May 14, 2021

tac0turtle commented May 17, 2021

tac0turtle commented May 20, 2021

faddat commented Jul 30, 2021

faddat commented Dec 11, 2021

panics on darwin arm #803

panics on darwin arm #803

Comments

mangas commented Apr 5, 2021 • edited Loading

Summary of Bug

Version

Steps to Reproduce

For Admin Use

gsora commented Apr 5, 2021 • edited Loading

faddat commented Apr 6, 2021 • edited Loading

tac0turtle commented Apr 6, 2021

tac0turtle commented Apr 6, 2021 • edited Loading

mangas commented Apr 6, 2021

mangas commented Apr 6, 2021 • edited Loading

shahankhatch commented Apr 7, 2021

faddat commented May 11, 2021

faddat commented May 11, 2021 • edited Loading

cyborgshead commented May 12, 2021

faddat commented May 12, 2021

faddat commented May 13, 2021

cyborgshead commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021

faddat commented May 14, 2021 • edited Loading

cyborgshead commented May 14, 2021 • edited Loading

faddat commented May 14, 2021

tac0turtle commented May 17, 2021

tac0turtle commented May 20, 2021

faddat commented Jul 30, 2021

faddat commented Dec 11, 2021

mangas commented Apr 5, 2021 •

edited

Loading

gsora commented Apr 5, 2021 •

edited

Loading

faddat commented Apr 6, 2021 •

edited

Loading

tac0turtle commented Apr 6, 2021 •

edited

Loading

mangas commented Apr 6, 2021 •

edited

Loading

faddat commented May 11, 2021 •

edited

Loading

faddat commented May 14, 2021 •

edited

Loading

cyborgshead commented May 14, 2021 •

edited

Loading