Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Complement fails to run - synapse_rust.abi3.so: invalid ELF header #13803

Closed
MadLittleMods opened this issue Sep 13, 2022 · 4 comments · Fixed by #13811
Closed

Complement fails to run - synapse_rust.abi3.so: invalid ELF header #13803

MadLittleMods opened this issue Sep 13, 2022 · 4 comments · Fixed by #13811
Labels
A-Testing Issues related to testing in complement, synapse, etc O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact

Comments

@MadLittleMods
Copy link
Contributor

MadLittleMods commented Sep 13, 2022 •
edited
Loading

  • macOS 12.5.1
  • Docker desktop 4.6.1 (76265)
    • Engine: 20.10.13
    • Compose: 1.29.2
  • cargo --version -> cargo 1.63.0 (fd9c4297c 2022-07-01)
  • rustc --version -> rustc 1.63.0 (4b91a6ea7 2022-08-08)
  • Synapse: latest develop (12dacec)
  • Complement: latest main (matrix-org/complement@d404c9f)
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages
...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.9/site-packages/synapse/__init__.py", line 23, in <module>
    from synapse.util.rust import check_rust_lib_up_to_date
  File "/usr/local/lib/python3.9/site-packages/synapse/util/rust.py", line 20, in <module>
    from synapse.synapse_rust import get_rust_file_digest
ImportError: /usr/local/lib/python3.9/site-packages/synapse/synapse_rust.abi3.so: invalid ELF header
Full terminal output 
COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages
[+] Building 2.1s (28/28) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                              0.1s
 => => transferring dockerfile: 6.12kB                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.1s
 => => transferring context: 35B                                                                                                                                                  0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                        0.8s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                   0.0s
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                0.4s
 => [internal] load build context                                                                                                                                                 0.3s
 => => transferring context: 8.79MB                                                                                                                                               0.3s
 => [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:031f73dcb29e616220ea3acf6cce822226f17971647cd682909d7c6a87feffd2                                             0.0s
 => CACHED [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update -qq && apt-get  0.0s
 => CACHED [builder  2/10] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update -qq && apt-ge  0.0s
 => CACHED [builder  3/10] RUN mkdir /rust /cargo                                                                                                                                 0.0s
 => CACHED [builder  4/10] RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable                                                           0.0s
 => CACHED [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked     apt-get update -qq &&   0.0s
 => CACHED [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry==1.2.0"                                                                 0.0s
 => CACHED [requirements 4/6] WORKDIR /synapse                                                                                                                                    0.0s
 => CACHED [requirements 5/6] COPY pyproject.toml poetry.lock /synapse/                                                                                                           0.0s
 => CACHED [requirements 6/6] RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then     /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_S  0.0s
 => CACHED [builder  5/10] COPY --from=requirements /synapse/requirements.txt /synapse/                                                                                           0.0s
 => CACHED [builder  6/10] RUN --mount=type=cache,target=/root/.cache/pip   pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt      0.0s
 => CACHED [builder  7/10] COPY synapse /synapse/synapse/                                                                                                                         0.0s
 => CACHED [builder  8/10] COPY rust /synapse/rust/                                                                                                                               0.0s
 => CACHED [builder  9/10] COPY pyproject.toml README.rst build_rust.py /synapse/                                                                                                 0.0s
 => CACHED [builder 10/10] RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then     pip install --prefix="/install" --no-deps --no-warn-script-location /synapse[all];   else  0.0s
 => CACHED [stage-2 3/5] COPY --from=builder /install /usr/local                                                                                                                  0.0s
 => CACHED [stage-2 4/5] COPY ./docker/start.py /start.py                                                                                                                         0.0s
 => CACHED [stage-2 5/5] COPY ./docker/conf /conf                                                                                                                                 0.0s
 => exporting to image                                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                                           0.0s
 => => writing image sha256:6a0da90e6f95458e86c9751fc1003ba023cab25e187162d46f0c80214354b52e                                                                                      0.0s
 => => naming to docker.io/matrixdotorg/synapse                                                                                                                                   0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 1.6s (28/28) FINISHED
 => [internal] load build definition from Dockerfile-workers                                                                                                                      0.0s
 => => transferring dockerfile: 2.59kB                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 35B                                                                                                                                                  0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                        0.2s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                   0.0s
 => [internal] load build definition from Dockerfile-workers                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse:latest                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/redis:6-bullseye                                                                                                               0.7s
 => [internal] load metadata for docker.io/library/debian:bullseye-slim                                                                                                           0.7s
 => [internal] load build context                                                                                                                                                 0.0s
 => => transferring context: 30.13kB                                                                                                                                              0.0s
 => [deps_base 1/2] FROM docker.io/library/debian:bullseye-slim@sha256:5cf1d98cd0805951484f33b34c1ab25aac7007bb41c8b9901d97e4be3cf3ab04                                           0.0s
 => [redis_base 1/1] FROM docker.io/library/redis:6-bullseye@sha256:664d7932b44306ea2943ffde0b2cce4596198b88b81d22c73607ce572d55c63a                                              0.0s
 => [stage-2  1/14] FROM docker.io/matrixdotorg/synapse:latest                                                                                                                    0.0s
 => CACHED [stage-2  2/14] RUN --mount=type=cache,target=/root/.cache/pip         pip install supervisor~=4.2                                                                     0.0s
 => CACHED [stage-2  3/14] RUN mkdir -p /etc/supervisor/conf.d                                                                                                                    0.0s
 => CACHED [stage-2  4/14] COPY --from=redis_base /usr/local/bin/redis-server /usr/local/bin                                                                                      0.0s
 => CACHED [deps_base 2/2] RUN        --mount=type=cache,target=/var/cache/apt,sharing=locked        --mount=type=cache,target=/var/lib/apt,sharing=locked       apt-get update   0.0s
 => CACHED [stage-2  5/14] COPY --from=deps_base /usr/sbin/nginx /usr/sbin                                                                                                        0.0s
 => CACHED [stage-2  6/14] COPY --from=deps_base /usr/share/nginx /usr/share/nginx                                                                                                0.0s
 => CACHED [stage-2  7/14] COPY --from=deps_base /usr/lib/nginx /usr/lib/nginx                                                                                                    0.0s
 => CACHED [stage-2  8/14] COPY --from=deps_base /etc/nginx /etc/nginx                                                                                                            0.0s
 => CACHED [stage-2  9/14] RUN rm /etc/nginx/sites-enabled/default                                                                                                                0.0s
 => CACHED [stage-2 10/14] RUN mkdir /var/log/nginx /var/lib/nginx                                                                                                                0.0s
 => CACHED [stage-2 11/14] RUN chown www-data /var/log/nginx /var/lib/nginx                                                                                                       0.0s
 => CACHED [stage-2 12/14] COPY ./docker/conf-workers/* /conf/                                                                                                                    0.0s
 => CACHED [stage-2 13/14] COPY ./docker/prefix-log /usr/local/bin/                                                                                                               0.0s
 => CACHED [stage-2 14/14] COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py                                                                           0.0s
 => exporting to image                                                                                                                                                            0.1s
 => => exporting layers                                                                                                                                                           0.0s
 => => writing image sha256:8cb8fa71f2b695e8b87e27887badf7df2505b40b0d0045a31a27d269aae64e9a                                                                                      0.0s
 => => naming to docker.io/matrixdotorg/synapse-workers                                                                                                                           0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 1.3s (25/25) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 37B                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                        0.1s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse-workers:latest                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/postgres:13-bullseye                                                                                                           0.6s
 => [postgres_base 1/4] FROM docker.io/library/postgres:13-bullseye@sha256:e19ef49637c613183f56be5fa99ec33bcab03fd1a4279445a2f5ac224668bf03                                       0.0s
 => [internal] load build context                                                                                                                                                 0.0s
 => => transferring context: 177B                                                                                                                                                 0.0s
 => [stage-1  1/11] FROM docker.io/matrixdotorg/synapse-workers:latest                                                                                                            0.0s
 => CACHED [stage-1  2/11] RUN adduser --system --uid 999 postgres --home /var/lib/postgresql                                                                                     0.0s
 => CACHED [postgres_base 2/4] RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password                                                                          0.0s
 => CACHED [postgres_base 3/4] RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single                                                             0.0s
 => CACHED [postgres_base 4/4] RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single                                                                               0.0s
 => CACHED [stage-1  3/11] COPY --from=postgres_base /var/lib/postgresql /var/lib/postgresql                                                                                      0.0s
 => CACHED [stage-1  4/11] COPY --from=postgres_base /usr/lib/postgresql /usr/lib/postgresql                                                                                      0.0s
 => CACHED [stage-1  5/11] COPY --from=postgres_base /usr/share/postgresql /usr/share/postgresql                                                                                  0.0s
 => CACHED [stage-1  6/11] RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql                                                                                    0.0s
 => CACHED [stage-1  7/11] RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2                                                                                                  0.0s
 => CACHED [stage-1  8/11] COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2                                                                                            0.0s
 => CACHED [stage-1  9/11] WORKDIR /data                                                                                                                                          0.0s
 => CACHED [stage-1 10/11] COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf                                                                               0.0s
 => CACHED [stage-1 11/11] COPY conf/start_for_complement.sh /                                                                                                                    0.0s
 => exporting to image                                                                                                                                                            0.1s
 => => exporting layers                                                                                                                                                           0.0s
 => => writing image sha256:451ac8c1a9fe37b0697225300be78841be83467a519f11c8e45856dedf22526e                                                                                      0.0s
 => => naming to docker.io/library/complement-synapse                                                                                                                             0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Images built; running complement
2022/09/13 17:45:02 config: &{BaseImageURI:complement-synapse BaseImageURIs:map[] DebugLoggingEnabled:false AlwaysPrintServerLogs:true BestEffort:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] PackageNamespace:fed CACertificate:0xc000666000 CAPrivateKey:0xc00080f080}
=== RUN   TestImportHistoricalMessages
2022/09/13 17:45:02 Sharing [SERVER_NAME=hs1 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/09/13 17:45:33 fed.hs_with_application_service.hs1 : failed to deployBaseImage: fed.hs_with_application_service.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container 517a6c0340f058fa21b4a3a538d0b9d0f468943de9bd74b0b702d3764d49985f => health: unhealthy
2022/09/13 17:45:33 ============================================


2022/09/13 17:45:33 fed.hs_with_application_service.hs1 : Server logs:
Complement Synapse launcher
  Args:
  Env: SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=
Generating RSA private key, 2048 bit long modulus (2 primes)
.............+++++
.................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = hs1
Getting CA Private Key
                DNS:hs1
Generating a random secret for SYNAPSE_REGISTRATION_SHARED_SECRET
Generating a random secret for SYNAPSE_MACAROON_SECRET_KEY
Generating synapse config file /data/homeserver.yaml
Generating log config file /data/log.config
Setting ownership on /data to 991:991
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.9/site-packages/synapse/__init__.py", line 23, in <module>
    from synapse.util.rust import check_rust_lib_up_to_date
  File "/usr/local/lib/python3.9/site-packages/synapse/util/rust.py", line 20, in <module>
    from synapse.synapse_rust import get_rust_file_digest
ImportError: /usr/local/lib/python3.9/site-packages/synapse/synapse_rust.abi3.so: invalid ELF header
Traceback (most recent call last):
  File "/start.py", line 276, in <module>
    main(sys.argv, os.environ)
  File "/start.py", line 216, in main
    return generate_config_from_template(
  File "/start.py", line 137, in generate_config_from_template
    subprocess.check_output(args)
  File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gosu', '991:991', '/usr/local/bin/python', '-m', 'synapse.app.homeserver', '--config-path', '/data/homeserver.yaml', '--keys-directory', '/data', '--generate-keys']' returned non-zero exit status 1.
Generating base homeserver config
Traceback (most recent call last):
  File "/configure_workers_and_start.py", line 619, in <module>
    main(sys.argv, os.environ)
  File "/configure_workers_and_start.py", line 594, in main
    generate_base_homeserver_config()
  File "/configure_workers_and_start.py", line 302, in generate_base_homeserver_config
    subprocess.check_output(["/usr/local/bin/python", "/start.py", "migrate_config"])
  File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '/start.py', 'migrate_config']' returned non-zero exit status 1.
2022/09/13 17:45:33 ============== fed.hs_with_application_service.hs1 : END LOGS ==============


    msc2716_test.go:64: Deploy: Failed to construct blueprint: ConstructBlueprintIfNotExist(hs_with_application_service): failed to ConstructBlueprint: errors whilst constructing blueprint hs_with_application_service: [fed.hs_with_application_service.hs1: failed to check server is up. timed out checking for homeserver to be up: inspect container 517a6c0340f058fa21b4a3a538d0b9d0f468943de9bd74b0b702d3764d49985f => health: unhealthy]
--- FAIL: TestImportHistoricalMessages (30.81s)
FAIL
FAIL	github.com/matrix-org/complement/tests	33.760s
2022/09/13 17:45:00 config: &{BaseImageURI:complement-synapse BaseImageURIs:map[] DebugLoggingEnabled:false AlwaysPrintServerLogs:true BestEffort:false EnvVarsPropagatePrefix:PASS_ SpawnHSTimeout:30s KeepBlueprints:[] HostMounts:[] PackageNamespace:csapi CACertificate:0xc000590580 CAPrivateKey:0xc000021980}
testing: warning: no tests to run
PASS
ok  	github.com/matrix-org/complement/tests/csapi	0.932s [no tests to run]
FAIL
@MadLittleMods MadLittleMods added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. A-Testing Issues related to testing in complement, synapse, etc and removed T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. labels Sep 13, 2022
@reivilibre reivilibre changed the title Complement fails to run - synapse_rust.abi3.so: invalid ELF header Complement fails to run - synapse_rust.abi3.so: invalid ELF header on Mac Sep 14, 2022
@reivilibre reivilibre changed the title Complement fails to run - synapse_rust.abi3.so: invalid ELF header on Mac Complement fails to run - synapse_rust.abi3.so: invalid ELF header Sep 14, 2022
@reivilibre
Copy link
Contributor

In your case I think the problem is that the .so is built on your Mac host and passed down into a Linux guest (afaict Docker for Mac uses virtualisation).

On my machine, I have a newer libc than the container, so I have a similar problem:

ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/local/lib/python3.9/site-packages/synapse/synapse_rust.abi3.so)

N.B. On the host, the .so is in the source folder (🤮), so it gets copied across into the container. It appears the container does not rebuild it.

@MatMaul
Copy link
Contributor

MatMaul commented Sep 14, 2022

As a workaround for now deleting the .so from synapse folder fixes the trouble.
We should either:

  • not build the .so here but perhaps it's a req, I haven't done much Python packaging
  • rm -f the .so in the Dockerfile

Thoughts ?

@reivilibre
Copy link
Contributor

One solution would be to add a pattern to .dockerignore so that the .so isn't copied into the container.

@reivilibre reivilibre added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. S-Major Major functionality / product severely impaired, no satisfactory workaround. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact O-Uncommon Most users are unlikely to come across this or unexpected workflow labels Sep 14, 2022
@MatMaul
Copy link
Contributor

MatMaul commented Sep 14, 2022

Oh but I didn't know that file, seems like the way to go indeed.

@MatMaul MatMaul mentioned this issue Sep 14, 2022
Merged
4 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Testing Issues related to testing in complement, synapse, etc O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Docker build when Rust .so has been build locally first matrix-org/synapse
3 participants
@MadLittleMods @MatMaul @reivilibre