Poetry 2.0 doesn't support the latest PyPy release

[henryiii]

Description

PyPy installs a few packages required for operation that are normally on PyPI into its stdlib folder. These are cffi, greenlet, hpy, and readline. In the latest release of PyPy, cffi (for PyPy 3.10 and 3.11) as well as hpy (only on PyPy 3.11) are unreleased versions that are not available on PyPI. Poetry doesn't seem to notice that these are not in site-packages, and even thinks it can remove them directly from the standard library (I think it doesn't actually uninstall them). I hit it on a minimal template using Poetry's versioning plugin.

Here's a MWE:

$ docker run --rm -it ubuntu:24.04
# apt update && apt install wget git bzip2 -y
# wget https://downloads.python.org/pypy/pypy3.10-v7.3.18-linux64.tar.bz2
# tar -xvf pypy3.10-v7.3.18-linux64.tar.bz2 -C /usr/local --strip-components=1
# pypy3 -m ensurepip
# pypy3 -m pip install poetry cookiecutter
# cookiecutter gh:scientific-python/cookie # Select Poetry backend (4) and all other defaults
# cd package
# poetry sync
Creating virtualenv package-ymOHZpyv-py3.10 in /root/.cache/pypoetry/virtualenvs
Ensuring that the Poetry plugins required by the project are available...
The following Poetry plugins are required by the project but are not installed in Poetry's environment:
  - poetry-dynamic-versioning[plugin] (>=1.0.0)
Installing Poetry plugins only for the current project...
Updating dependencies
Resolving dependencies... (7.1s)

Because poetry-project-instance depends on cffi (1.18.0.dev0) which doesn't match any versions, version solving failed.

You can see what's happening if you leave off the VCS plugin:

# cookiecutter gh:scientific-python/cookie # Select Poetry backend (4) and select "no" for vcs versioning
# cd package
# poetry sync
Creating virtualenv package-ymOHZpyv-py3.10 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (3.3s)

Package operations: 8 installs, 0 updates, 4 removals

  - Removing cffi (1.18.0.dev0)
  - Removing greenlet (0.4.13)
  - Removing hpy (0.9.0)
  - Removing readline (6.2.4.1)
  - Installing packaging (24.2)
  - Installing tomli (2.2.1)
  - Installing exceptiongroup (1.2.2)
  - Installing iniconfig (2.0.0)
  - Installing pluggy (1.5.0)
  - Installing coverage (7.6.12)
  - Installing pytest (8.3.4)
  - Installing pytest-cov (6.0.0)

Writing lock file

Installing the current project: package (0.1.0)

But these packages are in PyPy's standard library! You can still see and import cffi in the environment, even though it claims it's been "removed".

Too many of poetry's dependencies don't provide PyPy3.11 pre-built wheels to test this on 3.11, but I assume hpy might also complain in that case.

Using Poetry from pipx does the same thing.

Workarounds

I believe Poetry just needs to ignore any package that are in Python's standard library. I'm not sure the best case for locking with a package that includes one of these dependencies, but since you can't install these (the stdlib version takes precedence anyway), I think they should just be ignored - that's basically what already happens on older PyPy's, except Poetry is now tripping up over not being able to find the unreleased versions on PyPy.

Poetry Installation Method

pip

Operating System

ubuntu 24.04

Poetry Version

Poetry (version 2.0.1)

Poetry Configuration

cache-dir = "/root/.cache/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /root/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

sysconfig.log

Platform: "linux-x86_64"
Python version: "3.10"
Current installation scheme: "posix_prefix"

Paths:
  data = "/usr/local"
  include = "/usr/local/include/pypy3.10"
  platinclude = "/usr/local/include/pypy3.10"
  platlib = "/usr/local/lib/pypy3.10/site-packages"
  platstdlib = "/usr/local/lib/pypy3.10"
  purelib = "/usr/local/lib/pypy3.10/site-packages"
  scripts = "/usr/local/bin"
  stdlib = "/usr/local/lib/pypy3.10"

Variables:
  ABIFLAGS = ""
  AR = "ar"
  ARFLAGS = "rc"
  CC = "cc -pthread"
  CCSHARED = "-fPIC"
  CFLAGS = "-DNDEBUG -O2"
  CONFINCLUDEPY = "/usr/local/include/pypy3.10"
  CXX = "c++ -pthread"
  EXE = ""
  EXT_SUFFIX = ".pypy310-pp73-x86_64-linux-gnu.so"
  INCLUDEPY = "/usr/local/include/pypy3.10"
  LDCXXSHARED = "c++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions"
  LDFLAGS = "-Wl,-Bsymbolic-functions"
  LDLIBRARY = "libpypy3.9-c.so"
  LDSHARED = "cc -pthread -shared -Wl,-Bsymbolic-functions"
  LDVERSION = "3.10"
  LIBDIR = "/usr/local/bin"
  LIBRARY = ""
  MULTIARCH = "x86_64-linux-gnu"
  OPT = "-DNDEBUG -O2"
  Py_DEBUG = "0"
  Py_ENABLE_SHARED = "0"
  SHLIB_SUFFIX = ".so"
  SIZEOF_VOID_P = "8"
  SO = ".pypy310-pp73-x86_64-linux-gnu.so"
  SOABI = "pypy310-pp73"
  TZPATH = "/usr/local/share/zoneinfo:/usr/local/lib/zoneinfo:/usr/local/share/lib/zoneinfo:/usr/local/../etc/zoneinfo:/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo"
  VERSION = "3.10"
  abiflags = ""
  base = "/usr/local"
  exec_prefix = "/usr/local"
  implementation = "PyPy"
  implementation_lower = "pypy"
  installed_base = "/usr/local"
  installed_platbase = "/usr/local"
  platbase = "/usr/local"
  platlibdir = "lib"
  prefix = "/usr/local"
  projectbase = "/usr/local/bin"
  py_version = "3.10.16"
  py_version_nodot = "310"
  py_version_nodot_plat = ""
  py_version_short = "3.10"
  srcdir = "/usr/local/bin"
  userbase = "/root/.local"

Example pyproject.toml

[build-system]
requires = ["poetry-core>=2.0.0", "poetry-dynamic-versioning"]
build-backend = "poetry_dynamic_versioning.backend"


[project]
name = "package"
authors = [
  { name = "My Name", email = "[email protected]" },
]
description = "A great package."
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
  "Development Status :: 1 - Planning",
  "Intended Audience :: Science/Research",
  "Intended Audience :: Developers",
  "License :: OSI Approved :: BSD License",
  "Operating System :: OS Independent",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: 3.12",
  "Programming Language :: Python :: 3.13",
  "Topic :: Scientific/Engineering",
  "Typing :: Typed",
]
dynamic = ["version"]
dependencies = []

[project.optional-dependencies]
test = [
  "pytest >=6",
  "pytest-cov >=3",
]
dev = [
  "pytest >=6",
  "pytest-cov >=3",
]
docs = [
  "sphinx>=7.0",
  "myst_parser>=0.13",
  "sphinx_copybutton",
  "sphinx_autodoc_typehints",
  "furo>=2023.08.17",
]

[project.urls]
Homepage = "https://github.com/org/package"
"Bug Tracker" = "https://github.com/org/package/issues"
Discussions = "https://github.com/org/package/discussions"
Changelog = "https://github.com/org/package/releases"
[tool.poetry]
version = "0.0.0"

[tool.poetry.requires-plugins]
poetry-dynamic-versioning = { version = ">=1.0.0", extras = ["plugin"] }

[tool.poetry-dynamic-versioning]
enable = true
substitution.files = ["src/package/__init__.py"]

[tool.poetry.group.test.dependencies]
pytest = ">= 6"
pytest-cov = ">= 3"

Poetry Runtime Logs

poetry-runtime.log

Loading configuration file /root/.config/pypoetry/config.toml
Trying to detect current active python executable as specified in the config.
Found: /usr/local/bin/python
Using virtualenv: /root/.cache/pypoetry/virtualenvs/package-ymOHZpyv-py3.10
Ensuring that the Poetry plugins required by the project are available...
The following Poetry plugins are required by the project but are not installed in Poetry's environment:
- poetry-dynamic-versioning[plugin] (>=1.0.0)
Installing Poetry plugins only for the current project...
Updating dependencies
Resolving dependencies...
 1: fact: poetry-project-instance is 0
 1: derived: poetry-project-instance
 1: fact: poetry-project-instance depends on cffi (1.18.0.dev0)
 1: fact: poetry-project-instance depends on greenlet (0.4.13)
 1: fact: poetry-project-instance depends on hpy (0.9.0)
 1: fact: poetry-project-instance depends on readline (6.2.4.1)
 1: fact: poetry-project-instance depends on secretstorage (3.3.3)
 1: fact: poetry-project-instance depends on backports-tarfile (1.2.0)
 1: fact: poetry-project-instance depends on build (1.2.2.post1)
 1: fact: poetry-project-instance depends on cachecontrol (0.14.2)
 1: fact: poetry-project-instance depends on certifi (2025.1.31)
 1: fact: poetry-project-instance depends on charset-normalizer (3.4.1)
 1: fact: poetry-project-instance depends on cleo (2.1.0)
 1: fact: poetry-project-instance depends on crashtest (0.4.1)
 1: fact: poetry-project-instance depends on cryptography (44.0.1)
 1: fact: poetry-project-instance depends on distlib (0.3.9)
 1: fact: poetry-project-instance depends on dulwich (0.22.7)
 1: fact: poetry-project-instance depends on fastjsonschema (2.21.1)
 1: fact: poetry-project-instance depends on filelock (3.17.0)
 1: fact: poetry-project-instance depends on idna (3.10)
 1: fact: poetry-project-instance depends on importlib-metadata (8.6.1)
 1: fact: poetry-project-instance depends on installer (0.7.0)
 1: fact: poetry-project-instance depends on jaraco-classes (3.4.0)
 1: fact: poetry-project-instance depends on jaraco-context (6.0.1)
 1: fact: poetry-project-instance depends on jaraco-functools (4.1.0)
 1: fact: poetry-project-instance depends on jeepney (0.8.0)
 1: fact: poetry-project-instance depends on keyring (25.6.0)
 1: fact: poetry-project-instance depends on more-itertools (10.6.0)
 1: fact: poetry-project-instance depends on msgpack (1.1.0)
 1: fact: poetry-project-instance depends on packaging (24.2)
 1: fact: poetry-project-instance depends on pkginfo (1.12.0)
 1: fact: poetry-project-instance depends on platformdirs (4.3.6)
 1: fact: poetry-project-instance depends on poetry (2.0.1)
 1: fact: poetry-project-instance depends on poetry-core (2.0.1)
 1: fact: poetry-project-instance depends on pyproject-hooks (1.2.0)
 1: fact: poetry-project-instance depends on rapidfuzz (3.12.1)
 1: fact: poetry-project-instance depends on requests (2.32.3)
 1: fact: poetry-project-instance depends on requests-toolbelt (1.0.0)
 1: fact: poetry-project-instance depends on shellingham (1.5.4)
 1: fact: poetry-project-instance depends on tomli (2.2.1)
 1: fact: poetry-project-instance depends on tomlkit (0.13.2)
 1: fact: poetry-project-instance depends on trove-classifiers (2025.1.15.22)
 1: fact: poetry-project-instance depends on urllib3 (2.3.0)
 1: fact: poetry-project-instance depends on virtualenv (20.29.2)
 1: fact: poetry-project-instance depends on zipp (3.21.0)
 1: fact: poetry-project-instance depends on pip (25.0.1)
 1: fact: poetry-project-instance depends on setuptools (65.5.0)
 1: fact: poetry-project-instance depends on poetry-dynamic-versioning[plugin] (>=1.0.0)
 1: selecting poetry-project-instance (0)
 1: derived: poetry-dynamic-versioning[plugin] (>=1.0.0)
 1: derived: setuptools (==65.5.0)
 1: derived: pip (==25.0.1)
 1: derived: zipp (==3.21.0)
 1: derived: virtualenv (==20.29.2)
 1: derived: urllib3 (==2.3.0)
 1: derived: trove-classifiers (==2025.1.15.22)
 1: derived: tomlkit (==0.13.2)
 1: derived: tomli (==2.2.1)
 1: derived: shellingham (==1.5.4)
 1: derived: requests-toolbelt (==1.0.0)
 1: derived: requests (==2.32.3)
 1: derived: rapidfuzz (==3.12.1)
 1: derived: pyproject-hooks (==1.2.0)
 1: derived: poetry-core (==2.0.1)
 1: derived: poetry (==2.0.1)
 1: derived: platformdirs (==4.3.6)
 1: derived: pkginfo (==1.12.0)
 1: derived: packaging (==24.2)
 1: derived: msgpack (==1.1.0)
 1: derived: more-itertools (==10.6.0)
 1: derived: keyring (==25.6.0)
 1: derived: jeepney (==0.8.0)
 1: derived: jaraco-functools (==4.1.0)
 1: derived: jaraco-context (==6.0.1)
 1: derived: jaraco-classes (==3.4.0)
 1: derived: installer (==0.7.0)
 1: derived: importlib-metadata (==8.6.1)
 1: derived: idna (==3.10)
 1: derived: filelock (==3.17.0)
 1: derived: fastjsonschema (==2.21.1)
 1: derived: dulwich (==0.22.7)
 1: derived: distlib (==0.3.9)
 1: derived: cryptography (==44.0.1)
 1: derived: crashtest (==0.4.1)
 1: derived: cleo (==2.1.0)
 1: derived: charset-normalizer (==3.4.1)
 1: derived: certifi (==2025.1.31)
 1: derived: cachecontrol (==0.14.2)
 1: derived: build (==1.2.2.post1)
 1: derived: backports-tarfile (==1.2.0)
 1: derived: secretstorage (==3.3.3)
 1: derived: readline (==6.2.4.1)
 1: derived: hpy (==0.9.0)
 1: derived: greenlet (==0.4.13)
 1: derived: cffi (==1.18.0.dev0)
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Backend 'fail Keyring' is not suitable
No valid keyring backend was found
Creating new session for pypi.org
Source (PyPI): 14 packages found for poetry-dynamic-versioning >=1.0.0
Source (PyPI): 1 packages found for setuptools 65.5.0
Source (PyPI): 1 packages found for pip 25.0.1
Source (PyPI): 1 packages found for zipp 3.21.0
Source (PyPI): 1 packages found for virtualenv 20.29.2
Source (PyPI): 1 packages found for urllib3 2.3.0
Source (PyPI): 1 packages found for trove-classifiers 2025.1.15.22
Source (PyPI): 1 packages found for tomlkit 0.13.2
Source (PyPI): 1 packages found for tomli 2.2.1
Source (PyPI): 1 packages found for shellingham 1.5.4
Source (PyPI): 1 packages found for requests-toolbelt 1.0.0
Source (PyPI): 1 packages found for requests 2.32.3
Source (PyPI): 1 packages found for rapidfuzz 3.12.1
Source (PyPI): 1 packages found for pyproject-hooks 1.2.0
Source (PyPI): 1 packages found for poetry-core 2.0.1
Source (PyPI): 1 packages found for poetry 2.0.1
Source (PyPI): 1 packages found for platformdirs 4.3.6
Source (PyPI): 1 packages found for pkginfo 1.12.0
Source (PyPI): 1 packages found for packaging 24.2
Source (PyPI): 1 packages found for msgpack 1.1.0
Source (PyPI): 1 packages found for more-itertools 10.6.0
Source (PyPI): 1 packages found for keyring 25.6.0
Source (PyPI): 1 packages found for jeepney 0.8.0
Source (PyPI): 1 packages found for jaraco-functools 4.1.0
Source (PyPI): 1 packages found for jaraco-context 6.0.1
Source (PyPI): 1 packages found for jaraco-classes 3.4.0
Source (PyPI): 1 packages found for installer 0.7.0
Source (PyPI): 1 packages found for importlib-metadata 8.6.1
Source (PyPI): 1 packages found for idna 3.10
Source (PyPI): 1 packages found for filelock 3.17.0
Source (PyPI): 1 packages found for fastjsonschema 2.21.1
Source (PyPI): 1 packages found for dulwich 0.22.7
Source (PyPI): 1 packages found for distlib 0.3.9
Source (PyPI): 1 packages found for cryptography 44.0.1
Source (PyPI): 1 packages found for crashtest 0.4.1
Source (PyPI): 1 packages found for cleo 2.1.0
Source (PyPI): 1 packages found for charset-normalizer 3.4.1
Source (PyPI): 1 packages found for certifi 2025.1.31
Source (PyPI): 1 packages found for cachecontrol 0.14.2
Source (PyPI): 1 packages found for build 1.2.2.post1
Source (PyPI): 1 packages found for backports-tarfile 1.2.0
Source (PyPI): 1 packages found for secretstorage 3.3.3
Source (PyPI): 1 packages found for readline 6.2.4.1
Source (PyPI): 1 packages found for hpy 0.9.0
Source (PyPI): 1 packages found for greenlet 0.4.13
Source (PyPI): 0 packages found for cffi 1.18.0.dev0
 1: selecting setuptools (65.5.0)
 1: selecting pip (25.0.1)
 1: selecting zipp (3.21.0)
 1: fact: virtualenv (20.29.2) depends on distlib (>=0.3.7,<1)
 1: fact: virtualenv (20.29.2) depends on filelock (>=3.12.2,<4)
 1: fact: virtualenv (20.29.2) depends on platformdirs (>=3.9.1,<5)
 1: selecting virtualenv (20.29.2)
 1: selecting urllib3 (2.3.0)
 1: selecting trove-classifiers (2025.1.15.22)
 1: selecting tomlkit (0.13.2)
 1: selecting tomli (2.2.1)
 1: selecting shellingham (1.5.4)
 1: fact: requests-toolbelt (1.0.0) depends on requests (>=2.0.1,<3.0.0)
 1: selecting requests-toolbelt (1.0.0)
 1: fact: requests (2.32.3) depends on charset-normalizer (>=2,<4)
 1: fact: requests (2.32.3) depends on idna (>=2.5,<4)
 1: fact: requests (2.32.3) depends on urllib3 (>=1.21.1,<3)
 1: fact: requests (2.32.3) depends on certifi (>=2017.4.17)
 1: selecting requests (2.32.3)
 1: selecting rapidfuzz (3.12.1)
 1: selecting pyproject-hooks (1.2.0)
 1: selecting poetry-core (2.0.1)
 1: fact: poetry (2.0.1) depends on build (>=1.2.1,<2.0.0)
 1: fact: poetry (2.0.1) depends on cachecontrol[filecache] (>=0.14.0,<0.15.0)
 1: fact: poetry (2.0.1) depends on cleo (>=2.1.0,<3.0.0)
 1: fact: poetry (2.0.1) depends on dulwich (>=0.22.6,<0.23.0)
 1: fact: poetry (2.0.1) depends on fastjsonschema (>=2.18.0,<3.0.0)
 1: fact: poetry (2.0.1) depends on installer (>=0.7.0,<0.8.0)
 1: fact: poetry (2.0.1) depends on keyring (>=25.1.0,<26.0.0)
 1: fact: poetry (2.0.1) depends on packaging (>=24.0)
 1: fact: poetry (2.0.1) depends on pkginfo (>=1.12,<2.0)
 1: fact: poetry (2.0.1) depends on platformdirs (>=3.0.0,<5)
 1: fact: poetry (2.0.1) depends on poetry-core (2.0.1)
 1: fact: poetry (2.0.1) depends on pyproject-hooks (>=1.0.0,<2.0.0)
 1: fact: poetry (2.0.1) depends on requests (>=2.26,<3.0)
 1: fact: poetry (2.0.1) depends on requests-toolbelt (>=1.0.0,<2.0.0)
 1: fact: poetry (2.0.1) depends on shellingham (>=1.5,<2.0)
 1: fact: poetry (2.0.1) depends on tomli (>=2.0.1,<3.0.0)
 1: fact: poetry (2.0.1) depends on tomlkit (>=0.11.4,<1.0.0)
 1: fact: poetry (2.0.1) depends on trove-classifiers (>=2022.5.19)
 1: fact: poetry (2.0.1) depends on virtualenv (>=20.26.6,<21.0.0)
 1: fact: poetry (2.0.1) depends on xattr (>=1.0.0,<2.0.0)
 1: selecting poetry (2.0.1)
 1: derived: xattr (>=1.0.0,<2.0.0)
 1: derived: cachecontrol[filecache] (>=0.14.0,<0.15.0)
Source (PyPI): 3 packages found for xattr >=1.0.0,<2.0.0
 1: selecting platformdirs (4.3.6)
 1: selecting pkginfo (1.12.0)
 1: selecting packaging (24.2)
 1: selecting msgpack (1.1.0)
 1: selecting more-itertools (10.6.0)
 1: fact: keyring (25.6.0) depends on pywin32-ctypes (>=0.2.0)
 1: fact: keyring (25.6.0) depends on SecretStorage (>=3.2)
 1: fact: keyring (25.6.0) depends on jeepney (>=0.4.2)
 1: fact: keyring (25.6.0) depends on importlib_metadata (>=4.11.4)
 1: fact: keyring (25.6.0) depends on jaraco.classes (*)
 1: fact: keyring (25.6.0) depends on jaraco.functools (*)
 1: fact: keyring (25.6.0) depends on jaraco.context (*)
 1: selecting keyring (25.6.0)
 1: derived: pywin32-ctypes (>=0.2.0)
Source (PyPI): 4 packages found for pywin32-ctypes >=0.2.0
 1: selecting jeepney (0.8.0)
 1: fact: jaraco-functools (4.1.0) depends on more-itertools (*)
 1: selecting jaraco-functools (4.1.0)
 1: fact: jaraco-context (6.0.1) depends on backports.tarfile (*)
 1: selecting jaraco-context (6.0.1)
 1: fact: jaraco-classes (3.4.0) depends on more-itertools (*)
 1: selecting jaraco-classes (3.4.0)
 1: selecting installer (0.7.0)
 1: fact: importlib-metadata (8.6.1) depends on zipp (>=3.20)
 1: selecting importlib-metadata (8.6.1)
 1: selecting idna (3.10)
 1: selecting filelock (3.17.0)
 1: selecting fastjsonschema (2.21.1)
 1: fact: dulwich (0.22.7) depends on urllib3 (>=1.25)
 1: selecting dulwich (0.22.7)
 1: selecting distlib (0.3.9)
 1: fact: cryptography (44.0.1) depends on cffi (>=1.12)
 1: selecting cryptography (44.0.1)
 1: selecting crashtest (0.4.1)
 1: fact: cleo (2.1.0) depends on crashtest (>=0.4.1,<0.5.0)
 1: fact: cleo (2.1.0) depends on rapidfuzz (>=3.0.0,<4.0.0)
 1: selecting cleo (2.1.0)
 1: selecting charset-normalizer (3.4.1)
 1: selecting certifi (2025.1.31)
 1: fact: cachecontrol (0.14.2) depends on requests (>=2.16.0)
 1: fact: cachecontrol (0.14.2) depends on msgpack (>=0.5.2,<2.0.0)
 1: selecting cachecontrol (0.14.2)
 1: fact: build (1.2.2.post1) depends on packaging (>=19.1)
 1: fact: build (1.2.2.post1) depends on pyproject_hooks (*)
 1: fact: build (1.2.2.post1) depends on colorama (*)
 1: fact: build (1.2.2.post1) depends on tomli (>=1.1.0)
 1: selecting build (1.2.2.post1)
 1: derived: colorama
Source (PyPI): 43 packages found for colorama *
 1: selecting backports-tarfile (1.2.0)
 1: fact: secretstorage (3.3.3) depends on cryptography (>=2.0)
 1: fact: secretstorage (3.3.3) depends on jeepney (>=0.6)
 1: selecting secretstorage (3.3.3)
 1: selecting readline (6.2.4.1)
 1: fact: hpy (0.9.0) depends on setuptools (>=64.0)
 1: selecting hpy (0.9.0)
 1: selecting greenlet (0.4.13)
 1: fact: cachecontrol[filecache] (0.14.2) depends on cachecontrol (0.14.2)
 1: fact: cachecontrol[filecache] (0.14.2) depends on requests (>=2.16.0)
 1: fact: cachecontrol[filecache] (0.14.2) depends on msgpack (>=0.5.2,<2.0.0)
 1: fact: cachecontrol[filecache] (0.14.2) depends on filelock (>=3.8.0)
 1: selecting cachecontrol[filecache] (0.14.2)
 1: fact: no versions of cffi match 1.18.0.dev0
 1: conflict: no versions of cffi match 1.18.0.dev0
 1: ! cffi (==1.18.0.dev0) is satisfied by cffi (==1.18.0.dev0)
 1: ! which is caused by "poetry-project-instance depends on cffi (1.18.0.dev0)"
 1: ! thus: version solving failed
 1: Version solving took 2.019 seconds.
 1: Tried 1 solutions.

Stack trace:

4  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/puzzle/solver.py:166 in _solve
    164│
    165│         try:
  → 166│             result = resolve_version(self._package, self._provider)
    167│
    168│             packages = result.packages

3  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/mixology/__init__.py:18 in resolve_version
     16│     solver = VersionSolver(root, provider)
     17│
  →  18│     return solver.solve()
     19│

2  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/mixology/version_solver.py:173 in solve
    171│             next: str | None = self._root.name
    172│             while next is not None:
  → 173│                 self._propagate(next)
    174│                 next = self._choose_package_version()
    175│

1  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/mixology/version_solver.py:212 in _propagate
    210│                     # where that incompatibility will allow us to derive new assignments
    211│                     # that avoid the conflict.
  → 212│                     root_cause = self._resolve_conflict(incompatibility)
    213│
    214│                     # Back jumping erases all the assignments we did at the previous

SolveFailureError

Because poetry-project-instance depends on cffi (1.18.0.dev0) which doesn't match any versions, version solving failed.

at ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/mixology/version_solver.py:427 in _resolve_conflict
    423│             )
    424│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
    425│             self._log(f"! thus: {incompatibility}")
    426│
  → 427│         raise SolveFailureError(incompatibility)
    428│
    429│     def _choose_package_version(self) -> str | None:
    430│         """
    431│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


Stack trace:

14  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/application.py:327 in run
     325│
     326│             try:
   → 327│                 exit_code = self._run(io)
     328│             except BrokenPipeError:
     329│                 # If we are piped to another process, it may close early and send a

13  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/console/application.py:236 in _run
     234│
     235│         with directory(self._working_directory):
   → 236│             exit_code: int = super()._run(io)
     237│
     238│         return exit_code

12  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/application.py:431 in _run
     429│             io.input.interactive(interactive)
     430│
   → 431│         exit_code = self._run_command(command, io)
     432│         self._running_command = None
     433│

11  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/application.py:473 in _run_command
     471│
     472│         if error is not None:
   → 473│             raise error
     474│
     475│         return terminate_event.exit_code

10  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/application.py:457 in _run_command
     455│
     456│             if command_event.command_should_run():
   → 457│                 exit_code = command.run(io)
     458│             else:
     459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

 9  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/commands/base_command.py:117 in run
     115│         io.input.validate()
     116│
   → 117│         return self.execute(io) or 0
     118│
     119│     def merge_application_definition(self, merge_args: bool = True) -> None:

 8  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/cleo/commands/command.py:61 in execute
      59│
      60│         try:
   →  61│             return self.handle()
      62│         except KeyboardInterrupt:
      63│             return 1

 7  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/console/commands/install.py:114 in handle
     112│         from poetry.masonry.builders.editable import EditableBuilder
     113│
   → 114│         PluginManager.ensure_project_plugins(self.poetry, self.io)
     115│
     116│         if self.option("extras") and self.option("all-extras"):

 6  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/plugins/plugin_manager.py:69 in ensure_project_plugins
      67│     @classmethod
      68│     def ensure_project_plugins(cls, poetry: Poetry, io: IO) -> None:
   →  69│         ProjectPluginCache(poetry, io).ensure_plugins()
      70│
      71│     def load_plugins(self) -> None:

 5  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/plugins/plugin_manager.py:247 in ensure_plugins
     245│             f"Installing Poetry plugins only for the current project..."
     246│         )
   → 247│         self._install(missing_plugins, poetry_env, installed_packages)
     248│         self._io.write_line("")
     249│         self._write_config()

 4  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/plugins/plugin_manager.py:294 in _install
     292│         installer.update(True)
     293│
   → 294│         if installer.run() != 0:
     295│             raise RuntimeError("Failed to install required Poetry plugins")
     296│

 3  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/installation/installer.py:103 in run
     101│             self.verbose(True)
     102│
   → 103│         return self._do_install()
     104│
     105│     def dry_run(self, dry_run: bool = True) -> Installer:

 2  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/installation/installer.py:241 in _do_install
     239│                 source_root=self._env.path.joinpath("src")
     240│             ):
   → 241│                 solved_packages = solver.solve(
     242│                     use_latest=self._whitelist
     243│                 ).get_solved_packages()

 1  ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/puzzle/solver.py:85 in solve
      83│         with self._progress(), self._provider.use_latest_for(use_latest or []):
      84│             start = time.time()
   →  85│             packages = self._solve()
      86│             # simplify markers by removing redundant information
      87│             for transitive_info in packages.values():

SolverProblemError

Because poetry-project-instance depends on cffi (1.18.0.dev0) which doesn't match any versions, version solving failed.

at ~/.local/share/pipx/venvs/poetry/lib/pypy3.10/site-packages/poetry/puzzle/solver.py:172 in _solve
    168│             packages = result.packages
    169│         except OverrideNeededError as e:
    170│             return self._solve_in_compatibility_mode(e.overrides)
    171│         except SolveFailureError as e:
  → 172│             raise SolverProblemError(e)
    173│
    174│         return self._aggregate_solved_packages(packages)
    175│
    176│     def _aggregate_solved_packages(

[dimbleby]

I expect this comes via importlib.metadata

#!/usr/bin/env python3

import importlib.metadata as metadata

print([d.metadata.get("name") for d in metadata.distributions()])

with result

['hpy', 'greenlet', 'readline', 'cffi', 'pip']  

and if so I guess this either wants to go to importlib.metadata - why is it finding "standard library" packages here?

Or else back to pypy - what did they do wrong to cause importlib.metadata to behave this way?

[dimbleby]

either way this seems highly likely to cause confusion when folk try to install regular projects with regular versions of these packages, I wonder how pypy expect this to work...

I'm inclined to see this as a straight pypy bug. These packages are not part of the standard library - you can tell by reading the standard library docs - and my hot take would be that pypy has no business shipping them.

[henryiii]

Because PyPy is written in Python, and some of these as projects that were developed because of PyPy (at least I recognize hpy and cffi as born from PyPy), I expect PyPy has to have them to function. Normally, these just hard override whatever users do - you can pip install cffi=<other-version>, and pip just complains that it can't uninstall the one that is built in, but it thinks it installs it; you still always get the built in one (and pip can't uninstall the new one, too 🙄). For some reason, Poetry is picking up the 1.18.dev0; if it just ignored any sdtlib-installed packages and pretended they didn't exist, I think it would work (except for not actually being able to use whatever version it actually installs, but that's always been the case).

[Secrus]

My guess is the same as @dimbleby's, the fact that importlib.metadata returns those packages as installed distributions, makes Poetry think that those are 3rd party packages, not stdlib. I would expect PyPy to fix this on their end in their implementation of metadata.distributions(), to remove those packages from the list.

[henryiii]

I copied that into the PyPy issue, but I could see it being more complicated. Reporting that it is installed is correct, after all; the package is available and can be used, it's just incorrect to then think you can uninstall it or override it, which is what Poetry is doing.

[dimbleby]

I would add that if anything has changed recently here it is more likely to be at the pypy end than at the poetry end - poetry has been detecting installed packages by asking importlib.metadata.distributions() since forever.

[henryiii]

The thing that changed recently is that PyPy started using unreleased versions of cffi (and hpy, only for 3.11 though). I think upstream cffi development has nearly frozen since last year, so they've either been including unreleased changes or patches. Older versions of PyPy used released packages only, AFAIK.

[dimbleby]

does that adequately explain that readline and greenlet are also affected?

anyway this is sounding more and more like a pypy problem than a poetry problem!

[mattip]

my hot take would be that pypy has no business shipping them

PyPy has had vendored versions of greenlet, cffi, and hpy with built-in backend capabilities, and a pure-python readline "forever", so this problem is not new. For readline the situation is a little different. It is used in the pyrepl pure-python REPL module which is part of CPython3.13 as well. CPython vendors it under _pyrepl and does not report it in distributions(). I guess PyPy could do this as well, but that would only solve the problem for readline, not for the other three. Removing greenlet, cffi or hpy will require re-installing exactly those versions in order to use the packages. It seems no-one has tried to use poetry + pypy + (one of the packages removed), or at least has bothered to report the problem.

what did they do wrong to cause importlib.metadata to behave this way?

PyPy re-uses the CPython stdlib implementation of importlib.metadata, and up to now no one has complained that those packages appear in the distributions(). I don't think it would be correct to hide them, they are vendored into PyPy and should be reported.

I think we all agree poetry should not be removing them. What is the best way to get that to happen? If we do not report them at all in distributions(), that would be misleading to other tools. But if we do report them, poetry wants to remove them. Is there a way to mark the reported metadata in a way that poetry will say "ahh, no need to remove them"?

[dimbleby]

poetry sync tells poetry to make the installed packages match its lock file. If you are presenting greenlet etc as installed packages, then poetry is correct to try and remove them.

If that is not what you want - well my understanding of this thread is that pypy implementation details are leaking to user programs, and if that is right then I would think pypy maintainers will want to fix that

If anything, the bug here - though maybe a happy one - is that the package removal silently fails.

The title of this issue seems over-dramatic. So far as I can see the only actual symptom that is reported is some confusing output.

[mattip]

PyPy must present cffi, greenlet, and hpy as installed. Otherwise other package managers will try to install them. Poetry is the only one that insists on removing them. It would be nice if we could agree between us how to avoid that. I am not a packaging expert, but I can think of alternatives:

• internal poetry logic for PyPy to special case these packages, maybe by adding them to the lockfile?
• adding some flag to the package metadata that poetry would understand

Or we could continue to ignore the problem

[mattip]

What if we used the INSTALLER file spec to state "vendored", and poetry would then ignore any package with that INSTALLER in poetry sync?

[dimbleby]

Poetry is the only one that insists on removing them.

Nonsense. Which others did you try? Even pip will try to uninstall your vendored packages if given requirements that contradict them:

$ pip install cffi==1.17.1
...
  Attempting uninstall: cffi
    Found existing installation: cffi 1.18.0.dev0
    Not uninstalling cffi at /home/dch/pypy3.11-v7.3.18-linux64/lib/pypy3.11, outside environment /home/dch/foo/.venv
    Can't uninstall 'cffi'. No files were found to uninstall.

and as henryiii points out, it is likely that the only reason that uv is not encountering this is because of a bug which they will sooner or later fix.

The effect seems to be that pypy makes it impossible for users to use versions of these packages other than those that you have vendored - which would clearly be a bug: what if they want to rely on the behaviour of some older version? Or some newer version?

If you want to make a proposal about how python implementations should vendor packages in a way that does not interfere with packaging tools, then I suggest that the right place to do that would be at the packaging discourse, rather than working through the individual tools one at a time.

[mattip]

Thanks for the suggestion. https://discuss.python.org/t/how-to-vendor-a-package-into-a-python-distribution/80844

[henryiii]

This is on a package without a lock file (yet). Poetry is basically broken out of the box on PyPy, at least if there’s a plugin being used. Poetry doesn’t actually uninstall the packages it thinks it’s uninstalling, which is why it used to work. Poetry was “uninstalling” and installing new versions of these packages, but in fact, the built in version was overriding site packages. The problem is now that the unreleased version that is being reported is confusing Poetry and causing it to fail.

[dimbleby]

@henryiii I see this aspect of the conversation has been completely lost. At this point it probably wants breaking out into a separate issue so as not to confuse things.

So far we have been discussing: in a managed environment, poetry may try to remove packages that pypy has vendored

But it seems that what you are actually reporting is something quite different: that in poetry's own environment, when installing a plugin per the new mechanism, poetry locks too tightly to the packages that it sees. (And in a pypy environment this is unfortunate: poetry locks to a version of cffi that does not exist anywhere else)

That bug is likely fixed simply by removing these lines, maybe if you submit a pull request their author will chip in...

[mattip]

I am confused by two things, maybe they are connected.

1. Why is this line printed in the original MVR, where does the dependency come from?

Because poetry-project-instance depends on cffi (1.18.0.dev0) which doesn't match any versions, version solving failed.

2. Why is poetry removing cffi, greenlet, hpy, and readline in the second example?

[dimbleby]

1 is because of the lines I linked.

• poetry is trying to install plugins without breaking its existing environment
• to do that it is going "I should lock to everything that is in my existing environment"
• but since that includes an unreleased cffi it then cannot successfully perform resolution

2 is because poetry sync means "please arrange that the only packages in this managed environment are as specified in the poetry lock file" - cffi etc are not in the poetry lock file. However since poetry also fails to remove those things, the only real issue in this part is the confusing output.

[mattip]

but since that includes an unreleased cffi it then cannot successfully perform resolution

How can this step be told "that version already is installed, use the installed version"

To solve (2), can the lock file include packages already installed in the environment?

[dimbleby]

again repeating myself, 1 can likely be fixed simply by removing the lines that I linked. Again I invite a pull request as being a likely way to get the author of those lines to comment.

A lock file is a cross-platform artifact, not related to the current environment. But since the problem (2) is more or less benign I do not know that it needs a whole lot of solving.

[mattip]

I cannot in good faith submit a PR of things I don't understand. @radoering, @abn can you take a look?

[radoering]

That bug is likely fixed simply by removing these lines, maybe if you submit a pull request their author will chip in...

What happens if you remove these lines? I would have expected that another version of cffi is chosen and installed. Will this break the Poetry installation or does the vendored version has higher priority than the other version and this just adds more confusing output?

[dimbleby]

I would have expected that another version of cffi is chosen and installed.

no cffi is chosen because no package in the environment has a dependency on cffi

Edit: maybe there are cffi dependencies, I did not actually check.

But anyway then we see the other part of this bug, the vendored cffi would survive anyway. Just as it did when poetry was installed in the first place.

[radoering]

maybe there are cffi dependencies, I did not actually check.

cffi is a transitive dependency of Poetry.

I did not test, but I think removing the pinning would result in unwanted lib updates, when installing a project plugin. (Without the pins, the latest versions of all dependencies will be chosen and if they are not yet installed in Poetry's env they will be installed in the project in .poetry/plugins.)

I think what we may need is what has been removed in #9851: We do not want to ignore installed packages during solving in this special case!

[dimbleby]

#9851 should not be relevant, installed packages were only ever used as a last-case fallback in case the requirement could not be found in an index or whatever. Take it back, I see what you mean. Hmm, maybe.

another approach might be in the handling of InstalledRepository.load(), which could treat packages found in sysconfig.get_path("stdlib") or sysconfig.get_path("platstdlib") as invisible.

[abn]

I suspect @radoering is correct that we will end up with some unexpected updates if we don't pin.

This sounds similar to "distro managed" packages conceptually if I am groking this right on skimming the thread.

I also wonder how this impacts poetry projects outside of the plugin scenario.