Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pin datasets library version to avoid dependency conflict in llama test case #39

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

KeitaW
Copy link

@KeitaW KeitaW commented Jan 26, 2025

Installing llama nxd dependencies with requirement.txt on the latest Neuron SDK environment causes following dependency conflict error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.3 which is incompatible.

This requests version requirement comes from datasets library.

pip install requests==2.32.0
Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Collecting requests==2.32.0
  Downloading requests-2.32.0-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (1.26.20)
Requirement already satisfied: certifi>=2017.4.17 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (2024.12.14)
WARNING: The candidate selected for download or install is a yanked version: 'requests' candidate (version 2.32.0 at https://files.pythonhosted.org/packages/24/e8/09e8d662a9675a4e4f5dd7a8e6127b463a091d2703ed931a64aa66d00065/requests-2.32.0-py3-none-any.whl (from https://pypi.org/simple/requests/) (requires-python:>=3.8))
Reason for being yanked: Yanked due to conflicts with CVE-2024-35195 mitigation
Downloading requests-2.32.0-py3-none-any.whl (63 kB)
Installing collected packages: requests
  Attempting uninstall: requests
    Found existing installation: requests 2.32.3
    Uninstalling requests-2.32.3:
      Successfully uninstalled requests-2.32.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 3.1.0 requires requests>=2.32.2, but you have requests 2.32.0 which is incompatible.
neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.0 which is incompatible.

Thus we need to install datasets==3.0.2 (the latest which satisfies the constraint, released in Oct. 2024) instead of the latest.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Installing llama nxd dependencies with `requirement.txt` on the latest Neuron SDK environment causes following dependency conflict error:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.3 which is incompatible.
```

This `requests` version requirement comes from `datasets` library.

```
pip install requests==2.32.0
Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Collecting requests==2.32.0
  Downloading requests-2.32.0-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (1.26.20)
Requirement already satisfied: certifi>=2017.4.17 in /fsx/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from requests==2.32.0) (2024.12.14)
WARNING: The candidate selected for download or install is a yanked version: 'requests' candidate (version 2.32.0 at https://files.pythonhosted.org/packages/24/e8/09e8d662a9675a4e4f5dd7a8e6127b463a091d2703ed931a64aa66d00065/requests-2.32.0-py3-none-any.whl (from https://pypi.org/simple/requests/) (requires-python:>=3.8))
Reason for being yanked: Yanked due to conflicts with CVE-2024-35195 mitigation
Downloading requests-2.32.0-py3-none-any.whl (63 kB)
Installing collected packages: requests
  Attempting uninstall: requests
    Found existing installation: requests 2.32.3
    Uninstalling requests-2.32.3:
      Successfully uninstalled requests-2.32.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 3.1.0 requires requests>=2.32.2, but you have requests 2.32.0 which is incompatible.
neuronx-cc 2.16.372.0+4a9b2326 requires requests<2.32.0, but you have requests 2.32.0 which is incompatible.
```
Thus we need to install ` datasets==3.0.2` (the latest which satisfies the constraint, released in Oct. 2024) instead of the latest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant