Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Pypi.dependencies after update to package manager api #3040

Merged
merged 5 commits into from
Jan 25, 2023
Merged

Update Pypi.dependencies after update to package manager api #3040

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
aa2091b
Update Pypi.dependencies after update to package manager api
papiro Jan 24, 2023
15ac575
Implement smarter parsing logic according to PEP508 specification
papiro Jan 25, 2023
ef5746a
Fix name of constant
papiro Jan 25, 2023
afb5349
Fixes to dependency specification name regex and variable renames
papiro Jan 25, 2023
4f46915
single character packages now correctly parse with blank requirements
papiro Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion app/models/package_manager/pypi.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ class Pypi < Base
ENTIRE_PACKAGE_CAN_BE_DEPRECATED = true
SUPPORTS_SINGLE_VERSION_UPDATE = true
PYPI_PRERELEASE = /(a|b|rc|dev)[0-9]+$/.freeze
# Adapted from https://peps.python.org/pep-0508/#names
PYPI_508_NAME_REGEX = /^([A-Z0-9][A-Z0-9._-]*[A-Z0-9])/i

def self.package_link(db_project, version = nil)
# NB PEP 503: "All URLs which respond with an HTML5 page MUST end with a / and the repository SHOULD redirect the URLs without a / to add a / to the end."
Expand Down Expand Up @@ -123,14 +125,22 @@ def self.known_versions(name)
&.index_by { |v| v[:number] } || {}
end

# Simply parses out the name of a PEP 508 Dependency specification: https://peps.python.org/pep-0508/
# Leaves the rest as-is with any leading semicolons or spaces stripped
def self.parse_pep_508_dep_spec(dep)
name, requirement = dep.split(PYPI_508_NAME_REGEX, 2).last(2).map(&:strip)
requirement = requirement&.sub(/^[\s;]*/, "") || ""
return name, requirement
end

def self.dependencies(name, version, _mapped_project = nil)
Copy link
Contributor

@mikeyoung85 mikeyoung85 Jan 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we need this change. Is something calling this method without passing in the third argument?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the test.

api_response = get("https://pypi.org/pypi/#{name}/#{version}/json")
deps = api_response.dig("info", "requires_dist")
source_info = api_response.fetch("urls", [])
Rails.logger.warn("Pypi sdist (no deps): #{name}") unless source_info.any? { |rel| rel["packagetype"] == "bdist_wheel" }

deps.map do |dep|
name, version = dep.split(/ /, 2)
name, version = parse_pep_508_dep_spec(dep)
{
project_name: name,
requirements: version.nil? || version == ";" ? "*" : version.gsub(/\(|\)/, ""),
Expand Down
53 changes: 53 additions & 0 deletions spec/models/package_manager/pypi_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,5 +145,58 @@
)
end
end

# Copied from the tests of https://peps.python.org/pep-0508/#complete-grammar
[
["A", "A", ""],
["A.B-C_D", "A.B-C_D", ""],
["aa", "aa", ""],
["name", "name", ""],
["name<=1", "name", "<=1"],
["name>=3", "name", ">=3"],
["name>=3,<2", "name", ">=3,<2"],
["name@http://foo.com", "name", "@http://foo.com"],
[
"name [fred,bar] @ http://foo.com ; python_version=='2.7'",
"name",
"[fred,bar] @ http://foo.com ; python_version=='2.7'"
],
[
"name[quux, strange];python_version<'2.7' and platform_version=='2'",
"name",
"[quux, strange];python_version<'2.7' and platform_version=='2'"
],
[
"name; os_name=='a' or os_name=='b'",
"name",
"os_name=='a' or os_name=='b'"
],
[
"name; os_name=='a' and os_name=='b' or os_name=='c'",
"name",
"os_name=='a' and os_name=='b' or os_name=='c'"
],
[
"name; os_name=='a' and (os_name=='b' or os_name=='c')",
"name",
"os_name=='a' and (os_name=='b' or os_name=='c')"
],
[
"name; os_name=='a' or os_name=='b' and os_name=='c'",
"name",
"os_name=='a' or os_name=='b' and os_name=='c'"
],
[
"name; (os_name=='a' or os_name=='b') and os_name=='c'",
"name",
"(os_name=='a' or os_name=='b') and os_name=='c'"
],
].each do |test, expected_name, expected_requirement|
it "#{test} should be parsed correctly" do
expect(
PackageManager::Pypi.parse_pep_508_dep_spec(test)
).to eq([expected_name, expected_requirement])
end
end
end
end