x/pkgsite: self hosting - tracking issue

[marwan-at-work]

This is an umbrella issue that tracks all the possible blockers for having users self host the pkgsite codebase.

Issues:

• Make LICENSE detection optional in pkgsite codebase: x/pkgsite: make LICENSE detection optional in pkgsite codebase (not in pkg.go.dev) #39602
• Write self-hosting specific documentation: i.e. deployment steps, things to be aware of, etc.
  • Local quickstart docs. Running a local pkgsite with the bare minimum should be easy for any experienced Go developer.
  • Clarification of what features/requirements are needed or can be switched off for a local setup, e.g. GOPROXY or a search "engine".
• x/pkgsite: Remove hardcoded references to pkg.go.dev #48603
• Using a custom HTTPS proxy for the private git repositories.
• Using a proxy that acts as a gateway to the internet.
• x/pkgsite: Excluding private packages from external site lookups #55106

Please feel free to edit this issue to add/remove issues above.

[mvdan]

Two additions from what I've seen:

• Local quickstart docs. Running a local pkgsite with the bare minimum should be easy for any experienced Go developer.
• Clarification of what features/requirements are needed or can be switched off for a local setup, e.g. GOPROXY or a search "engine".

[julieqiu]

I opened #40371 to discuss issues related to running pkgsite locally. Let's use this issue to discuss self-hosting via a private GOPROXY.

[MicahParks]

I have a project that addresses the following use cases:

• Turning off license respecting, allowing the docs to be generated for projects with non-permissive licenses.
• Using a custom HTTPS proxy for the private git repositories.
• Using a proxy that acts as a gateway to the internet.

There is some documentation for these use cases in the project itself.

GitLab project
GitHub mirror

[myitcv]

Posting on behalf of @theckman via Slack

pkgsite is being proposed as part of the solution for the discoverability of new major module versions. This presents a challenge for groups that are using private repositories via a SaaS SCM without the means to run a shared module proxy or pkgsite.

[ryancurrah]

Looking forward to simplifying the deployability of the pkg site.

[l0nax]

Any updates on this?

[kevinburke]

Just to be clear, with godoc, you could dynamically load anything on the filesystem as long as it was present in $GOPATH/src/path/to/blah, and now you need to specify every package you want to load when you start the new binary?

[sfllaw]

Just to be clear, with godoc, you could dynamically load anything on the filesystem as long as it was present in $GOPATH/src/path/to/blah, and now you need to specify every package you want to load when you start the new binary?

pkgsite only serves things in its database, which it populates whenever you make a query for a package that it doesn't know about yet.

In addition, when I set it up for the first time, I go run ./devtools/cmd/seeddb in order to load the default set of packages, which includes the Go standard library, as specified in seed.txt.

Actually, I also put it our private repositories in seed.txt as well, but you don't have to do that.

[plo-]

I read the doc I found as well as some issues like #50229 but I'm still not sure how to properly run pkgsite for internal use.

I opened #40371 to discuss issues related to running pkgsite locally. Let's use this issue to discuss self-hosting via a private GOPROXY.

@julieqiu Is using a private GOPROXY the correct way to run pkgsite locally? So far, I run an instance of it using command like pkgsite <path1> <path2> <pathX> but it's kind of broken, functionalities like the "search", the "importedby" or even the "versions" (to select what version of a package is displayed) don't work at all.

[cezarguimaraes]

I have created a Helm chart that deploys most of pkgsite's architecture (except for the worker), plus a private module proxy, on kubernetes. Might be helpful to others who wish to host private pkgsite instances: https://github.com/cezarguimaraes/pkgsite

[mxk]

Here's another Helm chart that works around issues with the frontend server not supporting .netrc files, which results in invalid source code links with GitLab: https://github.com/mxk/go-pkgsite