Skip to content

Commit

Permalink
docs: remote pinning services (#369)
Browse files Browse the repository at this point in the history
Basic docs related to pin remote commands:
https://github.com/ipfs/go-ipfs/releases/v0.8.0

Co-authored-by: Johnny <[email protected]>
Co-authored-by: Marcin Rataj <[email protected]>
  • Loading branch information
3 people authored Feb 19, 2021
1 parent 1d12646 commit b18f003
Show file tree
Hide file tree
Showing 3 changed files with 152 additions and 5 deletions.
1 change: 1 addition & 0 deletions docs/.vuepress/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ module.exports = {
children: [
'/how-to/work-with-blocks',
'/how-to/pin-files',
'/how-to/work-with-pinning-services',
'/how-to/troubleshoot-file-transfers',
'/how-to/take-snapshot',
'/how-to/store-play-videos',
Expand Down
28 changes: 23 additions & 5 deletions docs/how-to/pin-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,11 @@ description: Learn how to pin files in IPFS in order to keep your files and othe

# Pin files using IPFS

Pinning is a very important concept in IPFS. IPFS semantics try to make it feel like every single object is local — there is no "retrieve this file for me from a remote server", just `ipfs cat` or `ipfs get`, which act the same way no matter where the actual object is located. While this is nice, sometimes you want to be able to control what you keep around. Pinning is the mechanism that allows you to tell IPFS to always keep a given object local. IPFS has a fairly aggressive caching mechanism that will keep an object local for a short time after you perform any IPFS operation on it, but these objects may get garbage-collected fairly regularly. To prevent that garbage collection, simply pin the hash you care about. Objects added through `ipfs add` are pinned recursively by default.
Pinning is a very important concept in IPFS. IPFS semantics try to make it feel like every single object is local — there is no "retrieve this file for me from a remote server", just `ipfs cat` or `ipfs get`, which act the same way no matter where the actual object is located.

While this is nice, sometimes you want to be able to control what you keep around. **Pinning** is the mechanism that allows you to tell IPFS to always keep a given object somewhere — the default being your local node, though this can be different if you use a [third-party remote pinning service](#using-a-pinning-service). IPFS has a fairly aggressive caching mechanism that will keep an object local for a short time after you perform any IPFS operation on it, but these objects may get garbage-collected regularly. To prevent that garbage collection, simply pin the CID you care about, or add it to [MFS](/concepts/file-systems/#mutable-file-system-mfs). Objects added through `ipfs add` are pinned recursively by default. Things in MFS are not pinned by default, but are protected from garbage-collection, so one can think about MFS as a mechanism for implicit pinning.

Let's look at this example to explore pinning to your local IPFS node in a bit more depth:

```
echo "ipfs rocks" > foo
Expand All @@ -19,11 +23,11 @@ ipfs pin ls --type=all

## Three kinds of pins

As you may have noticed, the first `ipfs pin rm` command didn't work — it should have warned you that the given hash was "pinned recursively". There are three types of pins in the IPFS world:
As you may have noticed in the example above, the first `ipfs pin rm` command didn't work — it should have warned you that the given hash was _pinned recursively_. What does this mean? There are three types of pins in the IPFS world:

- Direct pins, which pin just a single block, and no others in relation to it
- Recursive pins, which pin a given block and all of its children
- Indirect pins, which are the result of a given block's parent being pinned recursively
- **Direct pins**, which pin just a single block and no others in relation to it.
- **Recursive pins**, which pin a given block and all of its children.
- **Indirect pins**, which are the result of a given block's parent being pinned recursively.

A pinned object cannot be garbage-collected — try this for proof:

Expand All @@ -40,3 +44,17 @@ ipfs pin rm -r <foo hash>
ipfs repo gc
ipfs cat <foo hash>
```

## Local versus remote pinning

All the information above assumes that you're pinning items locally — that is, to your local IPFS node. That's the default behavior for IPFS, but it's also possible to pin your files to a _remote pinning service_. These third-party services give you the opportunity to pin files not to your own local node, but to nodes that they operate. You don't need to worry about your own node's available disk space or uptime.

While you can use a remote pinning service's own GUI, CLI, or other dev tools to manage IPFS files pinned to their service, you can also work directly with pinning services using your local IPFS installation — meaning that you don't need to learn a pinning service's unique API or other tooling.

- The [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec/) offers a specification that enables developers to integrate any pinning service that supports the spec, or create their own. Thanks to the OpenAPI spec format, both clients and servers can be [generated](https://github.com/ipfs/pinning-services-api-spec#code-generation) from the YAML spec file.

- If you use go-ipfs 0.8+ from the command line, you have access to `ipfs pin remote` commands acting as a client for interacting with pinning service APIs. Add your favorite pinning service(s), pin CIDs under human-readable names, get pin statuses, and more, straight from the CLI. [Learn how →](/how-to/work-with-pinning-services/)

<!-- commented out for now, ok to uncomment when https://github.com/ipfs/ipfs-gui/issues/91 is closed
- [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop) and its equivalent in-browser IPFS web interface, the [IPFS Web UI](https://github.com/ipfs-shipyard/ipfs-webui), both support remote pinning services, so you can pin to your favorite pinning service(s) straight from the UI. [Learn how →](/how-to/work-with-pinning-services/)
-->
128 changes: 128 additions & 0 deletions docs/how-to/work-with-pinning-services.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
---
title: Work with pinning services
description: Learn how to use or create remote pinning services with IPFS, the InterPlanetary File System.
---

# Work with remote pinning services

Depending on how you use IPFS, you might find it helpful to use a **remote pinning service** instead of, or in addition to, pinning files on your local IPFS node. Whether it happens remotely or locally, **pinning** an item in IPFS identifies it as something you always wish to keep available, exempting it from the routine _garbage collection_ that IPFS does on infrequently-used items in order to efficiently manage storage space. [Learn more about local pinning →](/how-to/pin-files)

If you've got just one local IPFS node that's always running, local pinning may be all you need to ensure your important items are persisted and never garbage-collected. However, using a remote pinning service — or creating your own — might be useful to you if:

- Your local node isn't always online, but you need items to be consistently available.
- You'd like to keep a persistent backup of your local node's files somewhere else.
- You don't have all the disk space you need on your local node.
- You run more than one IPFS node, and would like to use one of them as a "personal pinning service" as your preferred location for permanent storage.

There are a number of commercial pinning services that make it easy for you to purchase pinning capacity for your important files, some of which include Pinata, Temporal, Infura, and others. Each of these third-party services has its own unique interface for pinning files and managing those pins; this could include a GUI, an API, CLI commands, or other tooling.

However, you don't need to learn new commands or tools if your pinning service of choice supports the vendor-agnostic [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec/) specification. Those services are supported within IPFS itself through the command line: `ipfs pin remote --help`.

As of January 2021, [Pinata](https://pinata.cloud/) supports the [IPFS Pinning Service API endpoint](https://pinata.cloud/documentation#PinningServicesAPI), with more pinning services on the way! [Learn how to create your own →](#create-your-own-pinning-service)

## Use an existing pinning service

To add and use a remote pinning service directly in IPFS, you'll first need to have an account with that service. Once you've got an account, follow these steps to add and use it:

<!-- TODO: GUI section can be uncommented after https://github.com/ipfs/ipfs-gui/issues/91 is closed
### IPFS Desktop or IPFS Web UI
You can add your favorite pinning service(s) to IPFS Desktop/Web UI directly, enabling you to pin and unpin items from the Files screen in the same way as as you do local pins.
#### Adding a new pinning service
To add a new pinning service, open up IPFS Desktop or Web UI, navigate to the **Pinning Services** section of the **Settings** screen, and click the **Add Service** button:
![The Desktop/Web UI Settings screen, ready for adding a new pinning service](https://user-images.githubusercontent.com/1507828/102558464-b0c07700-408a-11eb-8ae4-cd30e3ce81fd.png)
Then, select your chosen pinning service from the modal that pops up. If the pinning service you'd like to add isn't listed in that modal, don't worry — you can add any remote pinning service that supports the IPFS Pinning Service API by clicking the **Add a custom one** link.
![Desktop/Web UI modal for selecting a pinning service to add](https://user-images.githubusercontent.com/1507828/102558471-b918b200-408a-11eb-9a28-b06d03f99121.png)
In the next screen, you’ll be asked for a few other details:
- A nickname for this service. This can be helpful if, for example, you want to add two accounts from the same service.
- The URL for your service's API endpoint. _Note: This field only appears if you've selected a custom pinning service!_
- Your secret API key. This is the unique token provided to you by the pinning service — check its documentation for more info.
- Whether you want to automatically add all files in your local node’s Files directory to the pinning service, or choose which ones you upload.
![Details modals for adding a new pinning service or a new custom pinning service in Desktop/Web UI](https://user-images.githubusercontent.com/1507828/102558910-ca15f300-408b-11eb-9fe3-742186c077c7.png)
After you hit **Save**, you’ll see your new pinning service added to the **Pinning Services** section of your **Settings** screen.
![Desktop/Web UI Settings screen with a new pinning service added](https://user-images.githubusercontent.com/1507828/102558530-db123480-408a-11eb-9e3b-58bbd59b2880.png)
#### Using a pinning service
Now that you’re set up, you can pin or unpin files to your new pinning service directly from the Files screen. Just right-click any file or click the **three dots** action icon in the files list, and select **Set pinning**:
![Desktop/Web UI Files screen with the action menu open and "Set pinning" visible](https://user-images.githubusercontent.com/1507828/102558546-e6656000-408a-11eb-97b8-5fdb060602d2.png)
-->

### IPFS CLI

Command-line users benefit from `ipfs pin remote` commands, which simplify remote pinning operations. The built-in pinning service API client executes all the necessary remote calls under the hood.

#### Adding a new pinning service

To add a new pinning service, use the following command:

```console
$ ipfs pin remote service add nickname https://my-pin-service.example.com/api-endpoint myAccessToken
```

- `nickname` is a unique name for this particular instantiation of a pinning service. This can be helpful if, for example, you want to add two accounts from the same service.
- `https://my-pin-service.example.com/api-endpoint` is the endpoint supplied to you by the pinning service. Check the service's documentation for more info.
- `myAccessToken` is the unique secret token provided to you by the pinning service. Check the service's documentation for more info.

#### Using a pinning service

Here are a few CLI commands to get you started. In all examples, replace `nickname` with the unique name you gave the pinning service when you added it.

To pin a CID under under a human-readable name:

```console
$ ipfs pin remote add --service=nickname --name=war-and-peace.txt bafybeib32tuqzs2wrc52rdt56cz73sqe3qu2deqdudssspnu4gbezmhig4
```

To list successful pins:

```console
$ ipfs pin remote ls --service=nickname
```

To list all "pending" pins:

```console
$ ipfs pin remote ls --service=nickname --status=queued,pinning,failed
```

For more commands and general help:

```console
$ ipfs pin remote --help
```

## Create your own pinning service

Obviously you aren't limited to a static list of pre-approved services. Any remote pinning service compatible with the [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec) can be added as a custom pinning service — which also means that you can create your own! This might be useful in circumstances like:

- Designating one of your own IPFS nodes to be a _personal pinning service_ as a preferred location for permanent storage.
- Running a private pinning service for your friends or company.
- Starting your own commercial pinning service.

As noted above, your service must use the [IPFS Pinning Service API](https://ipfs.github.io/pinning-services-api-spec) in order to be interoperable with client behind `ipfs pin remote` commands.


::: tip
If you're interested in creating your own pinning service for your own personal or shared use, you can [generate client and server from the OpenAPI spec](https://github.com/ipfs/pinning-services-api-spec#code-generation), reducing the development time.

You may also wish to read continuing details on how the API is evolving in the [Pinning Service API Spec GitHub repo](https://github.com/ipfs/pinning-services-api-spec), and be part of the discussion on its further development!
:::


<!-- TODO this call to action can be uncommented when https://github.com/ipfs/ipfs-gui/issues/91 is closed
If you'd like to make your custom pinning service available to every IPFS user, we welcome your submissions. Once you're ready to open the doors to the public, make a PR against the [IPFS Web UI GitHub repo](https://github.com/ipfs-shipyard/ipfs-webui) in order to add it to the default list of pinning services that are displayed in the Desktop/Web UI Settings screen, and one of the core maintainers will be in touch.
-->

0 comments on commit b18f003

Please sign in to comment.