S3 Gateway

Learn about Pachyderm's embedded S3 gateway, which is compatible with MinIO, AWS S3 CLI, and boto3.

March 24, 2023

Use the embedded S3 Gateway to send or receive data through the S3 protocol using object storage tooling such as Minio, boto3, or AWS s3 CLI. Operations available are similar to those officially documented for S3.

S3 Gateway Syntax #

The S3 gateway presents each branch from every Pachyderm repository as an S3 bucket. Buckets are represented via [<commit>.]<branch>.<repo>.<project>, with the commit being optional.

If auth is enabled, credentials must be passed with each S3 gateway endpoint as mentioned in the S3 Client configuration steps.

Command Examples #

The following command examples assume that you have upgraded to use the embedded proxy, which will become mandatory in future releases.

Put Data Into Pachyderm Repo #


Retrieve Data From Pachyderm Repo #


Port Forwarding #

You can pachctl port-forward to access the s3 gateway through the localhost:30600 endpoint, however, the Kubernetes port forwarder incurs substantial overhead and does not recover well from broken connections.