Overview

Key Features

Target Audience

Basic Concepts

Intro to Data Versioning

Intro to Pipelines

Get Started

Local Getting Started Guides

Docker Desktop

Minikube

Cloud Getting Started Guides

AWS + Pachyderm

Azure + Pachyderm

GCP + Pachyderm

Install Pachctl Auto-completion

Beginner Tutorial

Concepts

Advanced Concepts

Deferred Processing

Distributed Computing

Global Identifier

Pipeline Concepts

Datum

Cross & Union Inputs

Datum Processing

Datum Processing States

Glob Pattern

Group Input

Join Input

Metadata

Job

Pipeline

Cron Pipeline

Service

Spout

Versioned Data Concepts

Branch

Commit

File

History

Provenance

Repository

How Tos

Advanced Data Operations

Analyze Time-Windowed Data

Manage Secrets

Mount a Volume in a Pipeline

Skip Failed Datums

Transactions

Basic Data Operations

Export Data

Egress To An SQL Database

Export via Egress

Export via Pachctl

Mount a Repo Locally

Ingest Data

Manage Commits & Delete Data

SQL Ingest

Console Guide

View Project

View List

View Pipelines

View Jobs

View Outputs

View Inputs

Debugging Operations

View Audit Logs

View Kubernetes Logs

Developer Workflow

CI/CD Integration

Create a Machine Learning Workflow

The Push Images Flag

Working with Pipelines

JupyterLab Mount Extension

Docker Installation Guide

Local Installation Guide

User Guide

Troubleshooting

Pipeline Operations

Create a Pipeline

Delete a Pipeline

Jsonnet Pipeline Specifications

Update a Pipeline

Project Operations

Create a Project

Set a Project as Current

Add a Project Resource

Grant Project Access

Delete a Project

S3 Gateway Operations

Create S3 Bucket

Delete an S3 Object

Delete Empty S3 Bucket

Get an S3 Object

List S3 Buckets

List S3 Objects

Write an S3 Object

Deploy & Manage

Deploy

AWS Deployment

Azure

Configure Environment Variables

Configure Tracing with Jaeger

Console

Cloud Deployment

Local Deployment

Deploy Pachyderm with TSL (SSL, HTTPS)

Enable Logs Aggregation With Loki

Google Cloud Platform

Helm Deployment

Import a Kubernetes Context

Infrastructure Recommendations

Setup Ingress with Traefik

Monitor with Prometheus

Job Metrics

Pachd Metrics

Non-Default Namespaces

On Premises

Production Recommendations

Quickstart

RBAC

Set Up AWS Secret Manager

Upgrade to Embedded Proxy

Manage

Backup & Restore Your Cluster

Disable Usage Metrics

Manage Cluster Access

S3 Gateway

AWS CLI

Boto3

👉 Credentials

MinIO

Unsupported Operations

Sidecar S3 Gateway

Storage Use Optimization

Uninstall Pachyderm

Upgrade Pachyderm

Use GPUs

Using the Pachyderm Shell

Reference

PachCTL

Pachctl auth

Pachctl auth activate

Pachctl auth check

Pachctl auth check project

Pachctl auth check repo

Pachctl auth deactivate

Pachctl auth get config

Pachctl auth get groups

Pachctl auth get robot token

Pachctl auth get

Pachctl auth get cluster

Pachctl auth get enterprise

Pachctl auth get project

Pachctl auth get repo

Pachctl auth login

Pachctl auth logout

Pachctl auth revoke

Pachctl auth roles for permission

Pachctl auth rotate root token

Pachctl auth set config

Pachctl auth set

Pachctl auth set cluster

Pachctl auth set enterprise

Pachctl auth set project

Pachctl auth set repo

Pachctl auth use auth token

Pachctl auth whoami

Pachctl buildinfo

Pachctl completion

Pachctl completion bash

Pachctl completion zsh

Pachctl config

Pachctl config delete

Pachctl config delete context

Pachctl config get

Pachctl config get active context

Pachctl config get active enterprise context

Pachctl config get context

Pachctl config get metrics

Pachctl config import kube

Pachctl config list

Pachctl config list context

Pachctl config set

Pachctl config set active context

Pachctl config set active enterprise context

Pachctl config set context

Pachctl config set metrics

Pachctl config update

Pachctl config update context

Pachctl connect

Pachctl copy

Pachctl copy file

Pachctl create

Pachctl create branch

Pachctl create pipeline

Pachctl create project

Pachctl create repo

Pachctl create secret

Pachctl debug

Pachctl debug analyze

Pachctl debug binary

Pachctl debug dump

Pachctl debug profile

Pachctl delete

Pachctl delete all

Pachctl delete branch

Pachctl delete commit

Pachctl delete file

Pachctl delete job

Pachctl delete pipeline

Pachctl delete project

Pachctl delete repo

Pachctl delete secret

Pachctl delete transaction

Pachctl diff

Pachctl diff file

Pachctl draw

Pachctl draw pipeline

Pachctl edit

Pachctl edit pipeline

Pachctl enterprise

Pachctl enterprise deactivate

Pachctl enterprise get state

Pachctl enterprise heartbeat

Pachctl enterprise pause status

Pachctl enterprise pause

Pachctl enterprise register

Pachctl enterprise sync contexts

Pachctl enterprise unpause

Pachctl exit

Pachctl finish

Pachctl finish commit

Pachctl finish transaction

Pachctl fsck

Pachctl get

Pachctl get file

Pachctl glob

Pachctl glob file

Pachctl idp

Pachctl idp create client

Pachctl idp create connector

Pachctl idp delete client

Pachctl idp delete connector

Pachctl idp get client

Pachctl idp get config

Pachctl idp get connector

Pachctl idp list client

Pachctl idp list connector

Pachctl idp set config

Pachctl idp update client

Pachctl idp update connector

Pachctl inspect

Pachctl inspect branch

Pachctl inspect cluster

Pachctl inspect commit

Pachctl inspect datum

Pachctl inspect file

Pachctl inspect job

Pachctl inspect pipeline

Pachctl inspect project

Pachctl inspect repo

Pachctl inspect secret

Pachctl inspect transaction

Pachctl kube events

Pachctl license

Pachctl license activate

Pachctl license add cluster

Pachctl license delete all

Pachctl license delete cluster

Pachctl license get state

Pachctl license list clusters

Pachctl license update cluster

Pachctl list

Pachctl list branch

Pachctl list commit

Pachctl list datum

Pachctl list file

Pachctl list job

Pachctl list pipeline

Pachctl list project

Pachctl list repo

Pachctl list secret

Pachctl list transaction

Pachctl logs

Pachctl mount

Pachctl port forward

Pachctl put

Pachctl put file

Pachctl restart

Pachctl restart datum

Pachctl resume

Pachctl resume transaction

Pachctl run

Pachctl run cron

Pachctl run pfs load test

Pachctl run pps load test

Pachctl shell

Pachctl squash

Pachctl squash commit

Pachctl start

Pachctl start commit

Pachctl start pipeline

Pachctl start transaction

Pachctl stop

Pachctl stop job

Pachctl stop pipeline

Pachctl stop transaction

Pachctl subscribe

Pachctl subscribe commit

Pachctl unmount

Pachctl update

Pachctl update pipeline

Pachctl update project

Pachctl update repo

Pachctl version

Pachctl wait

Pachctl wait commit

Pachctl wait job

Configuration Specification

Helm Chart Values (HCVs)

Deploy Target HCVs

Global HCVs

Console HCVs

Enterprise Server HCVs

ETCD HCVs

Ingress HCVs

Loki HCVs

PachD HCVs

PachW HCVs

Kube Event Tail HCVs

PGBouncer HCVs

PostgreSQL Subchart HCVs

CloudSQL Auth Proxy HCVs

OpenID Connect HCVs

Test Connection HCVs

Proxy HCVs

Language Clients

Pachyderm Supported Releases & Features

Pipeline Scaling Limits (CE)

Pipeline Specification (PPS)

Autoscaling PPS

Datum Set Spec PPS

Datum Timeout PPS

Datum Tries PPS

Description PPS

Egress PPS

Input Cron PPS

Input Cross PPS

Input Group PPS

Input Join PPS

Input PFS PPS

Input Union PPS

Job Timeout PPS

Metadata PPS

Output Branch PPS

Parallelism Spec PPS

Pod Patch PPS

Pod Spec PPS

Reprocess Spec PPS

Resource Limits PPS

Resource Requests PPS

s3 Out PPS

Scheduling Spec PPS

Service PPS

Sidecar Resource Limits PPS

Spec Commit PPS

Spout PPS

Transform PPS

S3 Gateway API

Enterprise Edition

Activate Enterprise via Helm

Activate Enterprise via Pachctl

Authentication & Authorization

Authentication

Activate Authorization

Set Up IdP Connectors

Auth0

Okta

Log In via IdP

Check IdP User

Revoke User Access

Deactivate Authorization

Authorization

Add Roles to User

Add Roles to Group

Pachyderm IAM

Enterprise Server (ES)

Activate ES for Multi-Cluster

Activate ES for Single-Cluster

Register a Cluster via Helm

Register a Cluster via Pachctl

Server Management

Server Setup

Features Overview

Troubleshooting Guides

General Troubleshooting

Troubleshooting Deployments

Troubleshooting Pipelines

Contributing

Coding Conventions

Contributor Setup

Developing on Windows with VSCode

Documentation Style Guide

GitHub

Credentials

Learn how to configure an S3 client.

March 24, 2023

Before You Start #

You must configure an S3 Client (Boto3, AWS, MinIO)
You must have authentication enabled.

How to Set Your Credentials #

Auth:

Run the following command:

more ~/.pachyderm/config.json

Search for your session token: "session_token": "your-session-token-value".
Make sure to fill both fields Access Key ID and Secret Access Key with that same value.

You must set ACCESS_KEY_ID and SECRET_ACCESS_KEY to any matching, non-empty string.

For example, you could set both values to "x".

Robot Users #

Depending on your use case, it might make sense to pass the credentials of a robot-user or another type of user altogether. Refer to the authentication section of the documentation for more RBAC information.

👈 Boto3 MinIO 👉