Logo
Pachyderm SDK
2.9.72.10.52.11.4
Pachyderm
  • Get Started
    • First-Time Setup
      • Connect to Existing Instance
        • Language Clients
          • Beginner Tutorial
            • Overview
              • Create a Project
                • Create a Repo & Upload Data
                  • Create Pipelines
                    • Explore Resources, Data, & Logs
                    • Non-Bundled Database Setup
                    • Learn
                      • Key Features
                        • Target Audience
                          • Basic Concepts
                            • Intro to Data Versioning
                              • Intro to Console
                                • Set Cluster Defaults
                                  • Set Project Defaults
                                    • View Dashboard
                                      • View Project
                                        • Repo Actions
                                          • Create Repo
                                            • Manage Repo Metadata
                                              • Upload Files
                                                • View Input Files
                                                  • View Output Files
                                                    • Delete Repo
                                                    • Pipeline Actions
                                                      • Create Pipelines
                                                        • Duplicate Pipelines
                                                          • Pause/Restart Pipelines
                                                            • Rerun Pipelines
                                                              • Update Pipelines
                                                                • View Pipeline Details
                                                                  • List & Inspect Pipelines
                                                                    • Delete Pipelines
                                                                    • List & Inspect Jobs
                                                                    • Intro to Pipelines
                                                                      • Developer Workflow
                                                                        • CI/CD Integration
                                                                          • Create a Machine Learning Workflow
                                                                            • The Push Images Flag
                                                                              • Working with Pipelines
                                                                              • Diagrams
                                                                                • High-Level Architecture Diagram
                                                                                • Glossary
                                                                                  • Ancestry Syntax
                                                                                    • Branch
                                                                                      • Commit
                                                                                        • Commit Set
                                                                                          • Cron
                                                                                            • DAG
                                                                                              • Data Parallelism
                                                                                                • Datum
                                                                                                  • Deferred Processing
                                                                                                    • Distributed Computing
                                                                                                      • File
                                                                                                        • Glob Pattern
                                                                                                          • Global Identifier
                                                                                                            • History
                                                                                                              • Input Repository
                                                                                                                • Job
                                                                                                                  • NLP
                                                                                                                    • Output Repository
                                                                                                                      • Pachyderm Worker
                                                                                                                        • Pipeline
                                                                                                                          • Pipeline Inputs
                                                                                                                            • Pipeline Specification
                                                                                                                              • Project
                                                                                                                                • Provenance
                                                                                                                                  • Task Parallelism
                                                                                                                                    • User Code
                                                                                                                                  • Set Up
                                                                                                                                    • Cloud Deploy
                                                                                                                                      • AWS Deployment
                                                                                                                                        • Azure Deployment
                                                                                                                                          • GCP Deployment
                                                                                                                                            • Console Setup
                                                                                                                                              • Set Up AWS Secret Manager
                                                                                                                                              • Local Deploy
                                                                                                                                                • Docker Desktop
                                                                                                                                                  • Minikube
                                                                                                                                                  • On-Prem Deploy
                                                                                                                                                    • Pachctl
                                                                                                                                                      • Pachctl Auto-completion
                                                                                                                                                        • Unified Deployment
                                                                                                                                                          • Authentication & IdP Connectors
                                                                                                                                                            • MockIDP
                                                                                                                                                              • Auth0
                                                                                                                                                                • Okta
                                                                                                                                                                • Authorization (RBAC)
                                                                                                                                                                  • Access Control (RBAC) Roles & Permissions
                                                                                                                                                                    • Manage RBAC via Console
                                                                                                                                                                      • Add Roles to User via PachCTL
                                                                                                                                                                        • Add Roles to Group via PachCTL
                                                                                                                                                                        • Connection
                                                                                                                                                                          • Environment Variables
                                                                                                                                                                            • Kubernetes RBAC
                                                                                                                                                                              • Import a Kubernetes Context
                                                                                                                                                                                • Log Aggregation (Loki)
                                                                                                                                                                                  • Metadata
                                                                                                                                                                                    • Non-Default Namespaces
                                                                                                                                                                                      • Enterprise Edition
                                                                                                                                                                                        • Activate Enterprise via Helm
                                                                                                                                                                                          • Activate Enterprise via PachCTL
                                                                                                                                                                                            • Features Overview
                                                                                                                                                                                            • Enterprise Server (ES)
                                                                                                                                                                                              • Activate ES for Multi-Cluster
                                                                                                                                                                                                • Activate ES for Single-Cluster
                                                                                                                                                                                                  • Register a Cluster via Helm
                                                                                                                                                                                                    • Register a Cluster via PachCTL
                                                                                                                                                                                                      • Server Management
                                                                                                                                                                                                        • Server Setup
                                                                                                                                                                                                        • S3 Gateway API
                                                                                                                                                                                                          • TLS (SSL, HTTPS)
                                                                                                                                                                                                            • Tracing (Jaeger)
                                                                                                                                                                                                              • Blob/Object Storage
                                                                                                                                                                                                                • Global Config
                                                                                                                                                                                                                • Manage
                                                                                                                                                                                                                  • Helm Chart Values (HCVs)
                                                                                                                                                                                                                    • Deploy Target HCVs
                                                                                                                                                                                                                      • Determined HCVs
                                                                                                                                                                                                                        • Global HCVs
                                                                                                                                                                                                                          • Console HCVs
                                                                                                                                                                                                                            • Enterprise Server HCVs
                                                                                                                                                                                                                              • ETCD HCVs
                                                                                                                                                                                                                                • Loki HCVs
                                                                                                                                                                                                                                  • PachD HCVs
                                                                                                                                                                                                                                    • PachW HCVs
                                                                                                                                                                                                                                      • Kube Event Tail HCVs
                                                                                                                                                                                                                                        • PGBouncer HCVs
                                                                                                                                                                                                                                          • PostgreSQL Subchart HCVs
                                                                                                                                                                                                                                            • CloudSQL Auth Proxy HCVs
                                                                                                                                                                                                                                              • OpenID Connect HCVs
                                                                                                                                                                                                                                                • Test Connection HCVs
                                                                                                                                                                                                                                                  • Preflight Checks
                                                                                                                                                                                                                                                    • Proxy HCVs
                                                                                                                                                                                                                                                    • S3 Gateway
                                                                                                                                                                                                                                                      • AWS CLI
                                                                                                                                                                                                                                                        • Boto3
                                                                                                                                                                                                                                                          • Credentials
                                                                                                                                                                                                                                                            • MinIO
                                                                                                                                                                                                                                                              • Unsupported Operations
                                                                                                                                                                                                                                                              • Back Up & Restore
                                                                                                                                                                                                                                                                • Snapshots
                                                                                                                                                                                                                                                                  • Restoration
                                                                                                                                                                                                                                                                  • Upgrade
                                                                                                                                                                                                                                                                    • PachCTL Shell
                                                                                                                                                                                                                                                                      • Check IdP User
                                                                                                                                                                                                                                                                        • Supported Releases & Features
                                                                                                                                                                                                                                                                          • PostgresSQL Fine-Tuning
                                                                                                                                                                                                                                                                            • Cluster Access
                                                                                                                                                                                                                                                                              • Deactivate Authorization
                                                                                                                                                                                                                                                                                • GPUs
                                                                                                                                                                                                                                                                                  • Log In via IdP
                                                                                                                                                                                                                                                                                    • Revoke User Access
                                                                                                                                                                                                                                                                                      • Scaling Limits (CE)
                                                                                                                                                                                                                                                                                        • Secrets
                                                                                                                                                                                                                                                                                          • Sidecar S3 Gateway
                                                                                                                                                                                                                                                                                            • Storage Optimization
                                                                                                                                                                                                                                                                                              • Usage Metrics
                                                                                                                                                                                                                                                                                                • Monitor with Prometheus
                                                                                                                                                                                                                                                                                                  • Metrics
                                                                                                                                                                                                                                                                                                  • Uninstall
                                                                                                                                                                                                                                                                                                    • Disaster Recovery
                                                                                                                                                                                                                                                                                                    • Prepare Data
                                                                                                                                                                                                                                                                                                      • Data Caching (CDRs)
                                                                                                                                                                                                                                                                                                        • Datum Batching
                                                                                                                                                                                                                                                                                                          • Defer Processing via Staging Branch
                                                                                                                                                                                                                                                                                                            • Ingest Data
                                                                                                                                                                                                                                                                                                              • Mount Volumes
                                                                                                                                                                                                                                                                                                                • Skip Failed Datums
                                                                                                                                                                                                                                                                                                                  • Time-Windowed Data
                                                                                                                                                                                                                                                                                                                    • Transactions
                                                                                                                                                                                                                                                                                                                    • Build Pipelines & DAGs
                                                                                                                                                                                                                                                                                                                      • Pipeline Specification (PPS)
                                                                                                                                                                                                                                                                                                                        • Autoscaling PPS
                                                                                                                                                                                                                                                                                                                          • Datum Set Spec PPS
                                                                                                                                                                                                                                                                                                                            • Datum Timeout PPS
                                                                                                                                                                                                                                                                                                                              • Datum Tries PPS
                                                                                                                                                                                                                                                                                                                                • Description PPS
                                                                                                                                                                                                                                                                                                                                  • Determined
                                                                                                                                                                                                                                                                                                                                    • Egress PPS
                                                                                                                                                                                                                                                                                                                                      • Input Cron PPS
                                                                                                                                                                                                                                                                                                                                        • Input Cross PPS
                                                                                                                                                                                                                                                                                                                                          • Input Group PPS
                                                                                                                                                                                                                                                                                                                                            • Input Join PPS
                                                                                                                                                                                                                                                                                                                                              • Input PFS PPS
                                                                                                                                                                                                                                                                                                                                                • Input Union PPS
                                                                                                                                                                                                                                                                                                                                                  • Job Timeout PPS
                                                                                                                                                                                                                                                                                                                                                    • Maximum Expected Uptime
                                                                                                                                                                                                                                                                                                                                                      • Metadata PPS
                                                                                                                                                                                                                                                                                                                                                        • Output Branch PPS
                                                                                                                                                                                                                                                                                                                                                          • Parallelism Spec PPS
                                                                                                                                                                                                                                                                                                                                                            • Pod Patch PPS
                                                                                                                                                                                                                                                                                                                                                              • Pod Spec PPS
                                                                                                                                                                                                                                                                                                                                                                • Reprocess Spec PPS
                                                                                                                                                                                                                                                                                                                                                                  • Resource Limits PPS
                                                                                                                                                                                                                                                                                                                                                                    • Resource Requests PPS
                                                                                                                                                                                                                                                                                                                                                                      • s3 Out PPS
                                                                                                                                                                                                                                                                                                                                                                        • Scheduling Spec PPS
                                                                                                                                                                                                                                                                                                                                                                          • Service PPS
                                                                                                                                                                                                                                                                                                                                                                            • Sidecar Resource Limits PPS
                                                                                                                                                                                                                                                                                                                                                                              • Sidecar Resource Requests PPS
                                                                                                                                                                                                                                                                                                                                                                                • Spec Commit PPS
                                                                                                                                                                                                                                                                                                                                                                                  • Spout PPS
                                                                                                                                                                                                                                                                                                                                                                                    • Tolerations PPS
                                                                                                                                                                                                                                                                                                                                                                                      • Transform PPS
                                                                                                                                                                                                                                                                                                                                                                                        • Full Pipeline Specification
                                                                                                                                                                                                                                                                                                                                                                                        • Pipeline Ops
                                                                                                                                                                                                                                                                                                                                                                                          • Create a Pipeline
                                                                                                                                                                                                                                                                                                                                                                                            • Delete a Pipeline
                                                                                                                                                                                                                                                                                                                                                                                              • Draw a Pipeline
                                                                                                                                                                                                                                                                                                                                                                                                • Inspect a Pipeline
                                                                                                                                                                                                                                                                                                                                                                                                  • Jsonnet Pipeline Specifications
                                                                                                                                                                                                                                                                                                                                                                                                    • Update a Pipeline
                                                                                                                                                                                                                                                                                                                                                                                                      • View Pipeline Jobs & Runtimes
                                                                                                                                                                                                                                                                                                                                                                                                      • Project Ops
                                                                                                                                                                                                                                                                                                                                                                                                        • Create a Project
                                                                                                                                                                                                                                                                                                                                                                                                          • Set a Project as Current
                                                                                                                                                                                                                                                                                                                                                                                                            • Add a Project Resource
                                                                                                                                                                                                                                                                                                                                                                                                              • Grant Project Access
                                                                                                                                                                                                                                                                                                                                                                                                                • Delete a Project
                                                                                                                                                                                                                                                                                                                                                                                                                • Branch Ops
                                                                                                                                                                                                                                                                                                                                                                                                                  • Copy Files
                                                                                                                                                                                                                                                                                                                                                                                                                    • Process Specific Commits
                                                                                                                                                                                                                                                                                                                                                                                                                      • Set Branch Triggers
                                                                                                                                                                                                                                                                                                                                                                                                                        • Set Output Branch
                                                                                                                                                                                                                                                                                                                                                                                                                        • Datum Ops
                                                                                                                                                                                                                                                                                                                                                                                                                          • Get Metadata
                                                                                                                                                                                                                                                                                                                                                                                                                            • Inspect Datum
                                                                                                                                                                                                                                                                                                                                                                                                                            • Provenance Ops
                                                                                                                                                                                                                                                                                                                                                                                                                              • List Global Commits & Jobs
                                                                                                                                                                                                                                                                                                                                                                                                                                • List Global ID Sub Commits
                                                                                                                                                                                                                                                                                                                                                                                                                                  • Track Downstream
                                                                                                                                                                                                                                                                                                                                                                                                                                    • Delete Branch Head
                                                                                                                                                                                                                                                                                                                                                                                                                                      • Squash Non-Head Commits
                                                                                                                                                                                                                                                                                                                                                                                                                                        • Delete File From History
                                                                                                                                                                                                                                                                                                                                                                                                                                        • Tutorials
                                                                                                                                                                                                                                                                                                                                                                                                                                          • Standard ML Pipeline