Pipeline Scaling Limits in Community Edition¶
Our free Pachyderm Community Edition contains built-in scaling limitations and parallelism thresholds. If you want to scale beyond these limits, request your FREE 30-Day Pachyderm Enterprise token and enjoy unlimited scaling, JupyterHub integration, and more.
You might qualify for a free Enterprise license.
Pachyderm offers activation keys for proofs-of-concept, startups, academic, nonprofit, or open-source projects. Tell us about your project!
Scaling Limits¶
Number of pipelines deployed | Number of workers for each pipeline |
---|---|
Community Users can deploy up to 16 pipelines. | Community Users can run up to 8 workers in parallel on each pipeline. |
What happens when you exceed those limits?¶
As a general rule, Pachyderm provides an error message in the STDERR whenever a limit is encountered that prevents you from successfully running a command. In that case, the alert message links to a free trial request form.
Limit on the number of pipelines¶
When exceeding the number of pipelines:
-
pachctl create pipeline
fails once the maximum number of pipelines is reached. -
pachctl update pipeline
andpachctl edit pipeline
succeed on existing pipelines, fail when attempting to create pipelines beyond the limit.
Note
If update pipeline
fails for any other reason, it does not log any message related to pipeline limits.
pachctl restore
fails when trying to restore with too many pipelines.
All of the commands listed above create a distinct message to STDERR and to the pachd logs. This message includes information such as the limit on the number of pipelines in the Community Edition, the total number of pipelines deployed, and provides a link to request an Enterprise key to lift those limitations.
- all other
extract
,list
,run
,start
,stop pipeline
commands' behavior remains unchanged.
Limit on the number of workers per pipeline¶
Pachyderm has two parallelism strategies: constant and coefficient.
When constant
parallelism > 8:
pachctl create pipeline
andpachctl update pipeline
fail. A message to STDERR and pachd logs is generated. You will need to update your pipeline specification file accordingly or activate an Enterprise license.
The previous behavior differs for a coefficient
parallelism:
pachctl create, update, edit pipeline
will succeed. However, Pachyderm will scale the number of workers up to the Community Edition limit, and ignore the remaining workers if the value set oncoefficient
results in a request for more than 8 workers. A message is logged in pachd logs.
What happens when you upgrade to 1.13 and have more than 16 pipelines?¶
All existing pipelines continue to work after an upgrade to 1.13. pachd logs will mention the successful startup above the "update pipeline message". However, you will not be able to create additional pipelines.
Note
Pipelines automatically generated by the system (for example, build pipelines, cron...) are not considered when assessing the total number of pipelines deployed. The limit applies to user-created pipelines only.