Setup for contributors¶
General requirements¶
First, go through the general Local Installation Instructions. Additionally, make sure you have the following installed:
Bash helpers¶
To stay up to date, we recommend doing the following.
First clone the code: (Note, as of 07/11/19 pachyderm is using go modules and recommends cloning the code outside of the $GOPATH, we use the location ~/workspace as an example, but the code can live anywhere)
cd ~/workspace
git clone git@github.com:pachyderm/pachyderm
~/.bash_profile
by adding the line: source ~/workspace/pachyderm/etc/contributing/bash_helpers
Special macOS configuration¶
File descriptor limit¶
If you're running tests locally, you'll need to up your file descriptor limit. To do this, first setup a LaunchDaemon to up the limit with sudo privileges:
sudo cp ~/workspace/pachyderm/etc/contributing/com.apple.launchd.limit.plist /Library/LaunchDaemons/
launchctl limit maxfiles
256 unlimited
. After the change you'll see a much bigger number in the first field. This ups the system wide limit, but you'll also need to set a per-process limit. Second, up the per process limit by adding something like this to your ~/.bash_profile
:
ulimit -n 12288
ulimit
unlimited
, don't worry, it took effect. To make sure all of these settings are working, you can test that you have the proper setup by running:
make test-pfs-server
Timeout helper¶
You'll need the timeout
utility to run the make launch
task. To install on mac, do:
brew install coreutils
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
Dev cluster¶
Now launch the dev cluster: make launch-dev-vm
.
And check it's status: kubectl get all
.
pachctl¶
This will install the dev version of pachctl
:
cd ~/workspace/pachyderm
make install
pachctl version
And make sure that $GOPATH/bin
is on your $PATH
somewhere
Getting some images in place for local test runs¶
The following commands will put some images that some of the tests rely on in place in your minikube cluster:
For pachyderm_entrypoint
container:
make docker-build-test-entrypoint
./etc/kube/push-to-minikube.sh pachyderm_entrypoint
For pachyderm/python-build
container:
(cd etc/pipeline-build; make push-to-minikube)
Running tests¶
Now that we have a dev cluster, it's nice to be able to run some tests locally as we are developing.
To run some specific tests, just use go test
directly, e.g:
go test -v ./src/server/cmd/pachctl/cmd
We don't recommend trying to run all the tests locally, they take a while. Use CI for that.
Fully resetting¶
Instead of running the makefile targets to re-compile pachctl
and redeploy a dev cluster, we have a script that you can use to fully reset your pachyderm environment:
- All existing cluster data is deleted
- If possible, the virtual machine that the cluster is running on is wiped out
pachctl
is recompiled- The dev cluster is re-deployed
This reset is a bit more time consuming than running one-off Makefile targets, but comprehensively ensures that the cluster is in its expected state, and is especially helpful when you're first getting started with contributions and don't yet have a complete intuition on the various ways a cluster may get in an unexpected state. It's been tested on docker for mac and minikube, but likely works in other kubernetes environments as well.
To run it, simply call ./etc/reset.py
from the pachyderm repo root.