← All repos

pachyderm

Data-Centric Pipelines and Data Versioning

GitHubwww.pachyderm.com/
analyticsbig-datacontainersdata-analysisdata-sciencedistributed-systemsdockergokubernetespachyderm
Browse cluster: SQL databases and distributed systems →
22,752commits
203contributors
15languages

Tech stack & purpose

Pachyderm is a data versioning and pipeline orchestration system designed for data-centric analytics and big-data processing. Built with Go and Kubernetes, it enables users to create reproducible data pipelines with built-in version control for datasets, running on containerized infrastructure. The project includes a console interface built with Node.js, React, TypeScript, and Express, along with supporting infrastructure for deployment via Helm charts and integration with tools like Jaeger for distributed tracing.

Community & reference links

Languages

Go
52.0%
JavaScript
21.5%
TypeScript
17.5%
Starlark
3.2%
Python
2.4%
CSS
1.2%
Shell
0.9%
Mustache
0.5%
Jsonnet
0.2%
Makefile
0.2%
Jupyter Notebook
0.2%
HTML
0.1%
Awk
0.0%
Dockerfile
0.0%
Smarty
0.0%

Contributors (top 30 of 203)