disco

Tech stack & purpose

Disco is a distributed Map/Reduce and big-data framework that enables parallel computations over large datasets across unreliable computer clusters, handling technical concerns like communication protocols, load balancing, fault tolerance, and job scheduling automatically. The framework is built on Erlang and Python, with jobs written in Python using simple map and reduce functions. Disco can be deployed on cloud platforms including AWS and Google Cloud through provided automation scripts, and also supports containerization via Docker. The project includes integrations with tools like IPython notebooks and matplotlib, and provides a web interface for cluster management accessible at port 8989.

Community & reference links

• discoproject.org
• Disco Integrations
• Talks on Disco
• docker
• install docker

Languages

Contributors (top 30 of 54)

• Ville Tuulos1,059 commits
• 
  Prashanth Mundkur892 commits · 54 followers
• 
  Jared Flatow846 commits · 121 followers
• Shayan Pooya569 commits
• pmundkur62 commits
• tuulos33 commits
• Patrik Pettersson27 commits
• Taneli Mielikainen23 commits
• 
  Kostis Sagonas21 commits · 189 followers
• 
  J Seppänen14 commits · 10 followers
• Ville H. Tuulos14 commits
• 
  Ruslan Spivak14 commits · 2,310 followers
• dialtone8 commits
• Tait Clarridge8 commits
• Travis Cline7 commits
• Daniel Graña6 commits
• Jerry Zhou4 commits
• Roberto Aguilar4 commits
• Davin Potts4 commits
• Mike DeBoer3 commits
• Jens Rantil3 commits
• JohnEmhoff2 commits
• Evgeny Tataurov2 commits
• Gergely Erdelyi2 commits
• YAMAMOTO Takashi2 commits
• Patrick Butler2 commits
• Taneli Mielikäinen2 commits
• Gijs Molenaar2 commits
• Ryan Nowakowski2 commits
• Erik Dubbelboer2 commits