Uploaded image for project: 'Cookbooks'
  1. Cookbooks
  2. COOK-1

Extend hadoop::repo to support Apache Bigtop based custom distributions


    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: hadoop
    • Labels:
    • Rank:


      Apache Bigtop (http://bigtop.apache.org) is a framework for building and testing Apache big data stack distributions, initially derived from the CDH build infrastructure, now underpinning other distributions like HDP, WANDisco, etc. The killer feature of this framework is it provides users the ability to build RPM or DEB packages from arbitrary SCM repositories and versions of the respective projects, substituting custom or modified versions of the software, yet producing package names and a dependency structure that should more or less drop into hadoop::repo given the similarity to CDH and HDP.

      Suggested changes:

      Admit "bigtop" as a third distribution alternative in hadoop::repo.

      Allow for yum_repo_url , yum_repo_key_url, apt_repo_url, and apt_repo_key_url to be supplied by the user for the "bigtop" alternative.

      Provide an option for installing unsigned binaries (with a warning).

      This will be an essential feature for any developer of Hadoop ecosystem software who wishes to use Coopr for provisioning test clusters to test their work. Using Bigtop isn't too onerous. For example when building test HBase packages, typically these are the steps required to stage new HBase RPM packages for upgrade or reinstallation, assuming previously a repository was initialized by a one time build of all Bigtop packages:

      cd /path/to/hbase/src
      mvn -DskipTests clean install package assembly:single
      cp hbase-assembly/target/hbase-*.tar.gz /path/to/bigtop/dl
      cd /path/to/bigtop
      gradle hbase-rpm
      cp output/hbase/*.rpm /path/to/yum/repo
      cd /path/to/yum/repo
      createrepo .

      This is easy to automate with a dev script. It might then be easy to (re)deploy with Coopr.




            • Assignee:
              derek Derek Wood
              apurtell Andrew Purtell
            • Votes:
              0 Vote for this issue
              5 Start watching this issue


              • Created: