Cookbooks
  1. Cookbooks
  2. COOK-1

Extend hadoop::repo to support Apache Bigtop based custom distributions

    Details

    • Type: New Feature New Feature
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Component/s: hadoop
    • Labels:
      None
    • Rank:
      1|hzyf1r:

      Description

      Apache Bigtop (http://bigtop.apache.org) is a framework for building and testing Apache big data stack distributions, initially derived from the CDH build infrastructure, now underpinning other distributions like HDP, WANDisco, etc. The killer feature of this framework is it provides users the ability to build RPM or DEB packages from arbitrary SCM repositories and versions of the respective projects, substituting custom or modified versions of the software, yet producing package names and a dependency structure that should more or less drop into hadoop::repo given the similarity to CDH and HDP.


      Suggested changes:

      Admit "bigtop" as a third distribution alternative in hadoop::repo.

      Allow for yum_repo_url , yum_repo_key_url, apt_repo_url, and apt_repo_key_url to be supplied by the user for the "bigtop" alternative.

      Provide an option for installing unsigned binaries (with a warning).


      This will be an essential feature for any developer of Hadoop ecosystem software who wishes to use Coopr for provisioning test clusters to test their work. Using Bigtop isn't too onerous. For example when building test HBase packages, typically these are the steps required to stage new HBase RPM packages for upgrade or reinstallation, assuming previously a repository was initialized by a one time build of all Bigtop packages:

      cd /path/to/hbase/src
      mvn -DskipTests clean install package assembly:single
      cp hbase-assembly/target/hbase-*.tar.gz /path/to/bigtop/dl
      cd /path/to/bigtop
      gradle hbase-rpm
      cp output/hbase/*.rpm /path/to/yum/repo
      cd /path/to/yum/repo
      createrepo .
      

      This is easy to automate with a dev script. It might then be easy to (re)deploy with Coopr.

        Activity

        Hide
        Andrew Purtell added a comment -

        BTW, thanks a lot for adding support for Apache Bigtop and custom distributions based on it!

        Show
        Andrew Purtell added a comment - BTW, thanks a lot for adding support for Apache Bigtop and custom distributions based on it!
        Hide
        jay vyas added a comment -

        Ditto, thanks ! This is really cool .... !

        Show
        jay vyas added a comment - Ditto, thanks ! This is really cool .... !
        Hide
        Derek Wood added a comment -

        Great! Yeah, we'll just bump the default version to 0.8.0.

        Show
        Derek Wood added a comment - Great! Yeah, we'll just bump the default version to 0.8.0.
        Hide
        Chris Gianelloni added a comment -

        The cookbook is released to the wild, so I am going to RESOLVE this. Inclusion into Coopr is already out for review.

        https://github.com/caskdata/coopr/pull/675

        Show
        Chris Gianelloni added a comment - The cookbook is released to the wild, so I am going to RESOLVE this. Inclusion into Coopr is already out for review. https://github.com/caskdata/coopr/pull/675
        Hide
        Chris Gianelloni added a comment -

        Released with hadoop cookbook 1.6.1 version...

        Show
        Chris Gianelloni added a comment - Released with hadoop cookbook 1.6.1 version...

          People

          • Assignee:
            Derek Wood
            Reporter:
            Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: