Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-5094

ORC format not supported for explore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.2
    • Component/s: Explore
    • Labels:
    • Release Notes:
      Fixed a bug where the explore schema fileset property was being ignored unless an explore format was also present.
    • Rank:
      1|hzz7ov:

      Description

      There is no way to make a fileset use orc format, as there is no way to set the schema for orc filesets.

      You would expect to be able to do something like:

          createDataset("orcfiles", FileSet.class,
                        FileSetProperties.builder()
                          .setExploreFormat("orc")
                          .setExploreSchema("name string, item string")
                          .setEnableExploreOnCreate(true)
                          .build());
      

      This doesn't work because we explicitly limit it to parquet, text, or csv, but would be good to support other hive-native formats. At the very least, would expect to be able to do:

          createDataset("orctest", FileSet.class,
                        FileSetProperties.builder()
                          .setExploreInputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")
                          .setExploreOutputFormat("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat")
                          .setSerDe("org.apache.hadoop.hive.ql.io.orc.OrcSerde")
                          .setExploreSchema("name string, item string")
                          .setEnableExploreOnCreate(true)
                          .build());
      

      but when explore input/output format is used, the explore schema is ignored.

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: