Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-14486

using s3a:// as the scheme for S3 FS fails on dataproc cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1.0
    • Component/s: None
    • Labels:
      None
    • Rank:
      1|i00ikn:

      Description

      Create a pipeline with s3a as the scheme and run the pipeline on Dataproc cluster, the pipeline fails with the following error

       

      java.nio.file.AccessDeniedException: s3a://REDACTED: getFileStatus on s3a://REDACTED: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 144831DCF411DE81), S3 Extended Request ID: Vnyiz7avsrc4aF1BcG31RiQWb63Qr4osURJBN6/qwDIvZ9EEUXstT7S4eJ9Rbn7Um71ixvh1gaU= 

       

      using s3n as the scheme works. 

       

      For 5.1 change the scheme to s3n while going from dataprep to pipelines

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: