Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-2986

Spark program should be able to specify runtime arguments when reading or writing a datset

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: Datasets, Spark
    • Labels:
      None
    • Rank:
      1|hzyvxj:

      Description

      Currently, the readFromDataset() and writeToDataset() methods do not accept dataset arguments. They expect that the arguments are already set in the program arguments.

      But if a program is iterative, it may want to change the dataset's runtime arguments in every iteration (for example, write to a new partition in every iteration).

        Attachments

          Activity

            People

            • Assignee:
              rsinha Rohit Sinha
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: