Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-12774

Capabilities to use large schemas in CDAP

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.3
    • Component/s: Datasets
    • Labels:
      None
    • Release Notes:
      Increase the dataset changeset size and limit to integer max by default
    • Rank:
      1|i008xz:

      Description

      CDAP-12526 introduces several performance improvements, one of which is to limit the change set size to 10K by default, which can be controlled via the following parameter data.tx.changeset.count.limit.

      This causes a problem when registering schemas in lineage which is used for indexing purposes. The schema size that is used is 4K and it fails with the following error

      co.cask.cdap.api.dataset.DatasetManagementException: Failed to add instance REDACTED, details: Response code: 500, message: 'Internal Server Error', body: 'org.apache.tephra.TransactionSizeException: Change set for transaction 1509412791298000000 has 13350 entries and exceeds the limit of 10000'
      

      This used to work in 4.1.

      We need a way for large schemas to work.

        Attachments

          Activity

            People

            • Assignee:
              andreas Andreas Neumann
              Reporter:
              sree Sreevatsan Raman
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: