Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3833

Snapshot AVRO and Parquet Sink should work as Snapshot

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.1
    • Component/s: ETL
    • Labels:
      None
    • Release Notes:
      Fixed snapshot sinks so that they can be read safely.
    • Rank:
      1|hzz0r3:

      Description

      Currently Snapshot AVRO and PARQUET Sink don't actually creates Snapshots and link to latest version. This will cause an issue because if any job is processing a version which was current when it started is not more current, the job would fail.

      1. Create two pipelines - one writing to Snapshot AVRO and other reading from it.
      2. Schedule both jobs to run every one minute
      3. You will see at some time that reading job will fail. It doesn't happen immediately (atleast for me it didn't)

        Attachments

          Activity

            People

            • Assignee:
              ashau Albert Shau
              Reporter:
              nitin Nitin Motgi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: