Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-15152

Group-by aggregation causes cycles in field-level lineage

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 6.0.0
    • Fix Version/s: None
    • Component/s: Metadata, Pipeline Plugins
    • Labels:
      None
    • Rank:
      1|i00mmn:

      Description

      Using a group-by aggregation in a pipeline, when I click on the operations for the output field (not the group key, but the aggregate), the UI spins forever. Checking the logs for metadata service, I see:

      2019-03-28 13:56:56,613 - ERROR [metadata.service-executor-741:c.c.c.c.HttpExceptionHandler@70] - Unexpected error: request=GET /v3/namespaces/default/datasets/hits/lineage/fields/hits/operations?direction=incoming&end=now&start=now-7d user=<null>:
      java.lang.IllegalArgumentException: Cycle detected in graph for operations {Group By.Group landing_page=[Google BigQuery.Write]}
      	at co.cask.cdap.data2.metadata.lineage.field.FieldLineageInfo.getTopologicallySortedOperations(FieldLineageInfo.java:695) ~[na:na]
      	at co.cask.cdap.metadata.FieldLineageAdmin.computeFieldOperationInfo(FieldLineageAdmin.java:259) ~[na:na]
      	at co.cask.cdap.metadata.FieldLineageAdmin.processOperations(FieldLineageAdmin.java:214) ~[na:na]
      	at co.cask.cdap.metadata.FieldLineageAdmin.getOperationDetails(FieldLineageAdmin.java:200) ~[na:na]
      	at co.cask.cdap.metadata.LineageHTTPHandler.datasetFieldLineageDetails(LineageHTTPHandler.java:153) ~[na:na]
      	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) ~[na:na]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
      	at io.cdap.http.internal.HttpMethodInfo.invoke(HttpMethodInfo.java:82) ~[io.cdap.http.netty-http-1.2.0.jar:na]
      	at io.cdap.http.internal.HttpDispatcher.channelRead(HttpDispatcher.java:45) [io.cdap.http.netty-http-1.2.0.jar:na]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
      	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
      	at io.cdap.http.internal.NonStickyEventExecutorGroup$NonStickyOrderedEventExecutor.run(NonStickyEventExecutorGroup.java:254) [io.cdap.http.netty-http-1.2.0.jar:na]
      	at io.netty.util.concurrent.UnorderedThreadPoolEventExecutor$NonNotifyRunnable.run(UnorderedThreadPoolEventExecutor.java:277) [io.netty.netty-all-4.1.16.Final.jar:4.1.16.Final]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_181]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] 

      See screenshots for the pipeline:

       

       

        Attachments

          Activity

            People

            • Assignee:
              trishka Trishka
              Reporter:
              andreas Andreas Neumann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: