Uploaded image for project: 'CDAP'
  1. CDAP
  2. CDAP-3270

CLI set stream format documentation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0
    • Component/s: Docs
    • Labels:
      None
    • Release Notes:
      Improved the help provided in the CDAP-CLI for the setting of stream formats.
    • Rank:
      1|hzyxc7:

      Description

      In the CDAP CLI help, we currently have:

      set stream format <stream-id> <format> [<schema>] [<settings>]
      
      Sets the format of a stream. <schema> is a sql-like schema “column_name data_type, ...” or Avro-like JSON schema and <settings> is specified in the format “key1=v1 key2=v2”.
      

      However, it doesn't explain what format is, such as which formats are supported (such as 'grok').

      Perhaps the help should be something along the lines of:

      (supported formats = JSON).
      Sets the format of a stream (supported formats = Avro, grok, JSON). <schema> is either a sql-like schema “column_name data_type, ...” or an Avro-like JSON schema. <settings> is specified in the format “key1=v1 key2=v2”.
      

      The problem came up in this chat:

      [6:54 AM] David Finnegan: Can someone please provide an example for setting a stream format to grok?  I can do it from the gui, but I can't figure out the syntax to get it right from the cli.  I want %{COMBINEDAPACHELOG} with fields of "clientip string, ident string, auth string, timestamp string, verb string, request string, httpversion string, response int, bytes int, referrer string, agent string"
      		The syntax to place the '%{}' identifier eludes me.
      [6:55 AM] Joltie Neutrino: set stream format <name> grok "clientip string, ident string, auth string, timestamp string, verb string, request string, httpversion string, response int, bytes int, referrer string, agent string" 'pattern="%{COMBINEDAPACHELOG}"'
      [6:56 AM] David Finnegan: Thanks!
      		Was that in the docs somewhere? . . . I couldn't find it.
      [7:20 AM] David Finnegan: So, why is there no 'fields' spec?  . . . just the quoted, comma delimeted string of fields.
      

      However, it's not going to help if you don't know if

      'pattern="%{COMBINEDAPACHELOG}"

      is the "settings" parameter. Where should that information be kept so that people can find it? Is it because we are using logstash to parse the logs?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                alvin Alvin Wang
                Reporter:
                John John Jackson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: