INTEGRITY Cloudflare Docs

← Logs / logs / logpush / logpush-job

Log Output Options

Jobs in Logpush now have a new key, output_options, which replaces logpull_options and allows for more flexible formatting. You can modify output_options via the API.

Replace logpull_options

Previously, Logpush jobs could be customized by specifying the list of fields, sampling rate, and timestamp format in logpull_options as URL-encoded parameters. For example:

{
  "id": <JOB_ID>,
  "dataset": "http_requests",
  "enabled": false,
  "name": "<DOMAIN_NAME>",
  "logpull_options": "fields=ClientIP,EdgeStartTimestamp,RayID&sample=0.1&timestamps=rfc3339",
  "destination_conf": "s3://<BUCKET_PATH>?region=us-west-2"
}

We have replaced this with output_options as it is used for both Logpull and Logpush.

{
  "id": <JOB_ID>,
  "dataset": "http_requests",
  "enabled": false,
  "name": "<DOMAIN_NAME>",
  "output_options": {
    "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
    "sample_rate": 0.1,
    "timestamp_format": "rfc3339"
  },
  "destination_conf": "s3://<BUCKET_PATH>?region=us-west-2"
}

:::caution[Updates replace output_options in full]

When you update a Logpush job via PUT /accounts/{account_id}/logpush/jobs/{job_id} or PUT /zones/{zone_id}/logpush/jobs/{job_id}, the output_options object is replaced entirely. Any field that was previously set but omitted from the update payload is reset to its default value. For example, if the existing job sets timestamp_format: "rfc3339" and your update only includes field_names, the job will revert to the default timestamp_format (unixnano for API-created jobs). Always include the complete output_options object you want applied when updating a job.

Output types

By default Logpush outputs each record as a single line of JSON (also known as ndjson).

With output_options you can switch to CSV or single JSON object, further customize prefixes, suffixes, delimiters, or provide your own record template (in a stripped-down version of Go text/template ↗ syntax).

The output_options object has the following settings:

Examples

Specifying field_names and output_type will result in the remaining options being configured as below for the specified output_type:

ndjson

Default output_options for ndjson

{
	"record_prefix": "{",
	"record_suffix": "}\n",
	"field_delimiter": ","
}

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "ndjson"
}

Example output

{"ClientIP":"89.163.242.206","EdgeStartTimestamp":1506702504433000200,"RayID":"3a6050bcbe121a87"}
{"ClientIP":"89.163.242.207","EdgeStartTimestamp":1506702504433000300,"RayID":"3a6050bcbe121a88"}
{"ClientIP":"89.163.242.208","EdgeStartTimestamp":1506702504433000400,"RayID":"3a6050bcbe121a89"}

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "ndjson",
  "record_template": "\"client-ip\":{{.ClientIP}},\"timestamp\":{{.EdgeStartTimestamp}},\"ray-id\":{{.RayID}}"
}

Example output

{"client-ip":"89.163.242.206","timestamp":1506702504433000200,"ray-id":"3a6050bcbe121a87"}
{"client-ip":"89.163.242.207","timestamp":1506702504433000300,"ray-id":"3a6050bcbe121a88"}
{"client-ip":"89.163.242.208","timestamp":1506702504433000400,"ray-id":"3a6050bcbe121a89"}

Literal with double curly-braces ({{}}), that is, "double{{curly}}braces", can be inserted following go text/template convention, that is, "{{doublecurlybraces}}".

csv

Default output_options for CSV

{
	"record_suffix": "\n",
	"field_delimiter": ","
}

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "csv"
}

Example output

"89.163.242.206",1506702504433000200,"3a6050bcbe121a87"
"89.163.242.207",1506702504433000300,"3a6050bcbe121a88"
"89.163.242.208",1506702504433000400,"3a6050bcbe121a89"

csv/json variants

Based on above, other formats similar to csv or json are also supported:

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "csv",
  "batch_prefix": "ClientIP,EdgeStartTimestamp,RayID\n"
}

Example output

ClientIP,EdgeStartTimestamp,RayID
"89.163.242.206",1506702504433000200,"3a6050bcbe121a87"
"89.163.242.207",1506702504433000300,"3a6050bcbe121a88"
"89.163.242.208",1506702504433000400,"3a6050bcbe121a89"

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "csv",
  "batch_prefix": "ClientIP\tEdgeStartTimestamp\tRayID\n",
  "field_delimiter": "\t"
}

Example output

ClientIP EdgeStartTimestamp  RayID
"89.163.242.206"    1506702504433000200 "3a6050bcbe121a87"
"89.163.242.207"    1506702504433000300 "3a6050bcbe121a88"
"89.163.242.208"    1506702504433000400 "3a6050bcbe121a89"

Example output_options

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "output_type": "ndjson",
  "batch_prefix": "{\"events\":[",
  "batch_suffix": "\n]}\n",
  "record_prefix": "\n  {\"info\":{",
  "record_suffix": "}}",
  "record_delimiter": ","
}

Example output

{
	"events": [
		{
			"info": {
				"ClientIP": "89.163.242.206",
				"EdgeStartTimestamp": 1506702504433000200,
				"RayID": "3a6050bcbe121a87"
			}
		},
		{
			"info": {
				"ClientIP": "89.163.242.207",
				"EdgeStartTimestamp": 1506702504433000300,
				"RayID": "3a6050bcbe121a88"
			}
		},
		{
			"info": {
				"ClientIP": "89.163.242.208",
				"EdgeStartTimestamp": 1506702504433000400,
				"RayID": "3a6050bcbe121a89"
			}
		}
	]
}

How to migrate

In order to migrate your jobs from using logpull_options to the new output_options, take these steps:

  1. Change the &fields=ClientIP,EdgeStartTimestamp,RayID parameter to an array in output_options.field_names.
  2. Change the &sample=0.1 parameter to output_options.sample_rate.
  3. Change the &timestamps=rfc3339 parameter to output_options.timestamp_format.
  4. Change the &CVE-2021-44228=true parameter to output_options.CVE-2021-44228.

For example, if logpull_options are fields=ClientIP,EdgeStartTimestamp,RayID&sample=0.1&timestamps=rfc3339&CVE-2021-44228=true, the output_options would be:

"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "sample_rate": 0.1,
  "timestamp_format": "rfc3339",
  "CVE-2021-44228": true
}