Package google.cloud.bigquery.storage.v1

string

Required. The write_stream identifies the append operation. It must be provided in the following scenarios:

• In the first request to an AppendRows connection.

• In all subsequent requests to an AppendRows connection, if you use the same connection to write to multiple tables or change the input schema for default streams.

For explicitly created write streams, the format is:

• projects/{project}/datasets/{dataset}/tables/{table}/streams/{id}

For the special default stream, the format is:

• projects/{project}/datasets/{dataset}/tables/{table}/streams/_default.

An example of a possible sequence of requests with write_stream fields within a single connection:

• r1: {write_stream: stream_name_1}

• r2: {write_stream: /*omit*/}

• r3: {write_stream: /*omit*/}

• r4: {write_stream: stream_name_2}

• r5: {write_stream: stream_name_2}

The destination changed in request_4, so the write_stream field must be populated in all subsequent requests in this stream.

Authorization requires the following IAM permission on the specified resource writeStream:

• bigquery.tables.updateData

Int64Value

If present, the write is only performed if the next append offset is same as the provided value. If not present, the write is performed at the current end of stream. Specifying a value for this field is not allowed when calling AppendRows for the '_default' stream.

string

Id set by client to annotate its identity. Only initial request setting is respected.

map<string, MissingValueInterpretation>

A map to indicate how to interpret missing value for some fields. Missing values are fields present in user schema but missing in rows. The key is the field name. The value is the interpretation of missing values for the field.

For example, a map {'foo': NULL_VALUE, 'bar': DEFAULT_VALUE} means all missing values in field foo are interpreted as NULL, all missing values in field bar are interpreted as the default value of field bar in table schema.

If a field is not in this map and has missing values, the missing values in this field are interpreted as NULL.

This field only applies to the current request, it won't affect other requests on the connection.

Currently, field name can only be top-level column name, can't be a struct field path like 'foo.bar'.

ProtoData

Rows in proto format.

ProtoSchema

Optional. The protocol buffer schema used to serialize the data. Provide this value whenever:

• You send the first request of an RPC connection.

• You change the input schema.

• You specify a new destination table.

ProtoRows

Required. Serialized row data in protobuf message format. Currently, the backend expects the serialized rows to adhere to proto2 semantics when appending rows, particularly with respect to how default values are encoded.

TableSchema

If backend detects a schema update, pass it to user so that user can use it to input new type of message. It will be empty when no schema updates have occurred.

RowError

If a request failed due to corrupted rows, no rows in the batch will be appended. The API will return row level error info, so that the caller can remove the bad rows and retry the request.

string

The target of the append operation. Matches the write_stream in the corresponding request.

AppendResult

Result if the append is successful.

Status

Error returned when problems were encountered. If present, it indicates rows were not accepted into the system. Users can retry or continue with other append requests within the same connection.

Additional information about error signalling:

ALREADY_EXISTS: Happens when an append specified an offset, and the backend already has received data at this offset. Typically encountered in retry scenarios, and can be ignored.

OUT_OF_RANGE: Returned when the specified offset in the stream is beyond the current end of the stream.

INVALID_ARGUMENT: Indicates a malformed request or data.

ABORTED: Request processing is aborted because of prior failures. The request can be retried if previous failure is addressed.

INTERNAL: Indicates server side error(s) that can be retried.

Int64Value

The row offset at which the last append occurred. The offset will not be set if appending using default streams.

bytes

IPC-serialized Arrow RecordBatch.

int64

[Deprecated] The count of rows in serialized_record_batch. Please use the format-independent ReadRowsResponse.row_count instead.

bytes

IPC serialized Arrow schema.

CompressionCodec

The compression codec to use for Arrow buffers in serialized record batches.

bytes

Binary serialized rows in a block.

int64

[Deprecated] The count of rows in the returning block. Please use the format-independent ReadRowsResponse.row_count instead.

string

Json serialized schema, as described at https://avro.apache.org/docs/1.8.1/spec.html.

bool

Enable displayName attribute in Avro schema.

The Avro specification requires field names to be alphanumeric. By default, in cases when column names do not conform to these requirements (e.g. non-ascii unicode codepoints) and Avro is requested as an output format, the CreateReadSession call will fail.

Setting this field to true, populates avro field names with a placeholder value and populates a "displayName" attribute for every avro field with the original column name.

string

Required. Parent table that all the streams should belong to, in the form of projects/{project}/datasets/{dataset}/tables/{table}.

Authorization requires the following IAM permission on the specified resource parent:

• bigquery.tables.updateData

string

Required. The group of streams that will be committed atomically.

Timestamp

The time at which streams were committed in microseconds granularity. This field will only exist when there are no stream errors. Note if this field is not set, it means the commit was not successful.

StorageError

Stream level error if commit failed. Only streams with error will be in the list. If empty, there is no error and all streams are committed successfully. If non empty, certain streams have errors and ZERO stream is committed due to atomicity guarantee.

string

Required. The request project that owns the session, in the form of projects/{project_id}.

Authorization requires the following IAM permission on the specified resource parent:

• bigquery.readsessions.create

ReadSession

Required. Session to be created.

Authorization requires the following IAM permission on the specified resource readSession:

• bigquery.tables.getData

int32

Max initial number of streams. If unset or zero, the server will provide a value of streams so as to produce reasonable throughput. Must be non-negative. The number of streams may be lower than the requested number, depending on the amount parallelism that is reasonable for the table. There is a default system max limit of 1,000.

This must be greater than or equal to preferred_min_stream_count. Typically, clients should either leave this unset to let the system to determine an upper bound OR set this a size for the maximum "units of work" it can gracefully handle.

int32

The minimum preferred stream count. This parameter can be used to inform the service that there is a desired lower bound on the number of streams. This is typically a target parallelism of the client (e.g. a Spark cluster with N-workers would set this to a low multiple of N to ensure good cluster utilization).

The system will make a best effort to provide at least this number of streams, but in some cases might provide less.

string

Required. Reference to the table to which the stream belongs, in the format of projects/{project}/datasets/{dataset}/tables/{table}.

Authorization requires the following IAM permission on the specified resource parent:

• bigquery.tables.updateData

WriteStream

Required. Stream to be created.

string

Required. Name of the stream to finalize, in the form of projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}.

Authorization requires the following IAM permission on the specified resource name:

• bigquery.tables.updateData

int64

Number of rows in the finalized stream.

string

Required. The stream that is the target of the flush operation.

Authorization requires the following IAM permission on the specified resource writeStream:

• bigquery.tables.updateData

Int64Value

Ending offset of the flush operation. Rows before this offset(including this offset) will be flushed.

int64

The rows before this offset (including this offset) are flushed.

string

Required. Name of the stream to get, in the form of projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}.

Authorization requires the following IAM permission on the specified resource name:

• bigquery.tables.get

WriteStreamView

Indicates whether to get full or partial view of the WriteStream. If not set, view returned will be basic.

bytes

A sequence of rows serialized as a Protocol Buffer.

See https://developers.google.com/protocol-buffers/docs/overview for more information on deserializing this field.

Any

Descriptor for input message. The provided descriptor must be self contained, such that data rows sent can be fully decoded using only the single descriptor. For data rows that are compositions of multiple independent messages, this means the descriptor may need to be transformed to only use nested types: https://developers.google.com/protocol-buffers/docs/proto#nested

For additional information for how proto types and values map onto BigQuery see: https://cloud.google.com/bigquery/docs/write-api#data_type_conversions

string

Required. Stream to read rows from.

Authorization requires the following IAM permission on the specified resource readStream:

• bigquery.readsessions.getData

int64

The offset requested must be less than the last row read from Read. Requesting a larger offset is undefined. If not specified, start reading from offset zero.

int64

Number of serialized rows in the rows block.

StreamStats

Statistics for the stream.

ThrottleState

Throttling state. If unset, the latest response still describes the current throttling status.

AvroRows

Serialized row data in AVRO format.

ArrowRecordBatch

Serialized row data in Arrow RecordBatch format.

AvroSchema

Output only. Avro schema.

ArrowSchema

Output only. Arrow schema.

string

Output only. Unique identifier for the session, in the form projects/{project_id}/locations/{location}/sessions/{session_id}.

Timestamp

Output only. Time at which the session becomes invalid. After this time, subsequent requests to read this Session will return errors. The expire_time is automatically assigned and currently cannot be specified or updated.

DataFormat

Immutable. Data format of the output data. DATA_FORMAT_UNSPECIFIED not supported.

string

Immutable. Table that this ReadSession is reading from, in the form projects/{project_id}/datasets/{dataset_id}/tables/{table_id}

Authorization requires one or more of the following IAM permissions on the specified resource table:

• bigquery.tables.get
• bigquery.tables.getData

TableModifiers

Optional. Any modifiers which are applied when reading from the specified table.

TableReadOptions

Optional. Read options for this session (e.g. column selection, filters).

ReadStream

Output only. A list of streams created with the session.

At least one stream is created with the session. In the future, larger request_stream_count values may result in this list being unpopulated, in that case, the user will need to use a List method to get the streams instead, which is not yet available.

int64

Output only. An estimate on the number of bytes this session will scan when all streams are completely consumed. This estimate is based on metadata from the table which might be incomplete or stale.

int64

Output only. A pre-projected estimate of the total physical size of files (in bytes) that this session will scan when all streams are consumed. This estimate is independent of the selected columns and can be based on incomplete or stale metadata from the table. This field is only set for BigLake tables.

int64

Output only. An estimate on the number of rows present in this session's streams. This estimate is based on metadata from the table which might be incomplete or stale.

string

Optional. ID set by client to annotate a session identity. This does not need to be strictly unique, but instead the same ID should be used to group logically connected sessions (e.g. All using the same ID for all sessions needed to complete a Spark SQL query is reasonable).

Maximum length is 256 bytes.

AvroSchema

Output only. Avro schema.

ArrowSchema

Output only. Arrow schema.

Timestamp

The snapshot time of the table. If not set, interpreted as now.

string

Optional. The names of the fields in the table to be returned. If no field names are specified, then all fields in the table are returned.

Nested fields -- the child elements of a STRUCT field -- can be selected individually using their fully-qualified names, and will be returned as record fields containing only the selected nested fields. If a STRUCT field is specified in the selected fields list, all of the child elements will be returned.

As an example, consider a table with the following schema:

{ "name": "struct_field", "type": "RECORD", "mode": "NULLABLE", "fields": [ { "name": "string_field1", "type": "STRING", . "mode": "NULLABLE" }, { "name": "string_field2", "type": "STRING", "mode": "NULLABLE" } ] }

Specifying "struct_field" in the selected fields list will result in a read session schema with the following logical structure:

struct_field { string_field1 string_field2 }

Specifying "struct_field.string_field1" in the selected fields list will result in a read session schema with the following logical structure:

struct_field { string_field1 }

The order of the fields in the read session schema is derived from the table schema and does not correspond to the order in which the fields are specified in this list.

string

SQL text filtering statement, similar to a WHERE clause in a query. Aggregates are not supported.

Examples: "int_field > 5" "date_field = CAST('2014-9-27' as DATE)" "nullable_field is not NULL" "st_equals(geo_field, st_geofromtext("POINT(2, 2)"))" "numeric_field BETWEEN 1.0 AND 5.0"

Restricted to a maximum length for 1 MB.

ArrowSerializationOptions

Optional. Options specific to the Apache Arrow output format.

AvroSerializationOptions

Optional. Options specific to the Apache Avro output format

double

Optional. Specifies a table sampling percentage. Specifically, the query planner will use TABLESAMPLE SYSTEM (sample_percentage PERCENT). The sampling percentage is applied at the data block granularity. It will randomly choose for each data block whether to read the rows in that data block. For more details, see https://cloud.google.com/bigquery/docs/table-sampling)

string

Output only. Name of the stream, in the form projects/{project_id}/locations/{location}/sessions/{session_id}/streams/{stream_id}.

int64

Index of the malformed row in the request.

RowErrorCode

Structured error reason for a row error.

string

Description of the issue encountered when processing the row.

string

Required. Name of the stream to split.

Authorization requires the following IAM permission on the specified resource name:

• bigquery.readsessions.update

double

A value in the range (0.0, 1.0) that specifies the fractional point at which the original stream should be split. The actual split point is evaluated on pre-filtered rows, so if a filter is provided, then there is no guarantee that the division of the rows between the new child streams will be proportional to this fractional value. Additionally, because the server-side unit for assigning data is collections of rows, this fraction will always map to a data storage boundary on the server side.

ReadStream

Primary stream, which contains the beginning portion of |original_stream|. An empty value indicates that the original stream can no longer be split.

ReadStream

Remainder stream, which contains the tail of |original_stream|. An empty value indicates that the original stream can no longer be split.

StorageErrorCode

BigQuery Storage specific error code.

string

Name of the failed entity.

string

Message that describes the error.

Progress

Represents the progress of the current stream.

double

The fraction of rows assigned to the stream that have been processed by the server so far, not including the rows in the current response message.

This value, along with at_response_end, can be used to interpolate the progress made as the rows in the message are being processed using the following formula: at_response_start + (at_response_end - at_response_start) * rows_processed_from_response / rows_in_response.

Note that if a filter is provided, the at_response_end value of the previous response may not necessarily be equal to the at_response_start value of the current response.

double

Similar to at_response_start, except that this value includes the rows in the current response.

string

Required. The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

Type

Required. The field data type.

Mode

Optional. The field mode. The default value is NULLABLE.

TableFieldSchema

Optional. Describes the nested schema fields if the type property is set to STRUCT.

string

Optional. The field description. The maximum length is 1,024 characters.

int64

Optional. Maximum length of values of this field for STRINGS or BYTES.

If max_length is not specified, no maximum length constraint is imposed on this field.

If type = "STRING", then max_length represents the maximum UTF-8 length of strings in this field.

If type = "BYTES", then max_length represents the maximum number of bytes in this field.

It is invalid to set this field if type is not "STRING" or "BYTES".

int64

Optional. Precision (maximum number of total digits in base 10) and scale (maximum number of digits in the fractional part in base 10) constraints for values of this field for NUMERIC or BIGNUMERIC.

It is invalid to set precision or scale if type is not "NUMERIC" or "BIGNUMERIC".

If precision and scale are not specified, no value range constraint is imposed on this field insofar as values are permitted by the type.

Values of this NUMERIC or BIGNUMERIC field must be in this range when:

• Precision (P) and scale (S) are specified: [-10^(P-S) + 10^(-S), 10^(P-S) - 10^(-S)]
• Precision (P) is specified but not scale (and thus scale is interpreted to be equal to zero): [-10^P + 1, 10^P - 1].

Acceptable values for precision and scale if both are specified:

• If type = "NUMERIC": 1 <= precision - scale <= 29 and 0 <= scale <= 9.
• If type = "BIGNUMERIC": 1 <= precision - scale <= 38 and 0 <= scale <= 38.

Acceptable values for precision if only precision is specified but not scale (and thus scale is interpreted to be equal to zero):

• If type = "NUMERIC": 1 <= precision <= 29.
• If type = "BIGNUMERIC": 1 <= precision <= 38.

If scale is specified but not precision, then it is invalid.

int64

Optional. See documentation for precision.

string

Optional. A SQL expression to specify the default value for this field.

TableFieldSchema

Describes the fields in a table.

int32

How much this connection is being throttled. Zero means no throttling, 100 means fully throttled.

string

Output only. Name of the stream, in the form projects/{project}/datasets/{dataset}/tables/{table}/streams/{stream}.

Type

Immutable. Type of the stream.

Timestamp

Output only. Create time of the stream. For the _default stream, this is the creation_time of the table.

Timestamp

Output only. Commit time of the stream. If a stream is of COMMITTED type, then it will have a commit_time same as create_time. If the stream is of PENDING type, empty commit_time means it is not committed.

TableSchema

Output only. The schema of the destination table. It is only returned in CreateWriteStream response. Caller should generate data that's compatible with this schema to send in initial AppendRowsRequest. The table schema could go out of date during the life time of the stream.

string

Immutable. The geographic location where the stream's dataset resides. See https://cloud.google.com/bigquery/docs/locations for supported locations.