0

I use clickhouse server in docker with just 1 table and several rows in it. I can request all the data in default format with clickhouse client (over TCP) or with some GUI tool like DBeaver (over HTTP).

SELECT * FROM some_table;

Also I can change format to something special:

SELECT * FROM some_table FORMAT Pretty;

I want to request data from clickhouse in protobuf format. Query looks like this:

SELECT * FROM some_table FORMAT Protobuf SETTINGS format_schema = 'proto_file:ProtoStructure';

I have the proto_file.proto in the same directory with clickhouse-client, so I can made TCP request throw it (successful).

But I don't know data structure of TCP request to reproduce it in my program by myself. So I tried to execute the same request in HTTP (through DBeaver) to intercept request and reproduce it. Unfortunately I can't execute script in DBeaver properly, because it complains on proto_file.proto (File not found, I don't know where to place it to make it work). The only thing I known, that format is specified by X-Clickhouse-Format HTTP header, but I don't know and can't find any info about where in HTTP request I should place content of proto file.

So, the main question: Is there any examples of pure HTTP request to clickhouse for protobuf data output format?

1 Answer 1

3

SETTINGS format_schema = 'proto_file:ProtoStructure' -- is the feature of clickhouse-client application. It's only possible with clickhouse-client.

clickhouse-client is the reach client. It queries data from clickhouse-server using TPC/native protocol and forms Protobuf by itself using the schema file.

clickhouse-server is also able to form Protobuf using .proto files (using HTTP and GRPC protocols). But in this case .proto files should be placed at the clickhouse-server node into /var/lib/clickhouse/format_schemas/ folder.

https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings/#server_configuration_parameters-format_schema_path


for example I created .proto file

cat /var/lib/clickhouse/format_schemas/test.proto
syntax = "proto3";

message TestMessage {
  int64 id = 1;
  uint32 blockNo = 2;
  string val1 = 3;
  float  val2 = 4;
  uint32 val3 = 5;
};

made it available chown clickhouse.clickhouse test.proto

Now I can do this

curl -o out.protobuf 'localhost:8123/?format_schema=test:TestMessage&query=select+1+id+from+numbers(10)+format+Protobuf'
3
  • Sorry, but this answer is obvious. I already specified all this stuff in my question. I'm looking for the way of requesting data from the server in HTTP (the original recommended and only possible way). So I need to know how to produce Protobuf file to the server or how to use files from format_schemas in the query (because they included by default and can't be specified as input file in query AFAIK) Commented Jan 11, 2023 at 14:47
  • Also, it seems that DBeaver can request data in protobuf, because it's mentioned in official documentation and doesn't complain about FORMAT keyword, only about lack of file Commented Jan 11, 2023 at 14:48
  • Why do you need DBeaver here? I think Clickhouse JDBC driver is unable to use format Protobuf. It uses TSV or RowBinary as transport (it depends on a setting). Also Clickhouse java grpc driver uses Protobuf by default. Commented Jan 11, 2023 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.