Vector 调研测试同步至 MinIO

前言

继续接着《Vector 调研测试之环境安装》，本文旨在记录对 Vector 写入 MinIO 的功能性测试是否能够满足广泛的需求，以期能够为他人提供有用的参考和帮助。

准确工作

• 新建 Kafka Topic

  kafka-topics -–create -–bootstrap-server kafka-1:9092 -–replication-factor 1 -–partitions 6 -–topic bigdata-data-v1

• 查看 Kafka Topic

  	kafka-topics --list --bootstrap-server kafka-1:9092

• 测试数据

  • 测试数据来源

    https://www.kaggle.com/datasets/eliasdabbas/web-server-access-logs

  • 测试数据Demo

    	54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] "GET /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 HTTP/1.1" 200 30577 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)" "-"
    31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/60844/productModel/200x200 HTTP/1.1" 200 5667 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
    31.56.96.51 - - [22/Jan/2019:03:56:16 +0330] "GET /image/61474/productModel/200x200 HTTP/1.1" 200 5379 "https://www.zanbil.ir/m/filter/b113" "Mozilla/5.0 (Linux; Android 6.0; ALE-L21 Build/HuaweiALE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36" "-"
    40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/14925/productModel/100x100 HTTP/1.1" 200 1696 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
    91.99.72.15 - - [22/Jan/2019:03:56:17 +0330] "GET /product/31893/62100/%D8%B3%D8%B4%D9%88%D8%A7%D8%B1-%D8%AE%D8%A7%D9%86%DA%AF%DB%8C-%D9%BE%D8%B1%D9%86%D8%B3%D9%84%DB%8C-%D9%85%D8%AF%D9%84-PR257AT HTTP/1.1" 200 41483 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0)Gecko/16.0 Firefox/16.0" "-"
    40.77.167.129 - - [22/Jan/2019:03:56:17 +0330] "GET /image/23488/productModel/150x150 HTTP/1.1" 200 2654 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"

• 写入数据

  	kafka-console-producer --broker-list kafka-1:9092 --topic bigdata-data-v1 < ./access.log

• 消费数据查看

  	kafka-console-consumer --bootstrap-server kakfa-1:9092 --topic bigdata-data-v1 --from-beginning

MinIO 设置

• 创建 Bucket

  

• 创建 AK 信息

  

  Access Key: ofbHkAZT0CEsIgq4muR8

  Secret Key: 6T9ap5ywJfxad0oswUFVx0M3AdSCA5fv0cvH3Daq

写入 MinIO

• 测试从Kafka写入到MinIO

  vim /opt/vector/config/minio.toml

  [api]
  enabled = true
  address = "0.0.0.0:8686"
  
  [sources.src_kafka]
  type = "kafka"
  bootstrap_servers = "192.168.31.37:9092"
  auto_offset_reset = "beginning"
  group_id = "consumer-bigdata-es"
  topics = ["bigdata-data-v1"]
  decoding.codec = "bytes"
  
  [transforms.transform_log]
  type = "remap"
  inputs = ["src_kafka"]
  source = """
  . = parse_nginx_log!(string!(.message), "combined")
  .date, err = format_timestamp(.timestamp, "%Y-%m-%d")
  """
  
  [sinks.sink_minio]
  type = "aws_s3"
  inputs = ["transform_log"]
  region = "home"
  bucket = "log-bucket"
  auth.access_key_id = "ofbHkAZT0CEsIgq4muR8"
  auth.secret_access_key = "6T9ap5ywJfxad0oswUFVx0M3AdSCA5fv0cvH3Daq"
  encoding.codec = "native_json"
  compression = "none"
  endpoint = "http://192.168.31.37:9000"
  key_prefix = "access-log/{{ date }}/"
  filename_extension = "json"
  filename_time_format = "%s"
  batch.max_bytes = 100000000
  batch.timeout_secs = 60

• 运行

  	/opt/vector/bin/vector -c /opt/vector/config/minio.toml

• 查看文件是否生成

  

• 抽查文件内容

  

结论

经过部署测试，Vector 用 VRL 解析转换再写入 MinIO 在功能上是完全OK的。