初次使用 Kibana

發表於 Disqus：

以前就有看過 Kibana 的各種資料呈現功能。最近因工作需求，所以研究了一下 Kibana 用法，同時記錄這份筆記。

主要是因為手邊只有現成的 log 檔案，但沒辦法用簡單的 grep 指令來達成複雜的功能，因此才會參考 Kibana 的做法。

開始

首先當然是要架設環境，使用 Docker 即可輕鬆架設。參考官網所寫的方法，要先啟動 Elasticsearch 後，再啟動 Kibana 連線：

docker network create elastic
docker run --name es01-test --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.14.0

docker run --name kib01-test --net elastic -p 5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.14.0

把這段程式翻寫成 Docker Compose 格式：

version: "3.9"

services:
  es:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
    environment:
      - discovery.type=single-node
    networks:
      - elastic
  kibana:
    depends_on:
      - es
    image: docker.elastic.co/kibana/kibana:7.14.0
    environment:
      - ELASTICSEARCH_HOSTS=http://es:9200
    networks:
      - elastic
    ports:
      - 5601:5601
networks:
  elastic:
    driver: bridge

啟動會花點時間。一切就緒後，打開 http://localhost:5601 即可看到 Kibana 的首頁。再來就是上傳資料，新版 Kibana 的介面是可以上傳資料的，剛好符合我要本地查資訊的需求，只是有幾點要注意的：

  1. 資料餵進去之前需要先正規化，因為我是第一次做，所以就是各種 sed awk grep 土法煉鋼。
  2. 它有容量大小限制，預設 100MB，但可以調到 1GB。
  3. 上傳資料的型態因為是由 Kibana 自己判斷，大多數情況都沒什麼問題，只是如果剛好判斷成不想要的 type 就會有點麻煩。

以下給一個 sample 資料，有興趣可以把下面資料寫成 csv 後再載入：

datetime,ip,user,note,device,os,ua
"2021-07-16T12:00:40.695Z","29.71.123.41","2343750","訂閱頻道","Smartphone","Android","Mozilla/5.0 (Linux; Android 10; CPH1951 Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/91.0.4472.120 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/326.0.0.34.120;]"
"2021-07-30T13:23:56.274Z","49.16.179.240","1471261","訂閱頻道","Smartphone","Android","Mozilla/5.0 (Linux; Android 9; SM-J415GN) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"
"2021-07-16T05:53:44.045Z","39.12.16.107","81261443","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36"
"2021-07-10T17:30:38.196Z","116.26.74.115","1387719","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 6.3; WOW64; rv:63.0.247) Gecko/20100101 Firefox/63.0.247 Site24x7"
"2021-07-07T10:44:53.634Z","27.22.74.2","78792005","訂閱頻道","Smartphone","iOS","Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) GSA/166.0.381336632 Mobile/15E148 Safari/604.1"
"2021-07-16T17:39:34.053Z","223.37.213.65","13307474","訂閱頻道","Smartphone NativeAPP","Android","Mozilla/5.0 (Linux; Android 8.0.0; RNE-L02 Build/HUAWEIRNE-L02; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/91.0.4472.101 Mobile Safari/537.36"
"2021-07-12T07:26:53.363Z","","13140316","訂閱頻道","Other","Android","Dalvik/2.1.0 (Linux; U; Android 11; SM-T500 Build/RP1A.200720.012)"
"2021-07-27T07:00:42.502Z","36.232.50.24","1336836","加入粉絲頁","PC","Windows","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
"2021-07-04T23:41:17.608Z","27.240.162.9","5160097","訂閱頻道","Smartphone","iOS","Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"
"2021-07-11T08:37:53.447Z","194.45.52.28","677216","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
"2021-07-19T15:04:16.351Z","111.253.06.216","6742107","訂閱頻道","Smartphone","Android","Mozilla/5.0 (Linux; Android 10; MI 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36"
"2021-07-06T07:59:08.986Z","1.174.5.17","1237591","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
"2021-07-25T03:21:41.527Z","111.83.77.135","5545256","加入粉絲頁","Smartphone","Android","Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-A750GN) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/14.2 Chrome/87.0.4280.141 Mobile Safari/537.36"
"2021-07-04T03:35:38.372Z","106.1.149.148","12408024","加入粉絲頁","PC","OS X","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.64"
"2021-07-17T14:37:58.350Z","61.230.45.158","8441110","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
"2021-07-05T09:53:54.537Z","114.39.8.148","237038","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
"2021-06-30T17:16:44.360Z","180.177.77.137","471764","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
"2021-07-07T13:02:31.050Z","114.198.176.199","4583190","訂閱頻道","Smartphone NativeAPP","Android","Mozilla/5.0 (Linux; Android 11; J9210 Build/55.2.A.4.191; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/91.0.4472.120 Mobile Safari/537.36"
"2021-07-08T11:11:06.644Z","49.217.139.169","9441203","訂閱頻道","PC","Windows","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

每個上傳的資料都會建立唯一的 Index name，不同的資料名字不能重覆。而 index pattern 則是當有很多 index name 要混在一起搜尋的時候，它可以建立一個配對模式，讓符合條件的 index name 列入搜尋範圍中。比方說 order-server-* 就能夠找到下面幾個 index name：

  • order-server-01
  • order-server-03-ex1
  • order-server-03-ex2

建議在一開始規劃 index name 的時候要先考慮好。因為上傳資料後是沒辦法改 index name 的。如果資料又臭又長的話，那就會很悲劇。

接下來就能在 http://localhost:5601/app/discover 分頁查看資料了。以上面的資料來說，因為時間都在 2021 年 7 月，所以需要改成那段時間才能夠找得到資料。

以上，就是安裝並把部分資料餵給它的過程。

AdSense