ELK高级搜索二Elasticsearch使用

yangyanping20108

886人浏览 · 2020-07-09 09:51:17

yangyanping20108 · 2020-07-09 09:51:17 发布

ELK高级搜索二Elasticsearch使用

Elasticsearch介绍

百度百科

Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java语言开发的，并作为Apache许可条款下的开放源码发布，是一种流行的企业级搜索引擎。Elasticsearch用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。官方客户端在Java、.NET（C#）、PHP、Python、Apache Groovy、Ruby和许多其他语言中都是可用的。根据DB-Engines的排名显示，Elasticsearch是最受欢迎的企业搜索引擎，其次是Apache Solr，也是基于Lucene。

倒排索引

elasticsearch使用的是一种称为倒排索引的结构，采用Lucene倒排索引作为底层，这种结构适用于快速的全文搜索，一个索引有文档中所有不重复的列表构成，对于每一个词都有一个包含它的文档列表。elasticsearch的索引是有多个Lucene索引组成。

倒排索引建立的是分词（Term）和文档（Document）之间的映射关系，在倒排索引中，数据是面向词（Term）而不是面向文档的。

文档(document)： Elasticsearch是面向文档的，那么就意味着索引和搜索的最小单位就是文档。
类型：类型是文档的逻辑容器，就像关系型数据库一样，表格是行的容器。类型中对字段的定义称为映射。类型在最新的Elasticsearch中已经不是那么重要。
索引：索引是映射类型的容器，Elasticsearch中的索引是一个非常大的文档集合。索引存储了映射类型的字段和其他设置，然后它们被存储到各个分片上。
字段(field)：文档的一个片段，它包含两个部分：字段的名称和内容。
词项(term)：搜索时的一个单位，代表文本中的某个词。
词条(token)：词项在字段中的一次出现，包括词项的文本，开始和结束的位移以及类型。

该结构是一种将词项映射到文档的数据结构，其工作方式与传统的关系数据库不同，大可以认为倒排索引是面向词项而不是面向文档的。接下来我们看看简单的倒排索引是什么样子的。例如，我们有一些只包含title 字段的文档，如：
ElasticSearch Server （文档1）
SQL Server (文档2)
Apache solr (文档3)

词项	计数	文档
ElasticSearch	1	<1>
Server	2	<1><2>
SQL	1	<2>
Apache	1	<3>
solr	1	<3>

Elasticsearch概念

Elasticsearch 和 mysql 的类比

Elasticsearch	索引 index	类型 type	文档 document	字段 filed	约束 schema
mysql	数据库 database	表 table	数据行 row	数据列 column	映射 mapping

Elasticsearch的相关设计
Elasticsearch（集群）中可以包括多个索引（数据库），每个索引可以包含多个类型（表），每个类型可以包含多个文档（行），每个文档可以包含多个字段（列）

物理设计：
Elasticsearch在后台把每个索引划分为多个分片，每个分片可以在集群中的不同服务器上分布。（注意：Elasticsearch单机部署也是一个集群，其默认集群名称为elasticsearch。）
逻辑设计：
一个索引类型中可以包含多个文档，所以可以通过一系列顺序找到对应的文档信息，例如：索引 -> 类型 -> 文档ID，通过这个组合我们能够找到某个具体的文档。（注意：这里的文档ID不一定为整数，大部分情况这个ID是一个字符串。）

Elasticsearch目录结构

目录	说明
logs	日志目录
data	索引目录，存放索引文件的地方
config	配置文件目录
modules	模块目录，包括了es的功能模块
bin	脚本目录，包括：启动、停止等可执行脚本
lib	依赖的jar包，如：lucene-core-8.7.0.jar
plugins	插件目录，es支持插件机制

Elasticsearch启动

单机启动命令

ZBMAC-2f32839f6:bin yangyanping$ cd /Users/yangyanping/Downloads/server/elasticsearch/bin
ZBMAC-2f32839f6:bin yangyanping$ ./elasticsearch -d

关闭命令

ZBMAC-2f32839f6:bin yangyanping$ ps -ef | grep elasticsearch
ZBMAC-2f32839f6:bin yangyanping$ kill 81559

检查ES是否启动成功

GET  http://127.0.0.1:9200
{
    "name": "ZBMAC-15aba3b68",
    "cluster_name": "elasticsearch",
    "cluster_uuid": "ZyNrs0u7SmSAMHDLaDl3WA",
    "version": {
        "number": "7.10.1",
        "build_flavor": "default",
        "build_type": "tar",
        "build_hash": "1c34507e66d7db1211f66f3513706fdf548736aa",
        "build_date": "2020-12-05T01:00:33.671820Z",
        "build_snapshot": false,
        "lucene_version": "8.7.0",
        "minimum_wire_compatibility_version": "6.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1"
    },
    "tagline": "You Know, for Search"
}

名称	描述
name	node名称，取自机器的hostname
cluster_name	集群名称（默认的集群名称就是elasticsearch）
version.number	7.10.1，es版本号
version.lucene_version	封装的lucene版本号

查询集群状态

GET localhost:9200/_cluster/health

{
    "cluster_name": "elasticsearch",
    "status": "yellow",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 7,
    "active_shards": 7,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 1,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 87.5
}

名称	描述
Status	集群状态
Green	所有分片可用
Yellow	所有主分片可用
Red	主分片不可用，集群不可用

快速查看集群中有哪些索引

localhost:9200/_cat/indices?v

在这里插入图片描述

Elasticsearch集群

启动过程
故障检测
与ES通信
查询数据

ES的伪集群配置参数

##node-1
http.cors.enabled: true
http.cors.allow-origin: "*"

cluster.name: my-elastic
node.name: node-1
network.host: 127.0.0.1
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300","127.0.0.1:9301","127.0.0.1:9302"]

##node-2
http.cors.enabled: true
http.cors.allow-origin: "*"

cluster.name: my-elastic
node.name: node-2
network.host: 127.0.0.1
http.port: 9201
transport.tcp.port: 9301
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300","127.0.0.1:9301","127.0.0.1:9302"]

##node-3
http.cors.enabled: true
http.cors.allow-origin: "*"

cluster.name: my-elastic
node.name: node-3
network.host: 127.0.0.1
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300","127.0.0.1:9301","127.0.0.1:9302"]

Head插件安装

下载 elasticsearch-head 插件
跨域访问配置

http.cors.enabled: true
http.cors.allow-origin: "*"

启动命令：grunt server

索引

创建索引

使用postman 发送请求，创建索引

PUT   http://127.0.0.1:9200/book_index

请求返回：

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "blog"
}

删除索引

DELETE localhost:9200/book_index

{
    "acknowledged": true
}

查询索引settings

GET   http://127.0.0.1:9200/book_index/_settings


{
  "book_index" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "10s",
        "indexing" : {
          "slowlog" : {
            "level" : "info",
            "threshold" : {
              "index" : {
                "warn" : "200ms",
                "trace" : "20ms",
                "debug" : "50ms",
                "info" : "100ms"
              }
            },
            "source" : "1000"
          }
        },
        "translog" : {
          "sync_interval" : "5s",
          "durability" : "async"
        },
        "provided_name" : "book_index",
        "max_result_window" : "65536",
        "creation_date" : "1665542914136",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "5m"
          }
        },
        "number_of_replicas" : "2",
        "uuid" : "opdjZOy_QGuEpmsh9Ik9Lw",
        "version" : {
          "created" : "7140299"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot,data_warm,data_cold"
            }
          }
        },
        "search" : {
          "slowlog" : {
            "level" : "info",
            "threshold" : {
              "fetch" : {
                "warn" : "200ms",
                "trace" : "50ms",
                "debug" : "80ms",
                "info" : "100ms"
              },
              "query" : {
                "warn" : "500ms",
                "trace" : "50ms",
                "debug" : "100ms",
                "info" : "200ms"
              }
            }
          }
        },
        "number_of_shards" : "3",
        "merge" : {
          "policy" : {
            "auto_merge_enabled" : "false",
            "inactive_merge_enabled" : "false"
          }
        }
      }
    }
  }
}

查询索引mapping

Get /book_index/_mapping

{
  "book_index" : {
    "mappings" : {
      "dynamic" : "false",
      "dynamic_templates" : [
        {
          "message_full" : {
            "match" : "message_full",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 2048,
                  "type" : "keyword"
                }
              },
              "type" : "text"
            }
          }
        },
        {
          "message" : {
            "match" : "message",
            "mapping" : {
              "type" : "text"
            }
          }
        },
        {
          "strings" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        }
      ],
      "properties" : {
        "author" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "avgReadTime" : {
          "type" : "long"
        },
        "bookDesc" : {
          "type" : "text",
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookDetailId" : {
          "type" : "integer"
        },
        "bookHonors" : {
          "type" : "keyword"
        },
        "bookLabel" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookNameEscape" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookSound" : {
          "type" : "integer"
        },
        "bookSource" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookSourceId" : {
          "type" : "keyword"
        },
        "bookTag" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookTags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "bookType" : {
          "type" : "integer"
        },
        "bookWordType" : {
          "type" : "integer"
        },
        "categoryId" : {
          "type" : "integer"
        },
        "categoryName" : {
          "type" : "text",
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "channelType" : {
          "type" : "integer"
        },
        "chaptersCount" : {
          "type" : "integer"
        },
        "content" : {
          "type" : "keyword"
        },
        "copyFlag" : {
          "type" : "integer"
        },
        "copyright" : {
          "type" : "integer"
        },
        "cover" : {
          "type" : "keyword"
        },
        "createBy" : {
          "type" : "keyword"
        },
        "createTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "creator" : {
          "type" : "keyword"
        },
        "delFlag" : {
          "type" : "integer"
        },
        "detailContent" : {
          "type" : "keyword"
        },
        "downShelfTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "expire" : {
          "type" : "integer"
        },
        "expireTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "freeChaptersCount" : {
          "type" : "keyword"
        },
        "hotNum" : {
          "type" : "long"
        },
        "hotStatus" : {
          "type" : "integer"
        },
        "id" : {
          "type" : "integer"
        },
        "isPay" : {
          "type" : "integer"
        },
        "isbnNum" : {
          "type" : "keyword"
        },
        "level" : {
          "type" : "integer"
        },
        "listenCopyright" : {
          "type" : "integer"
        },
        "maxChapterId" : {
          "type" : "integer"
        },
        "onShelfTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "permission" : {
          "type" : "keyword"
        },
        "press" : {
          "type" : "text",
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "productType" : {
          "type" : "keyword"
        },
        "productTypes" : {
          "type" : "integer"
        },
        "readCount" : {
          "type" : "long"
        },
        "readType" : {
          "type" : "keyword"
        },
        "releaseTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "score" : {
          "type" : "double"
        },
        "scoreCount" : {
          "type" : "long"
        },
        "scoreType" : {
          "type" : "integer"
        },
        "serialStatus" : {
          "type" : "integer"
        },
        "serialTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "shelfStatus" : {
          "type" : "integer"
        },
        "startChapterId" : {
          "type" : "integer"
        },
        "threeCategoryId" : {
          "type" : "integer"
        },
        "threeCategoryName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "twoCategoryId" : {
          "type" : "integer"
        },
        "twoCategoryName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "ik_max_word",
          "search_analyzer" : "ik_smart"
        },
        "updateTime" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "updater" : {
          "type" : "keyword"
        },
        "wordNumber" : {
          "type" : "long"
        }
      }
    }
  }
}

文档

新增文档

POST /book_index_test/_doc/1000000
{
    "author" : "键盘侠",
    "bookDesc" : "人生，一定要拼尽全力！",
    "bookName" : "邻居",
    "bookNameEscape" : "邻居",
    "bookSource" : "文学小说",
    "bookSourceId" : "18433",
    "bookTag" : "暧昧,热血,扮猪吃虎",
    "bookTags" : [
      "暧昧",
      "热血",
      "扮猪吃虎"
    ],
    "bookWordType" : 4,
    "categoryName" : "",
    "channelType" : 1,
    "chaptersCount" : 840,
    "content" : "",
    "copyBookName" : "",
    "copyFlag" : 0,
    "copyright" : 0,
    "cover" : "https://read-img.beijzc.com/book/cover/20230511/content_1683781377638.jpg",
    "createBy" : "",
    "createTime" : 1683781378000,
    "delFlag" : 0,
    "downShelfTime" : 1683781377000,
    "freeChaptersCount" : "0",
    "hotNum" : 1183583,
    "id" : 26293,
    "onShelfTime" : 1683781377000,
    "productTypes" : [
      1
    ],
    "readCount" : 377869,
    "releaseTime" : 1683781377000,
    "score" : 9.7,
    "scoreCount" : 0,
    "serialStatus" : 2,
    "serialTime" : 1686281056000,
    "threeCategoryId" : 32,
    "threeCategoryName" : "都市生活",
    "twoCategoryId" : 14,
    "twoCategoryName" : "都市",
    "updateTime" : 1683949750000,
    "wordNumber" : 5473905,
    "startChapterId" : 10548020,
    "maxChapterId" : 10550946
  }


{
  "_index" : "book_index_test",
  "_type" : "_doc",
  "_id" : "1000000",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "_seq_no" : 775923,
  "_primary_term" : 1
}

修改文档

ES集群修改index副本数报错：index read-only / allow delete

PUT localhost:9200/_settings

Content-Type=application/json;charset=UTF-8
{
    "index":{
        "blocks":{
            "read_only_allow_delete":"false"
        }
    }
}

{
    "acknowledged": true
}

部分更新

POST /book_index_test/_update/1000000
{
  "doc": {
       "bookName" : "邻居续集" 
  }
}


{
  "_index" : "book_index_test",
  "_type" : "_doc",
  "_id" : "1000000",
  "_version" : 4,
  "result" : "updated",
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "_seq_no" : 775926,
  "_primary_term" : 1
}

全部替换

POST /book_index_test/_doc/1000000
{
    "author" : "键盘侠",
    "bookDesc" : "人生，一定要拼尽全力！",
    "bookName" : "邻居续集",
    "bookNameEscape" : "邻居续集",
    "bookSource" : "文学小说",
    "bookSourceId" : "18433",
    "bookTag" : "暧昧,热血,扮猪吃虎",
    "bookTags" : [
      "暧昧",
      "热血",
      "扮猪吃虎"
    ],
    "bookWordType" : 4,
    "categoryName" : "",
    "channelType" : 1,
    "chaptersCount" : 840,
    "content" : "",
    "copyBookName" : "",
    "copyFlag" : 0,
    "copyright" : 0,
    "cover" : "https://read-img.beijzc.com/book/cover/20230511/content_1683781377638.jpg",
    "createBy" : "",
    "createTime" : 1683781378000,
    "delFlag" : 0,
    "downShelfTime" : 1683781377000,
    "freeChaptersCount" : "0",
    "hotNum" : 1183583,
    "id" : 26293,
    "onShelfTime" : 1683781377000,
    "productTypes" : [
      1
    ],
    "readCount" : 377869,
    "releaseTime" : 1683781377000,
    "score" : 9.7,
    "scoreCount" : 0,
    "serialStatus" : 2,
    "serialTime" : 1686281056000,
    "threeCategoryId" : 32,
    "threeCategoryName" : "都市生活",
    "twoCategoryId" : 14,
    "twoCategoryName" : "都市",
    "updateTime" : 1683949750000,
    "wordNumber" : 5473905,
    "startChapterId" : 10548020,
    "maxChapterId" : 10550946
  }


{
  "_index" : "book_index_test",
  "_type" : "_doc",
  "_id" : "1000000",
  "_version" : 5,
  "result" : "updated",
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "_seq_no" : 775927,
  "_primary_term" : 1
}

检索文档

GET /book_index_test/_doc/1000000

{
  "_index" : "book_index_test",
  "_type" : "_doc",
  "_id" : "1000000",
  "_version" : 5,
  "_seq_no" : 775927,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "author" : "键盘侠",
    "bookDesc" : "人生，一定要拼尽全力！",
    "bookName" : "邻家有女",
    "bookNameEscape" : "邻家有女",
    "bookSource" : "暗夜文学",
    "bookSourceId" : "18433",
    "bookTag" : "暧昧,热血,扮猪吃虎",
    "bookTags" : [
      "暧昧",
      "热血",
      "扮猪吃虎"
    ],
    "bookWordType" : 4,
    "categoryName" : "",
    "channelType" : 1,
    "chaptersCount" : 840,
    "content" : "",
    "copyBookName" : "",
    "copyFlag" : 0,
    "copyright" : 0,
    "cover" : "https://read-img.beijzc.com/book/cover/20230511/content_1683781377638.jpg",
    "createBy" : "",
    "createTime" : 1683781378000,
    "delFlag" : 0,
    "downShelfTime" : 1683781377000,
    "freeChaptersCount" : "0",
    "hotNum" : 1183583,
    "id" : 26293,
    "onShelfTime" : 1683781377000,
    "productTypes" : [
      1
    ],
    "readCount" : 377869,
    "releaseTime" : 1683781377000,
    "score" : 9.7,
    "scoreCount" : 0,
    "serialStatus" : 2,
    "serialTime" : 1686281056000,
    "threeCategoryId" : 32,
    "threeCategoryName" : "都市生活",
    "twoCategoryId" : 14,
    "twoCategoryName" : "都市",
    "updateTime" : 1683949750000,
    "updater" : "暗夜文学",
    "wordNumber" : 5473905,
    "startChapterId" : 10548020,
    "maxChapterId" : 10550946
  }
}

删除文档

DELETE /book_index_test/_doc/1000000

{
  "_index" : "book_index_test",
  "_type" : "_doc",
  "_id" : "1000000",
  "_version" : 6,
  "result" : "deleted",
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "_seq_no" : 775928,
  "_primary_term" : 1
}

深开鸿技术专区

更多推荐

5分钟在macOS上运行Windows应用：Whisky免费兼容层终极指南

想在Apple Silicon Mac上无缝运行Windows应用和游戏吗？Whisky就是你的答案！这款基于SwiftUI构建的现代化Wine包装器，为macOS用户提供了优雅的Windows应用兼容解决方案。无需虚拟机或双系统，只需几分钟就能创建专业的Windows运行环境，让Office、Steam游戏、设计软件等Windows程序在Mac上流畅运行。## 🚀 快速入门：从零开始使用W