Online Casino VIP Programs on Playstar NJ High Roller Casino

The dedicated team is always happy to help and ensure you have the best possible gaming experience. While live chat offers the fastest response, all options provide prompt and professional assistance, so choose the one that works best for you. At 1,000 points, members receive exclusive gifts, and a daily spin on the prize wheel with potential wins up to $1,000. Attained with 250 points, offers a $50 Prizeout gift card, a free monthly voucher for various activities, and a daily scratch-off card with prizes up to $500.
Evolution’s Salon Privé Roulette is an example, with betting limits ranging from €1 to €5000 per spin. The high-roller version of roulette separates itself from standard versions by raising the betting limits and offering private tables reserved for select members. Such a rewards scheme adds a percentage from each bet you make to your very own cash pot, which is a claimable cash reward. We also check whether the rewards are tied to certain titles or if they apply across the entire library. We compare these features against standard VIP benefits to measure their true value. We examine how the programme is organised, including whether it follows a tiered system or an invite-only model.

CosmicSlot Loyalty Program

As players ascend, they can unlock more significant rewards, such as higher deposit and withdrawal limits, personalized birthday gifts and even more entries into exclusive draws. VIP and loyalty status in an online casino is about more than a fancy title; it’s about feeling valued and receiving tangible benefits that enhance the overall gaming experience. Move up the ranks by playing your favorite casino games, and unlock exclusive perks to take your vacation to the next level. An invitation-only level for high-spending players, providing personalized weekly cashback offers, high roller bonuses, personalized VIP gifts, and a chance to win up to $3,000 on the upgraded prize wheel. We believe in creating an inclusive and exclusive experience for all our players, and our PlayStar Club loyalty program does just that.
VIP points claimed can be converted into bonus funds, with added rewards released weekly and monthly. Complete the registration form to join the casino with a loyalty scheme. While casinos can’t speed up banks or payment providers, they can prioritise VIP withdrawals. From 10 to 100 or more, free spins are a classic way loyalty programmes reward regulars. At the top levels, some VIP programmes even pay cashback on total wagers, not just net losses. Check out the following list of top benefits you can unlock by collecting those loyalty points.

Top VIP Online Casinos with High Withdrawal Limits and Cashback Offers

You can log in, check promotions, use the cashier, and load games through the app, so it covers the important details you’d expect. There’s a live casino section with top providers running the tables, plus roulette, blackjack, and baccarat variants if you prefer something a bit more traditional. You’re getting the kind of library you’d expect from a casino run by Skill On Net casino, with enough variety to keep most people happy. The main menu gets you to games, promotions, and the VIP lounge without any fuss.

A closer look at the levels of the casino VIP program and their perks

The VIP programs offered by sweepstakes casinos differ from those at traditional real money online casinos. This guide explains how sweepstakes casino VIP programs operate, highlights which rewards deliver real value, and helps players decide if joining one is truly worthwhile. These programs are structured to make sure that players receive benefits that align with their level of engagement. VIP casino bonuses are specialised rewards offered to players who demonstrate consistent and significant gameplay. Beyond gaming, VIP programs curate entertainment experiences that appeal to high-stakes players' sophisticated tastes.

Online Casino Vip Program on Playstar: What high roller perks can you expect in NJ?

He’s the guy chatting with players, blackjack dealers, and asking pit bosses way too many questions, all in the name of “research,” of course. Whether you are playing on a sweepstakes casino or a real money casino site, always remember that gambling is all about fun and entertainment, and it is never a sure way to earn money. Don’t get caught up in chasing the next level if it means you have to play outside of your comfort zone.
Whether strolling the neon jungle called the Vegas Strip, or wandering into a smoky riverboat casino in the Midwest, Ziv’s in his element. When he’s not tracking gambling legislation or looking for the next breaking story, Ziv is living and dying with every pitch and play from his beloved Pittsburgh Steelers, Pirates, and Penguins. Since joining Covers, he's turned his sharp eye (and sharper keyboard) toward everything happening in the fast-moving world of online gambling.
To make it easier for you, we've listed all casinos with a VIP or loyalty programme here on this page. Some casinos may provide other forms of player rewards or incentives instead. The methods below are widely used in online casinos, each with clear strengths and drawbacks. When eligible, these are the games to play the best VIP offers like completely wager-free casino spins on. Some online casinos gift luxury items to their VIP customers to show their appreciation. With over 1,200 games from 35+ providers and strong licences, Wildz holds its place among established VIP casinos online.

The 44Aces app and mobile play

Everything is based on the player’s preference. There are no fixed rewards like in a typical tier system. You can manage your saved articles in your account and clicking the X located at the bottom right of the article. While specific tier names and benefits can vary, the structure is designed to be progressive. Enjoy members-only benefits like complimentary drinks at Casino Royale℠, discounts on WiFi packages, priority check-in and more, depending on your tier status.
Only slots count toward clearing that wagering requirement, so if you’re more of a table game player, those sessions won’t help you make progress on it. You’ve then got a 10x wagering requirement on whatever you win from those spins, meaning if you win £5 from your free spins, you would need to playthrough £50 worth of slots before those winnings become available to withdraw. Let’s compare the two to help you make an informed decision as to which bonus will work better for you.

If you collect fewer points next month than the level requires, you will drop to a lower level. At the start of the following month, your points will be reset to 0 at all levels. For example, if you bet 1 USDT, you will get 5 level points for the bet value, and if you bet 0.1 USDT, you will get 0.5 points for the bet value. At Bitcasino, the points are really easy to work out. ✔️ There are no limits on withdrawals, so users can still make use of the unlimited crypto withdrawal facility. You can visit the VIP page to monitor your progress with your current level.
With its multi-tiered structure, exclusive perks, dedicated support and the backing of the reputable Casino Rewards network, players are treated to a truly premium environment. As you play your favourite games, you’ll earn loyalty points which contribute to your VIP status within the Casino Rewards betzito casino program. The casino is part of Casino Rewards Highest Win Rate Guarantee, ensuring players always get the best available version of games, with many titles provided by Games Global and Pragmatic Play.
One of the most important things to understand about Baytree’s loyalty structure is that progression is strictly brand-specific. These credits act as a secondary in-game balance that can be used for wagers without directly drawing from a player’s cash balance. From a systems perspective, that means they share a common backend, including a nearly identical loyalty framework. From that point on, a dedicated account manager handles withdrawal prioritization and reviews the account directly. This data is tracked in the background to calculate overall account status, independent of individual game outcomes.

  • For players who already hold VIP status at other online or land-based casinos, Luxury Casino offers an attractive “Status Match” feature.
  • Widely recognised in online casinos, this e-wallet is valued for instant deposits and fast withdrawals, usually within 24 hours.
  • While that’s the case, it’s important to realise that all VIP programme casino solutions charge for high-tier membership.
  • From expedited check-in processes to priority restaurant reservations, these programs eliminate waiting times and bureaucratic delays.
  • Compared to standard offers, VIP reloads tend to come with higher caps and more generous conditions.
  • Check out the following list of top benefits you can unlock by collecting those loyalty points.
  • Evolution’s Salon Privé Roulette is an example, with betting limits ranging from €1 to €5000 per spin.

BitStarz Casino’s VIP program is all about the experience you receive as a player, and they work closely with each player to satisfy their needs and wants. We reveal how long it will take you to progress up the rewards ladder and earn some serious crypto rewards with each loyalty program. Many VIP programs are masked with promises of “tailored rewards” but no indication of what they may be or how much you have to spend.

  • Explore top-rated VIP casinos with enticing features, rewarding loyalty programs, and high-limit games from leading online casino game providers.
  • Yes, some online casinos offer one bonus for regular players and one for high rollers.
  • To view full Club Royale terms & conditions, visit RoyalCaribbean.com/content/dam/royal/resources/pdf/casino/club-royale-program-terms-and-conditions.pdf
  • We reveal how long it will take you to progress up the rewards ladder and earn some serious crypto rewards with each loyalty program.
  • Many VIP programs are masked with promises of “tailored rewards” but no indication of what they may be or how much you have to spend.
  • Betplay’s VIP program is designed to cater to each member of the platform
  • A key component of any high-quality loyalty and VIP program is exceptional support.

These status indicators serve both practical and psychological purposes, confirming the player's position within the gaming establishment's hierarchy while facilitating immediate recognition and service. Programs implement visible symbols of status through exclusive card designs, special access passes, and public recognition when appropriate. This includes specialized mobile applications for VIP members, real-time account management tools, and integrated communication platforms. This includes priority access to concerts, sporting events, and cultural experiences. Complimentary stays in premium suites, private jet services, and exclusive resort access become standard offerings.
A transparent path to higher tiers is a sign of a fair casino rewards VIP bonus system. Playzee also welcomes new players with a bonus of 100% up to €300 + 100 free spins. It’s always beneficial to see what other casino players and casino experts have to say. This might include reduced or even zero wagering requirements, or access to special high roller casino bonuses designed to maximise returns on bigger stakes. A personal VIP manager is a standout perk of joining an online casino’s elite club.
While that’s the case, it’s important to realise that all VIP programme casino solutions charge for high-tier membership. Relatively few VIP casinos charge a membership fee for joining a VIP club. The best casino loyalty club provides a steady accumulation of points for random daily activities.
To join the online casino VIP program at PlayStar, sign up for an account with PlayStar and ensure you are over the legal gambling age in New Jersey. With five loyalty levels, tailored rewards, and personalized perks, the VIP program is designed to make you feel like the star of the show. Our online casino VIP program is just one example of how PlayStar is revolutionizing the industry with a focus on personalization, engagement, and customer experience.

Elasticsearch Query DsL查询


Query DsL查询

一 Elasticsearch简介

Elasticsearch 是一个开源的搜索引擎,Elasticsearch 使用 Java 编写的,它的内部使用 Lucene 做索引与搜索,但是它的目的是使全文检索变得简单, 通过隐藏 Lucene 的复杂性,取而代之的提供一套简单一致的 RESTful API。

  • 一个分布式的实时文档存储,每个字段 可以被索引与搜索
  • 一个分布式实时分析搜索引擎
  • 能胜任上百个服务节点的扩展,并支持 PB 级别的结构化或者非结构化数据

 

二 安装并运行

已经在其他文档中详细介绍,此次仅做简单步骤介绍

# 安装:
$   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.3-linux-x86_64.tar.gz
$   tar -xzf elasticsearch-7.13.3-linux-x86_64.tar.gz 
$   cd elasticsearch-7.13.3/
# 运行
sh bin/elasticsearch

# 访问
$ curl http://192.168.3.14:9200/ 
{
  "name" : "87DNZWU",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "e3A3l85MSZuZlRhxj6IB2w",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

 

三 Query DSL 基本结构

查询表达式(Query DSL)是一种非常灵活又富有表现力的查询语言, Elasticsearch 使用它可以以简单的 JSON 接口来展现 Lucene 功能的绝大部分

// 查询
GET /_search  // 查找整个ES中所有索引的内容
{
  "query": {}, //具体的查询语句对象
  "from": 0,   //从第几条数据开始返回
  "size": 100, //返回的条数 默认ES最多返回10000条
  "highlight": { //高亮
    "pre_tags": {}, //高亮内容的前面标签 一般都是html比如<b> <p>这种
    "post_tags": {},//高亮内容的后面标签 一般都是html比如</b> </p>这种
    "fields": { //需要高亮的字段
    }
  },
  "sort": [{ //排序
    "FIELD": { //排序的字段(需要填上具体的字段名)
      "order": "desc"
    }
  }],
  "_source": "{field}" //指定返回的字段
}

// 结果
{
    "took": 350,  // 整个搜索请求消耗了多少毫秒
    "timed_out": false, // 表示本次查询是否超时,如果为true也会返回结果,只是数据可能不完整
    "_shards": { //  显示查询中参与的分片信息,成功多少分片失败多少分片等
        "total": 5, 
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 5, // 匹配到的文档总数
        "max_score": 1, // 为文档中所有_score的最大值
        "hits": [
            {
                "_index": "mysql-shop_trades-order_item_label_binds",
                "_type": "doc",
                "_id": "591935",
                "_score": 1,
                "_source": {
                    "id": 591935,
                    "updated_at": "2021-05-20T06:26:09.000Z",
                    "@version": "1",
                    "bind_item_label_id": 729,
                    "label_type": "brand",
                    "created_at": "2021-05-20T06:26:09.000Z",
                    "@timestamp": "2021-07-07T07:31:36.262Z",
                    "is_deleted": 0,
                    "table_name": "order_item_label_binds",
                    "bar_code": "6907925004486"
                }
            }
        ]
    }
}

 

四 指定索引搜索

 

上述查询会搜索ES中的所有索引,但通常只需要去固定一个或几个索引中搜索,搜索全部无疑会造成资源的浪费,在ES中可以通过以下几种方法来指定索引

  1. 指定一个固定的索引,ops-coffee-nginx-2019.05.15为索引名字
GET /mysql-shop_trades-order_statics/_search

以上表示在mysql-shop_trades-order_statics索引下查找数据

  1. 指定多个固定索引,多个索引名字用逗号分割
GET /mysql-shop_trades-order_statics,mysql-shop_trades-order_item_labels/_search

  1. 用*号匹配,在匹配到的所有索引下查找数据
GET /mysql-shop_trades-*/_search

这里也可以用逗号分割多个匹配索引

五 DSL查询

1、筛选字段

// 筛选_source的数据,单个字段
GET /_search
{
  "_source": "bar_code",  
  "query": {}
}

// 筛选_source的数据,多个字段
{
  "_source": {
    "includes": ["store_id", "sku_id"]
  },
  "query": {}
}

// 对字段进行转换
{
  "docvalue_fields": [
    {
      "field": "updated_at",
      "format": "yyyy-MM-dd HH:mm:ss"
    },
    {
      "field": "num",
      "format": "long" // 没有作用,懵逼...
    }
  ], 
  "query": {}
}

 

2、多条件查询 (where)

  1. constant_score:装另一个查询的查询,固定分数查询,支持filter查询,不支持match查询:
    {
        "constant_score": {
            "filter": {
                "match": {
                    "name": "小米"
                }
            },
            "boost": 10
        }
    }
    

     

  2. bool:主要与其他关键字组合使用,多条件的查询必须要用bool包在外层,然后再根据具体的业务来拼接。

{
  "query": {
    "bool": {
      "should": [{}], //满足其中一个对象查询条件就行 像sql里的or
      "must": [{}],   //必须满足所有对象的查询条件 就像sql里的and
      "must_not": [{}] //必须不满足所有对象的查询条件 就像sql里的and !=
    }
  }
}

 

  1. must: 类似于SQL中的AND,必须包含
  2. must_not: 类似于SQL中的NOT,必须不包含
  3. should: 满足这些条件中的任何条件都会增加评分_score,不满足也不影响,should只会影响查询结果的_score值,并不会影响结果的内容
  4. filter: 与must相似,但不会对结果进行相关性评分_score,大多数情况下我们对于日志的需求都无相关性的要求,所以建议查询的过程中多用filter

3、group by:

ES本身没有group关键词搜索,但支持聚合查询,,需要使用关键字aggs

// 单个字段 group by
{
  "query":{},//这里省略你的查询条件
  "aggs": {
    "age_group": {//这个是指你要返回字段名
      "terms": { //这里还可以用其它关键词 这里terms才能实现group by效果
        "field": "age",//groupby的字段
        "size":1 //返回的条数 相当于group by limit
      }
    }
  }
}

// 多字段group by (如 group by sku_id,store_id)
// 方法一:script
{
  "query":{},
  "aggs": {
    "age_group": {
      "terms": {
        "script":{
          "source": """ 's' + doc['store_id'] + '_s' + doc['sku_id'] """,
          "lang": "painless"
        },
        "size": 10
      }
    }
  }
}

// 方法二:copy to
1. 设置mapping中的多个字段,copy_to 为同一个字段(skuId_storeId)
2. 搜索新字段
{
  "query":{},
  "aggs": {
    "list": {
      "terms": { 
        "field": "skuId_storeId
        "size":1
      }
    }
  }
}

// 方法三:multi_terms (使用高版本,目前6.7不支持)
{
  "aggs": {
    "genres_and_products": {
      "multi_terms": {
        "terms": [{
          "field": "genre" 
        }, {
          "field": "product"
        }]
      }
    }
  }
}

4、order by

order by:注意日期格式和数值格式才支持排序;文本不支持,如果要排序, 需把字段设置为not analysis

// 单排序
{
    "query": {
        "sort": {
            "id": "desc"
        }
    }
}

// avg按照平均值排序
{
    "query": {
        "sort": [
            {
                "id": "desc"
            },
            {
                "price": {
                    "order": "asc",
                    "mode": "avg"
                }
            }
        ]
    }
}

5、count(distinct)

{
  "query":{},
  "aggs": {
    "total_sku_id": {
      "cardinality":{ "field": "sku_id"}
    },
     "total_entity_store_id": { // 非数字类型,无法使用field排序,可以对field增加fieldData = true,或者对field.keyword排序,建议使用后者,高效内存消耗低
      "cardinality":{ "field": "entity_store_id.keyword"}
    }
  }
}

 

6、SUM

{
  "query":{},
  "aggs": {
    "total_pay_num": {
      "sum": {"field": "num"}
    },
     "total_cost_fee": {
      "sum": {"field": "cost_fee"}
    }
  }
}

 

7、distinct :

select distinct(id) from table

{
  "query":{},
  "collapse": {
      "field": "id" //你需要distinct的字段
   }, 
}

 

8、limit

1. 分页:
    1. form:从第几个开始查询,最开始是0
    2. size:即limit
    3. 使用size,size最大可获取数量是xx个
2. 获取所有数据的三种方式
    1. scroll深度滚动需要根据scroll_id和循环取,取完后,需删除scroll,减小内存开销(深度滚动高效,用于处理大量数据,不适合实时获取)
    2.调整索引index.max_result_window的大小,默认10000 (大小与堆内存成正比,这个限制内存)
    3.search_after:请求需增加前一个结果的排序,(实时游标,可根据索引更新和删除而改变 )
    4。 如果是group by查询获取所有数据, 获取需要使用到cardinality查询预估总数,再使用partition、num_partitions分区依次获取数据

9、搜索关键字

  1. match:自定字段,根据字段关键字进行搜索,会分割关键词,匹配到含有一个多多个词的匹配
  2. query_string:全文搜索
  3. match_phrase:不分割关键词 {"match_phrase": {"name":"婴幼儿奶粉"}}
  4. term: 类似SQL where field = x,主要用于数字匹配;如果要匹配文本,会自动分词,不能精准查询,需把字段设置成not analysis
    {
      "query": {
        "term": {"bind_item_label_id": 729}
      }
    }
    

     

  5. terms: 类似SQL where field in (x,x),主要用于数字匹配,
    {
      "query": {
        "terms": {"bind_item_label_id": [703,729]}
      }
    }
    

     

  6. range:: 查询价格在1000-2000的商品
{
    "query": {
        "range": {
            "price": {
                "gte": 1000,
                "lte": 2000
            }
        }
    }
}

 

  1. filter:判断文档是否满足条件
{
    "query": {
        "bool": {
            "filter": {
                "term": {
                    "price": 1999
                }
            }
        }
    }
}

Elasticsearch:Aggregation

  1. metric:度量聚合,主要针对number类型的数据,需要es做较多的计算工作(类似SQL的SUM、MAX、AVG、MIN、Cardinality、stats<属于多值分析>等)
  2. bucket:桶聚合,划分不同步的桶,将数据分配到不同的桶,(类似SQL中的group by)
  3. Pipeline Aggregation:管道分析类型,对其他聚合结果进行二次聚合
  4. Matrix Aggregation:矩阵分析类型,支持对多个字段的操作并提供一个结果矩阵

term aggregation

  • size 可以通过size返回top size的文档,该术语聚合针对顶层术语(不包含嵌套词根),其搜索过程是将请求向所有分点发送请求,每个分片节点返回size条数据,然后聚合所有分片的结果(会对各分片返回的同样词根的数数值进行相加),最终从中挑选size条记录返回给客户端。从这个过程也可以看出,其结果并不是准确的,而是一个近似值。
  • Shard Size 为了提高该聚合的精确度,可以通过shard_size参数设置协调节点向各个分片请求的词根个数,然后在协调节点进行聚合,最后只返回size个词根给到客户端,shard_size >= size,如果shard_size设置小于size,ES会自动将其设置为size,默认情况下shard_size建议设置为(1.5 * size + 10)。
// 单个字段 group by
{
  "query":{},//这里省略你的查询条件
  "aggs": {
    "age_group": {//这个是指你要返回字段名
      "terms": { //这里还可以用其它关键词 这里terms才能实现group by效果
        "field": "age",//groupby的字段
        "size":1 //返回的条数 相当于group by limit
      }
    }
  }
}

// 返回结果格式
{
  ...
  "aggregations" : {
      "list" : {
        "doc_count_error_upper_bound" : 0, // 该值表示未进入最终术语列表的术语的最大潜在文档计数
        "sum_other_doc_count" : 90 // 该值表示未进入最终术语列表的术语的最大潜在文档计数
        "buckets" : [ // 返回doc_count排名最前的10个,受size参数的影响
          {
            "key" : "1",
            "doc_count" : 24,
            "total_refund_fee" : {
              "value" : 0.0
            },
            "total_cost_fee" : {
              "value" : -14976.0
            },
          }
        ]
      }
    }
    }
}

 

 

match_phrase :查询分析文本,创建词组查询

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html#query-dsl-match-query-phrase

 

 

举例子

GET mysql-shop_trades-order_item_label_binds/_search/?scroll=1m
{
  "docvalue_fields": [
    {
      "field": "updated_at",
      "format": "yyyy-MM-dd HH:mm:ss"
    }
  ], 
  "size": 1000,
  "sort": {"id":"desc"},
  "query": {
    "bool": {
      "must": [
        {"match": {"is_deleted": 0}},
        {"match": {"label_type": "brand"}},
        {
          "constant_score": {
            "filter": {
              "terms": {
                "bind_item_label_id": [703, 2, 729]
                }
              }
            }
        }
      ]
    }
    }
}




GET mysql-shop_trades-order_item_label_binds/_search/?scroll=1m
{
  "_source": "bar_code", 
  "query": {
    "bool": {
      "filter": [
        {"match": {"is_deleted": 0}},
        {"match_phrase": {"label_type": "brand"}},
        {"terms": {"bind_item_label_id": [703, 2, 729]}}
      ]
    }
  },
  "aggs": {
    "bar_code_group": {
      "terms": {
        "field": "bar_code.keyword",
        "size": 10 
      }
    }
  }
}


GET mysql-shop_trades-order_item_label_binds,mysql-shop_trades-order_statics/_search
{
  "query": {
    "bool": {
      "filter": [
        {"match_phrase": {"sys_name": "yiqigou"}},
        {"range": {"num": {"lte": 2000}}},
        {"range": {"return_num": {"gte": -1000}}},
        {"range": {"total_price": {"lte": 1000000}}},
          {"match": {"id": 60}},
        {"term": {"order_type": 0}},
        {"term": {"item_type": 0}},
        {"range": {"date": {
          "gte": "2020-01-21 00:00:00",
          "lte": "2021-07-22 00:00:00",
          "format": "yyyy-MM-dd HH:mm:ss",
          "time_zone": "+08:00"
        }}}
      ],
      "must_not": [
        {"terms": {"store_id": [165]}}
      ]
    }
  }
}

 

 

设置fieldData

// 第一步,创建索引 (如果已经有索引,直接看第二步)
PUT mysql-shop_trades-order_statics2
{
  "mappings": {
    "_doc": {
      "properties": {
        "entity_store_id": { 
          "type": "text",
          "fields": {
            "keyword": { 
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

// 第二步 设置fieldData为true
PUT mysql-shop_trades-order_statics/_mapping/_doc
{
  "properties": {
    "entity_store_id": {
      "type":     "text",
      "fielddata": true
    }
  }
}

// 第三步 可以查看该索引的Mapping结构,fieldData是否加上去
{
  "mapping": {
    "doc": {
      "properties": {
        "entity_store_id": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "fielddata": true
        }
      }
    }
  }
}

延伸

 

 

设置 max_result_window

PUT /mysql-shop_trades-order_statics/_settings
{
  "index": {
    "max_result_window": 100000
  }
}

 

Elasticsearch SQL

介绍

ES SQL 是x-pack的一个组件,它提供了一个到Elasticsearch的SQL接口,可以针对ES索引执行实时的SQL查询,轻松实时大规模的查询和处理数据,并以表格格式返回结果。它相当于一种翻译器,使用将sql语句转换为DSL语句,再搜索elasticsearch数据

SQL与Elasticsearch的映射关系

(只列常用的)

在 Elasticsearch 中,可用的索引集被分组在一个cluster,一个实例只有一个目录 |

SELECT语法

​ 同sql语法基本一致,基本所有的sql语法都支持

# select
SELECT [TOP [ count ] ] select_expr [, ...]
[ FROM table_name ]
[ WHERE condition ]
[ GROUP BY grouping_element [, ...] ]
[ HAVING condition]
[ ORDER BY expression [ ASC | DESC ] [, ...] ]
[ LIMIT [ count ] ]
[ PIVOT ( aggregation_expr FOR column IN ( value [ [ AS ] alias ] [, ...] ) ) ]

注意:

  1. FROM :目前只支持一张表,不支持连表查询,table_name可以是一个索引,也可以是一个模式(子语句、正则匹配的索引)
  2. WHERE: 从查询中过滤行,如果指定了子句,所有不满足条件的都将从输出中删除
  3. GROUP BY:如果指定了子句或者存在聚合函数调用,则输出将组合成匹配一个或多个值的行组,并计算聚合函数的结果,如果该HAVING子句存在,它将消除不满足给定条件的组
  4. 使用SELECT每个选定行或行组的输出表达式计算实际输出行,可以使用通配符*返回所有的列
  5. 如果ORDER BY指定了子句,则返回的行按指定的顺序排序。如果ORDER BY未给出,则以系统发现最快生成的任何顺序返回行
  6. 如果指定了LIMITor TOP(不能在同一个查询中同时使用两者),则该SELECT语句仅返回结果行的一个子集

缺点:

  1. 不支持复杂的sql: select count(distinct(field1, field2)),可以使用group by替代
  2. 但不支持连表查询,需分表查询

ES SQL 执行方式

  1. 在kabana console 面板中执行sql语句(经常用于调试)
// 方式一:直接搜索sql
GET _sql?format=json
{
"query": """
SELECT *  FROM "mysql-shop_trades-order_statics" where store_id = 165
"""
}

// 方式二:先将sql转换为DSL语句,再通过DSL语句查询结果,这样得到的数据格式比较清晰
// sql转换成DSL语句
GET _sql/translate
{
"query": """
SELECT *  FROM "mysql-shop_trades-order_statics" where store_id = 165
"""
}

// 使用DSL语句查询

GET /mysql-shop_trades-order_statics/_search
{
"size" : 100,
"query" : {
"term" : {
"store_id" : {
"value" : 165,
"boost" : 1.0
}
}
},
"sort" : [
{
"_doc" : {
"order" : "asc"
}
}
]
}

上述是查询一个简单的数据

  1. format:返回数据的格式,支持csv、json、tsv、text、yaml、cbor、smile
  2. sql搜索, 不支持 - .等特殊字符,因此需要转移,包裹sql使用""" 会自动转移特殊字符
  3. 在客户端使用SQL CLI执行
$ ./bin/elasticsearch-sql-cli http://192.168.3.53:9200
sql> select id from "mysql-shop_trades-order_statics" order by id asc limit 2;
  1. 使用 elasticsearch-PHP插件执行 (在业务中使用)
// 读取sql语句,不执行
$where = [
'is_deleted' => 0,
'label_type' => $type,
];
$this->dblink_trade_slave->select('bar_code')->where($where);
if ($is_having) {
$this->dblink_trade_slave->having($having);
}
if ($label_ids) {
$this->dblink_trade_slave->where_in('bind_item_label_id', $label_ids);
}
$sql = $this->dblink_trade_slave->limit(1)->get_compiled_select($index);

// 使用es搜索
$index = '"mysql-shop_trades-order_item_label_binds"'; // 索引需要加引号来转义特殊字符
$params = [
'body' => [
'query' => $sql,
'fetch_size' => 20 // 返回数据条数
]
];
$result = EsClient::sql($params);
// todo 处理结果

Elasticsearch6.3+后,开始支持SQL查询语言,但6.7之前SQL都是实验性质的,6.6进入beta特性,6.7后官方正常正式支持,因此elasticsearch-php 7.0+版本才支持查询x-pack sql,低版本不支持sql查询

需服务器手动安装elasticsearch-sql插件

./bin/elasticsearch-plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/6.7.0.0/elasticsearch-sql-6.7.0.0.zip

[Logstash] Jdbc input plugin


Jdbc input 插件

一 简介

目的:

需要定时将大量数据从mysql采集到ElasticSearch,这里使用Logstash作为数据采集器,使用Jdbc input plugin插件来提取数据到logstash

Jdbc input plugin作用

  • 将具有JDBC接口的任何数据库中的数据提取到Logstash中
  • 可以使用cront语法定期查询或者一次性的将数据加载到Logstash中
  • 结果集中的每一行都成为一个事件,结果集中的列被转换为事件中的字段

 

二 安装

是jdcb集成插件的一个组件,只需安装logstash即可,由于此插件没有和JDBC驱动程序库一起打包,因此需要使用jdbc_driver_library配置选项将所有的jdbc驱动程序传递给插件

环境要求:

  1. elasticsearch-7.13.2
  2. kibana-7.13.2-darwin-x86_64
  3. logstash-7.13.2
  4. logstash-input-jdbc 插件
  5. mysql-connector-java-8.0.25 (驱动)
  6. mysql数据库

 

三 使用

用法

编写logstash-mysql.conf

input{
     jdbc {
        # 驱动类名
        jdbc_driver_library => "/Users/www/elk/7.13.2/mysql-connector-java-8.0.25/mysql-connector-java-8.0.25.jar"
        jdbc_driver_class => "com.mysql.cj.jdbc.Driver" 
        jdbc_default_timezone => "Asia/Shanghai"
         # mysql 数据库链接,shop_trades为数据库名,zeroDateTimeBehaviro防止因时间格式为00-00导致报错
         jdbc_connection_string => "jdbc:mysql://192.168.3.53:3355/shop_trades?zeroDateTimeBehaviro=convertToNull"
         # 连接数据库用户名
         jdbc_user => "cishop"
         # 连接数据库密码
         jdbc_password => "*****"
         # 是否启用分页读取
         jdbc_paging_enabled => "true"
         jdbc_page_size => "1000" 
         # 设置监听间隔  各字段含义(由左至右) 分、时、天、月、年,全部为*默认含义为每分钟都更新
         schedule => "* * * * *"
         # 是否记录上次执行的结果
         record_last_run => "true"
        # 使用其它字段追踪,而不是用时间
         use_column_value => "true"
         tracking_column => "id"
         last_run_metadata_path => "/Users/www/elk/7.13.2/logstash-7.13.2/log-record/order_statics.txt"
         # 是否清除 last_run_metadata_path 的记录,如果为真那么每次都相当于从头开始查询所有的数据库记录
         clean_run => false
         # 直接写sql语句用这个
         statement => "SELECT * FROM `leiyuan` WHERE id > :sql_last_value"
         type => "order_statics" # 如果数据库字段含有type,此数据会被替换掉,不建议这么使用
         add_field => {table_name => "order_statics"}
       }
}
filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

output {
  # stdout {codec => rubydebug }
  stdout {codec => json_lines }
  elasticsearch {
    hosts => ["http://192.168.3.132:9200"]
    index => "mysql-shop_trades-order_statics"
    document_id => "%{id}"
  }
}

执行单个配置文件

# 校验是否有语法错误
bin/logstash -f config/logstash-mysql.conf --config.test_and_exit // 第一次校验配置是否正确

# 执行
bin/logstash -f config/logstash-mysql.conf

# 启动多个logstash文件:需要配置pipelines.yml

 

对多配置文件的引用


[root@VM235 config]# less pipelines.yml |grep -v "#"

 - pipeline.id: mysql
   pipeline.workers: 1
   pipeline.batch.size: 125
   path.config: "/opt/ci123/elk/logstash-6.7.0/config/config.d/mysql.d/*.conf"

 - pipeline.id: application
   pipeline.workers: 1
   pipeline.batch.size: 125
   path.config: "/opt/ci123/elk/logstash-6.7.0/config/config.d/application.conf"
   

 

# 校验
1. 开启config/logstash.yml中的 config.test_and_exit: true
2. bin/logstash
# (画外音:pipelines.yml启动不成功,或找不到pipelines.yml时,很有可能是语法错误,yml对语法邀请非常严格,需仔细检查)

# 启动
bin/logstash  --config.reload.automatic

 

显示以下结果说明,数据正在通过定时脚本导入

[2021-07-05T16:43:00,041][INFO ][logstash.inputs.jdbc     ][main][leiyuan] (0.001561s) SELECT * FROM `leiyuan` WHERE id > 0
{"@timestamp":"2021-07-05T08:43:00.213Z","dated":"2021-07-05T01:28:30.000Z","@version":"1","type":"jdbc","name":"第一名","id":1}
[2021-07-05T16:44:00,063][INFO ][logstash.inputs.jdbc     ][main][leiyuan] (0.002067s) SELECT * FROM `leiyuan` WHERE id > 1
[2021-07-05T16:45:00,066][INFO ][logstash.inputs.jdbc     ][main][leiyuan] (0.001203s) 
{"@timestamp":"2021-07-05T08:52:00.286Z","dated":"2021-07-03T23:19:18.000Z","@version":"1","type":"jdbc","name":"第二名","id":2}
{"@timestamp":"2021-07-05T08:52:00.299Z","dated":"2021-06-30T23:15:00.000Z","@version":"1","type":"jdbc","name":"第三名","id":3}
[2021-07-05T16:53:00,312][INFO ][logstash.inputs.jdbc     ][main][leiyuan] (0.002697s) SELECT * FROM `leiyuan` WHERE id > 3
[2021-07-05T16:53:00,312][INFO ][logstash.inputs.jdbc     ][main][leiyuan] (0.002697s) SELECT * FROM `leiyuan` WHERE id > 3

 

 

 


其他文件:

ELK安装使用手册


elk安装

一、为什么使用elk

一般大型系统是一个分布式部署的架构,不同的服务模块在不同的服务器上,出问题时,需要根据问题暴露的关键信息定位到具体的服务器和模块,构建一套集中式的日志系统,可以提高定位问题的效率
很多时候对于业务关键逻辑,通过file_put_content存储请求数据或者debug,数据量大的时候,对服务器和访问性能都具有不小的影响
一个完整的集中式日志系统包含的主要特点:

  • 收集 - 能够采集多种来源的日志数据
  • 传输 - 能够稳定的吧日志数据传输到中央系统
  • 存储 - 如何存储日志数据
  • 分析 - 可以支持UI分析
  • 警告 - 能够提供错误报告、监控机制

二、elk介绍

elk是三个开源软件的缩写:Elasticsearch , Logstash, Kibana

  • elasticsearch: 开源的日志搜索引擎,可搜集、分析、存储数据
  • kibana:web面板,可以为logstash和els提供的日志分析展示在web界面,
  • logstash:日志的搜索、分析、过滤日志的工具,支持大量的数据获取方式,工作方式是c/s架构,client端安装在需要收集日志的主机上,service端负责收集各个节点日志,并对日志进行过滤和修改等操作,并将其发往elasticsearch上
  • FileBeat:轻量级的日志收集处理工具,使用具在各个服务器上搜集日志后传输给logstash
  • filebeat数据beat,目前beat包含四种工具:
    • Packetbeat(搜集网络流量数据)
    • Topbeat (搜集系统、进程和文件系统级别低的CPU和内存使用情况等数据)
    • filebeat (搜集文件数据)
    • Winlogbeat (搜集windows事件日志数据)

一、安装elasticsearch

# 安装:
$   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.3-linux-x86_64.tar.gz
$   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.3-linux-x86_64.tar.gz.sha512
$   shasum -a 512 -c elasticsearch-7.13.3-linux-x86_64.tar.gz.sha512 
$   tar -xzf elasticsearch-7.13.3-linux-x86_64.tar.gz 
$   cd elasticsearch-7.13.3/ 
 

# 创建es用户 (不能使用root权限执行)
# 1、创建用户:elasticsearch
$ adduser elasticsearch
# 2、创建用户密码,需要输入两次
$ passwd elasticsearch
#3、将对应的文件夹权限赋给该用户
$ chown -R elasticsearch elasticsearch-7.17.3
#4、切换至elasticsearch用户
$ su elasticsearch

 
# config配置
$ less elasticsearch.yml  |grep '#' -v
network.host: 192.168.3.14
http.port: 9200
discovery.seed_hosts: ["192.168.3.14"]
cluster.initial_master_nodes: ["node-1"]
bootstrap.memory_lock: false  # 因运行时报错bootstrap checks failed,此处开启修复这个报错
bootstrap.system_call_filter: false # 因运行时报错bootstrap checks failed,此处开启修复这个报错
# xpack.security.transport.ssl.enabled: true # 开启安全验证,需要设置账号密码时可以开启
# xpack.security.enabled: true  # 开启安全验证,需要设置账号密码时可以开启

# 配置内存大小 (根据服务器内存大小设置适当的值)
$ less ./elasticsearch-7.13.3/config/jvm.options |grep '#' -v
-Xms4g
-Xmx4g
...


# 运行
$   su elasticsearch
$   sh bin/elasticsearch

# 后台运行
$ su elasticsearch
$ nohup bin/elasticsearch &

# 访问请求是否正常启动 (出现下列信息,则正常启动)
$ curl http://192.168.3.14:9200/ 
{
  "name" : "87DNZWU",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "e3A3l85MSZuZlRhxj6IB2w",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

二、安装kibana

# 安装
$   curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.13.3-linux-x86_64.tar.gz
$   curl https://artifacts.elastic.co/downloads/kibana/kibana-7.13.3-linux-x86_64.tar.gz.sha512 | shasum -a 512 -c - 
$   tar -xzf kibana-7.13.3-linux-x86_64.tar.gz
$   mv  kibana-7.13.3-linux-x86_64/  kibana-7.13.3/ 


# config配置 其中xpack 需要执行 (bin/kibana-encryption-keys generate)
$ less kibana.yml |grep '#' -v
server.port: 5602
server.host: "192.168.3.14"
elasticsearch.hosts: ["http://192.168.3.14:9200"]
i18n.locale: "zh-CN"

xpack.encryptedSavedObjects.encryptionKey: 58d5e678bf21278edeed84433f905663
xpack.reporting.encryptionKey: d0b608215432fc28ab1b17ed3906c95a
xpack.security.encryptionKey: 59819c05503d1364e3ec17c34839e6a1

monitoring.cluster_alerts.email_notifications.email_address: leiyuan@corp-ci.com

xpack.reporting.capture.browser.chromium.disableSandbox: true


# 运行
$   sh bin/kibana

# 后台运行
$   nohup bin/kibana &

# 查看是否正常启动
# 网页访问 http://192.168.3.53:5601/, 如果正常访问,则启动成功 

kibana 启动报错ersion GLIBC_2.14 not found 
https://blog.csdn.net/xinjing2015/article/details/93746179

三、安装JAVA

# 官方包内有自带的java环境,可以不用安装,如果想要自己配置的java环境,可以按照下放操作 (根据官方文档按照相应版本的java)
$ yum install java-1.8.0-openjdk.x86_64
$ java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)


四、安装logstash

# 下载 mysql-connector-java (用于连接数据库)
$   wget  http://search.maven.org/remotecontent?filepath=mysql/mysql-\
connector-java/5.1.32/mysql-connector-java-5.1.32.jar


# 官网下载安装包
$   wget https://artifacts.elastic.co/downloads/logstash/logstash-7.13.3-linux-x86_64.tar.gz
$ tar -xzf logstash-7.13.3-linux-x86_64.tar.gz  logstash-7.13.3



# 查询是否可以正常使用  (方式一)
$ bin/logstash -e 'input { stdin {} } output { stdout {} }'
#(画外音:选项 -e 的意思是允许你从命令行指定配置)
# 启动后 输入hello world,可返回json数据,即启动成功



# 接受redis数据   (方法二)
$ less application.conf  |grep -v '#'
input {
    redis {
        data_type => "list"
        key => "logstash-list"
        host => "192.168.3.53"
        port => 8003
        threads => 5
    }
}
output {
  elasticsearch {
    hosts => ["http://192.168.3.53:9200"]
    index => "application_log_%{+YYYY.MM.dd}"
  }
}

# 接受filebeat数据 (暂时不写,自行查询)   (方式三)
$   less  logstash-sample.conf | grep  -v '#'
input {
  beats {
    port => 5044
  }
}

output {
    stdout {codec => rubydebug } # 输出到页面
    elasticsearch {
        hosts => ["http://192.168.3.53:9200"] # 存储的elasticsearch
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

# 接受mysq 数据库数据 (具体写在Mysql>ElasticSearch.md文档中) (方式四)



# 启动单个配置
$ bin/logstash -f config/logstash-sample.conf --config.test_and_exit // 第一次校验配置是否正确
$ bin/logstash -e 'input { stdin { } } output { stdout {} }' // 普通输出
$ bin/logstash -f config/logstash-sample.conf --config.reload.automatic # 执行某个配置文件

# 同时启动多个配置文件 (需配置管道pipelines.yml)
1. 需配置管道pipelines.yml
$ less pipelines.yml |grep -v "#"

 - pipeline.id: mysql
   pipeline.workers: 1
   pipeline.batch.size: 125
   path.config: "/opt/ci123/elk/logstash-6.7.0/config/config.d/mysql.d/*.conf"

 - pipeline.id: application
   pipeline.workers: 1
   pipeline.batch.size: 125
   path.config: "/opt/ci123/elk/logstash-6.7.0/config/config.d/application.conf"

2. 校验
    开启config/logstash.yml中的 config.test_and_exit: true
3. 启动
    bin/logstash # (画外音:pipelines.yml启动不成功,或找不到pipelines.yml时,很有可能是语法错误,yml对语法邀请非常严格,需仔细检查)
    bin/logstash  --config.reload.automatic # 启动并自动加载修改的配置文件

 

五、安全设置

1. 为elk设置用户名和密码

注意:elastic 相当于超级管理员的账号,任何连接都可以使用此账号,但风险较高,建议为各自的模块设置自己的账号和权限

# step1 停止运行kibana和elasticsearch
# step2 在elasticsearch下增加配置 config/elasticsearch.yml
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

# step3 启动elasticsearch
./bin/elasticsearch
# step4 设置密码 (可选择 自动设置,也可以手动设置)
    # (画外音,设置的密码比较多,如果想设置不一样,需提前记录下)
./bin/elasticsearch-setup-passwords auto  #
./bin/elasticsearch-setup-passwords interactive # 手动设置 (注意,此命令有且只能操作一次)


# 修改某个账号密码
$ curl -H "Content-Type:application/json" -XPOST -u {user} 'http://192.168.1.123:9227/_xpack/security/user/{user}/_password' -d '{ "password" : "new_password" }'
Enter host password for user 'elastic':
{}

tip:
    elastic: 需要修改的账号名称
    new_password:新密码


elastic一个内置的超级用户,可用于连接elasticsearch、kibana

kibana_systemKibana 用于连接 Elasticsearch 并与之通信的用户。

logstash_systemLogstash 在 Elasticsearch 中存储监控信息时使用的用户。

beats_systemBeats 在 Elasticsearch 中存储监控信息时使用的用户。

apm_systemAPM 服务器在 Elasticsearch 中存储监控信息时使用的用户。

remote_monitoring_user在 Elasticsearch 中收集和存储监控信息时使用的用户 Metricbeat。它具有remote_monitoring_agentremote_monitoring_collector内置角色。

 

2. 访问elasticsearch

# 测试访问
es@ts_web_123 elasticsearch-7.13.3]$ curl http://192.168.1.123:9227 -u elastic 
Enter host password for user 'elastic': xxx
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "faDVR0zoS5CGGhDcm6TkIg",
  "version" : {
    "number" : "7.13.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5d21bea28db1e89ecc1f66311ebdec9dc3aa7d64",
    "build_date" : "2021-07-02T12:06:10.804015202Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}


3. kibana项目

#kibana: config/kibana
elasticsearch.username: "kibana_system"
elasticsearch.password: "你的密码"
(画外音:kibana密码需要与elastic密码一致,否则启动异常 待解决)

4. logstash 管道连接

为logstash配置的账号

  1. 使用Kibana 中的Management > Roles UI 或roleAPI 创建 logstash_writer角色。对于集群权限,添加manage_index_templates和monitor。对于指数的权限,添加write,create和create_index。
  2. 创建logstash_internal用户并为其分配logstash_writer角色。可以从Kibana 中的Management > Users UI中创建用户

 

output {
  elasticsearch {
    hosts => ["http://192.168.1.14:9200"]
    index => "application_log_%{+YYYY.MM.dd}"
    user => "logstash_internal"
    password => "xxx"
  }
}

 

5. kibana 后台 可设置各类用户角色

可通过内置角色,设置不同的账户

 

六、遇到的常见报错

1.【elasticsearch启动】sh bin/elasticsearch报错:can not run elasticsearch as root
# 创建es用户 (不能使用root权限执行)
# 1、创建用户:elasticsearch
$ adduser elasticsearch
# 2、创建用户密码,需要输入两次
$ passwd elasticsearch
#3、将对应的文件夹权限赋给该用户
$ chown -R elasticsearch elasticsearch-7.17.3
#4、切换至elasticsearch用户
$ su elasticsearch

 

2.【elasticsearch启动】bootstrap checks failed :

问题原因:因为Centos6不支持SecComp,而ES5.2.1默认bootstrap.system_call_filter为true进行检测,所以导致检测失败,失败后直接导致ES不能启动。详见 :https://github.com/elastic/elasticsearch/issues/22899

解决方法:

在elasticsearch.yml中配置bootstrap.system_call_filter为false,注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

 

3.【elasticsearch启动】 max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]

修改max user processes

错误说明: Linux系统为每个用户都设置了一个最大进程数, 这个特性可以让我们控制服务器上现有用户可以创建的进程数量.

(2) 查看max user processes:

# 与查看max open files类似, 可使用 ulimit -u查看max user processes:
ulimit -u

(3) 修改max user processes:

① 方案一: 修改/etc/security/limits.conf文件, 在文件最后添加下述内容:

*  soft      nproc      131072
*  hard      nproc      131072

② 方案二: 修改/etc/security/limits.d/90-nproc.conf文件, 在文件最后添加下述内容:

# 用户进程数的默认限制, 下面这个是对root外的其他用户限制max user processes, 要注释掉: 
# *          soft    nproc     1024
root       soft    nproc     131072

 

4.【elasticsearch启动】max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]**

修改虚拟内存大小

[root@localhost ~]# sysctl -w vm.max_map_count=262144

查看修改结果

[root@localhost ~]# sysctl -a|grep vm.max_map_countvm.max_map_count = 262144

 

5.【kibana启动】Index .kibana_7.13.2_001 belongs to a version of Kibana that cannot be automatically migrated. Reset it or use the X-Pack upgrade assistant.

原因:
遗留了旧版的es后,kibana还存放着原es数据索引。
解决方法:
修改kibana.yml里的,index.kibana 为index.kibana6.7
重启kibana

6. logstash报错:

本地出现报错,无法连接elasticsearch:9200时,关闭本地翻墙软件代理,再尝试。

 

Spring Boot 入门

一、开发基础

  • Java基础(两到三小时过一遍)
  • Java开发环境配置必须使用JDK1.8
  • IDE安装(优先使用IntelliJ IDEA)

二、名词解释

  • Spring:JAVA开发应用框架
  • Spring Boot:用来简化Spring应用的初始搭建以及开发过程的配置框架
  • Maven:Java项目构建工具,成熟的项目
  • Gradle:更简洁的Java项目构建工具,吸收了旧构建工具的优点。
  • JPA:是Sun官方提出的Java持久化规范,即数据库操作规范。
  • Hibernate:Hibernate是一个ORM框架,是JPA的默认实现方式,一般说JPA都是指Hibernate。
  • Mybatis:Mybatis是一个轻便的ORM框架。
  • Spring data jpa:是Spring基于ORM框架、JPA规范的基础上封装的一套JPA应用框架

三、新项目流程

  1. 新建gradle项目
    • File->new->Prroject->Spring Initializr

    • 填写Group、Artifact选择Gradle Project项目生成

    • 可以直接在生成项目的时候选择对应需要安装的插件,如:web、jpa、mybatis等,也可以在项目初始化完成之后在build.gradle中添加/配置

  2. 配置build.gradle(位于根目录)

    plugins {
        id 'org.springframework.boot' version '2.1.3.RELEASE'
        id 'java'
    }
    
    apply plugin: 'io.spring.dependency-management'
    
    group = 'com.duomai'
    version = '0.0.1-SNAPSHOT'
    sourceCompatibility = '1.8' // JDK最大兼容版本
    
    repositories {
        mavenCentral()
    }
    
    dependencies {
        // Spring Boot JPA 组件
        implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
        // Spring Boot Web组件
        implementation 'org.springframework.boot:spring-boot-starter-web'
        // Mybatis插件,注意暂时使用**1.1.1**版本,高版本的运行好像有问题
        implementation 'org.mybatis.spring.boot:mybatis-spring-boot-starter:1.1.1'
        // Mybatis分页插件
        implementation group: 'com.github.pagehelper', name: 'pagehelper-spring-boot-starter', version: '1.2.10'
        runtimeOnly 'org.springframework.boot:spring-boot-devtools'
        runtimeOnly 'mysql:mysql-connector-java'
        testImplementation 'org.springframework.boot:spring-boot-starter-test'
    }
    
    

    修改了build.gradle后,idea会自动安装/更新依赖包。
    参考:gradle官网Spring Boot Web服务搭建Spring Boot Mysql使用Spring Boot JPA使用

  3. 项目基础配置(位于src/resources/application.properties

    #运行配置
    server.port=9000
    #数据格式配置
    spring.jackson.time-zone=GMT+8 // 设置接口返回时区为东八区
    spring.jackson.date-format=yyyy-MM-dd HH:mm:ss // 自动将接口返回中的日期格式转换为标准格式
    #数据库连接配置
    spring.datasource.url=jdbc:mysql://192.168.0.235:3355/shop_balances?serverTimezone=Asia/Shanghai&tinyInt1isBit=false // serverTimezone选择Mysql东八区,tinyInt1isBit禁止Mysql自动将tinyint(1)类型数据映射为boolean类型
    spring.datasource.username=cishop
    spring.datasource.password=fuyuan1906
    #log配置
    logging.path=E:/java/demo/balance_card/log
    logging.level.com.favorites=DEBUG
    #logging.level.org.springframework.web=INFO
    logging.level.org.hibernate=ERROR
    #mybatis设置
    mybatis.type-aliases-package=com.duomai.balance_card.Model.Mapper
    mybatis.configuration.map-underscore-to-camel-case=true
    logging.level.com.duomai.balance_card.Model.Mapper=DEBUG
    #pagehelper插件设置
    pagehelper.helperDialect=mysql
    pagehelper.reasonable=false
    pagehelper.supportMethodsArguments=true
    pagehelper.params=pageNum=page;pageSize=limit
    #jpa 设置
    spring.jpa.properties.hibernate.hbm2ddl.auto=update
    spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
    spring.jpa.show-sql= true
    
  4. 项目启动
    • IDEA启动,运行[Prefix]Application.java文件即可

    • 命令行启动

      • Maven项目
        在springboot的应用的根目录下运行 mvn spring-boot:run

      • Grdale项目
        在springboot的应用的根目录下运行 gradle bootRungradlew bootRun
        (前者是使用本地的gradle版本运行,后者是使用代码仓库中的gradle运行)

    • 打包构建可执行文件运行

  5. 开发环境热更新设置
    热更新不是很好用,有一定的延迟时间

四、项目文件分层解析

// 业务代码 src/main
java 
    com
        duomai
            balance_card
                [Prefix]Application.java // 项目启动文件,可以做一些全局设置,如时区设置、Mapper扫描等
                Config // 配置类,用于注册一些全局配置,如拦截器注册等
                Middleware // 中间件,实现AOP功能
                Controller // 控制器,主要做路由功能
                    xxxController.java
                    BaseErrorController.java // 路由匹配失败时使用的控制器
                Service // 业务代码
                Model // 目录主要用于实体与数据访问层
                    Entity // 数据表实体类
                    Repository // JPA数据仓库
                    Mapper // Mybatis映射文件
                    Provider // Mapper的Sql生成器
                Library // 库类,存放公共类文件/纯定义文件等
                    ApiReturnDefines.java // 接口返回定义
                    ExceptionErrprDefines.java // 异常监听层定义
                Helper // 辅助函数类文件
                OutPut // 接口输出层
                    ApiResult.java // 最终的接口输出格式
                    AiReturn.java // 快速生成ApiResult类,供外部调用
                Exception // 统一的异常处理
                    ControllerHandler // 路由层异常监听
                    SqlHandler // 数据库层异常监听
// 配置项
resources
    appliaction.properties // 项目配置文件

五、控制器中间件

  1. 一般使用Spring过滤器或拦截器实现AOP切面编程

  2. 过滤器和拦截器的对比

    • 作用域不同
      过滤器依赖于servlet容器,只能在 servlet容器,web环境下使用。
      拦截器依赖于spring容器,可以在spring容器中调用,不管此时Spring处于什么环境。
    • 细粒度的不同
      过滤器的控制比较粗,只能在请求进来时进行处理,对请求和响应进行包装。
      拦截器提供更精细的控制,可以分为controller对请求处理之前、渲染视图之后、请求处理之后三个切面。
    • 中断链执行的难易程度不同
      拦截器可以 preHandle方法内返回 false 进行中断。
      过滤器就比较复杂,需要处理请求和响应对象来引发中断,需要额外的动作,比如将用户重定向到错误页面。
  3. 拦截器的使用
    • 编写自定义拦截器(Middleware/ControllerInterceptor.java
    public class ControllerInterceptor implements HandlerInterceptor {
        private Logger logger = LoggerFactory.getLogger(ControllerInterceptor.class);
        @Override
        public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
            logger.info("preHandle....");
            // token校验等
            String token = request.getHeader("token");
            //     Common.sendJson(response, ApiReturn.fail(1001, "token验证失败"));
            //      return false;
            return true;
        }
    
        @Override
        public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
            logger.info("postHandle...");
        }
    
        @Override
        public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
            // 接口日志记录等
            logger.info("afterCompletion...");
        }
     ```
    
    **说明**:
    preHandle:对客户端发过来的请求进行前置处理,如果方法返回true,继续执行后续操作,如果返回false,执行中断请求处理,请求不会发送到Controller。可以在这里校验一些权限信息,如token等,校验失败直接以JSON格式返回请求。
    
    postHandler:在请求进行处理后执行,也就是在Controller方法调用之后处理,前提是preHandle方法返回true。具体来说,postHandler方法会在DispatcherServlet进行视图返回渲染前被调用。
    
    afterCompletion: 该方法在整个请求结束之后执行,前提依然是preHandle方法的返回值为true。
    
    • 注册拦截器(Config/InterceptorConfig.java
    @Configuration
    public class InterceptorConfig implements WebMvcConfigurer {
        @Override
        // 核心方法
        public void addInterceptors(InterceptorRegistry registry) {
            registry.addInterceptor(ControllerInterceptor())
                    //配置拦截规则
                    .addPathPatterns("/**")
                    .order(1);
    
            // 多个拦截器按上述方法持续注册即可,同时也可以设置order值,从小到大执行。
        }
    
        @Bean
        public HandlerInterceptor ControllerInterceptor() {
            return new ControllerInterceptor();
        }
    }
    

六、Controller层

  1. 定义控制器文件
    @RestController
    在类文件头部定义,标明为控制器文件,且输出格式为JSON
    
  2. 路由和参数
    1. 定义路由名称,接收方法
      例:@RequestMapping(value = "/get", method = {RequestMethod.GET, RequestMethod.POST})
      
        可选参数:
      
        value:路由名称
      
        method:指定请求的method类型, GET、POST、PUT、DELETE、PATCH等,可多选
      
        consumes:指定处理请求的提交内容类型(Content-Type),例如application/json, text/html
      
        produces: 指定返回的内容类型,仅当request请求头中的(Accept)类型中包含该指定类型才返回
      
        params: 指定request中必须包含某些参数值
      
        headers: 指定request中必须包含某些指定的header值
      
    2. 参数接收
      例:@RequestParam(value = "fields", required = false, defaultValue = "*") String fields
      
          value:参数名称
      
          defaultValue:默认值
      
          required:是否是必要参数
      
    3. 自定义错误路由
      1. 在Controller层中添加BaseErrorController.java文件,用于监听路由匹配失败的情况
        @Controller
        public class BaseErrorController implements ErrorController {
            @Override
            public String getErrorPath() {
                System.out.print("错误页面");
                return "error/error";
        }
        
            @RequestMapping(value = "/error")
            public void error() throws Exception {
                throw new Exception("路由匹配失败");
            }
        }
        
      2. 在Exception文件夹中添加ControllerHandler.java,用于捕获路由报错并输出。
        @RestControllerAdvice
        public class ControllerHandler {
            // 缺少必选参数
            @ExceptionHandler({MissingServletRequestParameterException.class})
            @ResponseBody
            public ApiResult requestMissingServletRequest(MissingServletRequestParameterException e){
                return ApiReturn.fail(ExceptionErrorDefines.RequestMissingServletRequest, e.getMessage());
            }
        }
        未解决:抛出异常后访问404页面运行环境会报错,但是页面正常  
        

        参考:https://www.jianshu.com/p/393f70b55b1b

  3. Service层调用

    1. Service类成员注入
      • 使用@Autowired修饰符进行依赖注入
        @Autowired
        private final CardService cardService;
        
      • 用构造函数来做注入类成员(推荐使用)
        private StoreBalanceCardsRepository cardsRepository;
        public CardController(StoreBalanceCardsRepository cardsRepository) {
            this.cardsRepository = cardsRepository;
        }
        **注**:
        IntelliJ IDEA使用依赖注入会有IDE报错,但不影响实际编译运行,如需去除报错提示,需要在Dao层(Respository/Mapper)类开头添加注解 `@Repository`
        
    2. 调用
      cardService.get(id, fields);
      

七、Service层

  1. 定义Service文件
    @Service
    在类文件头部定义,标明为Service文件
    
  2. 注入Model层操作文件
    private final StoreBalanceCardsRepository storeBalanceCardsRepository;
    private final StoreBalanceCardsMapper storeBalanceCardsMapper;
    
        CardService(StoreBalanceCardsRepository storeBalanceCardsRepository, StoreBalanceCardsMapper storeBalanceCardsMapper) {
            this.storeBalanceCardsRepository = storeBalanceCardsRepository;
            this.storeBalanceCardsMapper = storeBalanceCardsMapper;
        }
    
  3. 调用Model层文件

八、Model层

  1. Repository

    • Spring中概念,概念类似于数据仓库,是Spring data jpa的实现。居于业务层和数据层之间,将两者隔离开来,在它的内部封装了数据查询和存储的逻辑。

    • Repository和传统意义上的DAO的区别:
      Repository蕴含着真正的OO概念,即一个数据仓库角色,负责所有对象的持久化管理。DAO则没有摆脱数据的影子,仍然停留在数据操作的层面上。Repository是相对对象而言,DAO则是相对数据库而言,虽然可能是同一个东西,但侧重点完全不同。

  2. Mapper

    存放Mybatis数据库关系映射方法

  3. Provider
    为Mapper层提供的SQL生成器,即将SQL的生成与映射解耦。

  4. Entity

    • 根据表结构自动生成实体类
      **注意**:
      i、多次自动生成不会覆盖,如需更新需要把旧的实体类文件删除
      
      ii、有需要的话可以在实体类文件右击generate一键生成构造函数、set get方法、@Autowired等
      
      iii、自动生成的实体类中datetime类型的字段会被转为Java的timestamp类型数据,存储的时候也使用timestamp类型即可
      
      iv、生成出来的实体类catalog = "" 报错,是IDE的报错,不影响使用
      

      参考文档:https://blog.csdn.net/chenju05244554/article/details/1009142081

  5. Hibernate 和 Mybatis 的对比

    • Hibernate优势:
      i、DAO层开发比MyBatis简单,Mybatis需要维护SQL和结果映射。
      
      ii、对对象的维护和缓存要比MyBatis好,对增删改查的对象的维护要方便。
      
      iii、数据库移植性很好,MyBatis的数据库移植性不好,不同的数据库需要写不同SQL。
      
      iv、有更好的二级缓存机制,可以使用第三方缓存。MyBatis本身提供的缓存机制不佳。
      
    • Mybatis优势:
      i、MyBatis可以进行更为细致的SQL优化,可以减少查询字段。
      
      ii、MyBatis容易掌握,而Hibernate门槛较高。
      
    • 选用Mybatis的原因
      i、Hibernate无法满足动态获取部分字段的需求,即使是使用Hibernate提供的原始SQL也无法实现
      
      ii、Hibernate的JPA查询只适用于一些简单的情况,如遇到复杂的SQL,Repository中的方法名会很长。这时候又将回到Hibernate的自定义SQL查询,即原生SQL。
      
      iii、Hibernate的维护成本比MyBatis高很多,MyBatis的SQL生成完全取决于开发者,所以SQL修改、维护、优化会比较便利。
      

九、MyBatis使用

  1. 引入MyBatis以及pagehelper分页插件
    dependencies {
        implementation 'org.mybatis.spring.boot:mybatis-spring-boot-starter:1.1.1'
        implementation group: 'com.github.pagehelper', name: 'pagehelper-spring-boot-starter', version: '1.2.10'
    }
    
  2. application.properties 添加相关配置
    #mybatis设置
    mybatis.type-aliases-package=com.duomai.balance_card.Model.Mapper // 项目中Mapper存放的包
    mybatis.configuration.map-underscore-to-camel-case=true // 自动将sql字段下划线转为驼峰,可以保证取出的数据格式就是数据库中存储的格式
    logging.level.com.duomai.balance_card.Model.Mapper=DEBUG // 开启DEBUG模式,用于开发环境,记录执行的SQL。
    #pagehelper插件设置
    pagehelper.helperDialect=mysql // 指定分页插件使用哪种数据库
    pagehelper.reasonable=false // 分页合理化参数,默认值为false。当该参数设置为 true 时,pageNum<=0 时会查询第一页, pageNum>pages(超过总数时),会查询最后一页。默认false 时,直接根据参数进行查询。
    pagehelper.supportMethodsArguments=true // 支持通过 Mapper 接口参数来传递分页参数,默认值false,分页插件会从查询方法的参数值中,自动根据上面 params 配置的字段中取值,查找到合适的值时就会自动分页,设置后无需手动启用分页插件。
    pagehelper.params=pageNum=page;pageSize=limit //  自定义Mapper 接口参数来传递分页参数的参数名
    
  3. 添加Mapper类扫描的两种方式
    • 在启动类中添加对mapper包扫描@MapperScan(推荐使用)
      @SpringBootApplication
      @MapperScan("com.duomai.balance_card.Model.Mapper")
      
    • 在具体的Mapper类上面添加注解 @Mapper

  4. Mapper类SQL用例

    public interface StoreBalanceCardsMapper {
        // select用例
        @SelectProvider(type = StoreBalanceCardsSqlBuilder.class, method = "getById")
        List<Map> getById(@Param("id") int id, @Param("fields") String fields, @Param("page") int page, @Param("limit") int limit);
    
        // 列表
        @SelectProvider(type = StoreBalanceCardsSqlBuilder.class, method = "list")
        Page<Map> list(@Param("fields") String fields, @Param("page") int page, @Param("limit") int limit);
    
        // insert用例
        @InsertProvider(type = StoreBalanceCardsSqlBuilder.class, method = "add")
        @Options(useGeneratedKeys=true, keyProperty="storeBalanceCard.id", keyColumn="id")
        int add(@Param("storeBalanceCard") StoreBalanceCards storeBalanceCard);
    
        // update用例
        @UpdateProvider(type = StoreBalanceCardsSqlBuilder.class, method = "updateName")
        @Options(useGeneratedKeys=true, keyProperty="storeBalanceCard.id", keyColumn="id")
        int updateName(@Param("storeBalanceCard") StoreBalanceCards storeBalanceCard, @Param("limit") int limit);
    
        // delete用例
        @DeleteProvider(type = StoreBalanceCardsSqlBuilder.class, method = "deleteById")
        int delete(@Param("id") int id, @Param("limit") int limit);
    }
    
    • 通过Provider的方式动态获取SQL

    • 需要分页时在具体的方法后多加page、limit参数,自动实现分页

      // Service层调用
      List storeBalanceCard = storeBalanceCardsMapper.list(fields, page, limit);
      // Mapper层实现
      Page<Map> list(@Param("fields") String fields, @Param("page") int page, @Param("limit") int limit);
      
    • 分页组件仅可用于查询,不可用于更新/删除,更新/删除需要另外实现

  5. Providerle类SQL用例

    public class StoreBalanceCardsSqlBuilder {
        private static final String STORE_BALANCE_CARDS = "store_balance_cards";
        public static String getById(int id, String fields) {
            return new SQL(){ { 
                SELECT(fields);
                FROM(STORE_BALANCE_CARDS);
                WHERE("store_balance_cards.id = #{id}");
            } }.toString();
        }
    
        public static String list(String fields) {
            return new SQL(){ {
                SELECT(fields);
                FROM(STORE_BALANCE_CARDS);
            } }.toString();
        }
    
        public static String add(@Param("storeBalanceCard") StoreBalanceCards storeBalanceCard) {
            return new SQL(){ {
                INSERT_INTO(STORE_BALANCE_CARDS);
                VALUES("sys_name", "#{storeBalanceCard.sysName}");
                VALUES("store_id", "#{storeBalanceCard.storeId}");
                VALUES("entity_store_id", "#{storeBalanceCard.entityStoreId}");
                VALUES("name", "#{storeBalanceCard.name}");
                VALUES("type", "#{storeBalanceCard.type}");
                VALUES("item_id", "#{storeBalanceCard.itemId}");
                VALUES("photo", "#{storeBalanceCard.photo}");
                VALUES("state", "#{storeBalanceCard.state}");
                VALUES("upgrade", "#{storeBalanceCard.upgrade}");
                VALUES("recharge_discount", "#{storeBalanceCard.rechargeDiscount}");
                VALUES("updated_at", "#{storeBalanceCard.updatedAt}");
                VALUES("created_at", "#{storeBalanceCard.createdAt}");
            } }.toString();
        }
    
        public static String updateName(@Param("storeBalanceCard") StoreBalanceCards storeBalanceCard, @Param("limit") int limit) {
            return new SQL(){ {
                UPDATE(STORE_BALANCE_CARDS);
                SET("name=#{storeBalanceCard.name}");
                if (storeBalanceCard.getUpdatedAt() != null) {
                    SET("updated_at=#{storeBalanceCard.updatedAt}");
                }
                WHERE("store_id = #{storeBalanceCard.storeId}");
            } }.toString() + " limit " + limit;
        }
    
        public static String deleteById(int id, int limit) {
            return new SQL(){ {
                DELETE_FROM(STORE_BALANCE_CARDS);
                WHERE("store_balance_cards.id = #{id}");
            } }.toString() + " limit " + limit;
        }
    }
    
  6. 分页组件详解

十、接口输出

  1. 接口统一输出格式

    state: // 状态位
    msg: // 接口输出提示信息
    data: { // 总的接口输出数据,可以为空
        data: // 接口数据
        other: // 其他返回数据,如total/page_total等
        ...:
    }
    
  2. 正常接口输出/异常监听输出统一使用OutPut/ApiReturn方法

十一、异常处理

  1. 项目运行异常统一监听
    现有:ControllrHandler 路由异常监听、SqlHandler 数据库操作异常监听
    需要持续添加
  2. 统一使用接口输出类进行返回,杜绝直接返回报错信息。

十二、单元测试

  1. 添加单元测试类

    src\test\java\com\duomai\balance_card\BalanceCardApplicationTests.java

  2. demo

    @RunWith(SpringRunner.class)
    @SpringBootTest
    @AutoConfigureMockMvc
    public class BalanceCardApplicationTests {
    
        @Autowired
        private MockMvc mockMvc;
    
        @Test
        @SuppressWarnings("unchecked")
        public void getCard() throws Exception {
            String card_id = "1";
            String fields = "store_id,item_id,recharge_discount,photo,type,name";
            String res = this.mockMvc.perform(get("/card/get")
                    .param("id", card_id).param("fields", fields)
                    )
                    .andDo(print())
                    .andExpect(status().isOk())
                    .andReturn()
                    .getResponse()
                    .getContentAsString();
           // 接口返回不为空
           assertThat(res).isNotNull();
           // 校验接口返回格式是否完整
           Map<String, Object> api_res = Common.jsonToMap(res);
           assertThat(api_res).isNotNull();
           assertThat(api_res).hasSize(3);
           assertThat(api_res).containsKeys("state", "msg", "data");
           // 校验接口返回state是否正确
           int state = (int) api_res.get("state");
           Map<String, Object> api_data = (HashMap<String, Object>) api_res.get("data");
           assertThat(state).isEqualTo(ApiReturnDefines.SUCCESS);
           // 校验data数据是否正确
           assertThat(api_data).containsOnlyKeys("data");
           ArrayList<Map> data = (ArrayList) api_data.get("data");
           String[] fieldsAll = fields.split(",");
           assertThat(data).hasSize(1);
           assertThat(data.get(0)).containsKeys((Object[]) fieldsAll);
        }
    }
    
  3. 详解
    • 使用 @RunWith(SpringRunner.class)@SpringBootTest 定义测试类
    • 添加注解 @AutoConfigureMockMvc ,使用 Spring MockMvc 模拟Spring的HTTP请求并将其交给控制器,实际上并没有真正地启动服务器,仅仅是Mock,节省了启动服务器的开销。
    • 使用 AssertJ 库类来验证接口返回内容
      在demo中,使用断言验证了接口返回格式、state状态、data数据格式等基本内容。

    参考文档:https://spring.io/guides/gs/testing-web/

十三、项目打包

  1. 不同的构建文件
  • 普通 jar 包 : 会将源码编译后以工具包(即将class打成jar包)的形式对外提供,此时,你的 jar 包不一定要是可执行的,只要能通过编译,可以被别的项目以 import 的方式调用。
  • 可执行 jar 包 : 能通过 java -jar 的命令运行。
  • 普通 war 包 : war 是一个 web 模块,其中包括 WEB-INF,是可以直接运行的 WEB 模块。做好一个 web 应用后,打成包部署到容器中。
    • 可执行 war 包 : 普通 war 包 + 内嵌容器 。
  1. 构建可执行 jar 包

    • IDE打包
      右侧Gradle -> Tasks -> build -> build
    • 命令行打包
      进入项目根目录,执行gradle build
  2. 运行可执行 jar 包
    java -jar build/libs/balance_card-0.0.1.jar
    运行时可带参数,同application.properties中的参数名
    例:

    java -Djava.security.egd=file:/dev/./urandom -jar /opt/ci123/www/html/java/balance_card/build/libs/${jarName}.jar --server.port=80
    

十四、容器化部署

  1. 创建docker镜像
  • Dockfile 编写

    FROM java:8
    MAINTAINER duomai
    # 设置镜像源
    COPY sources.list /etc/apt/sources.list
    # 安装扩展
    RUN apt-get update && apt-get install -y \
    wget \
    curl \
    vim \
    git \
    less
    # 配置系统时间
    RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
    && echo 'Asia/Shanghai' > /etc/timezone
    VOLUME /tmp
    # 设置工作目录
    ADD . /opt/ci123/www/html/java
    WORKDIR /opt/ci123/www/html/java
    # 指定输出端口
    EXPOSE 80

    • 快速构建镜像

      sh docker/build.sh [version]

  1. 环境变量配置
  • 配置

    #数据库连接配置 application.properties
    spring.datasource.url=jdbc:mysql://${APP_DB_HOST:192.168.0.235}:${APP_DB_PORT:3355}/${APP_DB_DATABASE:shop_balances}
    spring.datasource.username=${APP_DB_USER:cishop}
    spring.datasource.password=${APP_DB_PASSWORD:fuyuan1906}

    1. 启动容器( docker/run.sh )
  • 启动参数

    -p // 宿主机运行端口
    -d // 宿主机项目地址,缺省为执行run.sh的上一级目录
    -n // 容器名称
    -v // 镜像版本号
    -j // jar包版本,缺省为1.0

  • 环境变量

    # 运行配置
    APP_SERVER_PORT // 容器运行jar包的端口
    # 数据库配置
    APP_DB_HOST
    APP_DB_PORT
    APP_DB_DATABASE
    APP_DB_USER
    APP_DB_PASSWORD
    # 日志配置
    APP_LOG_PATH // 日志输出地址
    APP_LOG_SPRING_WEB_LEVEL // 对应logging.level.org.springframework.web,指org.springframework.web这个包下的日志输出等级,默认为ERROR,开发环境可配置为DEBUG
    APP_LOG_MYBATIS_LEVEL // 对应logging.level.com.duomai.balance_card.Model.Mapper,指balance_card.Model.Mapper包下的日志输出等级,默认为ERROR,开发环境可配置为DEBUG,开启DEBUG之后可在日志中查看DB操作

  • 启动容器

    docker run -d \
    -e APP_SERVER_PORT=8080 \
    -e APP_DB_HOST=192.168.0.235 \
    -e APP_DB_PORT=3355 \
    -e APP_DB_DATABASE=shop_balances \
    -e APP_DB_USER=cishop \
    -e APP_DB_PASSWORD=fuyuan1906 \
    -e APP_LOG_PATH=/opt/ci123/www/html/java/balance_card/log \
    -e APP_LOG_SPRING_WEB_LEVEL=DEBUG \
    -e APP_LOG_MYBATIS_LEVEL=DEBUG \
    -p $port:80 \
    -v $dir:/opt/ci123/www/html/java/balance_card \
    --restart=always \
    --name $name \
    harbor.oneitfarm.com/duomai/java-balance_card:$version \
    sh /opt/ci123/www/html/java/balance_card/docker/start.sh -j $jarName

通过mq消费者ip查找对应启动容器

  1. 首先先确定是否相应的消费者
  2. 3.14上有多个容器,如何确定是哪个容器运行的呢?
    • 由于在linux的docker中,容器里运行的进程会直接通过宿主机执行,所以可以通过查看是否有对应进程
      bash
      ps aux | grep cront/trades/Cashier_trade/payed

    • 获取对应进程的父级pid(也就是运行该脚本的pm2进程),比如这里选择第一个进程的pid为2163,获取到上级pid为29217
      bash
      # ps -ef|awk '$2 ~ /pid/{print $3}'
      ps -ef|awk '$2 ~ /2163/{print $3}'

    • 再获取pm2的上级(即运行容器的pid),获取到为24450
      bash
      ps -ef|awk '$2 ~ /29217/{print $3}'
    • 查看容器pid相对应的进程
      bash
      ps aux | grep 24450

    • 有一串容器编号,复制前一小段,查询对应容器
      bash
      docker ps | grep 0d7eccf5ab

Betpipo Mobil Uygulama İnceleme

Günümüzde mobil cihazlar üzerinden bahis oynama alışkanlığı hızla yaygınlaşmakta ve kullanıcıların büyük çoğunluğu artık masaüstü bilgisayar yerine akıllı telefonlarını tercih etmektedir. İstedikleri her an ve her yerden bahis yapabilme özgürlüğüne kavuşan kullanıcılar mobil deneyimden maksimum verim almak istemektedir. Mobil uygulama ile masaüstü deneyiminin tüm özelliklerine cep telefonunuzdan sorunsuz bir şekilde erişmek mümkün hale gelmiştir. Bu yazımızda mobil uygulamanın özelliklerini kurulum sürecini performans değerlendirmesini ve sunduğu avantajları detaylı olarak ele alacağız. Mobil bahis deneyiminizi en üst seviyeye taşımak için bu rehberi dikkatlice incelemenizi öneriyoruz.

Uygulama Kurulumu ve Sistem Gereksinimleri
Mobil uygulama hem Android hem de iOS işletim sistemlerinde sorunsuz ve stabil bir şekilde çalışmaktadır. Android kullanıcıları uygulamayı platformun resmi web sitesinden APK dosyası olarak indirebilmekte ve kurulum birkaç basit adımda tamamlanmaktadır. Kurulum öncesinde telefon ayarlarından bilinmeyen kaynaklardan uygulama yükleme izninin verilmesi gerekmektedir. iOS kullanıcıları ise doğrudan web sitesinden yönlendirme ile kurulum yapabilmektedir. Uygulama boyutu oldukça küçük olup cihaz hafızasında minimum yer kaplamaktadır. Minimum Android beş sürümü veya iOS on iki sürümü gereksinimi bulunmakta olup güncel cihazların tamamıyla uyumludur.

Arayüz ve Kullanım Kolaylığı
Betpipo mobil uygulaması son derece kullanıcı dostu ve modern bir arayüze sahiptir. Menü yapısı sade ve anlaşılır bir şekilde tasarlanmış olup aradığınız bölüme saniyeler içinde ulaşabilirsiniz. Tek elle kullanıma uygun buton yerleşimi ile hızlı ve rahat erişim sağlanmaktadır. Karanlık mod seçeneği düşük ışık ortamlarında göz yorgunluğunu önemli ölçüde azaltmaktadır. Favori maçlarınızı ve oyunlarınızı hızlı erişim listesine ekleyerek kişiselleştirilmiş bir deneyim yaşayabilirsiniz. Gelişmiş arama fonksiyonu ile istediğiniz müsabakaya veya oyuna anında ulaşabilirsiniz.

Mobil Uygulamanın Avantajları
Mobil uygulama kullanmanın tarayıcı üzerinden erişime göre birçok önemli avantajı bulunmaktadır. Push bildirimler sayesinde maç başlangıçlarından favori takımınızın müsabakalarından ve özel kampanyalardan anında haberdar olabilirsiniz. Adres değişikliklerinde uygulama otomatik güncelleme yaparak kesintisiz erişim sağlamakta ve kullanıcıların yeni adres aramasına gerek kalmamaktadır. Parmak izi ve yüz tanıma ile biyometrik hızlı giriş özelliği hem güvenliği artırırken hem de kullanım kolaylığı sunmaktadır. Veri tasarrufu modu ile mobil veri kullanımınızı minimize edebilir ve internet paketinizden tasarruf sağlayabilirsiniz.

Performans ve Hız Değerlendirmesi
Uygulama yüksek performans standartlarına göre geliştirilmiş olup akıcı bir kullanım deneyimi sunmaktadır. Sayfa geçişleri anlık olarak gerçekleşmekte ve bahis kuponu oluşturma işlemi saniyeler içinde tamamlanmaktadır. Canlı bahis bölümünde oran güncellemeleri minimum gecikme ile yansımakta ve bu durum kullanıcılara rekabet avantajı sağlamaktadır. Düşük internet hızlarında ve mobil veri bağlantısında bile stabil bir kullanım deneyimi sunulmaktadır. Uygulama düzenli olarak güncellenmekte performans iyileştirmeleri yapılmakta ve yeni özellikler eklenmektedir. Optimize edilmiş bellek yönetimi sayesinde arka planda minimum kaynak tüketimi sağlanmakta ve telefonunuzun performansını olumsuz etkilememektedir.

Mobil Özel Bonuslar
Mobil uygulama kullanıcılarına özel olarak tasarlanmış bonus kampanyaları düzenli aralıklarla düzenlenmektedir. İlk kez uygulama üzerinden bahis yapan kullanıcılara özel hoş geldin bonusu sunulmakta ve bu bonus web sitesi bonusundan farklı koşullara sahip olabilmektedir. Haftalık mobil özel freespin kampanyaları casino tutkunlarını memnun etmekte ve ücretsiz döndürme hakkı sağlamaktadır. Uygulama üzerinden yapılan yatırımlarda ekstra bonus oranları uygulanabilmekte ve bu durum mobil kullanımı daha cazip hale getirmektedir. Bu özel kampanyalar mobil kullanımı teşvik etmekte ve kullanıcılara ekstra değer katmaktadır. Bildirim ayarlarını açarak yeni kampanyalardan anında haberdar olabilir ve fırsatları kaçırmamış olursunuz.

【初阶】基于redis的抢购秒杀设计

秒杀系统架构思路

应用场景

限时抢购,10w人抢购100个商品。这种情况下如果直接走到数据存储层,瞬间会把数据库打挂掉,如果业务处理不好(流程过长),会出现超卖现象

优化方向

尽量将请求拦截在上游

充分利用缓存
1. 前端方案
- 页面静态
- 防重复提交
- 用户限流
2. 后端方案
- 网关限制
- 缓存
- 消息队列削峰

优化细节

这里我们由简单往细节深入(因为要考虑实际的用户量,先做最简单最有效的处理)

简单高效处理方式

  1. 前端防重复提交
    这是最基本的
  2. redis缓存校验
    如果没有缓存,由于业务校验时间过长,比如在200ms内10w个人同时读了数据库库存仅剩1,都是可购买的,然后同时走到下单流程就会超卖,另外就是如果并发量过高,数据库会挂掉。
    如果是缓存校验,则只有一个可下单。其余全部直接拦截。

redis数据类型选型
如果不考虑抢购的下单数量和购买限制大于1,那直接kv或者list都是可以的。否则可以考虑hash或者zset。

这里对比一下kv和list:
kv的一般处理方式是预加或者预减:累计购买数量超过总库存或者剩余库存小于0,则校验不通过,同时回退。
list:预存库存长度的list,不断pop,无法pop则校验不通过。

tip:注意原子性
判断最好走lua。
比如kv,如果库存1,10人同时下单,如果是程序判断,则全部不能下单。而如果用lua,用户判断和回退是原子性的,则有一个人可以下单成功。
如果使用list,当抢购数量大于1时,回退也需要用事务,不然会出现,比如库存3,A下单5,B下单2。B在回退push的过程中,又被Apop了1从而判断库存不足。导致两人都不能下单。

复杂完善处理方式

  1. redis集群,主从同步,读写分离(读多写少)。
  2. nginx负载均衡
  3. 前端资源静态化:只有少部分内容是动态的的
  4. 按钮控制
  5. 缓存预热
  6. mq削峰:队列下单

详细内容可以参考大佬们的文章,我只是一个搬运工

架构设计图:
秒杀架构设计

参考文档:
https://www.zhihu.com/question/54895548
https://yq.aliyun.com/articles/69704?utm_campaign=wenzhang&utm_medium=article&utm_source=QQ-qun&utm_content=m_10737

使用HAProxy实现RabbitMq集群负载均衡

一、目的

​ 在使用 RabbitMq集群时可能会遇到集群中某个节点出现异常或者连接数过多的情况,这个时候与该节点连接的Consumer将会断开,Publisher也会无法将消息发送至集群。为了解决这些问题,本文中将使用 HAProxy来代理集群,实现多个节点的负载均衡以及在某个节点异常时自动将连接切换至其他正常节点等功能。

二、HAProxy安装配置(Centos 7)

  1. 安装 HAProxy
    1. 下载最新稳定版2.0.8并解压
      // PWD:/opt/ci123/www/html/rabbitMq/
      wget https://www.haproxy.org/download/2.0/src/haproxy-2.0.8.tar.gz
      tar xf haproxy-2.0.8.tar.gz
      
    2. 查看系统内核版本来指定编译版本
      uname -r
      3.10.0-862.6.3.el7.x86_64
      

      版本参考:

      ```

    3. </ol>

      <ul>
      <li>linux22 for Linux 2.2</li>
      <li>linux24 for Linux 2.4 and above (default)</li>
      <li>linux24e for Linux 2.4 with support for a working epoll (> 0.21)</li>
      <li>linux26 for Linux 2.6 and above</li>
      <li>linux2628 for Linux 2.6.28, 3.x, and above (enables splice and tproxy)</li>
      <li>solaris for Solaris 8 or 10 (others untested)</li>
      <li>freebsd for FreeBSD 5 to 10 (others untested)</li>
      <li>netbsd for NetBSD</li>
      <li>osx for Mac OS/X</li>
      <li>openbsd for OpenBSD 5.7 and above</li>
      <li>aix51 for AIX 5.1</li>
      <li>aix52 for AIX 5.2</li>
      <li>cygwin for Cygwin</li>
      <li>haiku for Haiku</li>
      <li>generic for any other OS or version.</li>
      <li><p>custom to manually adjust every setting
      ```

      根据版本参考,这里我们选择linux2628版本进行编译。

      1. 编译到指定目录

        cd haproxy-2.0.8
        make TARGET=linux2628 PREFIX=/opt/ci123/haproxy
        
        // 这里出现报错,从2.0版本开始linux2628已被废弃
        Target 'linux2628' was removed from HAProxy 2.0 due to being irrelevant and
        often wrong. Please use 'linux-glibc' instead or define your custom target
        by checking available options using 'make help TARGET=<your-target>'.
        
        // 根据提示修改参数后编译
        make TARGET=linux-glibc PREFIX=/opt/ci123/haproxy
        make install PREFIX=/opt/ci123/haproxy
        
    4. 配置
      1. 复制 haproxy命令至全局变量
        cp /opt/ci123/haproxy/sbin/haproxy /usr/bin/
        
      2. 创建系统用户
        useradd -r haproxy
        
      3. 添加haproxy配置文件
        1. haproxy配置文件由五部分组成:
        • global: 参数是进程级的,通常和操作系统相关。这些参数一般只设置一次,如果配置无误,就不需要再次配置进行修改。
      • default:默认参数。
        • frontend:用于接收客户端请求的前端节点,可以设置相应的转发规则来指定使用哪个backend
        • backend:后端服务器代理配置,可实现代理多台服务器实现负载均衡、为请求添加额外报文数据等功能。
        • listen:是frontendbackend的结合,通常只对tcp流量有用。
      1. 添加配置文件/opt/ci123/haproxy/conf/haproxy.cfg

        ```
        # 全局配置
        global
        log 127.0.0.1 local3 # 设置日志
        pidfile /opt/ci123/haproxy/logs/haproxy.pid
        maxconn 4000 # 最大连接数
        user haproxy
        group haproxy
        daemon # 守护进程运行

        # 默认配置
        defaults
        log global
        mode tcp # 默认的模式mode { tcp|http|health },tcp是4层,http是7层,health只会返回OK
        option httplog # http 日志格式,仅在http模式下可用
        option dontlognull # 不记录健康检查日志信息;
        option redispatch # serverId对应的服务器挂掉后,强制定向到其他健康的服务器
        option http-server-close
        #option abortonclose # 当服务器负载很高的时候,自动结束掉当前队列处理比较久的链接;
        #option forwardfor # 如果后端服务器需要获得客户端真实ip需要配置的参数,可以从Http Header中获得客户端ip;
        #option httpclose # 主动关闭http通道,每次请求完毕后主动关闭http通道,ha-proxy不支持keep-alive,只能模拟这种模式的实现;<br />
        balance roundrobin # 负载均衡算法,轮询;
        retries 3 # 重试次数;

        <pre class="prism-highlight line-numbers" data-start="1"><code class="language-null"> timeout http-request 10s # 客户端建立连接但不请求数据时,关闭客户端连接;
        timeout queue 1m # 高负载响应haproxy时,会把haproxy发送来的请求放进一个队列中,timeout queue定义放入这个队列的超时时间;
        timeout connect 10s # 定义haproxy将客户端请求转发至后端服务器所等待的超时时间;
        timeout client 1m # 客户端非活动状态的超长时间(默认毫秒)
        timeout server 1m # 服务端与客户端非活动状态连接的超时时间。(默认毫秒)
        timeout http-keep-alive 10s # 定义保持连接的超时时长;
        timeout check 10s # 心跳检测超时;
        maxconn 3000 # 每个server最大的连接数;
        </code></pre>

        #前端配置
        frontend rabbitmq_cluster_front
        bind 0.0.0.0:10000 # http请求的端口,会被转发到设置的ip及端口
        default_backend rabbitmq_cluster_back

        # 后端配置
        backend rabbitmq_cluster_back
        #roundrobin 轮询方式
        balance roundrobin # 负载均衡的方式,轮询方式

        <pre class="prism-highlight line-numbers" data-start="1"><code class="language-null"> # 配置Rabbitmq连接负载均衡
        # 需要转发的ip及端口
        # inter 2000 健康检查时间间隔2秒
        # rise 3 检测多少次才认为是正常的
        # fall 3 失败多少次才认为是不可用的
        # weight 30 权重
        server clusterRabbit1 192.168.3.14:5672 check inter 2000 rise 3 fall 3 weight 30
        server clusterRabbit2 192.168.3.14:5673 check inter 2000 rise 3 fall 3 weight 30
        </code></pre>

        # 统计页面配置
        listen admin_stats<br />
        bind 0.0.0.0:10080 # 监听IP和端口,为了安全可以设置本机的局域网IP及端口;
        mode http
        option httplog # 采用http日志格式<br />
        stats refresh 30s # 统计页面自动刷新时间<br />
        stats uri /haproxy # 状态管理页面,通过/haproxy来访问
        stats realm Haproxy Manager # 统计页面密码框上提示文本<br />
        stats auth duomai:shijiemori@2012 # 统计页面用户名和密码设置<br />
        #stats hide-version # 隐藏统计页面上HAProxy的版本信息
        ```

        1. 配置日志 rsyslog
        vim /etc/rsyslog.conf
        # 取消如下2行注释
        $ModLoad imudp
        $UDPServerRun 51
        
        # 新增配置(自定义的日志设备)
        local3.*  /opt/ci123/haproxy/logs/haproxy.log
        
        # 重启rsyslog服务
        systemctl restart rsyslog
        
        1. 启动 haproxy
        haproxy -f /opt/ci123/haproxy/conf/haproxy.cfg
        

        访问统计页面出现如下界面:

        haproxy管理

    三、集群测试

    沿用上一篇【RabbitMq 镜像队列集群搭建】中的集群测试环境,在测试中将PubliserConsumer的连接替换为HAProxy的地址192.168.3.14:10000

    1. 测试环境
      1. 节点:
        1. 节点一:
        • clusterRabbit1
        • 端口:192.168.3.14:5672
          1. 节点二:
        • clusterRabbit2
        • 端口:192.168.3.14:5673
      2. 队列:
        1. 节点一 rabbit@clusterRabbit1
          1. 队列一:
          • nameclusterRabbit1Queue1
          • routing_keyclusterRabbit1key
          1. 队列二:
          • nameclusterRabbit1Queue2
          • routing_keyclusterRabbitCommonKey
        2. 节点二 rabbit@clusterRabbit2
          1. 队列三:
          • nameclusterRabbit2Queue1
          • routing_keyclusterRabbit2key
          1. 队列四:
          • nameclusterRabbit2Queue2
          • routing_keyclusterRabbitCommonKey(与队列二 相同)
    2. 启动消费者
      1. 消费者
        1. 消费者一:
        • 连接节点:节点一 rabbit@clusterRabbit1
        • 消费队列:队列一 clusterRabbit1Queue1
          1. 消费者二:
        • 连接节点:节点一 rabbit@clusterRabbit1
        • 消费队列:队列二 clusterRabbit1Queue2
          1. 消费者三:
        • 连接节点:节点二 rabbit@clusterRabbit2
        • 消费队列:队列三 clusterRabbit2Queue1
          1. 消费者四:
        • 连接节点:节点二 rabbit@clusterRabbit2
        • 消费队列:队列四 clusterRabbit2Queue2
      2. 启动结果:

        启动成功,但是在一分钟后客户端异常退出,原因是HAProxy设置了timeout client 1m 和 timeout server 1m,消费者在一分钟内都没有接收到消息导致被判定为不活跃连接从而被删除。

        由于HAProxy默认不支持长连接,上述问题可以使用pm2管理消费者的方法来解决,消费者进程在不活跃退出后pm2将自动重启此进程。

    3. 发布消息

      1. 发布clusterRabbit1key消息
      • 消费者一 成功收到消息。
        1. 发布clusterRabbitCommonKey消息
      • 消费者二/四 成功收到消息。
    4. 关闭节点二rabbit@clusterRabbit2后再次发布消息

    • 连接节点二的消费者先退出后重新使用HAProxy成功连接。
    • 发布的消息均能被成功消费。

    四、使用说明

    1. HAProxy默认不支持tcp 长连接,需要使用PM2之类的守护进程管理工具或者长连接技术来实现rabbitMq客户端持续连接。
    2. HAProxy通过活跃检测机制来判定负载均衡中的节点是否可用, 当rabbitMq集群中某个节点不可用时,在经过一段时间的活跃检测之后,HAProxy将弃用该节点直至节点恢复。在这种情况下,rabbitMq消费者将断开连接后选择剩余可用的节点再次启动,客户端发布时也会自动选择剩余可用的节点。
    3. rabbitMq集群中某个节点宕机之后,HAProxy会自动使用可用的集群节点,所以不会出现在HAProxy活跃检测期间发布消息出现一半成功一半失败的情况,所有的消息都将通过可用的集群节点发布至集群。