Service governance

• Overview

  • Architecture

  • Capacity planning [Todo]

  • Org architecture [Todo]

  • Microservice layers

  • Tech architecture [Todo]

  • Problems

  • Data distribution

  • Data join

  • Breakdown monolithic

  • Service mesh [Todo]

• Real world

  • Netflix

Overview

• Rate limit and fallback

  • https://time.geekbang.org/course/detail/100003901-2278)

  • https://time.geekbang.org/column/article/40908

• Circuit breaker and timeout

  • https://time.geekbang.org/column/article/41297

Architecture

• RPC vs message queue based: https://time.geekbang.org/column/article/73368

Capacity planning [Todo]

• https://time.geekbang.org/column/article/44118

Org architecture [Todo]

• TODO: Kangwei principle

• TODO: when to introduce microservices

• TODO: when to breakdown monolithic

• TODO: devops, management platform

• TODO: how to divide

• TODO: servicemesh在复杂环境下的落地

Microservice layers

• TODO: Alibaba's big Middletier, small front end

• TODO: Microservice divides to different layers

• TODO: Microservice challenges

  • https://time.geekbang.org/column/article/13891

  • https://time.geekbang.org/column/article/14222

Tech architecture [Todo]

• Routing architecture

• Components

• Microservices management

  • https://time.geekbang.org/course/detail/100003901-2275

  • https://time.geekbang.org/column/article/18651

  • https://time.geekbang.org/column/article/41758

Problems

Data distribution

• Definition: Double write. How to guarantee the ACID of two writes

Use case

• Data replication

• Database migration

• Implement CQRS or remove join from database

• Implement distributed transactions

Approaches

Transactional outbox

• Flowchart

MySQL HA github

• Implementation: Killbill common queue

MySQL HA github

CDC (Change data capture)

• Flowchart

MySQL HA github

• Implementation:

  • Alibaba Canal (recommended)

  • Redhat Debezium

  • Zendesk Maxell

  • Airbnb SpinalTap

Data join

Use case

• N+1 problem

• Data volume

• Performance

Approaches

CQRS

• Flowchart

MySQL HA github

• Possible problems: Eventual consistency

MySQL HA github

• Possible solutions:

MySQL HA github

Breakdown monolithic

Definition

• Velocity slow

• Hard to continue scale vertically

• Independent deployment capability

Steps to migration

Key difficult points

• Understand the APIs

• Database migration

  • Incremental and rollbackable

  • Data

• Use data distribution and remove join

Example

• StichFix - Scaling your architecture with services and events

Service mesh [Todo]

• Overview: https://time.geekbang.org/column/article/65132

• Istio: https://time.geekbang.org/column/article/67172

• Weibo implements service mesh:

  • https://time.geekbang.org/column/article/67548

  • https://time.geekbang.org/column/article/67940

Real world

Netflix

• Load shedding at Netflix: https://netflixtechblog.com/keeping-netflix-reliable-using-prioritized-load-shedding-6cc827b02f94

• Auto scaling: https://netflixtechblog.com/auto-scaling-production-services-on-titus-1f3cd49f5cd7

• Netflix Load balancing: https://netflixtechblog.com/netflix-edge-load-balancing-695308b5548c

• Netflix Application gateway:

  • https://netflixtechblog.com/open-sourcing-zuul-2-82ea476cb2b3

  • https://netflixtechblog.com/zuul-2-the-netflix-journey-to-asynchronous-non-blocking-systems-45947377fb5c

• Netflix failover: https://netflixtechblog.com/project-nimble-region-evacuation-reimagined-d0d0568254d4

• Netflix DDOS simulation: https://netflixtechblog.com/starting-the-avalanche-640e69b14a06

• Netflix Chaos: https://netflixtechblog.com/chap-chaos-automation-platform-53e6d528371f

• Netflix cluster management: https://netflixtechblog.com/dynomite-manager-managing-dynomite-clusters-dfb6874228e4

• Netflix resource scheduling: https://netflixtechblog.com/distributed-resource-scheduling-with-apache-mesos-32bd9eb4ca38

• Netflix automated failure testing: https://netflixtechblog.com/automated-failure-testing-86c1b8bc841f

• Netflix Chaos monkey: https://netflixtechblog.com/netflix-chaos-monkey-upgraded-1d679429be5d

• Netflix open source projects: https://netflixtechblog.com/evolution-of-open-source-at-netflix-d05c1c788429

• 知名博主的分布式系列：https://www.cnblogs.com/wt645631686/category/1793274.html

• https://time.geekbang.org/course/detail/100003901-2269

• Netflix's microservice architecture: https://time.geekbang.org/course/detail/100003901-2272