# Unread count

* [Badge count](#badge-count)
  * [Question1: Whether to store badge and conversation count separately](#question1-whether-to-store-badge-and-conversation-count-separately)
  * [Question2: Keep consistency between badge and conversation count](#question2-keep-consistency-between-badge-and-conversation-count)
    * [Problems](#problems)
    * [Solution](#solution)
      * [Distributed lock](#distributed-lock)
      * [Transaction](#transaction)
      * [Lua script](#lua-script)
* [Count of unread messages in a thread](#count-of-unread-messages-in-a-thread)
  * [Naive solution with SQL](#naive-solution-with-sql)
    * [Improve write requests with hash based sharding](#improve-write-requests-with-hash-based-sharding)
    * [Improve read requests with Redis](#improve-read-requests-with-redis)
      * [Not enough memory](#not-enough-memory)
  * [Avoid notification storm in large group chat](#avoid-notification-storm-in-large-group-chat)
    * [Solution: Aggregate and update](#solution-aggregate-and-update)
* [Count of unread message in newsfeed](#count-of-unread-message-in-newsfeed)
  * [How is the scenario different?](#how-is-the-scenario-different)
  * [Idea](#idea)

## Badge count

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-8778413409a061a3f69d1040a4159b6aea67a5fe%2FIm_badge_count.png?alt=media)

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-496f536c1da2c716b4519acfaf2e822ea303cc9f%2FIm_badge_count_conversation.png?alt=media)

### Question1: Whether to store badge and conversation count separately

* In theory, it is possible to calculate badge count from conversation count on the fly.
* In practice, badge counter is used in a much higher frequency than these internal counters. If it is always calculated on the fly, then it will be a performance penalty.
* So badge count and conversation count are usually stored separately.

### Question2: Keep consistency between badge and conversation count

#### Problems

* Total unread message increment and unread message against a specific person are two atomic operations. One could fail while the other one succeed. Or other clearing operations are being executed between these two operations.

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-8bf39639667ccb5d2952ddb78e66bad92866ba4e%2Fim_badgeCount_inconsistency_scenario_1.png?alt=media)

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-09acd9b8059b3c8c9c479ee91ce17039edcb2547%2Fim_badgeCount_inconsistency_scenario_2.png?alt=media)

#### Solution

**Distributed lock**

* MC add, Redis setNX

**Transaction**

* Redis's MULTI, DISCARD, EXEC and WATCH operations. Optimistic lock.

**Lua script**

## Count of unread messages in a thread

### Naive solution with SQL

* All attributes inside a table and using (thread id + user id) as the primary key

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-631140f8c9b3ec250e9e8b83b312234b4d52f966%2Funread_message_id.png?alt=media)

```
select unread_count from threadToUser table
```

#### Improve write requests with hash based sharding

* Hash sharding based on (thread id + user id) over time range based sharding because chat data obviously has hot / cold data. Using time range based sharding could lead to hot shards

#### Improve read requests with Redis

* Improve with Redis

**Not enough memory**

* Redis is designed for general data structures and could take much memory:
  * Revise the native data structure for Redis
  * For example, Key stored as string: 8 bit LONG type will be stored as 8 bit (sdshdr length)+ 19 bit (8 byte Long represent as string）+ 1(’\0’)=28; In addition, remove pointers
* Redis + SSD: Popularity of tweets usually calms down over time. For old data, persist them into SSD disk.

### Avoid notification storm in large group chat

* Suppose that there is a 5000 people group and there are 10 persons speaking within the group per second, then QPS for updating unread messges will be 50K; When there are 1000 such groups, the QPS will be 50M

#### Solution: Aggregate and update

1. There will be multiple queues A/B/C/... for buffering all incoming requests.
2. Two components will be pulling from queues
   * Timer: Will be triggered after certain time
   * Flusher: Will be triggered if any of the queue exceed a certain length
3. Aggregator service will pull msgs from Timer and Flusher, aggregate the read increment and decrement operations

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-a7dfca5587fe0fd6e8b3a478dfe3a4d5e37f33cf%2Fim_badgeCount_aggregator.png?alt=media)

* Cons:
  * Since there is no persistent on queues, if there is a restart, the number of unread messages will be inaccurate

## Count of unread message in newsfeed

### How is the scenario different?

* Counts of tweets repost, comments... will be based on user activity (follow, edit, etc.), counts of newsfeed need to happen for everyone without any user intervention. The former is triggered by user activity, the later always happen automatically.
* Counts of system notifications. Unread message number in newsfeed will vary among every user.

### Idea

* Record how many posts each user has made. Also record a snapshot of all posts.
* When need to check for how many unread messages there are in newsfeed, calculate the different between snapshots.

![](https://1010073591-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mk8dv8Mfudl_6ziUzDf%2Fuploads%2Fgit-blob-4b334850dc65c6210e80d40cd3e52684fda87c5d%2Ftrends_unreadNum_newsfeed.png?alt=media)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://eric-zhang-seattle.gitbook.io/mess-around/scenarios/productoverview/unreadcount.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.