AWS Kinesis Data Streams — Bare Essentials for DVA-C02

1. Kinesis Data Streams Basics

Kinesis Data Streams is used for real-time streaming data.

Examples: - IoT sensor data - Clickstream data - Logs - Financial transactions

Data is stored in Shards.

Think of a shard as an independent lane that stores and serves records.

2. What is a Shard?

Every shard provides fixed capacity.

Each shard supports:

• Write: 1 MB/sec OR 1000 records/sec
• Read (shared consumers): 2 MB/sec

More shards = - More write capacity - More read capacity - More parallel processing

3. Partition Key

Every record must have a Partition Key.

Kinesis hashes the partition key to decide which shard stores the record.

Example:

Customer123
Customer123
Customer123

All go to the same shard.

4. Hot Shard

If one partition key receives much more traffic than others,

→ One shard becomes overloaded.

This causes

• ProvisionedThroughputExceededException
• Write throttling

Fix:

• Add more shards
• Use random prefixes/suffixes (Key Sharding)

Example:

Instead of

Customer123

Use

1-Customer123
2-Customer123
3-Customer123

Traffic spreads across multiple shards.

5. Resharding

When traffic increases,

you can increase shards.

Example

4 shards
↓

8 shards

Important for the exam:

Lambda automatically detects new shards.

No redeployment.

No configuration change.

No manual updates.

6. Ordering Guarantee

Kinesis guarantees ordering

ONLY inside a single shard.

Not across shards.

Example

Shard A

1
2
3

Always processed in order.

Shard B

A
B
C

Also ordered.

But

1
A
2
B

can happen because shards are processed independently.

7. Lambda + Kinesis

Lambda polls Kinesis automatically.

Important rules:

• One Lambda invocation per shard (by default)
• Records inside a shard remain ordered
• Different shards process in parallel

Example

3 shards

↓

3 concurrent Lambda executions

8. Parallelization Factor

Normally

1 Lambda per shard

You can increase the Parallelization Factor.

Example

5 shards

Parallelization Factor = 2

↓

Up to 10 concurrent Lambda executions

Useful when consumers fall behind.

9. Iterator Age

Iterator Age tells you

How far behind the consumer is.

Small value

✅ Consumer is keeping up.

Growing value

❌ Consumer is too slow.

Fixes:

• Add shards
• Increase Parallelization Factor
• Make Lambda faster

10. Enhanced Fan-Out (EFO)

Normally,

all consumers share the shard's read throughput.

With Enhanced Fan-Out,

each consumer gets

• Dedicated 2 MB/sec per shard
• HTTP/2 push delivery

Best when

• Multiple consumers
• Fast independent reading
• Consumers falling behind because of shared throughput

11. Server-Side Encryption

Kinesis supports encryption at rest.

Enable

Server-Side Encryption (SSE)

using

AWS KMS

You can use

• AWS-managed key
• Customer-managed KMS key (CMK)

Exam answer:

Enable SSE with a KMS Customer Managed Key.

12. ProvisionedThroughputExceededException

Means a shard exceeded its capacity.

Common fixes:

✅ Add more shards

✅ Use better partition keys

✅ Use random partition key prefixes

Do NOT think of:

• Encryption
• Firehose
• SQS

Those do not solve hot shards.

13. DynamoDB Hot Partition (Often Asked Together)

Exactly the same idea as a hot shard.

One partition key receives most writes.

Symptoms:

• Table capacity looks fine
• Only one key gets throttled

Fix:

Write Sharding

Example

Instead of

Device123

Use

Device123-1
Device123-2
Device123-3

Reads combine results.

14. DynamoDB Query vs Scan

For

Partition Key + Sort Key

Always use

Query

Example

CustomerId = 100

OrderDate BETWEEN A AND B

Never use Scan unless absolutely necessary.

15. Exam Memory Tricks

Lambda after Resharding

✅ Automatically detects new shards

Ordering

✅ Ordered within a shard

❌ Not ordered across shards

Iterator Age Growing

Consumer is slow.

Fix:

• Add shards
• Increase Parallelization Factor
• Faster Lambda

Enhanced Fan-Out

Dedicated

2 MB/sec per consumer per shard

Hot Shard

One partition key sends too much traffic.

Fix:

• Add shards
• Better partition keys
• Random prefixes

Encryption

Enable

Server-Side Encryption + KMS CMK

Query vs Scan

Need Partition Key + Date Range?

✅ Query

❌ Scan