I'm excited to share CompactLog, a Certificate Transparency log implementation written in Rust that explores an alternative approach to CT scalability challenges.

Check it out here: GitHub - Barre/compact_log: RFC6962 certificate transparency log with LSM-tree based storage

The Scalability Challenge

Certificate Transparency logs face immense scale - millions of certificates daily, with logs growing to hundreds of millions of entries and multiple terabytes of data. Traditional implementations often struggle with:

• Database bottlenecks at multi-terabyte scale
• Expensive hardware requirements for adequate read performance
• Complex sharding and operational overhead
• Read operations that can overwhelm the system

New APIs vs Better Implementations

The CT community has been exploring different solutions to these challenges. One approach involves creating new APIs - moving from the traditional RFC 6962 dynamic endpoints to static, tile-based systems that can leverage CDNs and object storage more effectively. These new designs offer compelling benefits like efficient caching and horizontal scaling.

CompactLog explores a different question: Can we achieve similar scalability improvements while maintaining the existing RFC 6962 API?

The CompactLog Approach

Instead of changing the API, CompactLog rethinks the storage and implementation layers:

1. LSM-Tree Storage via SlateDB

Moving from relational databases to LSM-tree storage provides:

• Natural fit for append-only workloads
• Direct integration with object storage (S3, Azure Blob)
• No artificial database size limitations
• Cost-effective scaling with commodity storage

2. Zero Merge Delay Architecture

CompactLog eliminates merge delay entirely by incorporating certificates into the tree before issuing SCTs:

Submit → Add to tree → Issue SCT (all within 500ms)

This removes a common failure mode where logs fail to meet their merge delay commitments.

3. Smart Versioning Strategy

By versioning tree nodes only at Signed Tree Head boundaries rather than every update, storage overhead drops dramatically - potentially 1000x reduction for typical publishing intervals.

Benefits of Implementation-First Solutions

Focusing on implementation improvements offers several advantages:

• Ecosystem compatibility: Existing monitors, auditors, and tooling continue working unchanged
• Lower adoption friction: No need to update clients or specifications
• Proven interfaces: RFC 6962 has years of production experience
• Focused innovation: Solve the actual bottlenecks (storage) rather than redesigning everything

Try It Out

CompactLog supports both local storage for testing and cloud object storage for production deployments. The implementation is RFC 6962 compliant and ready for experimentation.

I'm curious to hear the community's thoughts on this architectural approach. Are there other systems where implementation improvements could obviate the need for protocol changes?

1 post - 1 participant

Read full topic