Now, if the working set of data and indexes is small, we can keep it in memory.īut if the data is sufficiently large that we can’t fit all (similarly fixed-size) pages of our B-tree in memory, then updating a random part of the tree can involve significant disk I/O as we read pages from disk into memory, modify in memory, and then write back out to disk (when evicted to make room for other B-tree pages). With an index, a query can quickly find a row with a specified ID (e.g., bank account number) without scanning the entire table or “walking” the table in some sorted order. Under most relational databases, a table is stored as a collection of fixed-size pages of data (e.g., 8KB pages in PostgreSQL), on top of which the system builds data structures (such as B-trees) to index the data. This is an old, common problem for relational databases. Eventually, our entire dataset will not fit in memory, which is why we’ll need to write our data and indexes to disk. While memory is faster than disk, it is much more expensive: about 20x costlier than solid-state storage like Flash, 100x more expensive than hard drives. Why databases do not normally scale up well: Swapping in/out of memory is expensiveĪ common problem with scaling database performance on a single machine is the significant cost/performance trade-off between memory and disk. Our motivations are twofold: for anyone facing similar problems, to share what we’ve learned and for those considering using TimescaleDB for time-series data (including the skeptics!), to explain some of our design decisions. How time-series data is unique, how one can leverage those differences to overcome the scaling problem, and some performance results.How LSM trees (typically used in NoSQL databases) do not adequately solve the needs of many time-series applications.
Why relational databases do not normally scale up well.(A scaling-out post will be published on a later date.) If each server is limited in its throughput or performance (i.e., unable to scale up), then the overall cluster throughput is greatly reduced. Why are both important? The most common approach to scaling out across a cluster of N servers is to partition, or shard, a dataset into N partitions. There are two separate ways to think about scaling: scaling up so that a single machine can store more data, and scaling out so that data can be stored across multiple machines.
But we also heard from skeptics, who found it hard to believe that one should (or could) build a scalable time-series database on a relational database (in our case, PostgreSQL). When we announced TimescaleDB two weeks ago, we received a lot of positive feedback from the community. One just needs to solve the scaling problem. We take a different, somewhat heretical stance: relational databases can be quite powerful for time-series data.
While relational databases have many useful features that most NoSQL databases do not (robust secondary index support complex predicates a rich query language JOINs, etc), they are difficult to scale.Īnd because time-series data piles up very quickly, many developers believe relational databases are ill-suited for it. Typically, the reason for adopting NoSQL time-series databases comes down to scale. NoSQL databases include: Elastic, InfluxDB, MongoDB, Cassandra, Couchbase, Graphite, Prometheus, ClickHouse, OpenTSDB, DalmatinerDB, KairosDB, RiakTS. Relational databases include: MySQL, MariaDB Server, PostgreSQL. Usage patterns are similar: a recent survey showed that developers preferred NoSQL to relational databases for time-series data by over 2:1. Most of these renounce the trappings of a traditional relational database and adopt what is generally known as a NoSQL model.
These days, time-series data applications (e.g., data center / server / microservice / container monitoring, sensor / IoT analytics, financial data analysis, etc.) are proliferating.Īs a result, time-series databases are in fashion ( here are 33 of them).
0 kommentar(er)
