I am creating a couple of bigger database tables with at least hundreds of millions of observations, but growing. Some tables are by minute, some by milliseconds. timestamps are not necessarily unique.

Should I create separate year, month, or date and time columns? Is one unique datetime column enough? At what size would you partition the tables?

Raw data is in csv.

Currently I aim for postgres and duckdb. Does timescaledb make a significant difference?

  • plumbus@feddit.org
    link
    fedilink
    English
    arrow-up
    5
    ·
    4 days ago

    I thought InfluxDB is the choice for such use cases. But I’m not an expert…