AWS Timestream Multi-Measures

While developing a Data Analytics solution using Apache Flink/ELK, my team faced some heavy load problems and indexes failing to handle the load we're supposed to handle and, after some research, we decided to go for AWS Timestream.

At first AWS Timestream didn't offer multi-measures and it's single measure model would cripple our queries and would inquire in a high cost, as our current measures are strongly tied to each other and our Grafana visualisations would require a lot of inner-joins, outer-joins to make the single-measures rows show the data as we expected but, last November, the multi-measure feature came into beta and, as a "platinum" client, me and my team were able to start playing with it's shiny new features. Now I'm going to show some code:

Versioning and Upserting

Timestream uses records dimensions and time as primary key so, as such, it's not possible to update dimensions or time after they were written to the database. What CAN be update are the measure values.

To do so, just send the same dimensions and time with the new measure values with the version field higher than the version field stored. This will update the measure values.

Cardinality and query performance

It's important to notice that dimensions should have a finite cardinality and fields with infinite cardinality (usually dates and counters) should be stored as measures. The higher the cardinality of a dimension, more storage it'll consume and bigger will be its index, making querying harder and slower.

Avoid BIGINTS on measures values

When designing your data model, if your measures will be used to define percentiles like in my case, note that if you use INTS and you divide two INTs, the result will be an INT. Yeah, 3/2 will be 1. Unfortunately, if you're planning to use the metrics in FLOAT/Double numbers, it's better to store them as such. It's possible to cast them when querying but it proofed to have a low performance and the cost of storing DOUBLES instead of BIGINTS was negligible.

Special thanks and references

My team's efforts were only possible because of the strong support from AWS's engineering team, the direct contact with Timestream's Architect and its consistent help and fast improvements. To know more about Timestream, refer to the docs https://docs.aws.amazon.com/timestream/latest/developerguide/what-is-timestream.html and feel free to get in touch: https://stackoverflow.com/users/5390316/eduardo-elias-sal%c3%a9h