ProductPromotion
Logo

Elixir

made by https://0x3d.site

GitHub - dalmatinerdb/mstore
Contribute to dalmatinerdb/mstore development by creating an account on GitHub.
Visit Site

GitHub - dalmatinerdb/mstore

GitHub - dalmatinerdb/mstore

mstore Build Status

MStore is a experimental metric store build in erlang, the primary functions are open, new, get and put.

A datastore is defined by:

  • The size of the consistant hashing ring.
  • The number of entreis per metrics.
  • The initial offset.

For each chunk a index is created (defining the position of the metrics) and a datafile which holds the values. This makes reading a number of metrics as simple as a calculation and a sequential read.

For a store holding 1000 metrics writing to the the numbers 0-999 would be in the file 0, 1000-1999 would be in the second file etc.

Idea

The basic idea is to take advantage of the special characteristics metrics have and modern filesystems. The following assumptions about metrics and filesystems are taken:

  • Metrics occour in a regular interval (i.e. every second) skips happen but are rare
  • Metrics are immutable. (i.e. once the cpu temperature was recorded for a measurement period it won't ever change again).
  • Reads are highly sequential, 'give me the values between X and Y'.
  • Metrics are written nearly sequentially, the delta of time between two metrics written will propably be small, this allows to limit the amount of open files.
  • Metrics can be represented as 64bit integers. (this might change!)
  • The filesystem uses checksums for data, this means we don't need to cehcksum values.
  • The filesystem allows compression. This means longer stratches of non written metrics don't have a big impact since a bunch fo 0's on the FS will easiely be compressed away.
  • The filesystem has a decent cacheing strategy (no need for mmap nonsense).
  • The filesystem actually is ZFS.

File Layout

Set

A set allows to group metrics into a hash ring, this limits the size of single files open. The directory layout will be like this:

<base dir>/<chash key>/<offset>.{mstore,idx} - data and store index files
<base dir>/mstore - set index file

Index File (for a set)

The index file is simply a Erlang file that can be read via consult:

{FileSize, CHashSize, Seed, Metrics}.
  • FileSize: The number of points per metric stored in the file.
  • CHashSize: The number of elements in the CHash ring.
  • Seed: A seed used to hash the metric keys, this is needed to allow putting a set behind another CHash ring (i.e. riak core). W/o the seed the distribution would not be even.
  • Metrics: A list of all metrics stored in this set, used for looking up metrics.

Store

Data File

Currently data is fixed to 64 bit (8 byte) integers this means a data file is layed out like this:

<metric 1:FileSize*8><metric 2:FileSize*8><metric 2:FileSize*8>

Index File (for a metric)

The index file is simply a Erlang file that can be read via consult:

{Offse, FileSize, [{Metric, Index}]}.
  • Offset: the base offset of the file.
  • FileSize: The number of points per metric stored in the file.
  • Metric and Index: A list of metricses and their indexes in the file.

Articles
to learn more about the elixir concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about Elixir.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory