ProductPromotion
Logo

Elixir

made by https://0x3d.site

GitHub - balance-platform/pillar: Elixir library client for work with ClickHouse
Elixir library client for work with ClickHouse. Contribute to balance-platform/pillar development by creating an account on GitHub.
Visit Site

GitHub - balance-platform/pillar: Elixir library client for work with ClickHouse

GitHub - balance-platform/pillar: Elixir library client for work with ClickHouse

Pillar

github.com hex.pm hex.pm hex.pm hex.pm github.com

Elixir client for ClickHouse, a fast open-source Online Analytical Processing (OLAP) database management system.

Features

Usage

Direct Usage with connection structure

conn = Pillar.Connection.new("http://user:password@localhost:8123/database")

# Params are passed in brackets {} in SQL query, and map strtucture does fill
# query by values.
sql = "SELECT count(*) FROM users WHERE lastname = {lastname}"

params = %{lastname: "Smith"}

{:ok, result} = Pillar.query(conn, sql, params)

result
#=> [%{"count(*)" => 347}]

Pool of workers

Recommended usage, because of limited connections and supervised workers.

defmodule ClickhouseMaster do
  use Pillar,
    connection_strings: [
      "http://user:password@host-master-1:8123/database",
      "http://user:password@host-master-2:8123/database"
    ],
    name: __MODULE__,
    pool_size: 15
end

ClickhouseMaster.start_link()

{:ok, result} = ClickhouseMaster.select(sql, %{param: value})

Async insert

connection = Pillar.Connection.new("http://user:password@host-master-1:8123/database")

Pillar.async_insert(connection, "INSERT INTO events (user_id, event) SELECT {user_id}, {event}", %{
  user_id: user.id,
  event: "password_changed"
}) # => :ok

Buffer for periodical bulk inserts

For this feature required Pool of workers.

defmodule BulkToLogs do
  use Pillar.BulkInsertBuffer,
    pool: ClickhouseMaster,
    table_name: "logs",
    # interval_between_inserts_in_seconds, by default -> 5
    interval_between_inserts_in_seconds: 5,
    # on_errors is optional
    on_errors: &__MODULE__.dump_to_file/2

  @doc """
  dump to file function store failed inserts into file 
  """
  def dump_to_file(_result, records) do
    File.write("bad_inserts/#{DateTime.utc_now()}", inspect(records))
  end

  @doc """
  retry insert is dangerous (but it is possible and listed as proof of concept)

  this function may be used in `on_errors` option
  """
  def retry_insert(_result, records) do
    __MODULE__.insert(records)
  end
end
:ok = BulkToLogs.insert(%{value: "online", count: 133, datetime: DateTime.utc_now()})
:ok = BulkToLogs.insert(%{value: "online", count: 134, datetime: DateTime.utc_now()})
:ok = BulkToLogs.insert(%{value: "online", count: 132, datetime: DateTime.utc_now()})
....

# All this records will be inserted with 5 second interval.

on_errors parameter allows you to catch any error of bulk insert (for example: one of batch is bad or clickhouse was not available )

Migrations

Migrations can be generated with mix task mix pillar.gen.migration migration_name.

Multi-statement migration Example for this UseCase

defmodule Pillar.Migrations.CreateMultipleTables do
  def up do
    # for MultiStatement migration result of this function should be List of Strings  
    (0..4) |> Enum.map(fn i ->
      "CREATE TABLE IF NOT EXISTS shard_#{i} (field FixedString(10)) ENGINE = Memory"
    end)
  end
end
mix pillar.gen.migration events_table

But for launching them we have to write own task, like this:

defmodule Mix.Tasks.MigrateClickhouse do
  use Mix.Task
  def run(_args) do
    connection_string = Application.get_env(:my_project, :clickhouse_url)
    conn = Pillar.Connection.new(connection_string)
    Pillar.Migrations.migrate(conn)
  end
end

And launch this via command.

mix migrate_clickhouse

Timezones

In order to be able to use Timezones add timezones database to your project and configure your app:

config :elixir, :time_zone_database, Tzdata.TimeZoneDatabase

Details here https://hexdocs.pm/elixir/1.12/DateTime.html#module-time-zone-database

HTTP Adapters

If you have problems with default Pillar HTTP Adapter (Mint over Tesla), you can use alternative one, based on :httpc or define your own and pass it through config.

config :pillar, Pillar.HttpClient, http_adapter: Pillar.HttpClient.TeslaMintAdapter

Adapter should define one function post/3 and return 2 possible results (%Pillar.HttpClient.Response{}, %Pillar.HttpClient.TransportError{})

Contribution

Feel free to make a pull request. All contributions are appreciated!

Articles
to learn more about the elixir concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about Elixir.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory