ProductPromotion
Logo

Elixir

made by https://0x3d.site

GitHub - seantanly/elixir-paratize: Elixir library providing some handy parallel processing facilities that supports configuring number of workers and timeout.
Elixir library providing some handy parallel processing facilities that supports configuring number of workers and timeout. - seantanly/elixir-paratize
Visit Site

GitHub - seantanly/elixir-paratize: Elixir library providing some handy parallel processing facilities that supports configuring number of workers and timeout.

GitHub - seantanly/elixir-paratize: Elixir library providing some handy parallel processing facilities that supports configuring number of workers and timeout.

Paratize

Build Status Hex.pm Version

Elixir library providing some handy parallel processing facilities that supports configuring number of workers and timeout.

This library is inspired by Parex.

Documentation

API documentation is available at http://hexdocs.pm/paratize

Adding Paratize To Your Project

To use Paratize with your projects, edit your mix.exs file and add it as a dependency:

defp deps do
  [
    {:paratize, "~> x.x.x"},
  ]
end

Examples

Paratize is designed to run slow tasks in parallel. There are two processor implementatons, first the chunk based implementation Paratize.Chunk and the second the worker pool based implementation Paratize.Pool. Both modules have the same API.

  • parallel_exec(fun_list, task_options)
  • parallel_map(arg_list, fun, task_options)
  • parallel_each(arg_list, fun, task_options)

To execute a list of functions in parallel,

import Paratize.Pool

function_list = [
  fn -> Math.fib(40) end,
  fn -> :timer.sleep(5000) end,
  fn -> HTTPotion.get("http://wwww.reddit.com") end
]

parallel_exec(function_list) # => [102334155, :ok, %HTTPotion.Response{body...}]

function_keyword_list = [
  fib: fn -> Math.fib(40) end,
  hang: fn -> :timer.sleep(5000) end,
  web_request: fn -> HTTPotion.get("http://wwww.reddit.com") end
]

parallel_exec(function_keyword_list) # => [fib: 102334155, hang: :ok, web_request: %HTTPotion.Response{body...}]

To execute a map in parallel,
(useful when results are needed for further processing)

import Paratize.Pool

slow_func = fn arg -> :timer.sleep(1000); arg + 1 end
workload = 1..100

{time, result} = :timer.tc fn -> workload |> parallel_map(slow_func) |> Enum.join(", ") end
time # => 13034452 (8 CPU cores system, running 8 workers)

To execute a each in parallel,
(useful when resultset is large, and can be processed individually to prevent memory hog)

import Paratize.Pool

lots_of_urls |> parallel_each(fn url ->
  HTTPotion.get(url) |> parse_page |> save_meta_data
end)

Task Options

Each function accepts task options to customize the parallel processing.

  • size - the number of parallel workers, defaults to the number of system cores given by :erlang.system_info(:schedulers)
  • timeout - in milliseconds, the minimum time given for a function to complete, defaults to 5000. If timeout happens, the entire parallel processing crashes with exit(:timeout,...). To disable timeout, use :infinity.

Considerations

To achieve maximum parallelism, %Paratize.TaskOptions{} size should be set to size of your workload,

alias Paratize.Pool

slow_func = fn arg -> :timer.sleep(1000); arg + 1 end
workload = 1..100

{time, result} = :timer.tc fn ->
    workload |> Pool.parallel_map(slow_func, size: Enum.count(workload)) |> Enum.join(", ")
end
time # => 1004370 (Running 100 workers)

The %Paratize.TaskOptions{} timeout should not be relied upon for precise timing out of each workload, because it is not strictly enforced. It is an implementation detail that reasonably crashes the processor if no further work is completed after the timeout period has lapsed.

LICENSE

This software is licensed under MIT LICENSE.

Articles
to learn more about the elixir concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about Elixir.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory