ProductPromotion
Logo

Elixir

made by https://0x3d.site

GitHub - joekain/bmark: A benchmarking tool for Elixir
A benchmarking tool for Elixir. Contribute to joekain/bmark development by creating an account on GitHub.
Visit Site

GitHub - joekain/bmark: A benchmarking tool for Elixir

GitHub - joekain/bmark: A benchmarking tool for Elixir

Bmark

Build Status Inline docs Package

Bmark is a benchmarking tool for Elixir. It allows easy benchmarking of Elixir functions. It also supports comparing sets of benchmarking results.

Comparing benchmarking results is a topic that I have struggled with for years. I run a benchmark several times and get varying results. Then, I make a change to my program and I want to decide if the change causes an improvement in the benchmark score. I rerun the benchmark several times and again get varying results. How do I compare these results? I can compare average score, but is that accurate? How do I tell if the mean of the second run is large enough to be meaningful? How do I know if it is "in the noise?"

Bmark answers this questions using statistical hypothesis testing. Given two sets of benchmark runs, bmark can show:

RunA:                                 RunB:
24274268                              6426990
24563751                              6416149
24492221                              6507946
24516553                              6453309
24335224                              6491314
24158102                              6405073
24357174                              6504260
24213098                              6449789
24466586                              6532929
24289248                              6509800

24366622.5 -> 6469755.9 (-73.45%) with p < 0.0005
t = 391.56626146910503, 18 degrees of freedom

This shows that RunA ran in an average of 24366622.5 ms and RunB ran in an average of 6469755.9 ms and that the runtime improved by 73.45% which is statistically meaningful with a confidence level of 99.95%.

Usage

Add Bmark as a depdency in your mix.exs file:

def deps do
  [ {:bmark, "~> 1.0.0"} ]
end

Writing Benchmarks

To create a benchmark with bmark, create a file ending in _bmark.ex. Put the file in a directory called bmark. Alltogether, that should look like this

Project Root
+-- bmark
|   +-- example_bmark.ex
+-- lib
|   +-- your_project_files
+-- mix.exs

In example_bmark.ex you should include a module and benchmark function created by using bmark like this:

defmodule Example do
  use Bmark

  bmark :runner do
    IO.puts ":runner test is running"
  end

  bmark :benchmark_with_runs, runs: 5 do
    IO.puts "test running 5 times"
  end
end

The :runner benchmark will be run 20 times, the default number of runs. :benchmark_with_runs specifies the :runs option and will be run only 5 times.

Running Benchmarks

To run all benchmarks run:

$ mix bmark

This will produce the files

Project Root
+-- results
|   +-- example.runner.results
|   +-- example.benchmark_with_runs.results

which will contain the run times, in miliseconds, for each run of the benchmark.

Comparing Benchmark Results

If you have two results files you can compare them by running

$ mix bmark.cmp results/RunA.results results/RunB.results

and bmark will print out the comparison. Here's an example of the comparison with explantions for each section:

RunA:                                 RunB:
24274268                              6426990
24563751                              6416149
24492221                              6507946
24516553                              6453309
24335224                              6491314
24158102                              6405073
24357174                              6504260
24213098                              6449789
24466586                              6532929
24289248                              6509800

The section above contains the raw result data presented side-by-side. This is the same data your would get by looking at RunA.results and RunB.results.

24366622.5 -> 6469755.9 (-73.45%) with p < 0.0005

This line shows the change in mean (average) between the two runs. Next, it shows the percentage change and finally the confidence value. You can interpret this as saying there is 1 - p, or a greater than 99.95% confidence that the change in means is statistically significant. That is, the smaller the value of p the more confident you can be in the change in performance.

t = 391.56626146910503, 18 degrees of freedom

The final line shows the t value and degrees of freedom. This is the raw statistical data used in Student's t-test to compute the confidence value.

More Information

Bmark development has been described in detail on my blog Learning Elixir you can find all the related posts on the Bmark page

Contributing

See Contributing

Articles
to learn more about the elixir concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about Elixir.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory