Jekyll2022-09-05T11:14:00+00:00https://aboobacker.in/feed.xmlAboobacker’s worldPersonal website of Aboobacker MK, Ruby on Rails developer in Kannur, KeralaAboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inPostgres transaction pitfalls for rails developers2022-04-12T07:02:00+00:002022-04-12T07:02:00+00:00https://aboobacker.in/2022/04/12/postgres-transaction-pitfalls-for-rails-developers<p>Rails abstract lot of database stuff away using Active record which is very convenient. But the convenience can bite back if we are not careful enough.</p>
<p>Here I am going to list some common mistakes rails developers make and how to avoid them.</p>
<ol>
<li>Having network calls on rails after_create/before_create callbacks</li>
</ol>
<p>Active record provides callback methods to perform some operations based on the changes in data. it includes <code class="language-plaintext highlighter-rouge">before_create</code>, <code class="language-plaintext highlighter-rouge">after_create</code>, <code class="language-plaintext highlighter-rouge">after_commit</code> etc. Rails wrap before_create and after_create inside the same transaction so that it can roll back when the exception is raised from any of these callbacks. after_commit is executed after the transaction is committed to the database. So if you have calls like</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyModel</span> <span class="o"><</span><span class="no">ApplicationRecord</span>
<span class="n">before_create</span> <span class="ss">:fetch_data_from_server</span>
<span class="k">def</span> <span class="nf">fetch_data_from_server</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">attribute</span> <span class="o">=</span> <span class="no">SlowApi</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">self</span><span class="p">).</span><span class="nf">call</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The transaction will remain open for long time and that will cause locks and potentially deadlocks on tables.</p>
<p>Solutions:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Make network calls before transaction begins, ie before the save method is called
2. Make the API call in after commit callback
</code></pre></div></div>
<ol>
<li>Scheduling Background jobs from after_create/before_create callbacks</li>
</ol>
<p>This is very similar to the previous point, as the backgrounds jobs typically use another data store like Redis and the communication will be over the network. This may not be significant on a small scale if the Redis is in the same infra and network latency is not significant. But any performance degradation on redis infra will cause a sudden spike in long transactions and can potentially cause cascading failures on the infrastructure.</p>
<p>Solutions:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Schedule background jobs from after_commit callback
2. Move job scheduling out of rails callbacks
</code></pre></div></div>
<ol>
<li>Nested transactions</li>
</ol>
<p>When you have nested services you might end up with nested transactions as well.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">OrderCreator</span>
<span class="k">def</span> <span class="nf">call</span>
<span class="no">Order</span><span class="p">.</span><span class="nf">transaction</span> <span class="k">do</span>
<span class="n">adjust_currency_conversion</span>
<span class="o">...</span>
<span class="no">InventoryUpdateService</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">call</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">InventoryUpdateService</span>
<span class="k">def</span> <span class="nf">call</span>
<span class="no">Inventory</span><span class="p">.</span><span class="nf">transaction</span><span class="p">(</span><span class="ss">require_new: </span><span class="kp">true</span><span class="p">)</span> <span class="k">do</span>
<span class="n">deduct_inventory_from_primary_store</span>
<span class="n">run_inventory_sync</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This can cause performance bottlenecks due to the SAVEPOINT behaviour, Skipping it here as Gilab wrote a great blog on this <a href="https://about.gitlab.com/blog/2021/09/29/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/">here</a></p>
<ol>
<li>Last but the not least is about adding transaction block where it is not necessary, for instance if there won’t any data corruption if the operations ran individually, or the impact is very minimal which gets fixed on background worker retry, then avoid using database transactions</li>
</ol>
<p>Also checkout these two great gems which allows us to reduce transaction burdern on database.</p>
<ol>
<li><a href="https://github.com/Envek/after_commit_everywhere">after_commit_everywhere</a></li>
<li><a href="https://github.com/palkan/isolator">isolator</a></li>
</ol>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inRails abstract lot of database stuff away using Active record which is very convenient. But the convenience can bite back if we are not careful enough.Bootsnap and Spring, Understanding rails boottime optimisations2021-06-28T08:33:00+00:002021-06-28T08:33:00+00:00https://aboobacker.in/2021/06/28/bootsnap-and-spring-understanding-rails-boottime-optimisations<p>Being the default the gem file entries in a rails project, you may have encountered spring and bootsnap. Here we are going through how they work and how they can improve boot time of your applications and why you get some weird errors in the app(Looking at you Spring).</p>
<p>Before diving into details, let’s go through some fundamentals to understand things better. Traditionally there were two kinds of languages, Compiled languages and interpreted languages. Compiled languages convert the code into binary format and that binary gets executed as the program. C, C++, Java etc are examples for this pattern. Another set of languages which were primarily designed for scripting read individual lines in the code and converted to machine instructions on runtime.</p>
<p>With ruby 1.9, a new virtual machine replaced the interpreter in ruby which was called Mat’s Ruby Interpreter(MRI). With a new virtual machine. Here instead of directly executing the code, ruby code will be converted to an intermediate representation called instruction sequence and the YARV virtual machine will execute them.</p>
<p>You can refer to the following diagram for reference.
<img src="/images/tokenize.png" alt="'tokenize.png'" /></p>
<p>Credit: Ruby under a microscope</p>
<p>There are also bulletin tools to inspect what is happening in the process, let’s check what happens for a 3 line code.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'ripper'</span>
<span class="nb">require</span> <span class="s1">'pp'</span>
<span class="n">code</span> <span class="o">=</span> <span class="o"><<</span><span class="no">CODE</span><span class="sh">
def add(x, y)
x + y
end
</span><span class="no">CODE</span>
</code></pre></div></div>
<p>Let’s split the code into tokens using ripper utility</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">043</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="no">Ripper</span><span class="p">.</span><span class="nf">tokenize</span> <span class="n">code</span>
<span class="o">=></span> <span class="p">[</span><span class="s2">"def"</span><span class="p">,</span> <span class="s2">" "</span><span class="p">,</span> <span class="s2">"add"</span><span class="p">,</span> <span class="s2">"("</span><span class="p">,</span> <span class="s2">"x"</span><span class="p">,</span> <span class="s2">","</span><span class="p">,</span> <span class="s2">" "</span><span class="p">,</span> <span class="s2">"y"</span><span class="p">,</span> <span class="s2">")"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"x"</span><span class="p">,</span> <span class="s2">" "</span><span class="p">,</span> <span class="s2">"+"</span><span class="p">,</span> <span class="s2">" "</span><span class="p">,</span> <span class="s2">"y"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"end"</span><span class="p">,</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">]</span>
</code></pre></div></div>
<hr />
<p>And see the parsed form using ripper sexp</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">042</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">pp</span> <span class="no">Ripper</span><span class="p">.</span><span class="nf">sexp</span><span class="p">(</span><span class="n">code</span><span class="p">)</span>
<span class="p">[</span><span class="ss">:program</span><span class="p">,</span>
<span class="p">[[</span><span class="ss">:def</span><span class="p">,</span>
<span class="p">[</span><span class="ss">:@ident</span><span class="p">,</span> <span class="s2">"add"</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">]],</span>
<span class="p">[</span><span class="ss">:paren</span><span class="p">,</span>
<span class="p">[</span><span class="ss">:params</span><span class="p">,</span>
<span class="p">[[</span><span class="ss">:@ident</span><span class="p">,</span> <span class="s2">"x"</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">8</span><span class="p">]],</span> <span class="p">[</span><span class="ss">:@ident</span><span class="p">,</span> <span class="s2">"y"</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">11</span><span class="p">]]],</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">]],</span>
<span class="p">[</span><span class="ss">:bodystmt</span><span class="p">,</span>
<span class="p">[[</span><span class="ss">:binary</span><span class="p">,</span>
<span class="p">[</span><span class="ss">:var_ref</span><span class="p">,</span> <span class="p">[</span><span class="ss">:@ident</span><span class="p">,</span> <span class="s2">"x"</span><span class="p">,</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">]]],</span>
<span class="p">:</span><span class="o">+</span><span class="p">,</span>
<span class="p">[</span><span class="ss">:var_ref</span><span class="p">,</span> <span class="p">[</span><span class="ss">:@ident</span><span class="p">,</span> <span class="s2">"y"</span><span class="p">,</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">]]]]],</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
</code></pre></div></div>
<p>We can also take look at how the Instruction sequence(yarv code) looks like</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">052</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="nb">puts</span> <span class="no">RubyVM</span><span class="o">::</span><span class="no">InstructionSequence</span><span class="p">.</span><span class="nf">compile</span><span class="p">(</span><span class="n">code</span><span class="p">).</span><span class="nf">disasm</span>
<span class="o">==</span> <span class="ss">disasm: </span><span class="c1">#<ISeq:<compiled>@<compiled>:1 (1,0)-(3,3)> (catch: FALSE)</span>
<span class="mo">0000</span> <span class="n">definemethod</span> <span class="ss">:add</span><span class="p">,</span> <span class="n">add</span> <span class="p">(</span> <span class="mi">1</span><span class="p">)[</span><span class="no">Li</span><span class="p">]</span>
<span class="mo">0003</span> <span class="n">putobject</span> <span class="ss">:add</span>
<span class="mo">0005</span> <span class="n">leave</span>
<span class="o">==</span> <span class="ss">disasm: </span><span class="c1">#<ISeq:add@<compiled>:1 (1,0)-(3,3)> (catch: FALSE)</span>
<span class="n">local</span> <span class="n">table</span> <span class="p">(</span><span class="ss">size: </span><span class="mi">2</span><span class="p">,</span> <span class="ss">argc: </span><span class="mi">2</span> <span class="p">[</span><span class="ss">opts: </span><span class="mi">0</span><span class="p">,</span> <span class="ss">rest: </span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="ss">post: </span><span class="mi">0</span><span class="p">,</span> <span class="ss">block: </span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="ss">kw: </span><span class="o">-</span><span class="mi">1</span><span class="err">@</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="ss">kwrest: </span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="p">[</span> <span class="mi">2</span><span class="p">]</span> <span class="n">x</span><span class="err">@</span><span class="mi">0</span><span class="o"><</span><span class="no">Arg</span><span class="o">></span> <span class="p">[</span> <span class="mi">1</span><span class="p">]</span> <span class="n">y</span><span class="err">@</span><span class="mi">1</span><span class="o"><</span><span class="no">Arg</span><span class="o">></span>
<span class="mo">0000</span> <span class="n">getlocal_WC_0</span> <span class="n">x</span><span class="err">@</span><span class="mi">0</span> <span class="p">(</span> <span class="mi">2</span><span class="p">)[</span><span class="no">LiCa</span><span class="p">]</span>
<span class="mo">0002</span> <span class="n">getlocal_WC_0</span> <span class="n">y</span><span class="err">@</span><span class="mi">1</span>
<span class="mo">0004</span> <span class="n">opt_plus</span> <span class="o"><</span><span class="n">calldata!mid</span><span class="p">:</span><span class="o">+</span><span class="p">,</span> <span class="n">argc</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span> <span class="no">ARGS_SIMPLE</span><span class="o">></span>
<span class="mo">0006</span> <span class="n">leave</span> <span class="p">(</span> <span class="mi">3</span><span class="p">)[</span><span class="no">Re</span><span class="p">]</span>
<span class="o">=></span> <span class="kp">nil</span>
</code></pre></div></div>
<p>Explaining the terms in the parsed and compiled code is beyond the scope of this blog post, and I recommend reading Ruby Under a Microscope.</p>
<p>Newer versions of ruby also allows us to compile the code to binary and execute them later. it is experimental and platform dependent.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>? <span class="nb">cat </span>example.rb
number <span class="o">=</span> 23
puts number + 23
? ruby <span class="nt">-e</span> <span class="s2">"File.write('example.bin',RubyVM::InstructionSequence.compile_file('example.rb')
.to_binary)"</span>
? <span class="nb">cat </span>example.bin
YARB@
?x86_64-darwin18%?#?%?gw
numberE+Eexampleputs?????????%
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">01</span><span class="mi">8</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="no">RubyVM</span><span class="o">::</span><span class="no">InstructionSequence</span><span class="p">.</span><span class="nf">load_from_binary</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s1">'example.bin'</span><span class="p">)).</span><span class="nf">eval</span>
<span class="mi">46</span>
</code></pre></div></div>
<hr />
<p>That is it for the ruby compilation process for now. Now let’s how the ruby’s <code class="language-plaintext highlighter-rouge">require</code> method works.</p>
<p>Contrary to what I thought initially, require is not a keyword in ruby, it is a method from ruby’s Kernal module.</p>
<p>Let’s look at the overly simplified version of <code class="language-plaintext highlighter-rouge">require</code> method for our context.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">require</span><span class="p">(</span><span class="n">file_name</span><span class="p">)</span>
<span class="nb">eval</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<hr />
<p>Two main issues with this implementation are</p>
<ol>
<li>Requiring same file again will load the file again</li>
<li>Only absolute paths are supported</li>
</ol>
<p>We can fix these issues in the following ways</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="vg">$LOADED_FEATURES</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">require</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">return</span> <span class="kp">false</span> <span class="k">if</span> <span class="vg">$LOADED_FEATURES</span><span class="p">.</span><span class="nf">include?</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="nb">eval</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="vg">$LOADED_FEATURES</span> <span class="o"><<</span> <span class="n">filename</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="vg">$LOAD_PATH</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1">#$LOAD_PATH = += gems_path + stdlib path + application code paths</span>
<span class="k">def</span> <span class="nf">require</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">full_path</span> <span class="o">=</span> <span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">take</span> <span class="k">do</span> <span class="o">|</span><span class="n">path</span><span class="o">|</span>
<span class="no">File</span><span class="p">.</span><span class="nf">exist?</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span>
<span class="k">end</span>
<span class="nb">eval</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>While above code snippets are dummy implementations, Ruby actually uses the constants <code class="language-plaintext highlighter-rouge">$LOADED_FEATURES</code> and <code class="language-plaintext highlighter-rouge">$LOAD_PATH </code>for the same use case. Here is a stat from one of our app for reference</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>irb<span class="o">(</span>main<span class="o">)</span>:054:0> <span class="nv">$LOADED_FEATURES</span>.count
<span class="o">=></span> 6552
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">05</span><span class="mi">8</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">count</span>
<span class="o">=></span> <span class="mi">779</span>
</code></pre></div></div>
<p>Another important method we need to recall is the fork system call in POSIX systems. <code class="language-plaintext highlighter-rouge">fork</code> allows the OS to create a new process as the child process with the same memory space. Modern hardware architectures like x86 allows the OS to optimise the fork with a mechanism called copy on write. In short it is very cheap to create a forked process of an app than loading that app from scratch.</p>
<p>Now let’s look at the gems from the title.</p>
<p>Spring is a rails only tool to speedup development and test environments. It creates your app process in the background for development and test environments and acts as a server. When you run a process like <code class="language-plaintext highlighter-rouge">bundle exec rails server</code> or <code class="language-plaintext highlighter-rouge">bundle exec rspec</code> it sends the data like command, ENV values, arguments etc to spring server, and spring server will fork the server process and run the task.</p>
<p>Since the server process already loaded the app, the loading time of the app will be negligible and your task will run faster.</p>
<p>But when the code changes, the server process needs to update and it may fail due to various reasons, like adding new directories. That is why you ended up having to run <code class="language-plaintext highlighter-rouge">spring stop</code> manually or restart the system to make the application behave as expected.</p>
<p>The serve method in Spring is provided as reference. You can see zipping io values, fetching arguments, env etc from client, forking the new process etc in the below snippet.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">serve</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">log</span> <span class="s2">"got client"</span>
<span class="n">manager</span><span class="p">.</span><span class="nf">puts</span>
<span class="n">_stdout</span><span class="p">,</span> <span class="n">stderr</span><span class="p">,</span> <span class="n">_stdin</span> <span class="o">=</span> <span class="n">streams</span> <span class="o">=</span> <span class="mi">3</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="n">client</span><span class="p">.</span><span class="nf">recv_io</span> <span class="p">}</span>
<span class="p">[</span><span class="no">STDOUT</span><span class="p">,</span> <span class="no">STDERR</span><span class="p">,</span> <span class="no">STDIN</span><span class="p">].</span><span class="nf">zip</span><span class="p">(</span><span class="n">streams</span><span class="p">).</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">|</span> <span class="n">a</span><span class="p">.</span><span class="nf">reopen</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">}</span>
<span class="n">preload</span> <span class="k">unless</span> <span class="n">preloaded?</span>
<span class="n">args</span><span class="p">,</span> <span class="n">env</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="nf">gets</span><span class="p">.</span><span class="nf">to_i</span><span class="p">)).</span><span class="nf">values_at</span><span class="p">(</span><span class="s2">"args"</span><span class="p">,</span> <span class="s2">"env"</span><span class="p">)</span>
<span class="n">command</span> <span class="o">=</span> <span class="no">Spring</span><span class="p">.</span><span class="nf">command</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="nf">shift</span><span class="p">)</span>
<span class="n">connect_database</span>
<span class="n">setup</span> <span class="n">command</span>
<span class="k">if</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">reloaders</span><span class="p">.</span><span class="nf">any?</span><span class="p">(</span><span class="o">&</span><span class="ss">:updated?</span><span class="p">)</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">reloader</span><span class="p">.</span><span class="nf">reload!</span>
<span class="k">end</span>
<span class="n">pid</span> <span class="o">=</span> <span class="nb">fork</span> <span class="p">{</span>
<span class="no">Process</span><span class="p">.</span><span class="nf">setsid</span>
<span class="no">IGNORE_SIGNALS</span><span class="p">.</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">sig</span><span class="o">|</span> <span class="nb">trap</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="s2">"DEFAULT"</span><span class="p">)</span> <span class="p">}</span>
<span class="nb">trap</span><span class="p">(</span><span class="s2">"TERM"</span><span class="p">,</span> <span class="s2">"DEFAULT"</span><span class="p">)</span>
<span class="k">unless</span> <span class="no">Spring</span><span class="p">.</span><span class="nf">quiet</span>
<span class="no">STDERR</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"Running via Spring preloader in process </span><span class="si">#{</span><span class="no">Process</span><span class="p">.</span><span class="nf">pid</span><span class="si">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">env</span><span class="p">.</span><span class="nf">production?</span>
<span class="no">STDERR</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"WARNING: Spring is running in production. To fix "</span> <span class="p">\</span>
<span class="s2">"this make sure the spring gem is only present "</span> <span class="p">\</span>
<span class="s2">"in `development` and `test` groups in your Gemfile "</span> <span class="p">\</span>
<span class="s2">"and make sure you always use "</span> <span class="p">\</span>
<span class="s2">"`bundle install --without development test` in production"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">ARGV</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="vg">$0</span> <span class="o">=</span> <span class="n">command</span><span class="p">.</span><span class="nf">exec_name</span>
<span class="c1"># Delete all env vars which are unchanged from before Spring started</span>
<span class="n">original_env</span><span class="p">.</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="no">ENV</span><span class="p">.</span><span class="nf">delete</span> <span class="n">k</span> <span class="k">if</span> <span class="no">ENV</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">==</span> <span class="n">v</span> <span class="p">}</span>
<span class="c1"># Load in the current env vars, except those which *were* changed when Spring started</span>
<span class="n">env</span><span class="p">.</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="no">ENV</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">||=</span> <span class="n">v</span> <span class="p">}</span>
<span class="c1"># requiring is faster, so if config.cache_classes was true in</span>
<span class="c1"># the environment's config file, then we can respect that from</span>
<span class="c1"># here on as we no longer need constant reloading.</span>
<span class="k">if</span> <span class="vi">@original_cache_classes</span>
<span class="no">ActiveSupport</span><span class="o">::</span><span class="no">Dependencies</span><span class="p">.</span><span class="nf">mechanism</span> <span class="o">=</span> <span class="ss">:require</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">config</span><span class="p">.</span><span class="nf">cache_classes</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
<span class="n">connect_database</span>
<span class="nb">srand</span>
<span class="n">invoke_after_fork_callbacks</span>
<span class="n">shush_backtraces</span>
<span class="n">command</span><span class="p">.</span><span class="nf">call</span>
<span class="p">}</span>
<span class="n">disconnect_database</span>
<span class="n">log</span> <span class="s2">"forked </span><span class="si">#{</span><span class="n">pid</span><span class="si">}</span><span class="s2">"</span>
<span class="n">manager</span><span class="p">.</span><span class="nf">puts</span> <span class="n">pid</span>
<span class="n">wait</span> <span class="n">pid</span><span class="p">,</span> <span class="n">streams</span><span class="p">,</span> <span class="n">client</span>
<span class="k">rescue</span> <span class="no">Exception</span> <span class="o">=></span> <span class="n">e</span>
<span class="n">log</span> <span class="s2">"exception: </span><span class="si">#{</span><span class="n">e</span><span class="si">}</span><span class="s2">"</span>
<span class="n">manager</span><span class="p">.</span><span class="nf">puts</span> <span class="k">unless</span> <span class="n">pid</span>
<span class="k">if</span> <span class="n">streams</span> <span class="o">&&</span> <span class="o">!</span><span class="n">e</span><span class="p">.</span><span class="nf">is_a?</span><span class="p">(</span><span class="no">SystemExit</span><span class="p">)</span>
<span class="n">print_exception</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="n">streams</span><span class="p">.</span><span class="nf">each</span><span class="p">(</span><span class="o">&</span><span class="ss">:close</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">client</span><span class="p">.</span><span class="nf">puts</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">if</span> <span class="n">pid</span>
<span class="n">client</span><span class="p">.</span><span class="nf">close</span>
<span class="k">ensure</span>
<span class="c1"># Redirect STDOUT and STDERR to prevent from keeping the original FDs</span>
<span class="c1"># (i.e. to prevent `spring rake -T | grep db` from hanging forever),</span>
<span class="c1"># even when exception is raised before forking (i.e. preloading).</span>
<span class="n">reset_streams</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Bootsnap is a gem released by shopify by extracting boot time improvements they made in their app. We can categorise them into two parts</p>
<h3 id="path-prescanning">Path prescanning</h3>
<ul>
<li>Kernel#require and Kernel#load are modified to eliminate $LOAD_PATH scans</li>
<li>ActiveSupport::Dependencies.{autoloadable_module?,load_missing_constant,depend_on} are overridden to eliminate scans of ActiveSupport::Dependencies.autoload_paths.</li>
</ul>
<h3 id="compilation-caching">Compilation Caching</h3>
<ul>
<li>RubyVM::InstructionSequence.load_iseq is implemented to cache the result of Ruby bytecode compilation</li>
<li>YAML.load_file is modified to cache the result of loading a YAML object in MessagePack format (or Marshal, if the message uses types unsupported by MessagePack)</li>
</ul>
<p>If you look at the pseudo code above to demonstrate the LOAD_PATH behaviour, you will see that we need to check file existence every time we do a require, which is an io operation and not very cheap to perform. What if we can do something like this?</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">require</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">if</span> <span class="vg">$CACHED_PATH</span><span class="p">[</span><span class="n">file_name</span><span class="p">]</span>
<span class="n">full_path</span> <span class="o">=</span> <span class="vg">$CACHED_PATH</span><span class="p">[</span><span class="n">filename</span><span class="p">]</span>
<span class="k">else</span>
<span class="n">full_path</span> <span class="o">=</span> <span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">take</span> <span class="k">do</span> <span class="o">|</span><span class="n">path</span><span class="o">|</span>
<span class="no">File</span><span class="p">.</span><span class="nf">exist?</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="nb">eval</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>load path for a library is not something that changes very often, especially for gem paths and standard library paths, bootsnap caches them to save redundant file checks. Not that cache duration and expiration vary with files, so caching them in a constant won’t work. <code class="language-plaintext highlighter-rouge">CACHED_PATH</code> is just for reference and not used by the gem.</p>
<p>Another important optimization by bootsnap is the compilation cache. We covered the ruby compilation process in the beginning and saw that every single file needed to go through the compilation process every single time they got called. Bootsnap addresses this by caching yarv code(instruction sequence) and compiling the code only if the code changes.</p>
<p>There are also other optimised YAML loading by caching in an optimised format. overall Bootsnap gives impressive benefits as it works on production as well. We got about 30% reduction in boottime for our production ecommerce app.</p>
<p>Note: This was originaly presented in internal tech talk in Sephora and made some tweaks to the content for wider audience.</p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inBeing the default the gem file entries in a rails project, you may have encountered spring and bootsnap. Here we are going through how they work and how they can improve boot time of your applications and why you get some weird errors in the app(Looking at you Spring).Intermediate Postgresql for rails developers, Part 0: Get your environment2021-04-30T17:42:00+00:002021-04-30T17:42:00+00:00https://aboobacker.in/2021/04/30/intermediate-postgresql-for-rails-developers-part-0-get-your-enviornoments-ready<p>Postgresql is one of the most advanced open source database available in the market. Adherence to the SQL standards and super cool extra features are few of the reason by Postgres being the most popular database among rails community. This is neither introduction to Postgres nor rails. This series is for the folks who have been using rails and Postgres for one or more years.</p>
<p>In part 0 we will walk through the tips for postgres setup to increase the productivity.</p>
<p>ActiveRecord orm which is built into rails provides a lot of cool methods to access the data in rubyistic way. But it is important to get familiarize with the native Postgres tooling for many tasks.</p>
<h2 id="psql">Psql</h2>
<p>Psql is the official commandline client for postgresql, rails does have shortcut to open <code class="language-plaintext highlighter-rouge">psql</code> using the configuration from database configuration as <code class="language-plaintext highlighter-rouge">rails dbconsole</code>. But you can also open by using <code class="language-plaintext highlighter-rouge">psql</code> command on your command line client.</p>
<p><code class="language-plaintext highlighter-rouge">psql -h localhost -U username databasename</code></p>
<p>Above is the basic syntax for opening psql console, Once you enter that you will be greeted with psql console if the credentials given are correct.</p>
<p>Now let’s go through few psql tips</p>
<h3 id="backward-slash-commands">Backward slash commands</h3>
<p><code class="language-plaintext highlighter-rouge">psql</code> supports a lot of useful configuration options to improve the experiance in postgresql shell.</p>
<ul>
<li>
<p><code class="language-plaintext highlighter-rouge">\timing</code></p>
<p>Timing command add the time taken to run the command, this is handy when optimizing query performance</p>
</li>
</ul>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="k">local</span><span class="p">]</span> <span class="n">abomk</span><span class="o">@</span><span class="n">cv_dump</span><span class="o">=#</span><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">from</span> <span class="n">users</span><span class="p">;</span>
<span class="o">?</span> <span class="mi">45129</span> <span class="o">?</span>
<span class="o">?????????</span>
<span class="p">[</span><span class="k">local</span><span class="p">]</span> <span class="n">abomk</span><span class="o">@</span><span class="n">cv_dump</span><span class="o">=#</span><span class="err">\</span><span class="n">timing</span>
<span class="n">Timing</span> <span class="k">is</span> <span class="k">on</span><span class="p">.</span>
<span class="p">[</span><span class="k">local</span><span class="p">]</span> <span class="n">abomk</span><span class="o">@</span><span class="n">cv_dump</span><span class="o">=#</span><span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">from</span> <span class="n">users</span><span class="p">;</span>
<span class="o">?</span> <span class="mi">45129</span> <span class="o">?</span>
<span class="o">?????????</span>
<span class="nb">Time</span><span class="p">:</span> <span class="mi">8</span><span class="p">.</span><span class="mi">412</span> <span class="n">ms</span>
</code></pre></div></div>
<ul>
<li>
<p><code class="language-plaintext highlighter-rouge">\s</code></p>
<p>Get posgtres commands history, this is handy for documenting the stuff after doing experiments with different queries and many more. You can also choose to save the history by providing filename as the parameter as <code class="language-plaintext highlighter-rouge">\s filename</code></p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">\i filename</code></p>
</li>
</ul>
<p>Load query from file. Big queries are often convent to write in a text editor, <code class="language-plaintext highlighter-rouge">\i</code> enables loading a query from a file and execute it.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">\e</code></li>
</ul>
<p>This another approach to solve the difficulty of writing multi line queries in a shell environment, When you give <code class="language-plaintext highlighter-rouge">\e</code> command, psql will open the text editor you set $EDITOR to open the query. You can edit and close the file, content will be copied to posql shell, and you can execute.</p>
<p>You can find more such commands from <a href="https://www.postgresqltutorial.com/psql-commands/">posgrestutorials</a> and <a href="https://pgdash.io/blog/postgres-psql-tips-tricks.html">pgdash</a></p>
<h3 id="managing-multiple-versions-of-postgres">Managing multiple versions of postgres</h3>
<p>It is important to have the same version of postgres in your development setup as the production version. If you have multiple apps with diffrent versions of postgres versions, you can use <a href="https://github.com/theory/pgenv">pgenv</a> to configure multiple versions of Postgres in your local. There are other options like <a href="https://github.com/smashedtoatoms/asdf-postgres">asdf-postgres</a></p>
<h3 id="use-different-pager">Use different pager</h3>
<p>The default pager is somewhat difficult to read when there are too many columns. <a href="https://github.com/okbob/pspg">pspg</a> is an alternate pager you can use with Postgres</p>
<h3 id="save-your-psql-settings-in-your-psqlrc">Save your psql settings in your psqlrc</h3>
<p>You can save the psql configurations we discussed above in a file so that you don’t need to repeat every time you open the psql shell. Here is my psql configuration for reference</p>
<pre><code class="language-SQL">\set ON_ERROR_ROLLBACK interactive
\set COMP_KEYWORD_CASE upper
\set HISTFILE ~/.psql/history- :DBNAME
\set VERBOSITY verbose
\set PROMPT1 '%[%033[1m%]%M %n@%/%R%[%033[0m%]%#'
\setenv PAGER pspg
\pset border 2
\pset linestyle unicode
\set null '(null)'
</code></pre>
<p>You can find explanation of individual configurations from <a href="https://thoughtbot.com/blog/an-explained-psqlrc">thoughtbot blog</a></p>
<h2 id="pgcli">Pgcli</h2>
<p><a href="https://www.pgcli.com/">Pgcli</a> is an alternate postgresql client with additional features like autocompletion of table and sql queries.</p>
<p>We will go through more tips in coming articles, while it is not mandatory to follow the above setup for cominig articles, it is good to have to get hands on with psql shell</p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inPostgresql is one of the most advanced open source database available in the market. Adherence to the SQL standards and super cool extra features are few of the reason by Postgres being the most popular database among rails community. This is neither introduction to Postgres nor rails. This series is for the folks who have been using rails and Postgres for one or more years.Intellectual property, Open source and mimemagic2021-03-28T18:47:00+00:002021-03-28T18:47:00+00:00https://aboobacker.in/2021/03/28/intellectual-property-open-source-and-mimemagic<p>The alleged GPL violation in a ruby library called mimemagic which broke builds of rails projects across the world brought discussion of Intellectual property in the software world one more time last week. It is important to have a general idea about these terms for a software developer to avoid getting into big troubles in the future.</p>
<p>This is my attempt to write down my understanding about open source and Intellectual property, I am not a lawyer and the implementation of IP is based on the local laws which is not universal.</p>
<p>Though the modern concept of IP is originated from the exclusive access to rights and grants by the British monarch, it soon became a mechanism to promote the creation of intellectual goods, by giving the inventors some rights to the work they made.</p>
<p>Though IP may sound an alien term, it is something critical for any company, for instance, company name, logo, design of the logo etc are <strong>trademarks</strong> which is a kind of intellectual property. This enables companies to protect reputation and goodwill of the brand by preventing others from copying same brand symbols to sell their products.</p>
<p>Content produced by the company is protected by <strong>copyright</strong>, this also includes the software they made, in almost all legislations content produced is inherently owned by the author and protected by copyright, this prevents unauthorized access and redistribution of content by other people.</p>
<p>Another important intellectual property is <strong>patent</strong> which provides protection for innovative ideas, unlike copyright patent requires registration within a particular timeframe to the government agency. In general Patent protects ideas and copyright protects manifestation of that idea</p>
<p>There are other IPs like trade secrets which I am not covering here to limit the scope of the article.</p>
<p>Historically hardware was the most expensive piece of a computer and program or the software that runs on them wasn’t valued as important. But as more software distribution patterns emerged people started distributing just binaries of the software which prevented users from changing the software as per their needs. Richard R Stallman who was working in MIT lab during this period had a big trouble because of this, they used to adjust the printer firmware as per their need of paper size, But the new printer firmware came with just binary format which prevented from making the modification which caused wastage of papers they already had, Stallman requested the firmware owner to share the source code, but he refused.</p>
<p>That incident hit him big as usage of a computer owned by the user is dictated by the firmware owner. He believed this is a big ethical and social issue. He launched the GNU project in 1983 to create an operating system consisting of only free software. As per his own words the word free of “free” as in “free speech,” not as in “free beer”. Many including Linux followed this approach and many free software programs became available in the market.</p>
<p>In late 90s some people in industry realized the advantage of having free software in the enterprise world, i.e. when the code is shared more people can learn and adapt from it and there by saving time and cost. They also a coined a term called “Open Source” since people may easily interpret free software as free of cost software.</p>
<p>While free softwares and open source softwares and practically largely the same, they are two movements with different motivations but almost the same path to achieve their goals. While free software movement is from the perspective of the user to have full control of his/her system, open source is from the perspective of software engineer or company to get benefit from shared source code.</p>
<p>Open source Initiative, which formalized the open source with 10 <a href="https://opensource.org/osd">rules</a> and listed software licenses compatible with these rules as open source licenses.</p>
<p>But why does an open source software need license?</p>
<p>The Answer is simple, as we discussed in the initial section, any content produced is owned by the author and others can’t redistribute without author’s permission. Licenses are contract from a software developer to the user to share some of his/her/their rights to the user, In the case of open source it will be freedom to redistribute, make derived work etc. In other words open source licenses Grants certain IP rights to users who are subject to license. This is different from EULA(End user license agreement) we see in the proprietary softwares which is to limit what the user can do with the software we purchased.</p>
<p>Also keep in mind that open source is not public domain which are the work which is not owned by anyone</p>
<p>There are many kinds of open source licenses ranging from permissive license which doesn’t ask the users anything more than attribution to copyleft licenses which mandate you to make your software of also open source if you use one in your software.</p>
<p>I’ll be writing another article about commonly used licenses and things to be taken care of while using such licenses in another post, till then you can refer <a href="https://tldrlegal.com/">this awesome website</a>.</p>
<p>Now let’s go back to the incident that happened in the last week. freedesktop is a project on interoperability of free software projects, they maintain the list of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types">mime types</a> and is licensed under <a href="https://tldrlegal.com/license/gnu-general-public-license-v3-(gpl-3)">GPL</a>. GPL is a very restrictive license and if you want to link your software to a GPLed software and distribute, then you will have to license your software with the same restrictions. In other words, you can’t directly link a software linked with a gpled software/code and distribute under proprietary or permissive open source license like MIT. The mimemagic gem which is MIT licensed used this GPLed code from freedesktop which is violation of GPL license. Initial fix to convert mimemagic license to GPL. But due to the nature of copyleft licenses, software which includes mimemagic will also have to comply GPL rules. So they later removed the GPLed file and made the library to use mime data from the user’s computer.</p>
<p>This may raise some questions about using GPL programs you use with your closed source(projects which are not open source) projects. First, you don’t have to care much about GPL in backend of software as service model like most websites does, as you are just distributing the output of the software, not the software itself to the end user, this is also called as <a href="https://resources.whitesourcesoftware.com/blog-whitesource/the-saas-loophole-in-gpl-open-source-licenses">saas loophole</a>. Even if you aren’t, it is unlikely that mysql being part of your source code, if you are just mysql as database, you are using its SQL interface and not as part of your programs.</p>
<p>Now let’s look at the common errors developers make due to lack of awareness about IP,</p>
<ul>
<li>
<p>Copy and paste snippets from stackoverflow (read more <a href="https://meta.stackexchange.com/questions/12527/do-i-have-to-worry-about-copyright-issues-for-code-posted-on-stack-overflow">here</a>).</p>
</li>
<li>
<p>Copy open source code snippets from GitHub</p>
<p>Even if the code is open source, some may require making your code also open source, some require attribution.</p>
</li>
<li>
<p>Copy code snippets from GitHub or other publically available places</p>
<p>If you found some code on GitHub or any publically accessible places, you shouldn’t assume that code is open source unless license is explicitly given. For instance projects without explicit license in GitHub, all you can do is read the code and fork, you aren’t supposed to change the code</p>
</li>
<li>
<p>Forking open source projects without changing trademarks</p>
<p>Even when the open source project is licensed under permissive open source license, it doesn’t grant you to use it’s trademark for redistribution. For example you can’t simply fork mysql and sell it as mysql</p>
</li>
<li>
<p>Open sourcing stuff without proper planning</p>
<p>When you are open sourcing certain part of codebase, you must have a plan for what to do with the contributions, For instance when you release a component of your code base as open source and someone from community send a pull request and you merged to your repo, by default the person who contributed will have the copyright of the code, company can only use that code as per the license given in the repository. <a href="https://en.wikipedia.org/wiki/Contributor_License_Agreement">CLA</a> Can be used as a workaround for this</p>
</li>
</ul>
<p>Do you have more examples? Feel free to comment below.</p>
<p>We covered the general IP concepts and it’s significanse in open source, I am not a lawyer and this blogpost is not a legal opinion. So talk to your lawyer before making legal decisons</p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inThe alleged GPL violation in a ruby library called mimemagic which broke builds of rails projects across the world brought discussion of Intellectual property in the software world one more time last week. It is important to have a general idea about these terms for a software developer to avoid getting into big troubles in the future.Let’s learn rack by implementing it from scratch2017-02-15T19:58:20+00:002017-02-15T19:58:20+00:00https://aboobacker.in/2017/02/15/lets-learn-rack-by-implementing-it-from-scratch<p>Rack is a framework for ruby web frameworks. If you developed apps in ruby frameworks like rails, hanami, Sinatra, you already used Rack. Almost all ruby web frameworks use rack under the hood, If you are already familiar with rack then you can skip next step and go to re building part</p>
<h2 id="introduction-to-rack">Introduction To Rack</h2>
<p>Rack provides minimalistic API to interact, First let’s have a look at rack</p>
<p>A rack app is an object which takes request environment hash and provides array of 3 elements the output, rack object should respond to the method <code class="language-plaintext highlighter-rouge">call</code></p>
<ul>
<li>The HTTP response code</li>
<li>Headers hash</li>
<li>response body object which responds to each</li>
</ul>
<p>The obvious question here is why does rack says about responding to the particular method. This one of the powerful paradigm available in ruby called <a href="https://en.wikipedia.org/wiki/Duck_typing">duck typing</a>. ie rack doesn’t care about the object or it’s implementation as long as it responds to the particular method</p>
<p>Let’s take a look at a simple example.</p>
<p>Install rack</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> gem <span class="nb">install </span>rack
</code></pre></div></div>
<p>create a file with name <code class="language-plaintext highlighter-rouge">config.ru</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config.ru</span>
<span class="n">run</span> <span class="no">Proc</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span> <span class="o">|</span><span class="n">env</span><span class="o">|</span> <span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'Hello World\'d'</span><span class="p">]]</span> <span class="p">}</span>
</code></pre></div></div>
<p>and run the command <code class="language-plaintext highlighter-rouge">rackup</code></p>
<p>Now open the browser and vist <code class="language-plaintext highlighter-rouge">localhost:9292</code></p>
<p>Congratulations !, You just made a rack app with just a single line</p>
<p>Now let’s have look at the above example</p>
<p>We have a proc object which responds to <a href="https://ruby-doc.org/core-2.2.0/Proc.html#method-i-call">call</a> method. <code class="language-plaintext highlighter-rouge">200</code>as status code , an array consist of
<code class="language-plaintext highlighter-rouge">{'Content-Type' => 'text/html'}</code> as Response header
and
<code class="language-plaintext highlighter-rouge">['Hello World \'d']</code> as body</p>
<p>Since rack do not care about the kind of rack object, we can do the same using class or an object</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">class</span> <span class="nc">SuperCoolApp</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'Hello World\'d'</span><span class="p">]]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">run</span> <span class="no">SuperCoolApp</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">CoolApp</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'Hello World\'d'</span><span class="p">]]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">run</span> <span class="no">SuperCoolApp</span><span class="p">.</span><span class="nf">new</span>
</code></pre></div></div>
<p>But this doesn’t do anything interesting, this will just display <code class="language-plaintext highlighter-rouge">Hello World</code> for all requests . Because we were returning same output without even considering the parameter <code class="language-plaintext highlighter-rouge">env</code> . Let’s have a look into by returning env hash as output</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">CoolApp</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="n">env</span><span class="p">.</span><span class="nf">inspect</span><span class="p">]]</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now run <code class="language-plaintext highlighter-rouge">rackup</code> and goto <code class="language-plaintext highlighter-rouge">localhost:9292/hello/world</code>
Output will be something like this</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="s2">"rack.version"</span><span class="o">=></span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="s2">"rack.errors"</span><span class="o">=></span><span class="c1">#>>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"puma 3.6.0 Sleepy Sunday Serenity", "GATEWAY_INTERFACE"=>"CGI/1.2", "REQUEST_METHOD"=>"GET", "REQUEST_PATH"=>"/hello/world", "REQUEST_URI"=>"/hello/world", "HTTP_VERSION"=>"HTTP/1.1", "HTTP_HOST"=>"localhost:9292", "HTTP_CONNECTION"=>"keep-alive", "HTTP_UPGRADE_INSECURE_REQUESTS"=>"1", "HTTP_USER_AGENT"=>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", "HTTP_ACCEPT"=>"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "HTTP_DNT"=>"1", "HTTP_ACCEPT_ENCODING"=>"gzip, deflate, sdch, br", "HTTP_ACCEPT_LANGUAGE"=>"en-US,en;q=0.8,ml;q=0.6", "SERVER_NAME"=>"localhost", "SERVER_PORT"=>"9292", "PATH_INFO"=>"/hello/world", "REMOTE_ADDR"=>"127.0.0.1", "puma.socket"=>#, "rack.hijack?"=>true, "rack.hijack"=>#, "rack.input"=>#>, "rack.url_scheme"=>"http", "rack.after_reply"=>[], "puma.config"=>#"development", :pid=>nil, :Port=>9292, :Host=>"localhost", :AccessLog=>[], :config=>"/home/tachyons/code/rack/config.ru"}, {:log_requests=>false, :environment=>"development", :binds=>["tcp://localhost:9292"], :app=>#, @content_length=nil>>, @logger=#>>>>}, {:environment=>"development"}, {}], @defaults={:min_threads=>0, :max_threads=>16, :log_requests=>false, :debug=>false, :binds=>["tcp://0.0.0.0:9292"], :workers=>0, :daemon=>false, :mode=>:http, :worker_timeout=>60, :worker_boot_timeout=>60, :worker_shutdown_timeout=>30, :remote_address=>:socket, :tag=>"rack", :environment=>"development", :rackup=>"config.ru", :logger=>#>, :persistent_timeout=>20}>, @plugins=#>, "rack.tempfiles"=>[]}</span>
</code></pre></div></div>
<p>Now change url and see the changes output</p>
<p>To make it clear Let’s build a simple app to hello</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">CoolApp</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span> <span class="s2">"Hi "</span> <span class="o">+</span> <span class="n">env</span><span class="p">[</span><span class="s1">'REQUEST_PATH'</span><span class="p">].</span><span class="nf">split</span><span class="p">(</span><span class="s1">'/'</span><span class="p">).</span><span class="nf">join</span><span class="p">(</span><span class="s2">" "</span><span class="p">)]]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">run</span> <span class="no">CoolApp</span>
</code></pre></div></div>
<p>Run rackup again, and go to <code class="language-plaintext highlighter-rouge">localhost:9292/aboobacker/mk</code></p>
<p>App will respond “Hi aboobacker mk”</p>
<p>You can implement your own logic using the env variable provided by rack</p>
<p>Rack also provides <a href="http://www.rubydoc.info/gems/rack/Rack/Request">Rack Request</a> Abstraction which provides a convenient interface to a Rack environment.</p>
<p>But that is not the end, rack also provides feature called middleware, which let you use multiple rack apps as pipeline . ie output of one rack app will feed as input to next rack app . Let’s check that by one example</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ReverseOutput</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span> <span class="o">=</span> <span class="vi">@app</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">body</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">msg</span><span class="o">|</span> <span class="n">msg</span><span class="p">.</span><span class="nf">reverse</span> <span class="p">}</span>
<span class="p">[</span><span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">CoolApp</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span> <span class="s2">"Hi "</span> <span class="o">+</span> <span class="n">env</span><span class="p">[</span><span class="s1">'REQUEST_PATH'</span><span class="p">].</span><span class="nf">split</span><span class="p">(</span><span class="s1">'/'</span><span class="p">).</span><span class="nf">join</span><span class="p">(</span><span class="s2">" "</span><span class="p">)]]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">use</span> <span class="no">ReverseOutput</span>
<span class="n">run</span> <span class="no">CoolApp</span>
</code></pre></div></div>
<p>Here we made a simple middleware <code class="language-plaintext highlighter-rouge">ReverseOutput</code> which will reverse the response body, You can add any number of middlewares like this, also you can use pre defined middlewares provided by the rack and open source general purpose middlewares. <a href="https://github.com/rack/rack/wiki/List-of-Middleware">List of middlewares</a></p>
<h2 id="building-from-scratch">Building from scratch</h2>
<p>Now let’s have a look at how the rack works by making a rack like library from scratch, Let’s name it Srack. But one obvious question here is why rackup file is <code class="language-plaintext highlighter-rouge">config.ru</code>, not <code class="language-plaintext highlighter-rouge">config.rb</code> ? . Also from where the methods like use, run etc are coming</p>
<p>Let’s look at our first code sample in a different way</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app.rb</span>
<span class="no">Rack</span><span class="o">::</span><span class="no">Builder</span><span class="p">.</span><span class="nf">app</span> <span class="k">do</span>
<span class="n">run</span> <span class="no">Proc</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span> <span class="o">|</span><span class="n">env</span><span class="o">|</span> <span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'Hello World\'d'</span><span class="p">]]</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here we can see that <code class="language-plaintext highlighter-rouge">config.ru</code> is a block that is to be passed to Rack::Builder.app method</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle gem srack
</code></pre></div></div>
<p>Now remove all TODOs from <code class="language-plaintext highlighter-rouge">srack.gemspec</code> So that we can run the test cases . Now if we run test cases it will show one failure message</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Failed examples:
rspec ./spec/srack_spec.rb:8 # Srack does something useful
</code></pre></div></div>
<p>And it is true, we haven’t done anything useful yet</p>
<p>First thing we have to do is to build an executable equiallant to rackup, Let’s call it srackup</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>touch exe/srackup
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s2">"srack"</span>
<span class="no">Srack</span><span class="o">::</span><span class="no">Server</span><span class="p">.</span><span class="nf">start</span>
</code></pre></div></div>
<p>I copied above file from <a href="https://github.com/rack/rack/blob/master/bin/rackup">rack repo</a> to make sure that we are following the same way . Since we haven’t implemenetd Srack::Server this won’t work yet . So let’s make that first</p>
<p>Since we have to make instance of <code class="language-plaintext highlighter-rouge">Rack::Server</code> we can make it as a <code class="language-plaintext highlighter-rouge">class</code> and define <code class="language-plaintext highlighter-rouge">start</code> as the class method</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now we have Srack::Server.start method. But it is doing nothing. Since we want Server object, we can delegate our start method to it’s instance method.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">start</span>
<span class="n">new</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now let’s set some default options for our app</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@options</span> <span class="o">=</span> <span class="n">default_options</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">start</span>
<span class="n">new</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">default_options</span>
<span class="p">{</span>
<span class="ss">environment: </span><span class="s2">"localhost"</span><span class="p">,</span>
<span class="no">Port</span><span class="p">:</span> <span class="s2">"9393"</span><span class="p">,</span>
<span class="no">Host</span><span class="p">:</span> <span class="s2">"localhost"</span>
<span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now we have to build the app from config.ru or the file specified as argument, we can store it in <code class="language-plaintext highlighter-rouge">@options</code> hash with the key <code class="language-plaintext highlighter-rouge">config</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@options</span> <span class="o">=</span> <span class="n">default_options</span>
<span class="vi">@options</span><span class="p">[</span><span class="ss">:config</span><span class="p">]</span> <span class="o">=</span> <span class="no">ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="no">ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">build_app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">start</span>
<span class="n">new</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">default_options</span>
<span class="p">{</span>
<span class="ss">environment: </span><span class="s2">"localhost"</span><span class="p">,</span>
<span class="no">Port</span><span class="p">:</span> <span class="s2">"9393"</span><span class="p">,</span>
<span class="no">Host</span><span class="p">:</span> <span class="s2">"localhost"</span><span class="p">,</span>
<span class="ss">config: </span><span class="s1">'config.ru'</span>
<span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">build_app</span>
<span class="no">Builder</span><span class="p">.</span><span class="nf">parse_file</span><span class="p">(</span><span class="vi">@options</span><span class="p">[</span><span class="ss">:config</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here we are we are using the <code class="language-plaintext highlighter-rouge">Srack::Builder</code> to parse the config file and load app from it . Let’s implement that logic in <code class="language-plaintext highlighter-rouge">Builder</code> factory</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Builder</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">parse_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">config_file</span> <span class="o">=</span> <span class="o">::</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">new_from_string</span><span class="p">(</span><span class="n">config_file</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">new_from_string</span><span class="p">(</span><span class="n">builder_script</span><span class="p">)</span>
<span class="nb">eval</span> <span class="s2">"Rack::Builder.new {</span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="n">builder_script</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">}.to_app"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The first method is self-explanatory, it just read the file and passes the file body to <code class="language-plaintext highlighter-rouge">new_from_string</code> . The method <code class="language-plaintext highlighter-rouge">new_from_string</code> takes the file contents, convert it into a proc and pass to <code class="language-plaintext highlighter-rouge">Rack::Builder.new</code> . So that we can execute the contents of config.ru in the context of the builder</p>
<p>Remember our first rack app ?</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config.ru</span>
<span class="n">run</span> <span class="no">Proc</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span> <span class="o">|</span><span class="n">env</span><span class="o">|</span> <span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'Hello World\'d'</span><span class="p">]]</span> <span class="p">}</span>
</code></pre></div></div>
<p>In order to execute this</p>
<ol>
<li>Builder class should accept block for initialize method</li>
<li>And execute it within the context of <code class="language-plaintext highlighter-rouge">Builder object</code></li>
<li>Builder class also should have methods run and to_app as setter and getter
Let’s see it in code
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Builder</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="nb">instance_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="nb">block_given?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">to_app</span>
<span class="vi">@app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">parse_file</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">config_file</span> <span class="o">=</span> <span class="o">::</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="n">new_from_string</span><span class="p">(</span><span class="n">config_file</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">new_from_string</span><span class="p">(</span><span class="n">builder_script</span><span class="p">)</span>
<span class="nb">eval</span> <span class="s2">"Srack::Builder.new {</span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="n">builder_script</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">}.to_app"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div> </div>
</li>
</ol>
<p>Now we have Builder class, But start method in <code class="language-plaintext highlighter-rouge">Srack::Server</code> class is still empty, In order to do that we have to connect to some real server. Remember when we mentioned rack is an interface to web servers?</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Srack</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="n">server</span><span class="p">.</span><span class="nf">run</span> <span class="vi">@app</span><span class="p">,</span> <span class="vi">@options</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">server</span>
<span class="vi">@server</span> <span class="o">||=</span> <span class="no">Srack</span><span class="o">::</span><span class="no">Handler</span><span class="p">.</span><span class="nf">default</span>
<span class="k">end</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Srack will have handlers for each type of servers, So that we can global api for handlers, ie all handlers should respond to <code class="language-plaintext highlighter-rouge">run</code> method with 2 arguments <code class="language-plaintext highlighter-rouge">@app</code> and <code class="language-plaintext highlighter-rouge">@options</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># lib/srack/handler.rb`</span>
<span class="k">module</span> <span class="nn">Srack</span>
<span class="k">module</span> <span class="nn">Handler</span>
<span class="nb">autoload</span> <span class="ss">:Thin</span><span class="p">,</span> <span class="s1">'srack/handler/thin'</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">default</span>
<span class="no">Handler</span><span class="o">::</span><span class="no">Thin</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># lib/srack/handler/thin.rb</span>
<span class="nb">require</span> <span class="s1">'thin'</span>
<span class="k">module</span> <span class="nn">Srack</span>
<span class="k">module</span> <span class="nn">Handler</span>
<span class="k">class</span> <span class="nc">Thin</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">run</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">options</span> <span class="o">=</span> <span class="p">{})</span>
<span class="n">host</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:Host</span><span class="p">]</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:Port</span><span class="p">]</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">[</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">app</span><span class="p">,</span> <span class="n">options</span><span class="p">]</span>
<span class="n">server</span> <span class="o">=</span> <span class="o">::</span><span class="no">Thin</span><span class="o">::</span><span class="no">Server</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="n">server</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here made handler module which can accomodate multple handlers, In this example we used thin as the default server . To use thin inside our app, we have to include it in our <code class="language-plaintext highlighter-rouge">srack.gemspec</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">spec</span><span class="p">.</span><span class="nf">add_dependency</span> <span class="s2">"thin"</span>
</code></pre></div></div>
<p>Now you can build the gem to test</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem build srack.gemspec
gem install srack-0.1.0.gem
</code></pre></div></div>
<p>Now our srack is capable for running our first rack app
Just goto the directory with the file <code class="language-plaintext highlighter-rouge">config.ru</code> and run <code class="language-plaintext highlighter-rouge">srackup</code></p>
<h2 id="implementing-middleware">Implementing middleware</h2>
<p>As discussed earlier one of the widely used feature in the rack is middleware. Let’s see how it works</p>
<p>A middleware will take the output(triplet) of rack app and modify it and give to next middleware or the app</p>
<p>We can make some tweaks in <code class="language-plaintext highlighter-rouge">Srackup::Builder</code> to accommodate this</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">...</span>
<span class="k">class</span> <span class="nc">Builder</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="vi">@use</span> <span class="o">=</span> <span class="p">[]</span>
<span class="nb">instance_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="nb">block_given?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@run</span> <span class="o">=</span> <span class="n">app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">use</span><span class="p">(</span><span class="n">middleware</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="vi">@use</span> <span class="o"><<</span> <span class="nb">proc</span> <span class="p">{</span><span class="o">|</span><span class="n">app</span><span class="o">|</span> <span class="n">middleware</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">to_app</span>
<span class="n">app</span> <span class="o">=</span> <span class="vi">@run</span>
<span class="n">app</span> <span class="o">=</span> <span class="vi">@use</span><span class="p">.</span><span class="nf">reverse</span><span class="p">.</span><span class="nf">inject</span><span class="p">(</span><span class="n">app</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">a</span><span class="p">,</span><span class="n">e</span><span class="o">|</span> <span class="n">e</span><span class="p">[</span><span class="n">a</span><span class="p">]</span> <span class="p">}</span>
<span class="n">app</span>
<span class="k">end</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="o">...</span>
</code></pre></div></div>
<p>Here we defined an extra method <code class="language-plaintext highlighter-rouge">use</code> which will accept middleware as input. Also we have a new instance variable array <code class="language-plaintext highlighter-rouge">@use</code> which will store procs which accept app as input and returns new middleware object in return</p>
<p>Also, we changed <code class="language-plaintext highlighter-rouge">to_app</code> in such way that middlewares will be executed in the reverse order of calling</p>
<p>Now our app can also handle middlewares</p>
<p>If something is missing, or getting some errors, you can cross check with my <a href="https://github.com/tachyons/srack">repo here</a></p>
<h3 id="references">References</h3>
<ul>
<li>https://github.com/rack</li>
<li>http://www.kavinder.com/blog/2014-10-10-rebuild-a-gem-rack/</li>
<li>http://www.kavinder.com/blog/2014-10-10-rebuild-a-gem-rack/</li>
</ul>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inRack is a framework for ruby web frameworks. If you developed apps in ruby frameworks like rails, hanami, Sinatra, you already used Rack. Almost all ruby web frameworks use rack under the hood, If you are already familiar with rack then you can skip next step and go to re building partBehind the scenes of hash table performance in ruby 2.42017-01-30T04:00:13+00:002017-01-30T04:00:13+00:00https://aboobacker.in/2017/01/30/behind-scenes-hash-table-performance-ruby-2-4<p>Ruby 2.4 got released this Christmas with lot of exciting features. One of the most underrated features in ruby 2.4 is hash table improvements. Before going into details about implementation, let’s first check the benchmark to know how this change gonna affect your ruby application.</p>
<ul>
<li>Getting keys and values of a hash</li>
</ul>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">h</span> <span class="o">=</span> <span class="p">{}</span>
<span class="mi">10000</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">h</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="k">end</span>
<span class="c1"># Get all hash values</span>
<span class="no">Benchmark</span><span class="p">.</span><span class="nf">measure</span> <span class="p">{</span> <span class="mi">50000</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">h</span><span class="p">.</span><span class="nf">values</span> <span class="p">}</span> <span class="p">}</span>
<span class="c1"># Get all hash keys</span>
<span class="no">Benchmark</span><span class="p">.</span><span class="nf">measure</span> <span class="p">{</span> <span class="mi">50000</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">h</span><span class="p">.</span><span class="nf">keys</span> <span class="p">}</span> <span class="p">}</span>
</code></pre></div></div>
<p>Ruby 2.3</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=> #<Benchmark::Tms:0x00000002a0f990 @label="", @real=2.8023432340005456, @cstime=0.0, @cutime=0.0, @stime=0.13000000000000012, @utime=2.6799999999999997, @total=2.8099999999999996>
#<Benchmark::Tms:0x00000002963398 @label="", @real=2.767410662000657, @cstime=0.0, @cutime=0.0, @stime=0.029999999999999805, @utime=2.729999999999997, @total=2.7599999999999967>
</code></pre></div></div>
<p>Ruby 2.4</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#<Benchmark::Tms:0x0000000226d700 @label="", @real=0.8854832770002758, @cstime=0.0, @cutime=0.0, @stime=0.08999999999999997, @utime=0.7999999999999998, @total=0.8899999999999998>
#<Benchmark::Tms:0x000000022b1018 @label="", @real=0.8476084579997405, @cstime=0.0, @cutime=0.0, @stime=0.06999999999999995, @utime=0.7799999999999994, @total=0.8499999999999993>
</code></pre></div></div>
<p>the above two operations executed ~ 3 times faster on my laptop. Though these numbers can vary with your machine and processor, the performance improvements will be significant on all modern processors. Not all operations became 3 times faster , average perfomence improvement is more than 50%.</p>
<h3 id="hash-table">Hash Table</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>In computing, hash table (hash map) is a data structure that is used to implement an associative array, a structure that can map keys to values. Hash table uses a hash function to compute an index into an array of buckets or slots, from which the desired value can be found. Wikipedia
</code></pre></div></div>
<p>In other words, it is a data structure that allows you to store key value pair and helps to fetch specific data using the key in an efficient way. Unlike arrays, you don’t have to iterate through all elements to find a given element in the hash. If you are new to this data structure, check this <a href="https://www.youtube.com/watch?v=h2d9b_nEzoA">s50</a> for the better understanding.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">hash</span> <span class="o">=</span> <span class="p">{</span><span class="ss">key1: </span><span class="n">value1</span><span class="p">,</span> <span class="ss">key2: </span><span class="n">value2</span><span class="p">}</span>
</code></pre></div></div>
<h3 id="pre-ruby-24">Pre Ruby 2.4</h3>
<p>Let’s first check how ruby implemented Hash in pre 2.4 old hash representation , hash table</p>
<p><img src="https://aboobacker.in/images/prehash.png" alt="Pre hash" /></p>
<p>Ruby internally uses a structure called st_table to represent hash. st_table contains type, the number of bins, the number of entries and pointers to bin array. Bin array is an array with 11 default size and can grow when required. Let’s take an example of hash and see how it will be represented using above diagram.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sample_hash</span> <span class="o">=</span> <span class="p">{</span><span class="ss">a: </span><span class="mi">10</span><span class="p">,</span> <span class="ss">b: </span><span class="mi">20</span><span class="p">,</span> <span class="ss">c: </span><span class="mi">30</span><span class="p">,</span> <span class="ss">d: </span><span class="mi">40</span><span class="p">,</span> <span class="ss">e: </span><span class="mi">50</span><span class="p">}</span>
</code></pre></div></div>
<p>Let’s take keys :c and :d</p>
<h4 id="step-1">Step 1:</h4>
<p>First thing ruby does is it will take the hash of the key using the internal hash function.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2.3.1 :075 > :c.hash
=> 2782
2.3.1 :076 > :d.hash
=> 2432
</code></pre></div></div>
<h4 id="step-2">Step 2:</h4>
<p>After getting the hash value, result ?it takes modulo of 11 to get figure in which bin ruby can store the given pair</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2.3.1 :073 > :c.hash % 11
=> 10
2.3.1 :074 > :d.hash % 11
=> 1
</code></pre></div></div>
<p>This means we can put :c => 30 in 10’th bin and :d in 1st bin</p>
<h4 id="step-3">Step 3</h4>
<p>What if there are multiple modulo operations that give the same result? This is called hash collision. To resolve this, ruby uses a separate chaining approach i.e, it will make a doubly linked list and add it to the existing value in the bin</p>
<h4 id="step-4">Step 4</h4>
<p>What if the hash is too large ?? Linked list will start growing and will make the hash slower. So, ruby will allocate more bins and do an operation called Rehashing to utilise newly added bins.</p>
<h5 id="improvements-in-20">Improvements in 2.0</h5>
<p>In ruby 2.0, Ruby eliminated the need for extra data structures for smaller hashes and uses linear search for better performance.</p>
<h5 id="improvements-in-22">Improvements in 2.2</h5>
<p>In 2.2.0, Ruby has used bin array sizes that correspond to powers of two (16, 32, 64, 128, …).</p>
<h3 id="changes-in-24">Changes in 2.4</h3>
<p>new hash structure in ruby 2.4 , hash table
<img src="https://aboobacker.in/images/prehash.png" alt="Pre hash" />
Source: https://github.com/ruby/ruby/blob/trunk/st.c</p>
<p>In ruby 2.4, Hash table is moved to open addressing model i.e, we no longer have the separate doubly linked list for collision resolution. Here, we will be storing records in the entries array itself i.e, there is no need of pointer chasing and data will be stored in the adjacent memory location (Data locality). The hash table has two arrays called bins and entries. Entry array contains hash entries in the order of insertion and the bin array provides access to entry the by their keys. The key hash is mapped to a bin containing the index of the corresponding entry in the entry array.</p>
<h3 id="inserting-entries-in-hash">Inserting entries in Hash</h3>
<h4 id="step-1-1">Step 1:</h4>
<p>Ruby will insert an entry to entries array in sequential order.</p>
<h4 id="step-2-1">Step 2:</h4>
<p>Ruby will identify the bin from which the entry is to be mapped. Ruby uses first few bits of the hash as the bin index, Explaining the whole process is beyond the scope of this article. You can check the logic in MRI source code <a href="https://github.com/ruby/ruby/blob/ruby_2_4/st.c#L962">here</a></p>
<h3 id="accessing-element-by-key">Accessing element by key</h3>
<p>Let’s examine it with a sample hash **</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sample_hash</span> <span class="o">=</span> <span class="p">{</span><span class="ss">a: </span><span class="mi">10</span><span class="p">,</span> <span class="ss">b: </span><span class="mi">20</span><span class="p">,</span> <span class="ss">c: </span><span class="mi">30</span><span class="p">,</span> <span class="ss">d: </span><span class="mi">40</span><span class="p">,</span> <span class="ss">e: </span><span class="mi">50</span><span class="p">}</span>
</code></pre></div></div>
<p>Here, ruby will create two internal arrays, entries and bins as shown below</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">entries</span> <span class="o">=</span> <span class="p">[[</span><span class="mi">2297</span><span class="p">,</span><span class="n">a</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span> <span class="p">[</span><span class="mi">451</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="mi">20</span><span class="p">],</span> <span class="p">[</span><span class="mi">2782</span><span class="p">,</span><span class="n">c</span><span class="p">,</span><span class="mi">30</span><span class="p">],</span> <span class="p">[</span><span class="mi">2432</span><span class="p">,</span><span class="n">d</span><span class="p">,</span><span class="mi">40</span><span class="p">],[</span><span class="mi">1896</span><span class="p">,</span><span class="n">e</span><span class="p">,</span><span class="mi">50</span><span class="p">]]</span>
</code></pre></div></div>
<p>each record in entries array will contain a hash, key, and value respectively</p>
<p>Default bin size in ruby is 16 so Bins array for the above hash will be somewhat like this</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">bins</span> <span class="o">=</span> <span class="p">[</span>
<span class="mi">3</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="mi">1</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="mi">5</span><span class="p">,</span>
<span class="mi">0</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="kp">nil</span><span class="p">,</span>
<span class="mi">2</span><span class="p">,</span>
<span class="kp">nil</span>
<span class="p">]</span>
</code></pre></div></div>
<p>Now what if we want to fetch an element from hash, say ‘c’</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sample_hash</span><span class="p">[</span><span class="ss">:c</span><span class="p">]</span>
</code></pre></div></div>
<h4 id="step-1-2">Step 1:</h4>
<p>Find the hash using ruby’s internal hash function</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:c.hash
2782
</code></pre></div></div>
<h4 id="step-2-2">Step 2</h4>
<p>Find the location in bin array by using find bin method</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find_bin(2782)
</code></pre></div></div>
<h4 id="step-3-1">Step 3</h4>
<p>Find the location entries array using bin array</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bins[14] => 2
</code></pre></div></div>
<h4 id="step-4-find-the-entry-using-the-key-we-got">Step 4. Find the entry using the key we got</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>entries[2] => [2782,c,30]
</code></pre></div></div>
<h3 id="deleting-an-item">Deleting an item</h3>
<p>Now we have value for the key ‘c’ without iterating through all the records
Deleting an item</p>
<p>If the item to be deleted is first one, then ruby will change the index of ‘current first entry ‘, otherwise ruby will mark the item as reserved using a reserved hash value.</p>
<p>In the ruby source code, DELETED is marked using 0 and EMPTY is marked using 1.</p>
<p>To summarise this approach made hash implementation in ruby faster because the new bins array stores much smaller reference to the entries instead of storing entries themselves. Hence, it can take advantage of the modern processor caching levels</p>
<p>** Small hashes will use the linear search to find entries from ruby 2.0 onwards to avoid extra overhead and improve performance. Given example is just for reference only.</p>
<p>References</p>
<ul>
<li><a href="">https://bugs.ruby-lang.org/issues/12142</a></li>
<li><a href="">https://blog.heroku.comruby-2-4-features-hashes-integers-rounding#hash-changes</a></li>
<li><a href="">https://en.wikipedia.org/wiki/Hash_table</a></li>
<li><a href="">http://patshaughnessy.net/ruby-under-a-microscope</a></li>
</ul>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inRuby 2.4 got released this Christmas with lot of exciting features. One of the most underrated features in ruby 2.4 is hash table improvements. Before going into details about implementation, let’s first check the benchmark to know how this change gonna affect your ruby application.Deploying Fedena Using Ubuntu 14.042015-11-18T04:00:13+00:002015-11-18T04:00:13+00:00https://aboobacker.in/2015/11/18/deploying-fedena-using-ubuntu-14-04<ul>
<li>Install Ruby Dependancies</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties
</code></pre></div></div>
<ul>
<li>Install Ruby Using RVM</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>libgdbm-dev libncurses5-dev automake libtool bison libffi-dev
curl <span class="nt">-L</span> https://get.rvm.io | bash <span class="nt">-s</span> stable
<span class="nb">source</span> ~/.rvm/scripts/rvm
<span class="nb">echo</span> <span class="s2">"source ~/.rvm/scripts/rvm"</span> <span class="o">>></span> ~/.bashrc
rvm <span class="nb">install </span>1.8.7
rvm use 1.8.7 <span class="nt">--default</span>
ruby <span class="nt">-v</span>
</code></pre></div></div>
<ul>
<li>Setting up MySQL server
Fedena uses mysql, so run,</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>libmysqlclient-dev mysql-server
</code></pre></div></div>
<p>Do remember the mysql password you set during this step, it is required later</p>
<ul>
<li>Checking Out modified version version of fedena</li>
</ul>
<p>I made some modifications in project fedena to make it easily installable</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
git clone https://github.com/tachyons/project_fedena.git
<span class="nb">cd </span>project_fedena
<span class="nb">cp </span>config/database.yml.example config/database.yml
bundle <span class="nb">install
</span>rake db:create
bundle <span class="nb">exec </span>rake fedena:plugins:install_all
</code></pre></div></div>
<ul>
<li>Installing passenger and nginix</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install passenger PGP key and add HTTPS support for APT</span>
<span class="nb">sudo </span>apt-key adv <span class="nt">--keyserver</span> hkp://keyserver.ubuntu.com:80 <span class="nt">--recv-keys</span> 561F9B9CAC40B2F7
<span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">-y</span> apt-transport-https ca-certificates
<span class="c"># Add passenger APT repository</span>
<span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main > /etc/apt/sources.list.d/passenger.list'</span>
<span class="nb">sudo </span>apt-get update
<span class="c"># Install Passenger + Nginx</span>
<span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">-y</span> nginx-extras passenger
</code></pre></div></div>
<p>Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby. For example, you may see this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># passenger_root /some-filename/locations.ini;</span>
<span class="c"># passenger_ruby /usr/bin/passenger_free_ruby;</span>
</code></pre></div></div>
<p>Remove the ‘#’ characters, like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>passenger_root /some-filename/locations.ini<span class="p">;</span>
passenger_ruby /usr/bin/passenger_free_ruby<span class="p">;</span>
</code></pre></div></div>
<p>Now you can validate the installation using the command</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>passenger-config validate-install
</code></pre></div></div>
<p>now create an entry in /etc/ngnix/sites.enabled</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>vim /etc/nginx/sites-enabled/project_fedena
</code></pre></div></div>
<p>Now modify and paste following text,ie edit your server name and project path</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>server <span class="o">{</span>
listen 80<span class="p">;</span>
server_name fedena.example.com<span class="p">;</span>
location / <span class="o">{</span>
root /your/fedena/directory/public<span class="p">;</span>
passenger_enabled on<span class="p">;</span>
<span class="o">}</span>
error_page 500 502 503 504 /50x.html<span class="p">;</span>
location <span class="o">=</span> /50x.html <span class="o">{</span>
root html<span class="p">;</span>
<span class="o">}</span>
error_page 413 /413.html<span class="p">;</span>
location <span class="o">=</span> /413.html<span class="o">{</span>
root html<span class="p">;</span>
allow all<span class="p">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>you can verify above configuration by adding <code class="language-plaintext highlighter-rouge">127.0.1.1 fedena.example.com
</code> in <code class="language-plaintext highlighter-rouge">/etc/hosts</code></p>
<p>Now restart the ngnix server and check fedena.example.com</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>service ngnix restart
</code></pre></div></div>
<h2 id="references">References</h2>
<ul>
<li>https://www.phusionpassenger.com/library/install/nginx/install/oss/trusty/</li>
<li>http://aboobacker.in/installing-project-fedena-in-14-04/</li>
</ul>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inInstall Ruby DependanciesInstalling Project Fedena in Ubuntu 14.042014-10-30T11:06:20+00:002014-10-30T11:06:20+00:00https://aboobacker.in/2014/10/30/installing-project-fedena-in-14-04<p>Modified version of Fedena Installation guide at project fedena website</p>
<ul>
<li>Install Ruby Dependancies</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties
</code></pre></div></div>
<ul>
<li>Install Ruby Using RVM</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>libgdbm-dev libncurses5-dev automake libtool bison libffi-dev
curl <span class="nt">-L</span> https://get.rvm.io | bash <span class="nt">-s</span> stable
<span class="nb">source</span> ~/.rvm/scripts/rvm
<span class="nb">echo</span> <span class="s2">"source ~/.rvm/scripts/rvm"</span> <span class="o">>></span> ~/.bashrc
rvm <span class="nb">install </span>1.8.7
rvm use 1.8.7 <span class="nt">--default</span>
ruby <span class="nt">-v</span>
</code></pre></div></div>
<ul>
<li>Install Rails 2.3.5</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem <span class="nb">install </span>rails <span class="nt">-v</span> 2.3.5
</code></pre></div></div>
<ul>
<li>Setting up MySQL server
Fedena uses mysql, so run,</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>libmysqlclient-dev mysql-server
</code></pre></div></div>
<p>Do remember the mysql password you set during this step, it is required later</p>
<ul>
<li>Clone Fedena Source Code
```bash</li>
</ul>
<p>git clone https://github.com/projectfedena/fedena.git</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
* Setup your database details in the database.yml
Open the file database.yml in the config folder of the fedena soucre. Change the following details:
database: fedena - The name of the database you want to use for fedena
username: root - Mysql username for fedena
password: mypass - The password for the above mysql user
* Install the rest of the gems
```bash
gem uninstall -i ~/.rvm/gems/ruby-1.8.7-head@global rake
gem install rake -v 0.8.7
gem install declarative_authorization -v 0.5.1
gem install i18n -v 0.4.2
gem install mysql
gem install rush -v 0.6.8
gem update --system 1.3.7
</code></pre></div></div>
<ul>
<li>Set up Fedena databases</li>
</ul>
<p>From the Fedena source directory in terminal run,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rake db:create
rake db:migrate
</code></pre></div></div>
<p>followed by,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rake fedena:plugins:install_all
</code></pre></div></div>
<ul>
<li>Set up pdf setings</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>wkhtmltopdf
<span class="nb">cd </span>config/initializers
<span class="nb">cp </span>wicked_pdf.rb.example wicked_pdf.rb
vi wicked_pdf.rb
</code></pre></div></div>
<p>now change :wkhtmltopdf => ‘/opt/wkhtmltopdf’, to :wkhtmltopdf => ‘/usr/bin/wkhtmltopdf’,
save the file</p>
<ul>
<li>Setup Email</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>config
<span class="nb">cp </span>smtp_settings.yml.example smtp_settings.yml
vi smtp_settings.yml
</code></pre></div></div>
<p>Add your settings and save the file</p>
<ul>
<li>Setup Sms</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>config
vi sms_settings.yml
</code></pre></div></div>
<p>Add your settings and save the file</p>
<ul>
<li>Change permissions for scripts</li>
</ul>
<p>From the same directory grant executable permissions for the files in script directory by,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">chmod</span> +x script/<span class="k">*</span>
</code></pre></div></div>
<ul>
<li>Run the inbuilt server</li>
</ul>
<p>If everything went fine till now, you are ready to run fedena server by running the following from fedena source folder,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>script/server
</code></pre></div></div>
<p>Note: This guide is for setting development environment , for fedena deployment refer <a href="http://aboobacker.in/deploying-fedena-using-ubuntu-14-04/">Deploying Fedena in 14.04</a>.</p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inModified version of Fedena Installation guide at project fedena websiteApertium Malayalam-english Pair Tools Usage and Directory Structure2014-08-20T02:03:25+00:002014-08-20T02:03:25+00:00https://aboobacker.in/2014/08/20/apertium-malayalam-english-pair-tools-usage-and-directory-structure<p>Apertium malayalam english pair contain two directories</p>
<ol>
<li>apertium-mal -> conatins malayalam specific rules and resources</li>
<li>apertium-mal-eng ->contains translation rules</li>
</ol>
<ul>
<li>Apertium-mal</li>
</ul>
<ol>
<li>apertium-mal.mal.lexc -> contain mono lingual dictionary in lexec format , it is contain morphotactics rules</li>
<li>apertium-mal.mal.twol -> contain morphophonology rules (changes when morphemes are joined together)</li>
<li>apertium-mal.mal.tsx ->tag set to train the tagger</li>
<li>apertium-mal.mal.rlx -> constarint grammar rules</li>
</ol>
<ul>
<li>Apertium-mal-eng</li>
</ul>
<ol>
<li>apertium-mal-eng.eng.dix -> mono lingual dictionary for english in lttoolbox format</li>
<li>apertium-mal-eng.post-eng.dix -> English post generator</li>
<li>apertium-mal-eng.eng-mal.t1x – > english to malayalam chunker</li>
<li>apertium-mal-eng.eng-mal.t2x -> english to malayalam interchunk</li>
<li>apertium-mal-eng.eng-mal.t3x ->english to malayalam post chunk</li>
</ol>
<h2 id="translation">Translation</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>apertium-mal-eng
<span class="nb">echo</span> “മലയാളം വാക്യം ” | apertium <span class="nt">-d</span> <span class="nb">.</span> mal-eng
</code></pre></div></div>
<p>where</p>
<p>-d . -> directory=current directory</p>
<p>mal-eng -> malayalam english mode</p>
<h2 id="morphological-analyser">Morphological analyser</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> “മലയാളം “| lt-proc mal-eng.automorf.bin
</code></pre></div></div>
<h2 id="gui-for-translator">GUI for translator</h2>
<p><img src="https://aboobacker.in/images/mltranslator.png" alt="Qt GUI" /></p>
<h2 id="step-by-step-process-view-developer--modes-viewer">Step by step process view (Developer ->Modes viewer)</h2>
<p><img src="https://aboobacker.in/images/modes.png" alt="Modes viewer" /></p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inApertium malayalam english pair contain two directoriesInstall Apertium Malayalam-english Pair Development Environment in Ubuntu2014-08-20T01:43:36+00:002014-08-20T01:43:36+00:00https://aboobacker.in/2014/08/20/install-apertium-malayalam-english-pair-development-environment-in-ubuntu<p>Apertium is an open source machine translation toolkit .It is an extensible platform so that new language pairs can be added easily . It is currently in incubator state , we have to do a lot to to make it useful for translation of real life examples . I created ppas for the tools to make the process easier</p>
<p>Platform : ubuntu :14.04</p>
<ul>
<li>Add repositories</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget http://apertium.projectjj.com/apt/install-nightly.sh <span class="nt">-O</span> - | <span class="nb">sudo </span>bash
<span class="nb">sudo </span>add-apt-repository ppa:tinodidriksen/cg3
</code></pre></div></div>
<ul>
<li>Install tools</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nt">-f</span> <span class="nb">install </span>locales build-essential automake subversion pkg-config gawk libtool apertium-all-dev
</code></pre></div></div>
<ul>
<li>Install English Malayalam pair data</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
svn co https://svn.code.sf.net/p/apertium/svn/incubator/apertium-mal-eng/
svn co https://svn.code.sf.net/p/apertium/svn/incubator/apertium-mal
</code></pre></div></div>
<ul>
<li>Compile</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> ~/apertium-mal
./autogen.sh
make
<span class="nb">cd</span> ~/apertium-mal-eng
./autogen.sh <span class="nt">--with-lang1</span><span class="o">=</span>../apertium-mal
make
</code></pre></div></div>
<ul>
<li>Installation is over , now you can test the system using</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">" അവന് നല്ല കുട്ടിയാണ് "</span> | apertium <span class="nt">-d</span> <span class="nb">.</span> mal-eng
</code></pre></div></div>
<p>This will print
he is nice child</p>
<ul>
<li>Prefer Graphical user interface ? , you can try my simple gui for it</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>qt5-default espeak
git clone https://github.com/tachyons/mltranslator.git
<span class="nb">cd </span>mltranslator
qmake
make
<span class="nb">sudo </span>make <span class="nb">install</span>
</code></pre></div></div>
<p>Stuck ? Don’t worry just comment below . :-)</p>Aboobacker MKaboobackervyd@gmail.comhttps://aboobacker.inApertium is an open source machine translation toolkit .It is an extensible platform so that new language pairs can be added easily . It is currently in incubator state , we have to do a lot to to make it useful for translation of real life examples . I created ppas for the tools to make the process easier