Roman’s blog2014-12-20T01:16:07+00:00http://www.rovrov.comRoman Lavrovblah@email.testPass By Rvalue Reference Or Pass By Value2014-12-19T23:00:00+00:00http://www.rovrov.com/blog/2014/12/19/pass-by-rvalue-reference-or-pass-by-value
<p>Passing by rvalue reference is typically an optimization for the case when data can be “stolen” from the parameter instead of copying from it.</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">S</span>
<span class="p">{</span>
<span class="kt">void</span> <span class="n">init</span><span class="p">(</span><span class="k">const</span> <span class="n">SomeType</span><span class="o">&</span> <span class="n">param</span><span class="p">);</span> <span class="c1">// copy from param</span>
<span class="kt">void</span> <span class="nf">init</span><span class="p">(</span><span class="n">SomeType</span><span class="o">&&</span> <span class="n">param</span><span class="p">);</span> <span class="c1">// steal from param</span>
<span class="p">};</span></code></pre></div>
<p>People noticed that if <code>SomeType</code> is cheaply movable, then we don’t have to write two overloads and can get away with a single pass-by-value:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">S</span>
<span class="p">{</span>
<span class="kt">void</span> <span class="n">initByVal</span><span class="p">(</span><span class="n">SomeType</span> <span class="n">param</span><span class="p">);</span> <span class="c1">// can now steal from param</span>
<span class="p">};</span>
<span class="n">SomeType</span> <span class="n">t</span><span class="p">;</span>
<span class="n">s</span><span class="p">.</span><span class="n">initByVal</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">t</span><span class="p">));</span> <span class="c1">// move data to param, then can steal from it</span></code></pre></div>
<p>Compared to the first version, this will result in one more invocation of SomeType’s move constructor to construct <code>param</code>, but as it is cheap to move it does not matter (much).</p>
<p>It is often mentioned that the first approach would result in <code>2^N</code> overloads in case of <code>N</code> arguments, and passing by value magically solves this issue. However, passing by reference solves it just as well (see below). Also, passing by value <a href="http://stackoverflow.com/questions/24814696/move-semantics-and-function-order-evaluation">can easily lead to unspecified behavior</a> and, in my opinion, this alone makes it worth considering alternative approaches.</p>
<!-- more -->
<p>First, note that similarly to <em>only passing by value</em> we might equally well <em>only pass by rvalue reference</em>:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">S</span>
<span class="p">{</span>
<span class="kt">void</span> <span class="n">initByRef</span><span class="p">(</span><span class="n">SomeType</span><span class="o">&&</span> <span class="n">param</span><span class="p">);</span>
<span class="p">};</span>
<span class="n">SomeType</span> <span class="n">t</span><span class="p">;</span>
<span class="n">s</span><span class="p">.</span><span class="n">initByRef</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">t</span><span class="p">));</span> <span class="c1">// this works, and even better (less moves)</span>
<span class="n">s</span><span class="p">.</span><span class="n">initByRef</span><span class="p">(</span><span class="n">SomeType</span><span class="p">());</span> <span class="c1">// this also works</span></code></pre></div>
<p>The only case when it “does not work” is when the argument passed to <code>initByRef()</code> is an lvalue. Note, however, that in this case the pass-by-value approach will do no magic neither - it will copy-construct (not move!) <code>param</code> and this copy cannot be elided. The question of the day is: do you actually want such copies to be created implicitly? If the code above was <code>initByVal(t);</code>, did the author actually mean to copy the object or did they simply forget the <code>std::move()</code>?</p>
<p>Anyway, how to deal with non-movable lvalues? Apart from <code>std::move()</code>, there is a second simple way of turning an lvalue into an rvalue: build a temporary object. Actually, we are already doing this implicitly when passing by value, and of course nothing stops us from doing this explicitly now:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="o">></span>
<span class="n">T</span> <span class="n">copy_to_temp</span><span class="p">(</span><span class="k">const</span> <span class="n">T</span><span class="o">&</span> <span class="n">t</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">t</span><span class="p">;</span> <span class="p">}</span>
<span class="n">SomeType</span> <span class="n">t</span><span class="p">;</span> <span class="c1">// I'll need t later, so cannot std::move(t)</span>
<span class="n">s</span><span class="p">.</span><span class="n">initByRef</span><span class="p">(</span><span class="n">copy_to_temp</span><span class="p">(</span><span class="n">t</span><span class="p">));</span> <span class="c1">// but can copy</span>
<span class="n">s</span><span class="p">.</span><span class="n">initByVal</span><span class="p">(</span><span class="n">t</span><span class="p">);</span> <span class="c1">// note that here t is also copied, it just happens implicitly</span></code></pre></div>
<p>While <code>copy_to_temp</code> may not be a great name, I picked it just to show how exactly the same idea can be applied to move-only types (such as <code>unique_ptr<T></code>). Obviously, we cannot <code>copy_to_temp</code> something that is not copyable, but then we can <code>move_to_temp</code> just as simply:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="o">></span>
<span class="n">T</span> <span class="n">move_to_temp</span><span class="p">(</span><span class="n">T</span><span class="o">&</span> <span class="n">t</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">t</span><span class="p">);</span> <span class="p">}</span>
<span class="kt">void</span> <span class="n">foo</span><span class="p">(</span><span class="n">unique_ptr</span><span class="o"><</span><span class="n">SomeType</span><span class="o">>&&</span> <span class="n">param</span><span class="p">);</span>
<span class="n">unique_ptr</span><span class="o"><</span><span class="n">SomeType</span><span class="o">></span> <span class="n">p</span><span class="p">;</span>
<span class="n">foo</span><span class="p">(</span><span class="n">move_to_temp</span><span class="p">(</span><span class="n">p</span><span class="p">));</span> <span class="c1">// note: this does move, p will become empty</span></code></pre></div>
<p>But wait a second - why moving to a temporary when we can simply turn it into rvalue <code>foo(std::move(p));</code>? Does not it do the same thing but just less efficiently? Well, it turns out there is a subtle difference: <code>foo(move_to_temp(p))</code> guarantees that after the operation <code>p</code> will be empty (because it was “moved from”). Herb Sutter called this “the killer argument” for passing by value vs passing by reference in the comments to Scott Meyers’ <a href="http://scottmeyers.blogspot.ca/2014/07/should-move-only-types-ever-be-passed.html">Should move-only types ever be passed by value?</a>. Well, here is a way to provide the same guarantee when passing by reference <em>if this actually is a requirement</em>: <code>move_to_temp</code>.</p>
<p>On a side note, this is a very weak guarantee. All you get is that data was “moved from” the object that was passed to the function. The data might be copied somewhere, but it might as well be just ignored and destroyed when the temporary goes out of scope. My expectation is that in most cases <code>foo(std::move(p))</code> will be just as good as <code>foo(move_to_temp(p))</code> because <em>no one will notice the difference</em>.</p>
<p>To sum up, here is what <em>only passing by reference</em> does compared to <em>only passing by value</em>:</p>
<ol>
<li>Saves one move construction (modulo move elisions)</li>
<li>Avoids the mentioned <a href="http://stackoverflow.com/questions/24814696/move-semantics-and-function-order-evaluation">unspecified behavior situation</a></li>
<li>Makes copies as explicit as moves. When you pass an lvalue to the function accepting rvalue references only, you get a compilation error. Is it a bad thing? In my opinion it is actually good - it forces you to decide whether you mean to <em>move</em> or to <em>copy</em> and indicate it explicitly for better maintainability.</li>
<li>Turns adding a <code>const T&</code> overload into an <em>optimization for the case when you cannot move</em>.</li>
</ol>
Limiting File Download Speed In C#2014-12-06T17:00:00+00:00http://www.rovrov.com/blog/2014/12/06/limiting-file-download-speed-in-c#
<p>Apparently, limiting file download speed it is not straightforward with the standard web helper classes provied by .NET - <a href="http://www.codeproject.com/Questions/163737/Webclient-download-speed-limit">WebClient</a> or <a href="http://stackoverflow.com/questions/15449147/limiting-httpwebresponse-stream-reading-speed">HttpWebResponse</a>. Someone even re-implemented a <a href="http://www.codeproject.com/Articles/100769/Socket-based-HTTP-Client-with-Bandwidth-Limit">socket-based HTTP client</a> to add this feature. While the latter might be a solution, relying on an incomplete and potentially buggy HTTP implementation is not something I like. <a href="http://en.wikipedia.org/wiki/CURL#libcurl">Libcurl</a>, on the other hand, has been around for ages and also has an option to limit the download speed. The library is ported to many OSes including Windows and wrapped for .NET as well, so I gave it a try.</p>
<!-- more -->
<p><a href="https://github.com/masroore/CurlSharp">CurlSharp</a>, an improved fork of the official <a href="http://sourceforge.net/projects/libcurl-net/">libcurl.NET</a>, looked like a good start. The use is pretty straightforward - the wrapper comes with a few examples and pre-compiled libcurl binaries including dependencies:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">libcurl.dll libeay32.dll msvcr120.dll zlib1.dll
libcurlshim.dll libssh2.dll ssleay32.dll</code></pre></div>
<p>The curl option for limiting download speed was missing from the wrapper at the time of writing, but it can be easily added. According to <a href="http://curl.haxx.se/libcurl/c/CURLOPT_MAX_RECV_SPEED_LARGE.html">CURLOPT_MAX_RECV_SPEED_LARGE docs</a>, the corresponding data type is <code>curl_off_t</code> which <a href="http://curl.haxx.se/dev/readme-curl_off_t.html">seems to match</a> the signed 64-bit type, i.e. C# <code>long</code>.</p>
<div class="highlight"><pre><code class="language-c#" data-lang="c#"><span class="c1">// ----- CurlEasy.cs -----</span>
<span class="k">private</span> <span class="kt">long</span> <span class="n">_maxRecvSpeedLarge</span><span class="p">;</span>
<span class="k">public</span> <span class="kt">long</span> <span class="n">MaxRecvSpeedLarge</span>
<span class="p">{</span>
<span class="k">get</span> <span class="p">{</span> <span class="k">return</span> <span class="n">_maxRecvSpeedLarge</span><span class="p">;</span> <span class="p">}</span>
<span class="k">set</span> <span class="p">{</span> <span class="n">setLongOption</span><span class="p">(</span><span class="n">CurlOption</span><span class="p">.</span><span class="n">MaxRecvSpeedLarge</span><span class="p">,</span> <span class="k">ref</span> <span class="n">_maxRecvSpeedLarge</span><span class="p">,</span> <span class="k">value</span><span class="p">);</span> <span class="p">}</span>
<span class="p">}</span>
<span class="c1">// ----- CurlOption.cs -----</span>
<span class="n">MaxRecvSpeedLarge</span> <span class="p">=</span> <span class="m">30146</span><span class="p">,</span> <span class="c1">// CURLOPT_MAX_RECV_SPEED_LARGE</span></code></pre></div>
<p>Now this can be used, for example in the <a href="https://github.com/masroore/CurlSharp/blob/master/Samples/EasyGet/EasyGet.cs">EasyGet</a> sample:</p>
<div class="highlight"><pre><code class="language-c#" data-lang="c#"><span class="n">easy</span><span class="p">.</span><span class="n">BufferSize</span> <span class="p">=</span> <span class="m">4096</span><span class="p">;</span>
<span class="n">easy</span><span class="p">.</span><span class="n">MaxRecvSpeedLarge</span> <span class="p">=</span> <span class="m">10</span><span class="p">*</span><span class="m">1024</span><span class="p">;</span></code></pre></div>
<p>Setting a buffer size smaller than default makes it easier to see the data chunks going in at the reduced rate. Also, as the speed limit is most likely implemented through throttling, a big buffer with a small speed limit would yield in a large delay between downloading the chunks and the connection might timeout.</p>
<p>It is worth noting that when the speed is limited this way, the actual network usage is likely to be non-constant over time (contain bursts) because of the additional network buffering by OS. For example, here is what it looks like on my Windows 8.1 laptop:</p>
<p><img src="/files/2014-12-06-limiting-file-download-speed-in-csharp/network_usage.png" class="imgmb" alt="Network usage" /></p>
Rvo And Copy Elision Failing2014-11-21T20:00:00+00:00http://www.rovrov.com/blog/2014/11/21/RVO-and-copy-elision-failing
<p>Following this interesting <a href="http://stackoverflow.com/questions/25963685/why-does-visual-studio-not-perform-return-value-optimization-rvo-in-this-case/">StackOverflow question about RVO failure</a>, I’ve experimented a little to see in which cases copies are elided by the compiler. The question is about RVO failing in case when NRVO is performed properly; it is hard to explain <em>why</em> this happens without knowing the internals of the MSVC compiler, but here are a few observations on <em>when</em> this happens.</p>
<!-- more -->
<p>The same behavior can be reproduced with simple classes instead of the rather complex <code>std::vector</code>:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="cp">#include <iostream></span>
<span class="k">struct</span> <span class="n">C</span>
<span class="p">{</span>
<span class="n">C</span><span class="p">(</span><span class="k">const</span> <span class="n">C</span><span class="o">&</span> <span class="n">in</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"C(&) <-- copy</span><span class="se">\n</span><span class="s">"</span><span class="p">;}</span>
<span class="n">C</span><span class="p">(</span><span class="n">C</span><span class="o">&&</span> <span class="n">in</span><span class="p">)</span> <span class="p">{}</span>
<span class="n">C</span><span class="p">()</span> <span class="p">{}</span>
<span class="o">~</span><span class="n">C</span><span class="p">()</span> <span class="p">{}</span>
<span class="p">};</span>
<span class="k">struct</span> <span class="n">D</span>
<span class="p">{</span>
<span class="n">C</span> <span class="n">c</span><span class="p">;</span>
<span class="n">D</span><span class="p">(</span><span class="n">C</span> <span class="n">_c</span><span class="p">)</span> <span class="o">:</span> <span class="n">c</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">_c</span><span class="p">))</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"D(C)</span><span class="se">\n</span><span class="s">"</span><span class="p">;}</span>
<span class="p">};</span>
<span class="n">D</span> <span class="nf">test_RVO</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">C</span> <span class="n">c</span><span class="p">;</span>
<span class="k">return</span> <span class="n">D</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">c</span><span class="p">));</span>
<span class="p">}</span>
<span class="n">D</span> <span class="nf">test_NRVO</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">C</span> <span class="n">c</span><span class="p">;</span>
<span class="n">D</span> <span class="n">d</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">c</span><span class="p">));</span>
<span class="k">return</span> <span class="n">d</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"RVO</span><span class="se">\n</span><span class="s">"</span><span class="p">;</span>
<span class="n">test_RVO</span><span class="p">();</span> <span class="c1">// or auto val = test_RVO();</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"</span><span class="se">\n</span><span class="s">NRVO</span><span class="se">\n</span><span class="s">"</span><span class="p">;</span>
<span class="n">test_NRVO</span><span class="p">();</span>
<span class="p">}</span></code></pre></div>
<p>When built and run with MSVC 2013 with optimizations enabled, it produces the following output:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">RVO
D<span class="o">(</span>C<span class="o">)</span>
C<span class="o">(</span><span class="p">&</span><span class="o">)</span> <-- copy
NRVO
D<span class="o">(</span>C<span class="o">)</span></code></pre></div>
<p><a href="http://rextester.com/YJL85582">try online</a> (as long as their compiler is not upgraded to 2014+)</p>
<p>In <code>test_NRVO()</code>, <code>D</code> is not copied due to the NRVO optimization, but the equivalent RVO optimization fails for <code>test_RVO()</code>. Note that in debug mode (<code>/Od</code>) the results might be different as some optimizations are disabled and not all copies are elided. This has been changed in MSVC 2014/15 where even in debug mode all copies seem to be elided (this RVO behavior is fixed too). Gcc and clang elide all copies including the non-optimized builds (there is <code>-fno-elide-constructors</code> option though).</p>
<h3 id="taking-a-closer-look">Taking a closer look</h3>
<p>First, note that the same issue occurs when instead of <code>std::move</code> the object is passed directly to the D’s constructor:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="n">D</span> <span class="nf">test_RVO</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">D</span><span class="p">(</span><span class="n">C</span><span class="p">());</span>
<span class="p">}</span></code></pre></div>
<p>Outputs</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">D<span class="o">(</span>C<span class="o">)</span>
C<span class="o">(</span><span class="p">&</span><span class="o">)</span> <-- copy</code></pre></div>
<p>Next, adding traces to move constructors</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="c1">//...</span>
<span class="n">C</span><span class="p">(</span><span class="n">C</span><span class="o">&&</span> <span class="n">in</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"C(&&)</span><span class="se">\n</span><span class="s">"</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">//...</span>
<span class="n">D</span><span class="p">(</span><span class="n">D</span><span class="o">&&</span> <span class="n">in</span><span class="p">)</span> <span class="o">:</span> <span class="n">c</span><span class="p">(</span><span class="n">in</span><span class="p">.</span><span class="n">c</span><span class="p">)</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"D(&&)</span><span class="se">\n</span><span class="s">"</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">//...</span></code></pre></div>
<p>produces the following ouput for <code>D(C())</code></p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">RVO
C<span class="o">(&&)</span>
D<span class="o">(</span>C<span class="o">)</span>
C<span class="o">(</span><span class="p">&</span><span class="o">)</span> <-- copy
D<span class="o">(&&)</span></code></pre></div>
<p><code>C(&&)</code> is printed from the <code>D(C _c)</code> constructor as per <code>c(std::move(_c))</code>, which is then followed by the constructor’s <code>D(C)</code> trace. Then, however, the <code>D</code> instance is being moved when returning from <code>test_RVO()</code>. The move in its turn calls <code>c(in.c)</code> and this invokes <code>C</code>’s copy constructor. This <code>C</code> copy could be turned into a move by doing <code>c(std::move(in.c))</code> in the <code>D(D&&)</code> constructor, but this is somewhat irrelevant: the question is why the whole <code>D</code>’s move is not elided by RVO in the first place. NRVO does elide it (there is no <code>D(&&)</code>):</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">NRVO
C<span class="o">(&&)</span>
C<span class="o">(&&)</span>
D<span class="o">(</span>C<span class="o">)</span></code></pre></div>
<h3 id="constructor-parameter-type-matters">Constructor parameter type matters</h3>
<p>It turns out that the reason of the RVO failure is linked to the properties of the type that is being passed in the constructor <code>D(C _c)</code>. For example, using a different type in the constructor magically “fixes” RVO:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">X</span>
<span class="p">{</span>
<span class="p">};</span>
<span class="k">struct</span> <span class="n">D</span>
<span class="p">{</span>
<span class="c1">//...</span>
<span class="n">D</span><span class="p">(</span><span class="n">X</span> <span class="n">_x</span><span class="p">)</span> <span class="o">:</span> <span class="n">c</span><span class="p">()</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"D(X)</span><span class="se">\n</span><span class="s">"</span><span class="p">;</span> <span class="p">}</span>
<span class="p">};</span>
<span class="n">D</span> <span class="nf">test_RVO</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">D</span><span class="p">(</span><span class="n">X</span><span class="p">());</span>
<span class="p">}</span></code></pre></div>
<p>Outputs</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">RVO
D<span class="o">(</span>X<span class="o">)</span></code></pre></div>
<p>There is no <code>D(&&)</code> meaning no move is performed, and RVO works. How is <code>D(X _x)</code> different from <code>D(C _c)</code>? Let’s add a few things to <code>X</code>:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">X</span>
<span class="p">{</span>
<span class="n">X</span><span class="p">()</span> <span class="p">{}</span>
<span class="o">~</span><span class="n">X</span><span class="p">()</span> <span class="p">{}</span>
<span class="n">X</span><span class="p">(</span><span class="n">X</span> <span class="k">const</span><span class="o">&</span><span class="p">)</span> <span class="p">{}</span>
<span class="p">};</span></code></pre></div>
<p>And now RVO suddenly fails:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">RVO
D<span class="o">(</span>X<span class="o">)</span>
C<span class="o">(</span><span class="p">&</span><span class="o">)</span> <-- copy
D<span class="o">(&&)</span></code></pre></div>
<p>Wait. What?</p>
<p>It turns out that RVO is performed correctly with the <code>D(X)</code> constructor, but it starts failing when <code>(-a- and (-b- or -c-))</code> are uncommented below:</p>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">X</span>
<span class="p">{</span>
<span class="n">X</span><span class="p">()</span> <span class="p">{}</span>
<span class="c1">//~X() {} -a-</span>
<span class="c1">//X(X const&) {} -b-</span>
<span class="c1">//X(X&&) {} -c-</span>
<span class="p">};</span></code></pre></div>
<p>In other words, <strong>a constructor of type <code>D</code> with signature <code>D(T)</code> leads to RVO failure when used as above, when the type <code>T</code> has a non-default destructor and a non-default copy (or move) constructor</strong>. Certainly, this applies to that SO example where <code>T</code> is <code>std::vector</code> implementing all of those methods.</p>
<p>A couple of final notes:</p>
<ul>
<li>
<p>This is not specific to values returned from functions. For example, the same applies to expressions like <code>D d = D(T());</code>, where MSVC <2014 does not elide the copy/move (under the same conditions).</p>
</li>
<li>
<p>There is no RVO failure when the constructor is replaced with a static method returning by value:</p>
</li>
</ul>
<div class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="k">struct</span> <span class="n">D</span>
<span class="p">{</span>
<span class="c1">//...</span>
<span class="k">static</span> <span class="n">D</span> <span class="n">D</span><span class="o">::</span><span class="n">construct</span><span class="p">(</span><span class="n">X</span> <span class="n">_x</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">D</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">D</span> <span class="nf">test_RVO</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">D</span><span class="o">::</span><span class="n">construct</span><span class="p">(</span><span class="n">X</span><span class="p">());</span> <span class="c1">// <-- RVO is performed correctly</span>
<span class="c1">// return D(X()); // <-- but not this way</span>
<span class="p">}</span></code></pre></div>
Objects As Data In D32014-11-13T16:18:00+00:00http://www.rovrov.com/blog/2014/11/13/objects-as-data-in-d3
<p><a href="http://d3js.org/">D3</a> uses a very powerful data binding approach. However, many D3 examples only show how to deal with very simple data, such as <code>[1,2,3]</code>. Let’s have a look at what happens when data items are objects, for example</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span><span class="nx">x</span><span class="o">:</span> <span class="mi">70</span><span class="p">,</span> <span class="nx">y</span><span class="o">:</span> <span class="mi">20</span><span class="p">,</span> <span class="nx">label</span><span class="o">:</span> <span class="s2">"first"</span><span class="p">},</span>
<span class="p">{</span><span class="nx">x</span><span class="o">:</span> <span class="mi">10</span><span class="p">,</span> <span class="nx">y</span><span class="o">:</span> <span class="mi">60</span><span class="p">,</span> <span class="nx">label</span><span class="o">:</span> <span class="s2">"second"</span><span class="p">}</span>
<span class="p">];</span></code></pre></div>
<!-- more -->
<p>The binding itself does not change when going from numbers to objects - just bind data to your selection as before and go ahead with the usual D3 functions-as-values approach</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">selection</span>
<span class="p">.</span><span class="nx">data</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span>
<span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s2">"x"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">x</span><span class="p">;</span> <span class="p">});</span></code></pre></div>
<p>Inspecting a DOM element from the selection and navigating to its properties shows that the <code>__data__</code> property is set to the corresponding item from the list (this is how d3 data binding internally works):</p>
<p><img src="/files/2014-11-13-objects-as-data-in-d3/data_devtools.png" class="imgmb" alt="Data in developer tools" /></p>
<p>Now let’s visualize our data as <code>SVG Text</code> objects with text coming from <code>d.label</code> positioned at <code>(d.x, d.y)</code>, and add a couple of transitions:</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">function</span> <span class="nx">render</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">svg</span> <span class="o">=</span> <span class="nx">d3</span><span class="p">.</span><span class="nx">select</span><span class="p">(</span><span class="s2">"#viz svg"</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">ditems</span> <span class="o">=</span> <span class="nx">svg</span><span class="p">.</span><span class="nx">selectAll</span><span class="p">(</span><span class="s2">"text"</span><span class="p">).</span><span class="nx">data</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="c1">// enter</span>
<span class="nx">ditems</span><span class="p">.</span><span class="nx">enter</span><span class="p">()</span>
<span class="p">.</span><span class="nx">append</span><span class="p">(</span><span class="s2">"text"</span><span class="p">);</span>
<span class="c1">// update</span>
<span class="nx">ditems</span><span class="p">.</span><span class="nx">transition</span><span class="p">().</span><span class="nx">duration</span><span class="p">(</span><span class="mi">500</span><span class="p">)</span>
<span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s2">"x"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">x</span><span class="p">;</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s2">"y"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">y</span><span class="p">;</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">text</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">label</span><span class="p">;</span> <span class="p">});</span>
<span class="c1">// exit</span>
<span class="nx">ditems</span><span class="p">.</span><span class="nx">exit</span><span class="p">()</span>
<span class="p">.</span><span class="nx">transition</span><span class="p">()</span>
<span class="p">.</span><span class="nx">style</span><span class="p">(</span><span class="s2">"opacity"</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">.</span><span class="nx">remove</span><span class="p">();</span>
<span class="p">}</span></code></pre></div>
<p>Here, the <code>update</code> transition will interpolate object coordinates, and the <code>exit</code> transition makes the text fade out before it is removed from the DOM. Here is what it looks like:</p>
<style type="text/css">
.d3ex {
margin-left :30px;
margin-bottom: 20px;
margin-top: 20px;
}
.svgExample {
background-color: #eee;
width: 130px;
height: 80px;
}
#viz {
display: inline-block;
vertical-align: top;
}
.rblock {
display: inline-block;
vertical-align: top;
margin-left: 10px;
}
.dataPreview {
height: 45px;
padding: 2px;
font-family: Menlo, Monaco, Consolas, 'Courier New', monospace;
font-size: 13px;
}
</style>
<div id="ex1" class="d3ex">
<div id="viz"></div>
<div class="rblock">
<div class="dataPreview">text</div>
<div>
<button type="button" class="btn btn-sm btn-default" id="bmove">Move</button>
<button type="button" class="btn btn-sm btn-default" id="brm">Remove</button>
<button type="button" class="btn btn-sm btn-default" id="breset">Reset</button>
</div>
</div>
</div>
<p>But what happens if we remove the first object? Give it a try:</p>
<div id="ex2" class="d3ex">
<div id="viz"></div>
<div class="rblock">
<div class="dataPreview">text</div>
<div>
<button type="button" class="btn btn-sm btn-default" id="brm">Remove first</button>
<button type="button" class="btn btn-sm btn-default" id="breset">Reset</button>
</div>
</div>
</div>
<p>That’s weird. How about swapping the objects in the <code>data</code> array without changing their contents?</p>
<div id="ex3" class="d3ex">
<div id="viz"></div>
<div class="rblock">
<div class="dataPreview">text</div>
<div>
<button type="button" class="btn btn-sm btn-default" id="bswap">Swap</button>
</div>
</div>
</div>
<p>As we only re-ordered the objects and still see some changes, it has something to do with indexes. In fact, by default D3 assumes that the objects are identified by their index in the data array, and this works well in many scenarios. Here, however, we don’t want to use indexes and need to identify objects differently, which can be done by passing a <code>key</code> function to <a href="https://github.com/mbostock/d3/wiki/Selections#data">selection.data(values, key)</a>. For example, if we know the <code>label</code> is going to be unique then we can simply make the key function return it:</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">selection</span><span class="p">.</span><span class="nx">data</span><span class="p">(</span><span class="nx">data</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">label</span><span class="p">;</span> <span class="p">})</span></code></pre></div>
<p>With that, swapping the objects will now have no effect and removing the first element will behave as expected:</p>
<div id="ex4" class="d3ex">
<div id="viz"></div>
<div class="rblock">
<div class="dataPreview">text</div>
<div>
<button type="button" class="btn btn-sm btn-default" id="bswap">Swap</button>
<button type="button" class="btn btn-sm btn-default" id="brm">Remove first</button>
<button type="button" class="btn btn-sm btn-default" id="breset">Reset</button>
</div>
</div>
</div>
<p>In case when labels are not necessarily unique, we would need to use something else as an identifier. For example, an <code>id</code> field could be added to each object and used as the key.</p>
<p>Happy data-binding!</p>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://cdnjs.cloudflare.com/ajax/libs/d3/3.4.13/d3.min.js"></script>
<script type="text/javascript">
function d3ex(id, dataId) {
dataId = dataId || function(d, i) { return i; }
var data0 = [{x: 70, y:20, label: "first"}, {x: 10, y:60, label: "second"}];
var data;
function init() {
data = JSON.parse(JSON.stringify(data0));
var svg = d3.select("#" + id + " #viz")
.html("<svg class='svgExample'></svg>");
}
function render() {
var svg = d3.select("#" + id + " #viz svg");
var ditems = svg.selectAll("text").data(data, dataId);
// enter
ditems.enter()
.append("text");
// update
ditems.transition().duration(500)
.attr("x", function(d) { return d.x; })
.attr("y", function(d) { return d.y; })
.text(function(d) { return d.label; });
// exit
ditems.exit()
.transition()
.style("opacity", 0)
.remove();
updPreview();
}
function reset() {
init();
render();
}
function removeFirst() {
if (data.length != 2)
return;
if (data[0].label == "first")
data.shift();
else
data.pop();
render();
}
function removeLast() {
data.pop();
render();
}
function exMove() {
for (var i=0; i<data.length; i++) {
data[i].x = 10 + Math.floor(Math.random() * 60);
data[i].y = Math.floor(15 + i* 30 + Math.random() * 15);
}
render();
}
function exSwap() {
data.reverse();
render();
}
function updPreview() {
$("#" + id + " .dataPreview").html(JSON.stringify(data).replace("},", "},<br> "));
}
reset();
return {
reset: reset,
render: render,
removeFirst: removeFirst,
exMove: exMove,
removeLast: removeLast,
exSwap: exSwap
};
}
</script>
<script type="text/javascript">
$(document).ready(function() {
var ex1 = d3ex("ex1");
$("#ex1 #bmove").click(ex1.exMove);
$("#ex1 #brm").click(ex1.removeLast);
$("#ex1 #breset").click(ex1.reset);
var ex2 = d3ex("ex2");
$("#ex2 #brm").click(ex2.removeFirst);
$("#ex2 #breset").click(ex2.reset);
var ex3 = d3ex("ex3");
$("#ex3 #bswap").click(ex3.exSwap);
var ex4 = d3ex("ex4", function(d, i) { return d.label; });
$("#ex4 #bswap").click(ex4.exSwap);
$("#ex4 #brm").click(ex4.removeFirst);
$("#ex4 #breset").click(ex4.reset);
});
</script>
Github Pages 302 Redirect2014-11-11T02:00:00+00:00http://www.rovrov.com/blog/2014/11/11/github-pages-302-redirect
<p>My blog is on a <a href="https://help.github.com/articles/setting-up-a-custom-domain-with-github-pages/">custom domain with GitHub Pages</a>. As it turns out, this configuration can cause some issues, for example, when the site is indexed by <em>a</em> search engine.</p>
<p>A few people have already posted about this, and <a href="http://instantclick.io/github-pages-and-apex-domains">here</a> an update on the top of the post says: “This is no longer true since at least August 2014. GitHub fixed this!”. This note might refer to the bigger issue about the 5 second delay described in that post, although it made me think maybe GitHub does not do redirects anymore. It still does sometimes:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>curl -I rovrov.com
HTTP/1.1 <span class="m">302</span> Found
Connection: close
Pragma: no-cache
cache-control: no-cache
Location: /</code></pre></div>
<p>Note: this does not happen every time, but after a certain timeout the first request oftens gets redirected.</p>
<!-- more -->
<p>Basically, GitHub Pages server replies with a <code>302</code> temporary redirect pointing to the same location as requested. The browser will have to request the same exact url once again and hope not to see a <code>302</code> this time. When I asked GH support if there is something can I do about this, I got this not very encouraging response:</p>
<blockquote>
<p>you will sometimes receive a 302 Found response in place of a 200 OK response from GitHub Pages sites because of DDoS mitigation software we have in place for all Pages sites</p>
<footer><cite>GitHub support</cite>, November 10, 2014</footer>
</blockquote>
<p>Why is this bad for users? Apart from the delay due to the additional round-trip(s) to the server, this may cause issues with searching engines. I would expect crawlers to be smart enough to handle this kind of redirects. However, as for some reason my blog was not being properly indexed by google, I submitted a link to <a href="https://support.google.com/webmasters/answer/6066467?hl=en">Fetch as Google</a>. Well, guess what:</p>
<p><img src="/files/2014-11-09-github-pages-302-redirect/fetch2.png" class="imgmb" alt="Fetch as Google" /></p>
<p>While it obviously did not follow the redirect, it displays as ✔<strong>Complete</strong> in the list and offers to <em>Submit to Index</em> without any warning or whatsoever:</p>
<p><img src="/files/2014-11-09-github-pages-302-redirect/fetch1.png" class="imgmb" alt="Fetch as Google" /></p>
<p>Right, submitting empty content to the index sounds like the way to go. I had to fetch some of my urls twice and verify that they got a proper response before submitting. I hope crawlers are a bit smarter in this regard, but you never know <em>what they might have on their mind</em>.</p>
<p>According to some posts on this topic, a workaround is to configure a sub-domain such as <code>www.</code> through a CNAME pointing to <code>user.github.io</code>. I guess I will stick with this if it works better, though I am not a huge fun of the <code>www</code> prefix. <em>* Sigh *</em></p>
Rsync Windows To Linux2014-11-09T02:00:00+00:00http://www.rovrov.com/blog/2014/11/09/rsync-windows-to-linux
<p>How can I synchronize a directory on my Windows system to a remote Linux machine, and then update the remote copy efficiently when files in the directory are changed? Nowadays a common answer is to use Dropbox or the like - install it on both systems and let it do its magic. It works, but what are other options without using a third-party serivce?</p>
<p>One way is to set up a git repository on Linux, a working directory on Windows and then <code>git push</code> changes to the repo. A drawback is that a commit must be done before each synchronization, and every version of each file will be stored in the repository which is not always necessary.</p>
<h2 id="rsync">rsync</h2>
<p>Another approach is to use the <a href="http://en.wikipedia.org/wiki/Rsync">rsync</a> tool originally created for Unix-like OS but a couple of Windows ports exist such as <a href="http://google.com/search?q=cwRsync+free">cwRsync</a> (proprietary) and <a href="http://google.com/search?q=deltacopy+rsync">DeltaCopy</a> (presumably open source). Both are cygwin-based, although do not require a complete cygwin installation being bundled with the required dlls.</p>
<!-- more -->
<p>Here is the command line I ended up using to synchronize a directory to a remote <code>linuxhost</code> where <code>myuser</code> is an existing user on that system for the ssh connection:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">rsync --stats --delete --chmod<span class="o">=</span><span class="nv">Du</span><span class="o">=</span>rwx,Dg<span class="o">=</span>rx,Do<span class="o">=</span>rx,Fu<span class="o">=</span>rw,Fg<span class="o">=</span>r,Fo<span class="o">=</span>r --modify-window<span class="o">=</span><span class="m">5</span> --exclude<span class="o">=</span><span class="s2">".git"</span> -c -r /cygdrive/C/win/path/ myuser@linuxhost:/home/user/linux_path</code></pre></div>
<p><code>--stats</code> provides some stats about the transfer, which I find useful:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">...
Total bytes sent: 3,573
Total bytes received: 130
sent 3,573 bytes received <span class="m">130</span> bytes 1,481.20 bytes/sec
total size is 341,524 speedup is 92.23</code></pre></div>
<p><code>--delete</code> is required to ensure files deleted from local directory are deleted remotely as well. Otherwise, a file synced once and then removed from the local directory will not be deleted remotely.</p>
<p><code>--chmod</code> specifies permissions assigned to directories and files created during synchronization.</p>
<p><code>--modify-window=5</code> is a workaround for differences in handling timestamps between Linux and Windows file systems. Otherwise, rsync thinks all files have changed and <a href="http://serverfault.com/questions/291818/linux-ntfs-to-ntfs-rsync-repeatedly-recopying-files">sends the entire contents</a>. Stats above indicate this actually works.</p>
<p><code>--exclude=".git"</code> excludes the <code>.git</code> sub-directory from synchronization. My local directory is a git repo and I only need the contents synced, without the metadata.</p>
<p><code>-c</code> <em>skips</em> files from synchronization based on checksum instead of the default timestamp/size. I first thought that it should resolve the issue described in <code>modify-window</code>, but it didn’t.</p>
<p><code>-r</code> for recursive mode, of course.</p>
<h2 id="ssh-authentication">ssh authentication</h2>
<p>The rsync command above works over an ssh connection, and thus requires to authenticate each time it is run which is not extremely convenient. In case of key-based authentication which I am using, it prompts for a passphrase</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">Enter passphrase <span class="k">for</span> key <span class="s1">'/cygdrive/c/Users/winuser/.ssh/id_rsa'</span>:</code></pre></div>
<p>Now, this can be avoided by either setting an empty passphrase or using an <a href="http://en.wikipedia.org/wiki/Ssh-agent">ssh-agent</a> (pageant for PuTTY). Empty passphrase adds risk of identity theft if the key file is exposed. Using ssh-agent is somewhat better, but this utility is not included in the rsync distributions I mentioned. My git distribution has <code>ssh-agent.exe</code> in it (msys-based), but for some reason it didnt’t work with cwRsync when I tried. Maybe will find something later, typing the passphrase is not that big of a deal for now.</p>
Starting This Blog With Jekyll2014-11-08T23:00:00+00:00http://www.rovrov.com/blog/2014/11/08/starting-this-blog-with-jekyll
<p>A quick search for blogging platforms and hostings led me to GitHub Pages offering to host a site directly from a GitHub repo using Jekyll and Markdown. After seeing a few people migrating their blogs from Wordpress to Jekyll and not being too unhappy, I figured I should give it a try. Of course, then I had to write this mandatory <em>look-now-I-am-a-Jekyll-hacker</em> post.</p>
<p>I wish there were a little more structure to this blogging platform. For now what I’ve seen is people cloning another blogs and then patching all around the code using snippets from the web. Having some sort of components and being able to inlcude them as building blocks would be nicer… well, it is not there yet I guess.</p>
<p><a href="http://jekyllbootstrap.com/">Jekyll Bootstrap</a> has some structure, and even though it seemed unfinished and somewhat abandoned, I decided to use it as a starting point and modify to my needs. First thing I did was <code>rake theme:switch name="bootstrap-3"</code>.</p>
<!-- more -->
<p>A lot of things come with JB out-of-the-box, such as RSS/Atom, sitemap, tags and categories (not sure if I need both of those though). Enabling Disqus and GA was only a matter of modifying <code>_config.yml</code>. GA was not working for me when served locally with <code>jekyll serve</code>, so I had to find out that it is only enabled when <code>site.safe</code> is set (e.g. <code>jekyll serve --safe</code>).</p>
<p>First I thought code highlighting was not working but <em>I just couldn’t see it :)</em> The css classes were actually applied to the code, but the css with the corresponding style definitions file <code>pygments.css</code> was missing from the distribution for some reason.</p>
<p>Post previews and pagination did not come with JB. <a href="http://truongtx.me/2013/05/01/jekyll-read-more-feature-without-any-plugin/">Read More without plugin</a> describes an easy way to get post previews. Pagination support comes directly from Jekyll, so I just enabled it in <code>_config.yaml</code></p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="l-Scalar-Plain">paginate</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">5</span>
<span class="l-Scalar-Plain">paginate_path</span><span class="p-Indicator">:</span> <span class="s">"blog/page:num/"</span> <span class="c1"># my blog lives at /blog/</span></code></pre></div>
<p>and used <a href="http://jekyllrb.com/docs/pagination/#render-the-paginated-posts">this pagination snippet</a> replacing the pangination links part with this bootstrap 3 version:</p>
<div class="highlight"><pre><code class="language-html" data-lang="html"><span class="c"><!-- Pagination links --></span>
<span class="nt"><nav></span>
<span class="nt"><ul</span> <span class="na">class=</span><span class="s">"pager"</span><span class="nt">></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"previous{% unless paginator.previous_page %} disabled{% endunless %}"</span><span class="nt">><a</span> <span class="na">href=</span><span class="s">"{{paginator.previous_page_path | replace: '/index.html', '/'}}"</span><span class="nt">></span><span class="ni">&larr;</span> Older<span class="nt"></a></li></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"next{% unless paginator.next_page %} disabled{% endunless %}"</span><span class="nt">><a</span> <span class="na">href=</span><span class="s">"{{paginator.next_page_path}}"</span><span class="nt">></span>Newer <span class="ni">&rarr;</span><span class="nt"></a></li></span>
<span class="nt"></ul></span>
<span class="nt"></nav></span></code></pre></div>
<p>I also modified a few visual things here and there, probably had to figure out and change more than I originally expected but that was fun.</p>
<h3 id="trailing-slashes">Trailing slashes</h3>
<p>It is better to have trailing slashes in all “pretty” urls such as <code>rovrov.com/blog/</code>. Otherwise, when browser is navigated to <code>rovrov.com/blog</code>, GH pages will respond <code>301 Moved Permanently</code> and redirect to the version with the trailing slash. That’s why I have <code>paginate_path: blog/page:num/</code> and <code>permalink: /blog/:year/:month/:day/:title/</code> with trailing slashes.</p>