---

title: simple_interpolation


keywords: fastai
sidebar: home_sidebar

summary: "Interpolation rocks, but it can screw the original features of your data series if done poorly. One algorithm that at least preserves the original volatility is **Brownian Bridge**, which uses it as an input. This is an implementation of the Brownian Bridge to be used with **Pandas**."
description: "Interpolation rocks, but it can screw the original features of your data series if done poorly. One algorithm that at least preserves the original volatility is **Brownian Bridge**, which uses it as an input. This is an implementation of the Brownian Bridge to be used with **Pandas**."
nb_path: "00_core.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: 00_core.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>/home/navarro/.local/lib/python3.8/site-packages/pandas/compat/__init__.py:120: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="interpolate_gaps" class="doc_header"><code>interpolate_gaps</code><a href="https://github.com/pnmartinez/simple_interpolation/tree/master/simple_interpolation/core.py#L12" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>interpolate_gaps</code>(<strong><code>df_to_patch</code></strong>, <strong><code>method</code></strong>=<em><code>'wiener'</code></em>, <strong><code>fixed_freq</code></strong>=<em><code>'min'</code></em>, <strong><code>plot</code></strong>=<em><code>False</code></em>)</p>
</blockquote>
<p>Searches for gaps and interpolates with a Brownian Bridge,
which preserves volatility of the whole trend.</p>
<p>::returns:: patched, interpolated DataFrame with 3 columns:
                "X_original_name",
                "Y_original_name",
                "interpolated" (flag to mark interpolated rows)</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">df</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>X</th>
      <th>Y</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>11.844657</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>9.800014</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>7.690769</td>
    </tr>
    <tr>
      <th>3</th>
      <td>6</td>
      <td>9.601032</td>
    </tr>
    <tr>
      <th>4</th>
      <td>7</td>
      <td>9.723581</td>
    </tr>
    <tr>
      <th>5</th>
      <td>8</td>
      <td>11.044739</td>
    </tr>
    <tr>
      <th>6</th>
      <td>9</td>
      <td>10.988900</td>
    </tr>
    <tr>
      <th>7</th>
      <td>10</td>
      <td>9.685440</td>
    </tr>
    <tr>
      <th>8</th>
      <td>11</td>
      <td>9.287421</td>
    </tr>
    <tr>
      <th>9</th>
      <td>12</td>
      <td>10.877917</td>
    </tr>
    <tr>
      <th>10</th>
      <td>13</td>
      <td>10.431381</td>
    </tr>
    <tr>
      <th>11</th>
      <td>14</td>
      <td>11.607592</td>
    </tr>
    <tr>
      <th>12</th>
      <td>15</td>
      <td>11.600781</td>
    </tr>
    <tr>
      <th>13</th>
      <td>16</td>
      <td>11.329190</td>
    </tr>
    <tr>
      <th>14</th>
      <td>19</td>
      <td>8.718075</td>
    </tr>
    <tr>
      <th>15</th>
      <td>20</td>
      <td>9.301708</td>
    </tr>
    <tr>
      <th>16</th>
      <td>21</td>
      <td>9.355985</td>
    </tr>
    <tr>
      <th>17</th>
      <td>22</td>
      <td>9.580390</td>
    </tr>
    <tr>
      <th>18</th>
      <td>23</td>
      <td>9.732817</td>
    </tr>
    <tr>
      <th>19</th>
      <td>24</td>
      <td>7.719776</td>
    </tr>
    <tr>
      <th>20</th>
      <td>25</td>
      <td>11.907776</td>
    </tr>
    <tr>
      <th>21</th>
      <td>26</td>
      <td>10.568831</td>
    </tr>
    <tr>
      <th>22</th>
      <td>27</td>
      <td>8.139414</td>
    </tr>
    <tr>
      <th>23</th>
      <td>28</td>
      <td>10.942830</td>
    </tr>
    <tr>
      <th>24</th>
      <td>45</td>
      <td>9.851524</td>
    </tr>
    <tr>
      <th>25</th>
      <td>46</td>
      <td>12.324436</td>
    </tr>
    <tr>
      <th>26</th>
      <td>47</td>
      <td>11.604481</td>
    </tr>
    <tr>
      <th>27</th>
      <td>48</td>
      <td>10.379039</td>
    </tr>
    <tr>
      <th>28</th>
      <td>49</td>
      <td>10.806593</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">patched_df</span> <span class="o">=</span> <span class="n">interpolate_gaps</span><span class="p">(</span> <span class="n">df</span> <span class="p">)</span> <span class="c1">#, plot = True )</span>
<span class="n">patched_df</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>No datetime column: assuming first column &#39;X&#39; as X-axis
std() built with Wiener method
Will interpolate if X-column interval is more than 1.7675
Processed 0.00% of gaps
Ended succesfully
Ended execution
</pre>
</div>
</div>

<div class="output_area">


<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>X</th>
      <th>Y</th>
      <th>interpolated</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.0000</td>
      <td>11.844657</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.0000</td>
      <td>9.800014</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2.0000</td>
      <td>7.690769</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3.0000</td>
      <td>8.514332</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4.0000</td>
      <td>9.240396</td>
      <td>1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>5.0000</td>
      <td>9.612237</td>
      <td>1</td>
    </tr>
    <tr>
      <th>6</th>
      <td>6.0000</td>
      <td>9.601032</td>
      <td>0</td>
    </tr>
    <tr>
      <th>7</th>
      <td>7.0000</td>
      <td>9.723581</td>
      <td>0</td>
    </tr>
    <tr>
      <th>8</th>
      <td>8.0000</td>
      <td>11.044739</td>
      <td>0</td>
    </tr>
    <tr>
      <th>9</th>
      <td>9.0000</td>
      <td>10.988900</td>
      <td>0</td>
    </tr>
    <tr>
      <th>10</th>
      <td>10.0000</td>
      <td>9.685440</td>
      <td>0</td>
    </tr>
    <tr>
      <th>11</th>
      <td>11.0000</td>
      <td>9.287421</td>
      <td>0</td>
    </tr>
    <tr>
      <th>12</th>
      <td>12.0000</td>
      <td>10.877917</td>
      <td>0</td>
    </tr>
    <tr>
      <th>13</th>
      <td>13.0000</td>
      <td>10.431381</td>
      <td>0</td>
    </tr>
    <tr>
      <th>14</th>
      <td>14.0000</td>
      <td>11.607592</td>
      <td>0</td>
    </tr>
    <tr>
      <th>15</th>
      <td>15.0000</td>
      <td>11.600781</td>
      <td>0</td>
    </tr>
    <tr>
      <th>16</th>
      <td>16.0000</td>
      <td>11.329190</td>
      <td>0</td>
    </tr>
    <tr>
      <th>17</th>
      <td>17.5000</td>
      <td>10.023632</td>
      <td>1</td>
    </tr>
    <tr>
      <th>18</th>
      <td>19.0000</td>
      <td>8.718075</td>
      <td>0</td>
    </tr>
    <tr>
      <th>19</th>
      <td>20.0000</td>
      <td>9.301708</td>
      <td>0</td>
    </tr>
    <tr>
      <th>20</th>
      <td>21.0000</td>
      <td>9.355985</td>
      <td>0</td>
    </tr>
    <tr>
      <th>21</th>
      <td>22.0000</td>
      <td>9.580390</td>
      <td>0</td>
    </tr>
    <tr>
      <th>22</th>
      <td>23.0000</td>
      <td>9.732817</td>
      <td>0</td>
    </tr>
    <tr>
      <th>23</th>
      <td>24.0000</td>
      <td>7.719776</td>
      <td>0</td>
    </tr>
    <tr>
      <th>24</th>
      <td>25.0000</td>
      <td>11.907776</td>
      <td>0</td>
    </tr>
    <tr>
      <th>25</th>
      <td>26.0000</td>
      <td>10.568831</td>
      <td>0</td>
    </tr>
    <tr>
      <th>26</th>
      <td>27.0000</td>
      <td>8.139414</td>
      <td>0</td>
    </tr>
    <tr>
      <th>27</th>
      <td>28.0000</td>
      <td>10.942830</td>
      <td>0</td>
    </tr>
    <tr>
      <th>28</th>
      <td>29.0625</td>
      <td>10.612403</td>
      <td>1</td>
    </tr>
    <tr>
      <th>29</th>
      <td>30.1250</td>
      <td>10.259143</td>
      <td>1</td>
    </tr>
    <tr>
      <th>30</th>
      <td>31.1875</td>
      <td>9.728705</td>
      <td>1</td>
    </tr>
    <tr>
      <th>31</th>
      <td>32.2500</td>
      <td>9.665752</td>
      <td>1</td>
    </tr>
    <tr>
      <th>32</th>
      <td>33.3125</td>
      <td>9.766997</td>
      <td>1</td>
    </tr>
    <tr>
      <th>33</th>
      <td>34.3750</td>
      <td>9.694638</td>
      <td>1</td>
    </tr>
    <tr>
      <th>34</th>
      <td>35.4375</td>
      <td>9.658282</td>
      <td>1</td>
    </tr>
    <tr>
      <th>35</th>
      <td>36.5000</td>
      <td>9.638956</td>
      <td>1</td>
    </tr>
    <tr>
      <th>36</th>
      <td>37.5625</td>
      <td>9.611720</td>
      <td>1</td>
    </tr>
    <tr>
      <th>37</th>
      <td>38.6250</td>
      <td>9.259513</td>
      <td>1</td>
    </tr>
    <tr>
      <th>38</th>
      <td>39.6875</td>
      <td>9.235943</td>
      <td>1</td>
    </tr>
    <tr>
      <th>39</th>
      <td>40.7500</td>
      <td>9.435961</td>
      <td>1</td>
    </tr>
    <tr>
      <th>40</th>
      <td>41.8125</td>
      <td>9.248638</td>
      <td>1</td>
    </tr>
    <tr>
      <th>41</th>
      <td>42.8750</td>
      <td>9.225836</td>
      <td>1</td>
    </tr>
    <tr>
      <th>42</th>
      <td>43.9375</td>
      <td>9.502167</td>
      <td>1</td>
    </tr>
    <tr>
      <th>43</th>
      <td>45.0000</td>
      <td>9.851524</td>
      <td>0</td>
    </tr>
    <tr>
      <th>44</th>
      <td>46.0000</td>
      <td>12.324436</td>
      <td>0</td>
    </tr>
    <tr>
      <th>45</th>
      <td>47.0000</td>
      <td>11.604481</td>
      <td>0</td>
    </tr>
    <tr>
      <th>46</th>
      <td>48.0000</td>
      <td>10.379039</td>
      <td>0</td>
    </tr>
    <tr>
      <th>47</th>
      <td>49.0000</td>
      <td>10.806593</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="check_dataframe" class="doc_header"><code>check_dataframe</code><a href="https://github.com/pnmartinez/simple_interpolation/tree/master/simple_interpolation/core.py#L124" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>check_dataframe</code>(<strong><code>df</code></strong>)</p>
</blockquote>
<p>Collection of tests done first on the dataframe
to ensure it's properly formatted for interpolation,
and identification of x, y columns to be used.</p>
<p>Checks:</p>
<ul>
<li>2 columns only (or 1 + datetime index)</li>
<li>looks for datetime presence (affects the next steps)</li>
</ul>
<p>:returns: params_deduced = (
            dataframe_valid :: boolean,
            params_deduced :: dict, stores flags and X,Y colums names
            )</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="plot_results" class="doc_header"><code>plot_results</code><a href="https://github.com/pnmartinez/simple_interpolation/tree/master/simple_interpolation/core.py#L178" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>plot_results</code>(<strong><code>patched</code></strong>)</p>
</blockquote>
<p>Plot the original series and the interpolated data.</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="bb" class="doc_header"><code>bb</code><a href="https://github.com/pnmartinez/simple_interpolation/tree/master/simple_interpolation/core.py#L196" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>bb</code>(<strong><code>x0</code></strong>, <strong><code>y0</code></strong>, <strong><code>x1</code></strong>, <strong><code>y1</code></strong>, <strong><code>params_deduced</code></strong>, <strong><code>variance</code></strong>, <strong><code>interpolated</code></strong>, <strong><code>fixed_freq</code></strong>)</p>
</blockquote>
<p>Brownian Bridge algo implementation.</p>
<ul>
<li><p>The hard part (calculating the std() to be used) is not done here.</p>
</li>
<li><p>If datetime_present = True, we round x_midpoint to ensure 'fixed_freq' timesteps,
which defaults to 'min'. Valid options from Pandas, see link:
<a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases">https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases</a></p>
</li>
</ul>
<p>::returns:: nothing, but adds (xm, ym) tuples to an interpolated list.</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_local_std" class="doc_header"><code>get_local_std</code><a href="https://github.com/pnmartinez/simple_interpolation/tree/master/simple_interpolation/core.py#L261" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_local_std</code>(<strong><code>delta</code></strong>, <strong><code>df_to_patch</code></strong>, <strong><code>params_deduced</code></strong>, <strong><code>gap_index</code></strong>, <strong><code>method</code></strong>=<em><code>'wiener'</code></em>)</p>
</blockquote>
<p>Calculates variance to be ussed in the interpolation. Must
be different depending on the gap length.</p>
<p>Two methods: local_scan and weiner. 'local_scan' is a naive
approach mostly kept for learning purposes; weiner is default
and is always recommended.</p>
<p>Assuming a Wiener process (like in stock market theory)
then std ~ sqrt(delta_t), and in the gap case
delta_t := delta_t_gap, so:</p>

<pre><code>std_gap = std_whole / sqrt( delta_t_gap )

</code></pre>
<p>::returns:: std, float to be used in the brownian bridge</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

</div>