
<% df = load_pkg_stat_snapshot() 
make_heaviness_plot()

bioc_version = ALL_BIOC_RELEASES$Release[ALL_BIOC_RELEASES$Date == pkgndep_opt$heaviness_db_version]
lta = load_from_heaviness_db(qq("adjusted_heaviness_select_a_@{bioc_version}.rds"))
%>

<div style="width:1100px;">

<% if(as.Date(pkgndep_opt$heaviness_db_version) < as.Date("2007-01-01")) { %>
<div class="method"><p>Note: if the version of the heaviness database is too early, the heaviness analysis might make less sense due to the small size of the ecosystem.</p></div>

<% } %>

<br>
<p>In this report, we study two questions on the package dependencies:</p>
<ul>
<li>How does the dependency heaviness spread from parent to child packages?</li>
<li>How do packages contribute the dependency heaviness to their remote downstream packages?</li>
</ul>



<h3>Heaviness from parent packages</h3>

<p>For each package, we first look at the maximal heaviness from its parents. Following plots show the relation between number of parents
and max heaviness from parents. Generally, on the border of the point cloud, there is a trend that max heaviness from parents drops as 
numbers of parents increase. This is because when a package has more parents, additional dependency packages brought by each parent 
would have more overlap (<i>i.e.</i>, dependencies from parent <i>A</i> overlap to the dependencies from parent <i>B</i>). Since heaviness measures number of unique dependencies that a single parent brings in, or in other words, the number of dependencies that are mutually exclusive to those brought by all other parents, thus with more parents, the max heaviness from parents would decrease.</p>

<p>In the plot, we can see there are several packages far away from the cloud (highlighted in red and orange). These packages can be thought as those having extreamly heavy parents compared to most of the others. To
capture these packages with heavy parents, we define <b>"adjusted max heaviness on parent packages"</b> as follows.</p>

<div class="method"><p>For a package <i>P</i>, denote $h$ as the max heaviness from its parent packages. The adjusted heaviness is calculated as $ h^\mathrm{adj} = h \cdot a $
where $a$ is a zooming factor. $a$ is calculated as $a = (n+<%=lta$cutoff$max_heaviness_from_parents%>)/n_\mathrm{max}$ where $n$ is the number of parents for package <i>P</i> and $n_\mathrm{max}$ is the maximal number of parents of all packages (<i>i.e.</i>, all CRAN/Bioconductor packages). The value of <%=lta$cutoff$max_heaviness_from_parents%> is selected emperically to balance the zooming rate on different $n$.</p></div>

<p>The zooming factor $a$ decreases the heaviness faster for small number of parents, thus, it actually transforms the original distribution of point cloud more horizontal so that it is easy to set a cutoff to mark extream points. The plot of adjusted heaviness verse number of parents can be seen by clicking the radio button <b>"Adjusted max heaviness verse number of parent packages"</b> below. 
We simply mark a package as <span style="color:red">having highly heavy parents</span> if the adjusted heaviness larger than <%=CUTOFF$adjusted_max_heaviness_from_parents%>. The packages with highly heavy parents are listed
on the right of the following figure.</p>


<div class="form-box">
<input type="radio" name="plot-parent-max-heaviness-radio" onchange="$('#plot-parent-max-heaviness').show();$('#plot-parent-adjusted-max-heaviness').hide();" checked>
  <label >
    Max heaviness verse number of parent packages
  </label>
<input type="radio" name="plot-parent-max-heaviness-radio" onchange="$('#plot-parent-max-heaviness').hide();$('#plot-parent-adjusted-max-heaviness').show();">
  <label>
    Adjusted max heaviness verse number of parent packages
  </label>
</div>

<% df2 = df[df$adjusted_max_heaviness_from_parents >= CUTOFF$adjusted_max_heaviness_from_parents, , drop = FALSE]
	top_pkgs = rownames(df2)[order(df2$adjusted_max_heaviness_from_parents, decreasing = TRUE)]
	generate_top_pkgs_html = function(top_pkgs, caption = "Top packages") {
		nr = max(15, ceiling(length(top_pkgs)/3))
		n_col = ceiling(length(top_pkgs) / nr) 
  		html_tb = matrix("", nrow = nr, ncol = n_col)
		for(i in seq_along(top_pkgs)) { 
		    i_col = ceiling(i/nr - 1/(nr+1))
		    i_row = (i-1) %% nr + 1
		  html_tb[i_row, i_col] = qq("<a href='package?package=@{top_pkgs[i]}'>@{top_pkgs[i]}</a>")
		} 
		kable(html_tb, format = "html", row.names = FALSE, escape = FALSE, table.attr = "class='table'", caption = caption)
	}
%>

<div id="plot-parent-max-heaviness" style="position:relative">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-parent-max-heaviness.png"), style="height:500px")%></p>
</div>

<div id="plot-parent-adjusted-max-heaviness" style="display:none;position:relative;">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-parent-adjusted-max-heaviness.png"), style="height:500px")%></p>
</div>

<h3>Heaviness on child packages</h3>

<p>Generally, the heaviness on child packages has a trend to decrease with increasing the number of child packages, since it is averaged on
	the heaviness of all children. To
highlight the packages that heavily affect large numbers of children, the original definition of heaviness
is adjusted. The original definition of <b>heaviness on child packages</b> is defined as:</p>

<div class="method"><p>For a package <i>P</i>,
assume it has $K$ child packages and the $k^{th}$ child is denoted
as $A_k$. Denote $n_{1k}$ as the number of strong dependencies of package
$A_k$ and $n_{2k}$ as the number of strong dependencies of $A_k$ if moving <i>P</i>
to its <code>Suggests</code>, the heaviness of <i>P</i> on its child packages denoted as $h$ is
calculated as $h = \frac{1}{K} \sum_k^K(n_{1k} - n_{2k})$, which is the average heaviness to all its child packages.</p>
</div>

<p>Since the original heaviness is scaled by the number of children, it is possible that large $K$ generates a small heaviness. The heaviness on child package is adjusted by adding a small constant $a$ to $K$, so that heaviness for small $K$ decreases more quickly than large $K$.</p>

<div class="method"><p>
$h^\mathrm{adj} = \frac{1}{K + a} \sum_k^K(n_{1k} - n_{2k})$
</p></div> 

<p>We emperically select <%=lta$cutoff$heaviness_on_children%> for $a$. Clicking on the following title to see how $a$ is selected.</p>

<details><summary>How is <i>a</i> selected?</summary>

<br>
<p>It is easy to see that a decreases $h$ faster for smaller $K$ than larger $K$. To select an optimized value for $a$, we took $a$ as integers in the set {1, 2, …, 29, 30}; and for a specific package indexed as $k$ and a value of $a$, we calculated the adjusted heaviness on its child packages, denoted as $h_{k,a}^\mathrm{adj}$ and the vector for all packages is denoted as $h_a^\mathrm{adj}$. $a$ is selected as the value by which the ranking of adjusted heaviness of all packages becomes stable. To measure the stability of the ranking of $h_a^\mathrm{adj}$ compared to $h_{a-1}^\mathrm{adj}$, we calculate the stability score denoted as $s_a$ as $s_a=\frac{1}{N}\sum_k^NI(|R_{k,a}-R_{k,a-1}|\le<%=lta$children$rank_diff[1]%>)$,
where $N$ is the total number of packages in the ecosystem, $R_{k,a}$ and $R_{k,a-1}$ are the ranks of package $k$’s adjusted heaviness in $h_a^\mathrm{adj}$ and $h_{a-1}^\mathrm{adj}$ respectively, and $I()$ is the indicator function.</p>

<p>$s_a$, or its general denotation $s$, measures the proportion of packages whose ranking difference of adjusted heaviness is no larger than <%=lta$children$rank_diff[1]%> in the two neighboring values of $a$ (<%=lta$children$rank_diff[1]%> is a small value compared to the total number of R packages in the ecosystem). When $s$ becomes stable with $a$, we can conclude increasing $a$ won't greatly change the ranking of $s$. Thus, based on the “knee” rule, $a$ was selected to <%=lta$cutoff$heaviness_on_children%>.</p>

<p><%= img(paste0(env$figure_dir, "/plot-select-a-adjusted-heaviness-children.png"), style="height:500px")%></p>
</details>
<br>

<p>The plot of adjusted heaviness verse number of children can be seen by clicking the radio button <b>"Adjusted heaviness verse number of child packages"</b>. 
We simply set a package <span style="color:red">having a highly heavy impact on its children</span> if the adjusted heaviness larger than <%=CUTOFF$adjusted_heaviness_on_children%>.</p>

<div class="form-box">
<input class="form-check-input" type="radio" name="plot-child-heaviness-radio" onchange="$('#plot-child-heaviness').show();$('#plot-child-adjusted-heaviness').hide();" checked>
  <label >
    Heaviness verse number of child packages
  </label>
<input class="form-check-input" type="radio" name="plot-child-heaviness-radio" onchange="$('#plot-child-heaviness').hide();$('#plot-child-adjusted-heaviness').show();">
  <label>
    Adjusted heaviness verse number of child packages
  </label>
</div>

<% df2 = df[df$adjusted_heaviness_on_children >= CUTOFF$adjusted_heaviness_on_children, , drop = FALSE]
	top_pkgs = rownames(df2)[order(df2$adjusted_heaviness_on_children, decreasing = TRUE)]
%>

<div id="plot-child-heaviness" style="position:relative;">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-child-heaviness.png"), style="height:500px")%></p>
</div>

<div id="plot-child-adjusted-heaviness" style="display:none;position:relative;">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-child-adjusted-heaviness.png"), style="height:500px")%></p>
</div>

<p>The analysis of heaviness on child packages is more useful for developers because it tells when adding a new direct dependency
package to their package, the expected number of additional dependency package it brings to. <i>E.g.</i>, if they add
<a href="package?package=lumi">lumi</a> in <code>Imports</code> of their package, the package will likely 
have 111 more extra dependency packages.</p>

<h3>Heaviness on <b>indirect</b> downstream packages</h3>

<p>We next look at the indirect effects to the downstream packages' dependencies. Note here we only look at the downstream packages
with excluding the child packages. A comparison of with including child packages can be found in the next section of this report.</p>

<p>Similar to the heaviness on child packages, heaviness on indirect downstream packages also decrease as the number of downstream packages increase.
We also define <b>"adjusted heaviness on indirect downstream packages"</b>. The original definition of heaviness on indirect downstream packages is as follows:</p>

<div class="method">
<p>For a package <i>P</i>, assume it has $K$ downstream packages (also include child packages) and the $k^{th}$ downstream package is denoted
as $B_k$. Denote $n_{1k}$ as the number of strong dependencies of package
$B_k$. Since <i>P</i> can affect its downstream in an indirect manner, 
we re-calculate the global dependency relations for all packages by moving 
<i>P</i> to all its child packages' <code>Suggests</code>. Then we denote 
$n_{2k}$ as the number of strong dependencies of $B_k$ in the modified dependency graph.
Next we denote $S_\mathrm{c}$ as the set of child packages of <i>P</i> and $K_\mathrm{c}$ as the number of its child packages, thus $K \geq K_\mathrm{c}$.
The adjusted heaviness of <i>P</i> on its indirect downstream packages (excluding child packages) denoted as $h$ is
calculated as: $h = \frac{1}{K-K_\mathrm{c}} \sum_{k}^K(n_{1k} - n_{2k}) \cdot I(B_{k} \notin S_\mathrm{c})$ where $I()$ is an indicator function. $h$ is set to 0 when $K = K_\mathrm{c}$.

</div>

<p>Then a small constant $a$ is added to $K - K_\mathrm{c}$ to adjust the original heaviness:</p>

<div class="method"><p>
$h^\mathrm{adj} = \frac{1}{K-K_\mathrm{c} + a} \sum_{k}^K(n_{1k} - n_{2k}) \cdot I(B_{k} \notin S_\mathrm{c})$</p>
</div> 

<p>We emperically select <%=lta$cutoff$heaviness_on_indirect_downstream%> for $a$. Clicking on the following title to see how $a$ is selected.</p>

<details><summary>How is <i>a</i> calculated?</summary>
<br>
<p>Similar method as in the section <b>Heaviness on child packages</b>. A value of <%=lta$cutoff$heaviness_on_indirect_downstream%> is taken as the optimized value of $a$.</p>

<p><%= img(paste0(env$figure_dir, "/plot-select-a-adjusted-heaviness-downstream-no-children.png"), style="height:500px")%></p>
</details>
<br>

<p>The plot of adjusted heaviness verse number of downstream can be seen by clicking the radio button <b>"Adjusted heaviness verse number of indirect downstream packages"</b>. 
We simply set a package <span style="color:red">having a highly heavy impact on its indirect downstream packages</span> if the adjusted heaviness larger than <%=CUTOFF$adjusted_heaviness_on_indirect_downstream%>.</p>

<div class="form-box">
<input class="form-check-input" type="radio" name="plot-downstream-heaviness-radio" onchange="$('#plot-downstream-heaviness').show();$('#plot-downstream-adjusted-heaviness').hide();" checked>
  <label >
    Heaviness verse number of indirect downstream packages
  </label>
<input class="form-check-input" type="radio" name="plot-downstream-heaviness-radio" onchange="$('#plot-downstream-heaviness').hide();$('#plot-downstream-adjusted-heaviness').show();">
  <label>
    Adjusted heaviness verse number of indirect downstream packages
  </label>
</div>

<% df2 = df[df$adjusted_heaviness_on_indirect_downstream >= CUTOFF$adjusted_heaviness_on_indirect_downstream, , drop = FALSE]
	top_pkgs = rownames(df2)[order(df2$adjusted_heaviness_on_indirect_downstream, decreasing = TRUE)]
%>

<div id="plot-downstream-heaviness" style="position:relative;">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-downstream-no-children-heaviness.png"), style="height:500px")%></p>
</div>

<div id="plot-downstream-adjusted-heaviness" style="display:none;position:relative;">
<div style="position:absolute;left:1140px;">
<%=generate_top_pkgs_html(top_pkgs)%>
</div>
<p><%= img(paste0(env$figure_dir, "/plot-downstream-no-children-adjusted-heaviness.png"), style="height:500px")%></p>
</div>


<h3>Why child packages are removed from downstream packages in the analysis?</h3>

<p>Each of The following two plots visualizes the ranking of all packages based on their heaviness on child packages
and on downstream packages. For each plot, the left and right panels contain sorted heaviness for children
and downstream respectively. In the middle panel are lines connecting the same package in the two rankings. The two ends of
a line are assigned with the same color.
There is a "Venn diagram" at the bottom panel which shows the overlap of the top 500 packages with the highest
heaviness on children and the highest heaviness on downstream.</p>

<p>The left plot shows top 500 packages with the highest heaviness on children almost also have the highest heaviness on 
downstream (474 out of 500), and the right plot shows if only considering the indirect downstream packages, the overlap to 
packages with top heaviness on children have very small overlap.
</p>

<div>
<p><%= img(paste0(env$figure_dir, "/plot-compare-downstream-and-downstream2.png"), style="height:500px")%></p>
</div>

<p>We think why there is such a huge overlap for the top packages with the highest heaviness on children and on downstream
is because the downstream packages are mainly composed of child packages. To demonstrate this, for 474 packages that are both in the lists of top 500 packages having the highest heaviness on children and top 500 packages having the highest heaviness on downstream, we plot
the fraction of their child packages in downstream packages. The following plot clearly shows for these top packages,
their downstream packages are mostly child packages. With 76.4% of them, their downstream packages are completely child packages, and with 91.1% of them,
more than 60% of their downstream packages are child packages.</p>

<div>
<p><%= img(paste0(env$figure_dir, "/plot-top-500-children-downstream-pct.png"), style="height:500px")%></p>
</div>


</div>


<script type="text/x-mathjax-config">
  MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});
</script>
<script type="text/javascript" async
  src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-MML-AM_CHTML">
</script>
