Updates to disaggregation to support electricity

[bl-young]

Updates to support sequential disaggregations and methods necessary for electricity including:

• Addition of aggregationFunctions.R
• Refactor of disaggregateFunctions.R
• Functions to enable disaggregation of satellite tables by flow based on supplied ratios
• new specs for electricity disaggregation
• update model crosswalk to utilize USEEIO column for active sector list for NAICS -> BEA
• Draft model spec 2.1 for national model with waste and electricity disaggregation (satellite tables mimic 2.0.1). Disaggregation of satellite tables for electricity is not yet fully complete pending updates to FlowBySector methods to better handle allocation to 6-digit NAICS

[bl-young]

@MoLi7, we have made a number of updates including:

• refactoring the disagg and aggregation specs to be separate objects (and a separate folder for the aggregation specs)
• consolidated the aggregation of satellite tables throughout the package to be more consistent (happens in a few places)
• updated the crosswalk to use "USEEIO' for the active crosswalk list

I'm going to spend a bit more time reviewing the satellite table disaggregation given some of the bugs we found recently, but hopefully nothing new crops up.

[MoLi7]

A potential issue: disagg elec sectors are mapped to both utilities and gov sectors in model$crosswalk but only to utilities in model$Commodities.

I think the latter is correct mapping, because S00101 and S00202 are aggregated to 221100 therefore should be removed from model. Plus, having 22XXXX mapped to 22 and G is creating problems in visualization - colors are completely off.

Hope I'm not missing/mistaking anything here. @bl-young @jvendries

[bl-young]

@MoLi7 keep in mind that in this case the two gov't sectors are only industries and not commodities. but yes, they are maintained in the crosswalk partly as a record of the agg/disagg process (so the original BEA sectors are maintained). I did not consider how this would impact the visualization functions.

This could be problematic in future cases of aggregating across summary levels like we are doing here, but perhaps that is unlikely. This situation is also tricky because there are no NAICS associated with federal and state utilities.

[bl-young]

@MoLi7 as a solution to the visualization problem you raised, I updated the aggregation of the crosswalk to also update the sector and summary data such that it matches the aggregated sector (in this case '22' and '22')

That way when assigning sectors for visualization, all those data will be assigned to utilities (22) and we won't have duplicates. Yes the crosswalk maintains the BEA_Detail showing the original sectors still as s00101 and s00202.

see 85a3c38

[MoLi7]

@bl-young that's a fair solution. I was unsure whether assigning S00101 and S00102 to 22 would create more confusion, but now I'm convinced it should be just fine.

[MoLi7]

All looks good at this point. Here is an updated plot of D matrix - nothing abnormal to me.

[MoLi7]

There is only one suggestion: this log info of aggregation is quite detailed and noticeable, compared to the very generalized log info of disaggregation, as if we wanted users to pay special attention to it.

2021-10-13 21:36:47 INFO::Loading aggregation specs for ElectricityAggregationDetail...
2021-10-13 21:36:47 INFO::Aggregating [S00101/US, S00202/US] to 221100/US...
2021-10-13 21:36:47 INFO::Loading disaggregation specs for ElectricityDisaggregationDetail...
2021-10-13 21:36:47 INFO::Loading disaggregation specs for WasteDisaggregationDetail...
2021-10-13 21:36:48 INFO::Initializing Disaggregation of IO tables...

I find the generalized one informative enough (maybe I'm just used to it), unless you think disclosing the aggregation details are much needed here. @bl-young @jvendries

[bl-young]

@MoLi7 - we updated it based on your comment above,

Consider give more informative message? It'd be ideal to print sector names for transparency.

perhaps I misunderstood 😄

[MoLi7]

Sorry I was not being clear enough...

My previous comment about "more informative" was referring to "print sector names", and it was made when I was exclusively focused on aggregation. Now with improved understanding of the agg and disagg processes, I'm more inclined to having them two consistent in terms of the log info we give to users, therefore I suggested simplifying as the disagg part does not disclose any details.

But it's your call, I'm okay with either simplifying or informative as long as agg and disagg are consistent. Hope this makes sense 😊

[bl-young]

I'm more inclined to having them two consistent in terms of the log info we give to users, therefore I suggested simplifying as the disagg part does not disclose any details.

Ok updated with c515469
This should be good to go then