Introduction
- The Trade Module operates on one year at a time
- To create a time series, it has to be run multiple times
- Each run of the module takes at least 80 to 100 minutes
- Reading data from and writing back to the Statistical Working System takes a considerable share of this time
- The period 2000-2014 is targeted for dissemination
- Test runs with varying number of parallel processes are performed in order to determine the minimum time to update the whole time series
Setup
The following code is to set up a log to see CPU and memory consumption for all current Rserve processes:
echo ' DATE PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND' > testRservelog.txt
top -bd 30 -n 720 | grep Rserve | awk '{ print strftime("%Y-%m-%d_%H:%M:%S"), $0; fflush() }' >> testRservelog.txt &
The log is available at file://///hqlprsws1.hq.un.fao.org/sws_r_share/testRservelog.txt
2 Processes
Compare CPU and RAM usage of 2 simultaneous runs on the SWS for the data file sws_rserve_log.txt.
The table lists all processes consuming more than 5% of CPU
| DATE | PID | USER | PR | NI | VIRT | RES | SHR | S | X.CPU | X.MEM | TIME. | COMMAND |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2016-07-21 18:22:14 | 24115 | Ruser | 20 | 0 | 2059m | 1.7g | 4360 | R | 99.8 | 11.1 | 19:15.25 | Rserve |
| 2016-07-21 18:25:14 | 24116 | Ruser | 20 | 0 | 2132m | 1.8g | 4360 | R | 99.8 | 11.6 | 20:25.62 | Rserve |
4 Processes different years
Compare CPU and RAM usage of 4 simultaneous runs on the SWS for the data file sws_rserve_log.txt.
- years: 2009-2012
- separate sessions
- interval: 180 seconds
The table lists all processes consuming more than 5% of CPU
| DATE | PID | USER | PR | NI | VIRT | RES | SHR | S | X.CPU | X.MEM | TIME. | COMMAND |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2016-07-22 16:27:35 | 24120 | Ruser | 20 | 0 | 2977m | 2.6g | 6152 | R | 99.5 | 16.7 | 67:40.87 | Rserve |
| 2016-07-22 14:17:27 | 24118 | Ruser | 20 | 0 | 367m | 82m | 7936 | R | 97.3 | 0.5 | 0:08.18 | Rserve |
| 2016-07-22 16:24:35 | 24111 | Ruser | 20 | 0 | 3400m | 3.0g | 6172 | R | 68.8 | 19.5 | 63:15.53 | Rserve |
| 2016-07-22 15:37:32 | 24110 | Ruser | 20 | 0 | 2799m | 2.4g | 5088 | R | 52.8 | 15.7 | 39:08.30 | Rserve |
4 Processes same year
Compare CPU and RAM usage of 4 simultaneous runs on the SWS for the data file sws_rserve_log.txt.
- year: 2009
- interval: 60 seconds
The table lists all processes consuming more than 5% of CPU
| DATE | PID | USER | PR | NI | VIRT | RES | SHR | S | X.CPU | X.MEM | TIME. | COMMAND |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2016-07-25 17:15:33 | 20633 | Ruser | 20 | 0 | 2697m | 2.4g | 6192 | R | 99.4 | 15.1 | 64:03.18 | Rserve |
| 2016-07-25 17:15:33 | 20697 | Ruser | 20 | 0 | 2849m | 2.5g | 6192 | R | 99.1 | 16.1 | 63:48.04 | Rserve |
| 2016-07-25 15:08:55 | 20610 | Ruser | 20 | 0 | 382m | 102m | 8732 | R | 79.2 | 0.6 | 0:24.32 | Rserve |
| 2016-07-25 17:13:03 | 20626 | Ruser | 20 | 0 | 2902m | 2.6g | 6188 | R | 66.8 | 16.4 | 62:03.81 | Rserve |
Messages
- The plugin started Mon, 25 Jul 2016 15:09:03 +0200 and ended Mon, 25 Jul 2016 17:22:02 +0200 with the following message:
- Module completed in 132.95 minutes.
Values inserted: 29424
appended: 3912
ignored: 1420760
discarded: 1000 - The plugin started Mon, 25 Jul 2016 15:08:48 +0200 and ended Mon, 25 Jul 2016 17:21:22 +0200 with the following message:
- Module completed in 132.53 minutes.
Values inserted: 29532
appended: 5176
ignored: 1427786
discarded: 1000 - The plugin started Mon, 25 Jul 2016 15:08:38 +0200 and ended Mon, 25 Jul 2016 17:18:47 +0200 with the following message:
- Module completed in 130.15 minutes.
Values inserted: 29600
appended: 6078
ignored: 1434237
discarded: 1000 - The plugin started Mon, 25 Jul 2016 15:08:28 +0200 and ended Mon, 25 Jul 2016 17:17:35 +0200 with the following message
- Module completed in 129.11 minutes.
Values inserted: 30068
appended: 7589
ignored: 1453135
discarded: 1000
8 Processes different years
Compare CPU and RAM usage of 8 simultaneous runs on the SWS for the data file sws_rserve_log.txt.
- years: 4 times each 2009 and 2010
- separate sessions
- interval: 30 seconds
The table lists all processes consuming more than 5% of CPU
| DATE | PID | USER | PR | NI | VIRT | RES | SHR | S | X.CPU | X.MEM | TIME. | COMMAND |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2016-07-25 20:43:11 | 20610 | Ruser | 20 | 0 | 2345m | 1.9g | 5784 | R | 99.0 | 12.3 | 101:27.05 | Rserve |
| 2016-07-25 20:43:11 | 20626 | Ruser | 20 | 0 | 2405m | 2.0g | 5784 | R | 64.6 | 12.7 | 101:48.85 | Rserve |
| 2016-07-25 18:10:01 | 20386 | Ruser | 20 | 0 | 396m | 108m | 7076 | R | 61.2 | 0.7 | 0:34.63 | Rserve |
| 2016-07-25 18:10:31 | 11020 | Ruser | 20 | 0 | 2012m | 1.4g | 3772 | R | 56.4 | 8.7 | 64:17.97 | Rserve |
| 2016-07-25 20:43:11 | 20633 | Ruser | 20 | 0 | 2377m | 1.9g | 5788 | S | 54.3 | 12.1 | 102:48.09 | Rserve |
| 2016-07-25 18:10:31 | 17627 | Ruser | 20 | 0 | 371m | 88m | 8696 | R | 50.4 | 0.6 | 0:15.15 | Rserve |
| 2016-07-25 20:39:41 | 20697 | Ruser | 20 | 0 | 2234m | 1.9g | 5780 | R | 49.7 | 12.0 | 101:29.17 | Rserve |
| 2016-07-25 18:10:31 | 17631 | Ruser | 20 | 0 | 378m | 94m | 8696 | R | 37.5 | 0.6 | 0:18.99 | Rserve |
Required ETL Time
QA Complete TF CPC 2000-2008
Session IDs
- 2009
- ca58cea9-78c1-464b-8336-40acb3f9f895
- 2008
- 3c41eedc-0291-45b2-b23e-3a94e6722592
- 2007
- fda971eb-bce6-4d84-be22-395d7fa905fa
- 2006
- 013f2448-cb88-458f-8dae-287be2823e72
- 2005
- c0aa8c8e-ec80-425c-8eab-c550c3188e4a
- 2004
- 934189da-3026-4980-b5a3-047a76c85d14
- 2003
- fbdbf8ea-b1fe-4a9e-bb67-439f250c995d
- 2002
- cd51f707-d38f-432a-8af5-8f62d5124906
- 2001
- dd4b613e-7567-49ff-a18e-3e8162626b40
- 2000
- 63734bf5-9ecf-4fa7-a697-5784b0e94c8c
Example Error Message
fclunitsobject in faoswsTrade repository: /data/#dataframe-faoswstrade-fclunits created using /data-raw/fclunits.R
The plugin was launched over the following datasets:
- completed_tf_cpc_m49 in domain trade
and with the following parameters:
- year = 2003
- out_coef = 1.5
The plugin started Fri, 5 Aug 2016 11:51:15 +0200 and ended Fri, 5 Aug 2016 12:34:26 +0200 with the following message:
Error message from R: Error: cannot join on columns 'fcl' x 'fcl': index out of bounds
Stacktrace:
stop(list(message = "cannot join on columns 'fcl' x 'fcl': index out of bounds ", .Call("dplyr_left_join_impl", PACKAGE = "dplyr", x, y, by_x, left_join_impl(x, y, by$x, by$y) left_join.tbl_df(., fclunits, by = "fcl") left_join(., fclunits, by = "fcl")
function_list[[k]](value)
withVisible(function_list[[k]](value))
freduce(value, `_function_list`)
`_fseq`(`_lhs`)
eval(expr, envir, enclos)
eval(quote(`_fseq`(`_lhs`)), env, env)
withVisible(eval(quote(`_fseq`(`_lhs`)), env, env)) esdata %>% left_join(fclunits, by = "fcl")