Overview of Available Suites - Revision history

Carlo at 16:00, 27 February 2020

2020-02-27T16:00:25Z

Schuschu: updated number of source and destination nodes in several benchmark suites, replaced sink with destination to keep a common naming scheme.

2019-12-05T13:56:07Z

updated number of source and destination nodes in several benchmark suites, replaced sink with destination to keep a common naming scheme.

Carlo at 13:25, 5 December 2019

2019-12-05T13:25:39Z

Carlo at 15:47, 28 November 2019

2019-11-28T15:47:19Z

Carlo at 15:38, 28 November 2019

2019-11-28T15:38:11Z

Carlo at 15:37, 28 November 2019

2019-11-28T15:37:49Z

Carlo at 14:43, 10 November 2019

2019-11-10T14:43:11Z

Carlo at 14:41, 10 November 2019

2019-11-10T14:41:26Z

Carlo at 14:41, 10 November 2019

2019-11-10T14:41:07Z

Carlo at 14:40, 10 November 2019

2019-11-10T14:40:38Z

@@ Line 1: / Line 1: @@
 <br />
-D-Cube currently supports three benchmark suites. Each suite is specified by: (i) a given hardware platform, (ii) a given application scenario, and (iii) a set of performance metrics. We describe next the benchmark suites on a high level:
+D-Cube currently supports four benchmark suites. Each suite is specified by: (i) a given hardware platform, (ii) a given application scenario, and (iii) a set of performance metrics. We describe next the benchmark suites on a high level:
 <ol>
 <li><strong>SkyDC_1 (Tmote Sky Data Collection v1).</strong> This benchmark resembles the first category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].

@@ Line 13: / Line 13: @@
 <ul>
 <li><i>Hardware platform</i>: Tmote Sky.</li>
-<li><i>Application scenario</i>: In this benchmark, a fixed number of source nodes needs to disseminate actuation commands of different length to a specific set of destination nodes over a multi-hop network. This benchmark hence focuses on point-to-multipoint traffic. In particular, each source node is associated with a specific set of destinations (at most eight), which will be injected as an input parameter into the firmware under test. A destination can receive messages from only a single source node, cannot act as a source at the same time, and allows out-of-order delivery.</li>
+<li><i>Application scenario</i>: In this benchmark, a fixed number of source nodes (at most eight) needs to disseminate actuation commands of different length to a specific set of destination nodes over a multi-hop network. This benchmark hence focuses on point-to-multipoint traffic. In particular, each source node is associated with a specific set of destinations (at most eight), which will be injected as an input parameter into the firmware under test. A destination can receive messages from only a single source node, cannot act as a source at the same time, and allows out-of-order delivery.</li>
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
@@ Line 21: / Line 21: @@
 <ul>
 <li><i>Hardware platform</i>: nRF52840.</li>
-<li><i>Application scenario</i>: This benchmark focuses on multipoint-to-point traffic: up to 48 source nodes generate raw sensor values of different lengths, which should be communicated to the same destination. The latter may be located several hops away from a given source node, even when making use of the coded PHY layers available on the nRF52. The messages containing the raw sensor values should be forwarded to the intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the sink should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost.</li>
+<li><i>Application scenario</i>: This benchmark focuses on multipoint-to-point traffic: up to 48 source nodes generate raw sensor values of different lengths, which should be communicated to the same destination. The latter may be located several hops away from a given source node, even when making use of the coded PHY layers available on the nRF52. The messages containing the raw sensor values should be forwarded to the intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the destination should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost.</li>
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
@@ Line 29: / Line 29: @@
 <ul>
 <li><i>Hardware platform</i>: nRF52840.</li>
-<li><i>Application scenario</i>: In this benchmark, a fixed number of source nodes needs to disseminate actuation commands of different length to a specific set of destination nodes over a multi-hop network. This benchmark hence focuses on point-to-multipoint traffic. In particular, each source node is associated with a specific set of destinations, which will be injected as an input parameter into the firmware under test. A destination can receive messages from only a single source node, cannot act as a source at the same time, and allows out-of-order delivery. The messages should be disseminated to all intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the sink should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost.</li>
+<li><i>Application scenario</i>: In this benchmark, a single source nodes needs to disseminate actuation commands of different length to a specific set of up to 48 destination nodes over a multi-hop network. This benchmark hence focuses on point-to-multipoint traffic. The identity of the source node and its associated destinations nodes be injected as an input parameter into the firmware under test. A destination cannot act as a source at the same time, but allows out-of-order delivery. The messages should be disseminated to all intended destinations as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the destination should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost.</li>
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
 The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/4 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available [https://iti-testbed.tugraz.at/wiki/images/d/db/NRFDC_1.zip here].</li><br/>
 </ol>

@@ Line 25: / Line 25: @@
 </ul>
 The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available [https://iti-testbed.tugraz.at/wiki/images/d/db/NRFDC_1.zip here].</li><br/>
 </ol>

@@ Line 8: / Line 8: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.</li>
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available [https://iti-testbed.tugraz.at/wiki/images/9/97/SkyDC_1.zip here].</li><br/>
 <li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1).</strong> This benchmark resembles the second category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].
@@ Line 16: / Line 16: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available [https://iti-testbed.tugraz.at/wiki/images/9/97/SkyDC_1.zip here].</li><br/>
 <li><strong>nRFDC_1 (nRF52840 Timely Data Collection v1). </strong> This benchmark runs on the nRF52840 platform and resembles a data collection with bounded delays within a multi-hop network.
@@ Line 24: / Line 24: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available [https://iti-testbed.tugraz.at/wiki/images/d/db/NRFDC_1.zip here].</li><br/>
 </ol>

@@ Line 8: / Line 8: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.</li>
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here]. An example of the pre-defined configuration struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
 <li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1).</strong> This benchmark resembles the second category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].
@@ Line 16: / Line 16: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here]. An example of the pre-defined configuration struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
 <li><strong>nRFDC_1 (nRF52840 Timely Data Collection v1). </strong> This benchmark runs on the nRF52840 platform and resembles a data collection with bounded delays within a multi-hop network.
@@ Line 24: / Line 24: @@
 <li><i>Performance metrics</i>: The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to bootstrap the network.
 </ul>
-The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here]. An example of the pre-defined configuration struct for binary patching for this benchmark suite is available here.</li><br/>
+The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here]. An example of the pre-defined config struct for binary patching for this benchmark suite is available here.</li><br/>
 </ol>

@@ Line 2: / Line 2: @@
 D-Cube currently supports three benchmark suites. Each suite is specified by: (i) a given hardware platform, (ii) a given application scenario, and (iii) a set of performance metrics. We describe next the benchmark suites on a high level:
 <ol>
-<li><strong>SkyDC_1 (Tmote Sky Data Collection v1)</strong><br />
+<li><strong>SkyDC_1 (Tmote Sky Data Collection v1).</strong> This benchmark resembles the first category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].
 <ul>
 <li><i>Hardware platform</i>: Tmote Sky.</li>
@@ Line 11: / Line 10: @@
 The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here].</li><br/>
-<li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1)</strong><br />
+<li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1).</strong> This benchmark resembles the second category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].
 <ul>
 <li><i>Hardware platform</i>: Tmote Sky.</li>
@@ Line 20: / Line 18: @@
 The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here].</li><br/>
-<li><strong>nRFDC_1 (nRF52840 Timely Data Collection v1)</strong>
+<li><strong>nRFDC_1 (nRF52840 Timely Data Collection v1). </strong> This benchmark runs on the nRF52840 platform and resembles a data collection with bounded delays within a multi-hop network.
-This benchmark runs on the nRF52840 platform and resembles a data collection with bounded delays within a multi-hop network. This benchmark hence focuses on multipoint-to-point traffic. In particular, in this benchmark, up to 48 source nodes generate raw sensor values of different length, which should be communicated to the same destination. The latter may be located several hops away from a given source node, even when making use of the coded PHY layers available on the nRF52. The messages containing the raw sensor values should be forwarded to the intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the sink should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost. The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to boostrap the network. The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here].</li><br/>
+<ul>
 </ol>

@@ Line 10: / Line 10: @@
 The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/1 here].</li><br/>
-<li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1).</strong> This benchmark runs on the Tmote Sky platform and resembles the second category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition]. In this benchmark, a fixed number of source nodes needs to disseminate actuation commands of different length to a specific set of destination nodes over a multi-hop network. This benchmark hence focuses on point-to-multipoint traffic. In particular, each source node is associated to a specific set of destinations (at most eight), which will be injected as input parameter into the firmware under test. A destination can receive messages from only a single source node, cannot act as a source at the same time, and allows out-of-order delivery. The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, (ii) the average end-to-end latency in communicating each message to its intended destination(s), and (iii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to boostrap the network. The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/2 here].</li><br/>
+<li><strong>SkyDD_1 (Tmote Sky Data Dissemination v1).</strong> This benchmark resembles the second category of the [http://ewsn2019.thss.tsinghua.edu.cn/competition-scenario.html EWSN 2019 dependability competition].
 <li><strong>nRFDC_1 (nRF52840 Timely Data Collection v1).</strong> This benchmark runs on the nRF52840 platform and resembles a data collection with bounded delays within a multi-hop network. This benchmark hence focuses on multipoint-to-point traffic. In particular, in this benchmark, up to 48 source nodes generate raw sensor values of different length, which should be communicated to the same destination. The latter may be located several hops away from a given source node, even when making use of the coded PHY layers available on the nRF52. The messages containing the raw sensor values should be forwarded to the intended destination as efficiently as possible within a maximum per-message delay bound ∂, i.e., the end-to-end delay of every message from its generation to its reception at the sink should be lower than ∂ (which is fixed and applies to all messages exchanged during a test run). If a message has been received with an end-to-end delay greater than ∂, it is considered to be lost. The measured performance metrics are: (i) the reliability of transmissions, i.e., the number of messages correctly reported to each intended destination, and (ii) the average energy consumption on all nodes in the network. Note that, on each run, during the first 60 seconds no data is generated and the energy consumption is not measured, so to allow the firmware under test to boostrap the network. The results of this benchmark are available [https://iti-testbed.tugraz.at/leaderboard/benchmarksuite/3 here].</li><br/>
 </ol>