Performance metrics and scores
If nothing is measured during an experiment, the experiment makes little sense. Experimenters characterize the performance of their solution under test by recording a set of performance metrics. This overview presents a selection of metrics that may be of use for experimenters during the execution of their experiments.
This section provides both descriptions of performance metrics and of scores, which are higher level abstractions of one or multiple performance metrics. There are two main reasons why sharing these metrics is relevant:
- it helps experimenters to build their experiments using unambiguously defined metrics (less effort, better definition).
- it improves the comparability of results between experiments carried out by different experimenters (or at different times).
The metrics and below are organized in different categories.
1. Metrics
1.1 Spectrum characterization and Spectrum sensing
| Name | Unit | Explanation / relevance | Formula or methodology | Comments |
| Channel duty cycle | ms | The channel duty cycle is defined as the ratio of time a channel is "occupied" over a given interval. A channel is considered "occupied" when the measured energy is above a certain threshold value Eth. | T(E>Eth)/Ti, where E is the measured energy over a certain channel for the specified interval, Eth is the energy threshold threshold, T(E>Eth) denotes the time when measured energy is above the treshold and Ti is the selected time interval | |
| Received Signal Strength Indicator (RSSI) for VESNA | dBm | The RSSI indicates the power of the received signal as measured by the device. If only noise is present on the communication channel, this value equals the noise floor. This metric is relevant from two points of view: 1) it helps monitoring the sensitivity of the receiver over time 2) characterize the transmission channel for known transmitters. | The value is retrieved from the receiver (the CC chips). | RSSI data from sensing devices can be processed to obtain a periodogram (received power level vs. frequency and time). This holds for energy based detection methods and applies to VESNA sensor nodes. In order to characterize the radio environment, these will be compared to the transmitting part that is under the control of the testbed. |
| Probability of false transmission detection (PFTD) | % | The probability of false transmission detection represents the ratio between the number of false transmission detection reports vs the total number of transmission detection reports. It reflects the probability that the device reports that a transmission is present on the monitored channel without this transmission actually being present. In other words, the probability of the device erroneously reporting a transmission. | This value is computed as the ratio between false detection reports vs the total number of detection reports over a predefined time interval. | PFTD is dependent on the RSSI values as these are the values based on which the device decides whether a transmission is present ot not. This holds for energy based detection methods and applies to VESNA sensor nodes. |
| Probability of correct transmission detection (PCTD) | % | The probability of correct transmission detection represents the ratio between number of correct detection reports vs the total number of detection reports. | 1-PFTD | |
| Probability of missed transmission detection (PMTD) | % | The probability of missed transmission detection represents the ratio between false silence reports vs the total number of silence reports. It reflects the probability that the device reports that a transmission is not present on the monitored channel while a transmission is actually taking place on the channel. In other words, the probability of the device erroneously reporting a free channel. | This value is computed as the ratio between false silence reports vs the total number of silence reports over a predefined time interval. | PMTD is dependent on the RSSI values as these are the values based on which the device decides whether a transmission is present ot not. This holds for energy based detection methods and applies, among other things, to VESNA sensor nodes. |
| Probability of correct silence detection (PCSD) | % | The probability of correct silence detection represents the ratio between number of correct silence reports vs the total number of silence reports. | 1-PMTD | |
| Time to detection (TD) | ms | The time required that the sensing device detects an existing transmission in a given band. This metric is useful, among others, in estimating how long a sensing device from the secondary user's equipment needs to detect the transmission coming from a primary user. | Assuming band B is of interest, this value is measured a follows. 1) The sensing device listens to the activity in band B. 2) A known transmitter starts transmitting on B at time T1. 3) The sensing device detects the transmission at time T2. 4) The value of TD is: TD = T2 - T1. | TD is also dependent of RSSI values. This holds for energy based detection methods and applies, among other things, to VESNA sensor nodes. |
| Time to clear the spectrum (TCS) | ms | The time required that a transmitted vacates a given band. This is particularly useful for the cases when the secondary user needs to move away from a band dedicated to the primary user. | Assuming band B is of interest, this value is measured a follows. 1) A sensing device listens to the activity in band B where a known transmitter T is transmitting. 2) The transmitter T receives a command instructing it to stop transmission on B at time T1. 3) The sensing device detects that transmitter T stopped transmitting in B at time T2. 4) The value of TCS is: TCS = T2 - TD - T1. | TCS is also dependent of RSSI values. This holds for energy based detection methods and applies, among other things, to VESNA sensor nodes. |
| Probability of correct hidden node localization (PCHNL) | % | The probability that the hidden node is correctly localized. | The value is computed as the ratio between the number of times the known location of a hidden node has been detected correctly versus the total number of detections. | |
| Probability of incorrect hidden node localization (PICHNL) | % | The probability that the hidden node is incorrectly localized. | The value is computed as the ratio between the number of times the known location of a hidden node has been detected incorrectly versus the total number of detections. |
1.2 Networking
| Name | Unit | Explanation / relevance | Formula or methodology | Comments |
| Application layer packet delay | ms | measures the time needed for a packet to arrive at the application layer at a certain node B after being sent from the application layer at node A. | 1/ record timestamp as packet leaves the application layer of node A. 2/ record (synchronized) timestamp when packet is received by node B. 3/ calculate the difference | |
| Link packet reception rate (LPRR) | The Link packet reception rate over a single link is defined as the ratio of received packets at the end node of the link over the number of transmitted packets transmitted by the source node of the link. The End-to-end PRR is calculated as the product of the reception rates on each link along a path in the network | LPRR_i = num_of_packet_received_i /num_of_packets_transmited (i is the index of the link segments in a multihop routes), LPRR_e2e= PRR_1*PRR_2…*PRR_N (N is the total number of link) | ||
| Link packet error rate (LPER) | The packet error rate over a single link is defined as the ratio of lost packets over the total amount of transmitted packets over that link. | LPER = sum(lost packets) /num_tx_packets = 1- LPRR | ||
| Packet Jitter | ms | We define jitter as the variation of a packet delay with respect to the application layer packet delay. In CR can be high and therefore cause problems to the steraming applications. | Based on the application layer packet delay measurements it is possible to calulate average and variance of the metric. |
1.3 Metrics specifically related to the imec Sensing Agent
| Name | Unit | Explanation / relevance | Formula or methodology | Comments |
|
FFT based RSSI (IMEC Sensing Engine) |
dBm |
RSSI value based on FFT on time domain signal which is retrieved from the Sensing Engine. The bandwidth to which the values apply depends on the number of points selected in the FFT and bandwidth of the RF front-end. Currently the SE API supports 20 MHz bandwidth and 128 point FFT which results in a corresponding bandwidth for each value of 156.25kHz |
See (1) |
1) Since the FFT uses the baseband time domain signal saturation effects in the RF sections may become invisible and corrupt the result 2) The dBm values have been calibrated for the maximum gain setting of the RF front-end |
|
Baseband time domain power measurements (IMEC Sensing Engine) |
|
Depending on the configuration (WLAN G, ZigBEE, WLAN A, BlueTooth, 2 or 4MHz in ISM band) a power measurement is executed for the desired channel bandwidth and RF frequency. The result is a unitless number which is the sum of the powers of 32 time-domain samples. |
See (1) |
Since value heavily depends on the used gain settings in the RF section, reference curves are available to convert the unitless values to dBm |
|
Cyclostationary detection for DVB-T signal (IMEC Sensing Engine) |
|
Outcome of cyclostationary detection on DVB-T signal. Two modes are supported, 2k and 8k carriers and multiple cyclic prefix lengths are supported (1/4, 1/8, 1/16 and /1/32) |
See (1) and (2) |
|
- http://www.crew-project.eu/sites/default/files/SensingEngine_UserManual.pdf
- “Performance evaluation of sensing solutions for LTE and DVB-T”, Van Wesemael, P.; Pollin, S.; Lopez Estraviz, E. and Dejonghe, A., IEEE Symposium on New Frontiers in Dynamic Spectrum Access Networks – DySPAN 2011
1.4 Other
| Name | Unit | Explanation / relevance | Formula or methodology | Comments |
| Node energy consumption (P) | mW | The node energy consumption indicates how much energy is consumed by this node at a certain moment. In wireless sensor networks, there is a trade off between the desired network performance and energy consumption. Hence it is wise to deliberately put sensor in sleep mode and only wake up from time to time. | 1/ Direct measurement: P=U*I (measure the output voltage U and the current I provided by the battery or other power source) 2/ Indirect measurement: P=Q*P1+(1-Q)*P2 (Where Q is the sleeping duty cycle of the sensor node), P1 is the power consumption during sleep mode and P2 is the power consumption when it's in full operation |
There are many additional ways to measure energy consumption, the provided formulas and methodology only shows the most common ones. |
| Memory consumption | byte | Program memory (flash) consumption as well as RAM size of a software component are especially important in the context of evaluating resource-constraint platforms, such as sensor networks. The numbers are given in byte and can either be for specific software components (e.g. representing a protocol) or for the entire image. | Typically the compiler will list the memory usage (Flash/RAM) of the overall application. Otherwise the binary may be inspected with tools such as objdump to retrieve memory consumption. | |
| Physical Source Lines Of Code | Sometimes not only memory consumption but also the physical number of source lines of code (PSLOC) are an important metric to evaluate a solution. For example, PSLOC may give an indication how complex/difficult it may be to use a certain service from another client. | PSLOC are typically computed by a dedicated script that counts relevant line numbers for a specific programming language, more precisely "a physical source line of code is a line ending in a newline or end-of-file marker, and which contains at least one non-whitespace non-comment character.'' (http://www.dwheeler.com/sloc/redhat71-v1/redhat71sloc.html) | The following tool supports a large number of program languages: http://www.dwheeler.com/sloc (Wheeler, D.A.: Counting source lines of code (SLOC)) |
2. Scores
A score is an abstract representation of the experiment result. A score is a higher level abstraction of performance metrics. Within the CREW benchmarking concept, the score of an experiment is scaled between 0 to 10, with 10 representing the perfect performance for a certain characteristic, and 0 for the opposite performance.
| Name | Description | Script |
| Score for large scale CTP sensor network performance | A large scale collection tree protocol (CTP) sensor network typically consists of multiple routes and each route has multi hops, it is difficult to say how the overall network is performing. Therefore we calculate end-to-end packet reception rate for each sensor node. For a node that is only one hop away from the sink node, the end-to-end PRR is the PRR on the final link. For a node located two hop from the sink, the end-to-end PRR refers to the product of the PRR on two links, etc. Finally the end-to-end PRR is averaged among all the none-sink nodes. The averaged PRR can maximum reach 1, therefore the scaling factor is 10. The MATLAB script is attached for clarification | |
| Pre/Post experiment threshold score |
Pre and Post experiments are used to estimate environment interference and thus easily select outlier experiments. Inside CREW experiment execution tool when pre/post experimentation is selected, a threshold score will be calculated after the end of each experiment run. The tool generates 9 threshold scores starting from -55dbm to -95dbm with a step of 5dbm level. A very high score (closer to 10), on any threshold level indicates that the environment is not interfered by a signal with strength above the given threshold dbm value. |
|
