An Evaluation of Provider-Sharing Networks of Patients Using Prescription Opioids

IMPORTANCE: Patients using prescription opioid are embedded in a network due provider-sharing and living in the same community. As a result, they may exert influence on each other’s treatment preferences and share attitudes towards prescription opioid use and misuse. OBJECTIVE: To determine patient characteristics associated with the observed pattern of shared prescribers in a network and identify influential patients in the network. DESIGN, SETTING, AND PARTICIPANTS: We conducted a cross-sectional network-based study using the Rhode Island (RI) Prescription Drug Monitoring Program (PDMP) data for the 2015 calendar year. All patients who filled at least one opioid prescription at a retail pharmacy were eligible. The analysis was limited to patients who were on a stable opioid regimen and used only one source of payment, and filled only one type of opioid medication (oxycodone, hydrocodone or buprenorphine/naloxone) from ≥ 3 prescribers, and visited ≥ 3 pharmacies during the year. To minimize the influence of less relevant network connections, we excluded institutional providers and providers who issued opioid prescriptions to ≤ 6 patients. We applied social network analysis (SNA) methods to a sample of 372 patients connected to each other through provider-sharing. We used the exponential random graph model (ERGM) assuming conditional dyadic independence to examine the relationship between patient attributes and the likelihood of forming network ties. Homophily was defined as the tendency of patients to associate with others who have similar characteristics. Three centrality measures (degree, closeness, and betweenness) were used to identify patients with potential influence in the opioid prescription network. MAIN OUTCMES AND MEASURES: We provide a visual and descriptive characterization of the network, used centrality measures to identify influential patients, and ERGM to assess homophily and differential homophily. RESULTS: The mean age of patients included in the analysis was 51 years; 53% were female; 57% took oxycodone, 34% took hydrocodone and 9% took buprenorphine/naloxone. On average, 53% of patients received less than 50 morphine milligram equivalents (MME) daily, and the mean (standard deviation [SD]) number of opioid prescriptions per patient was 14.4 (6.6). Sixty-four percent of patients had commercial insurance, 28% had Medicaid, 5% had Medicare, and almost 2.5% used cash payment only. All three centrality measures were in agreement on the identification of the most influential patient in the opioid prescription network but overall correlation between the measures was low. After controlling for the main effects in the ERGM model, homophily was associated with age group, method of payment, number and type of opioid prescription filled, mean daily MME, and number

x  year. To minimize the influence of less relevant network connections, we excluded institutional providers and providers who issued opioid prescriptions to ≤ 6 patients.

LIST OF TABLES
We applied social network analysis (SNA) methods to a sample of 372 patients connected to each other through provider-sharing. We used the exponential random graph model (ERGM) assuming conditional dyadic independence to examine the relationship between patient attributes and the likelihood of forming network ties.
Homophily was defined as the tendency of patients to associate with others who have similar characteristics. Three centrality measures (degree, closeness, and betweenness)

KEY POINTS
Questions: What patient characteristics explain the pattern of shared-provider connections among patients in an opioid prescription network and can we identify influential patients as potential targets for opioid misuse prevention interventions?
Findings: In this social network analysis of PDMP data, we found extensive homophily that was associated with age group, method of payment, number and type of opioid prescription filled, mean daily dose, and number of prescribers ordering opioid prescriptions. All three commonly used centrality measures identified the same individual as the most influential patient in the network.
Meaning: Some patients in an opioid prescription network occupy influential positions through a large number of shared providers or by virtue of their location on paths between other patients in the network. Patients with similar characteristics tend to share providers with each other. These findings suggest that interventions targeted at influential, well-connected patients in the network may alter social norms around prescription opioid use and misuse in a community.

INTRODUCTION
The United States is experiencing an unprecedented prescription opioid overdose crisis driven in part by few patients who possibly engage in doctor shopping which may be identified in this context as patients obtaining opioid prescriptions from multiple providers without the prescribers' knowledge of other opioid prescriptions. [1][2][3] Prescribers may be sought by patients using opioids because of their reputation around opioid prescribing patterns such as use of high daily dose, use of combination opioids, and frequent refills. Knowledge about individual prescriber clinical practices and preferences may be shared among patients during co-visitation or social encounters in the community. A recent study demonstrated that health care providers tended to share patients with providers who have similar patients in their practice. 4 This suggests that patients prescribed opioids in a single state could be conceptualized as a network of patients with connections through shared providers which we define in an opioid prescription network. We hypothesized that patients within an opioid prescription network may exert influence on each other's opioid prescription utilization, including opioid misuse as a result of living in the same community or sharing a common opioid prescriber in the network, thereby impacting their network member's opioid prescription utilization and social norms around opioid use and misuse. 5 Limited data suggests that a few high-intensity prescribers play a central role in sustaining the prescription opioid epidemic. 6,7 The pattern of provider-sharing may help identify corresponding influential or central patients in a network, thereby providing a clearer picture of where doctor shopping for prescription opioids may be occurring. This understanding can inform the implementation of targeted interventions designed to improve prescription opioid utilization, prevent misuse, and treat opioid use disorder among patients in a network. A network-based perspective has been used to study a wide range of relational processes involving the flow of information between network members connected to each other in a social network. This perspective provides a framework that can be used to understand the structure of a network and how it influences the behavior of individual members in the network. [8][9][10] Landon et al. recently used network-based methods to demonstrate that characteristics of patient-sharing networks and the position of providers in the network are associated with healthcare resource utilization and cost. 11 Another study used network analysis to show racial differences in referral patterns for total hip replacement between communities with low and high concentrations of back residents. 12 Similar studies have not been done using an opioid prescription network.
There is a dearth of knowledge about characteristics of patients possibly engaged in doctor shopping for opioid prescriptions and methods to identify prescription opioid doctor shopping behavior are limited. To the best of our knowledge, network analysis has not yet been used to study an opioid prescription network within any state. The purpose of this study was to explore and characterize a patient-based opioid prescription network using social network analysis (SNA) methods. Specifically, we described patterns of relationships between patients within an opioid network, identified patients who have an influential role in the network, and examined patient characteristics that may explain the observed pattern of providersharing relationships. We used the exponential random graph model (ERGM) assuming conditional dyadic independence to examine the influence of some characteristics of individual patients in the network on their likelihood to form network connections through provider sharing.

BACKGROUND
Over the past three decades opioid prescribing has increased tremendously in the United States, with a corresponding rise in opioid misuse and opioid overdose-related deaths. 13,14 An important feature of this opioid epidemic is the association between increasing rates of opioid prescribing and opioid-related morbidity and mortality. [15][16][17] Among people who died of opioid overdoses, up to 66% used prescription opioid analgesics originally prescribed for someone else; with doctor shopping being an important means for acquiring these prescription opioids for misuse. 18 Standard statistical approaches often assume independence of patients and/or providers and ignore contextual relationships between providers and patients, and among patients due geographic proximity, social influence, and local medical practice norms; thereby, limiting our ability to evaluate prescription opioid doctor shopping behavior. The goal of this study was to incorporate relational information using SNA.
These findings will better inform future intervention policies designed to improve social norms around prescription opioid use and prevent potential misuse among patients within a community of patients using prescription opioids.

METHODS
Data source: We conducted a cross-sectional network-based study using the Rhode Island (RI) PDMP data for the 2015 calendar year when the opioid crisis was a major statewide concern to patients, prescribers and public health regulators.

Network-based framework
A network may be defined as a collection of points (i.e., vertices, nodes) and lines (i.e., edges, ties, links, connections) joining them. In a social network, these vertices represent people or groups of people and edges represent a kind of interaction between them. PDMP data links each patient who received at least one opioid prescription to one or more providers who ordered the opioid prescription(s). The receipt of one or more opioid prescriptions from a prescriber was used as a proxy for a relationship or interaction between a patient and a provider because state regulation requires a physician visit for a written opioid prescription. These prescription records were used We first modeled a simple random graph (i.e., null model) which contained only an edges term to capture the network density. 53 A simulated network of the same size and density as the observed opioid prescription network was compared to the observed network in order to identify important differences between the two networks.
The main effects and pairwise homophily interaction terms were added sequentially to the null model to represent attributes of patients in the network. Homophily was defined as the tendency for patients to connect with others like themselves. To examine the influence of node attributes on the likelihood of having a shared provider in the network, patient attributes were added to the model as main effects. We hypothesized that specific patient attributes, including number of opioid prescriptions, sex, age group, type of insurance coverage, type of opioid prescription, number of prescribers and pharmacies, explain the pattern of patient connections through provider sharing.
Homophily or assortative mixing is a tendency of patients to associate with similar patients, while disassortative mixing is the tendency to associate with dissimilar patients. Two types of dyadic interaction terms were added to the main effects model to assess assortative and disassortative mixing in the network leading to patterns of homophily or heterophily, respectively. First, we assessed the likelihood of provider sharing when both patients in a dyad had the same level of a categorical attribute. The number of opioid prescriptions was added as a continuous attribute. We hypothesized that two patients with a similar number of opioid prescriptions filled during the study year were more likely to form a network connection based on having a shared provider. Secondly, we assessed the likelihood of provider sharing when both patients in a dyad had different levels (i.e., dissimilar) of a categorical attribute such as type of opioid prescription (differential homophily).
We limited this analysis to ERGM models that assume dyadic independence of network connections. 54 This assumption specified that patients sharing a provider were dependent but independent if they had no provider in common. The null and main effects models with and without homophily and differential homophily terms were compared using Log L and related measures of deviance (-2LogL), the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). 55,56 All tests of statistical significance were two-sided and performed at the 0.05 significance level.
Data manipulation was performed with SAS, version 9.4 (SAS Institute, Cary, NC) and network analysis was implemented with R statistical software, version 3.2.3 (R Core Team 2016). The study was approved by the Institutional Review Board at University of Rhode Island.

RESULTS
A total of 372 patients prescribed opioids by 746 providers during a one-year study period met the inclusion criteria for meaningful involvement in the opioid prescription network in RI. Table 1   Characteristics of the full opioid network and its LCC are presented in Table 2.
The LCC had an overall density of 5% compared with 3% for the whole network. The average number of shared providers was higher among those in the LCC (13.8) compared to the whole network (10.6). The average path length and longest path were the same (≈ 10) suggesting that the rate of flow of information diffusion in the LCC would be similar to that of the full network. However, the density around the most central patient was 65%, average number of shared providers was 25 (SD=9) and the average path length only 1.4 ( Figure 6). Assortative mixing and the fraction of transitive triples (transitivity) were higher for the whole cohort. About 85% of patients who shared a provider were connected to other patients who also shared a provider with each of them. Seventy-five of patients were connected to one or more patients with at least one similar characteristic.
The number of shared providers was quite heterogeneous across patients ( Figure 7). While there are many patients with few shared providers, there was a nontrivial number with many shared providers. In particular, there are 28 patients with 29 shared prescribers. This may correspond to providers in the same practice or on-call group. Given the nature of the decay in the degree distribution, a log-log scale was used to assess the results. The middle panel in Figure 7 shows a somewhat linear decay in the log-frequency as a function of the log-degree. A plot of the average neighbor degree versus vertex degree suggests that while there is a tendency for patients with many shared providers to connect to each other, those with fewer shared providers tend to connect with both patients having lower and higher number of shared providers (assortative degree network). This is illustrated by the high network density around the most influential patient ( Figure 6). Tertiles of standardized centrality measures estimated from the LCC are presented in Table 3. Age group, type of opioid used, average daily dose, and number of opioid prescribers were associated with at least one standardized centrality measure while gender, method of payment, and number of pharmacies were not associated with any centrality measure. Based on multivariable logistic regression model, age group, type of opioid used, average daily dose, and number of opioid prescribers were associated with the highest tertile of at least one standardized centrality measure, after adjusting for other covariates in the model (Table 4). Patients aged 45-64 years were most likely to be classified as having the highest levels of standardized degree centrality tertile as compared to those ages 65 years and older. Furthermore, patients who took on average > 90 daily MME were 6.7 times more likely to have the highest standardized degree centrality tertile compared to those on < 50 MME per day. This suggests that patients on higher daily doses of opioids tend to have more shared providers. Based on standardized closeness and betweenness centralities, patients who had ≥ 4 providers were more likely to be classified in the highest tertile of their respective standardized centrality measures. As compared with patients on hydrocodone, patients on buprenorphine/naloxone were less likely to be in the highest standardized degree centrality tertile and more likely to be in the highest standardized betweenness centrality tertile. This suggests that patients with few connections may be crucial for the diffusion of information and prescription opioids in the network.
Network connectivity: Graphical examination of triangles, degree, DSP, and ESP were used to understand the network structure ( Figure 8). The observed LCC of the network had many more completed triples than a randomly-generated network of the same size and density. Similarly, the LCC had many patients with few shared providers (lowdegree nodes) and few patients with many shared providers (high-degree nodes) compared with a random network of the same size and density. Edgewise and dyadwise shared partner distributions also differed in the observed LCC and random networks with the observed LCC having more patients with multiple ESP and DSP compared with the random network, which indicated a large number of patients with one or two shared partners, and hardly any higher level multiples.
Mixing matrices is presented in patients who also took hydrocodone (i.e., homophily of opioid prescribing). From Table 6 opioid type, number of providers, average daily MME and age group had the highest modularity score and may explain some of the clustering observed in the network. From the perspective of the network connections and patient attributes, the GC was a reasonable representation of the full opioid prescription network.

Results from ERGMs:
Based on a null model with only the edge term to account for the number of connections in the network, the probability of a connection between any randomly selected two patients was 0.06 (i.e., density of the network). This baseline model was compared to models with more constraints. Although the null model  Model with main effects and differential homophily terms: The results of the model with differential homophily terms are presented in Table 9.          Excluding patients who saw < 3 providers or used < 3 pharmacies per year

FUTURE RESEARCH WORK
In the future we plan to incorporate structural properties of the observed opioid network as model covariates thereby allowing the observed network to be conditioned on observed degree distribution and level of transitivity. This will be implemented in R using a Bayesian approach which exhibits better convergence properties than non-Bayesian method used in this analysis. We also plan to analyze corresponding providerand pharmacy-based networks where connections between providers or pharmacies represent patient-sharing. These analyses will also use centrality measures to identify the most influential providers and pharmacies and evaluate whether patterns of opioid prescribing or dispensing vary by communities in the network.

A. NETWORK CHARACTERIZATION AND MEASUREMENTS
This Technical Appendix provides statistical formulations of key terms used in this thesis. The notations and definitions are adapted from Kolaczyk. 1 In general, let uppercase letters denote random variables and lowercase denote realizations of those random variables. We assume the observed network is fixed and does not vary over time.
We also assumed that we ascertained the full network sample. pre-specified subset of vertices and ⊆ E is the collection of edges to be found in G among that subset of vertices. A simple graph has no edges for which both ends connect to a single vertex (i.e., no loops) and no pairs of vertices with more than one edge between them (i.e., no multi-edges). Two vertices u,v ∈ V are said to be adjacent if joined by an edge in E, and two edges e 1 ,e 2 ∈ E are adjacent if joined by a common endpoint in A network therefore consists of a set of nodes and the relationships (ties, links, edges, connections) among them. The relationship can be directed or undirected and dichotomous (present or absent). All pairs of nodes in a network are dyads while all sets 3 nodes form triads. These dyads and triads can be linked or unlinked. A group of k nodes taking a star format with a node at the center linked to all others in the group is called k-star. An undirected network has two types of dyads (null or present) and four types of triads defined by the number of connected edges (0, 1, 2 or 3) and may have a 3star, 4-star, and 5-star formats. We also distinguish edgewise shared partnership (ESP) and dyad-wise shared partnership (DSP). A DSP is a linked or unlinked dyad where both members of the dyad are linked to a third network member. ESP is a subset of DSP with a linked dyad in which both members of the dyad also have a link to a third network member. The distribution of ESP in a network shows how many dyads have one shared partner, two shared partners, and so on. Similarly, the distribution of DSP shows the number of dyads in the network with one shared partner, two shared partners, and so on.
A bipartite network is a graph G = (V, E) such that the vertex set V may be partitioned into two disjoint sets, say V 1 and V 2 , and each edge in E has one endpoint in The graph G is said to be connected if every vertex is reachable from every other. A component of a graph is a maximally connected subgraph. Geodesic (distance) is the length of the shortest path(s) between the vertices (which we set equal to infinity if no such path exists). The diameter of the graph is the value of the longest distance in a graph.
Algebraic graph theory has several applications in social network analysis. The connectivity of a graph G may be captured and stored in an N v ×N v binary, symmetric, adjacency matrix A with entries: where A is non-zero for entries whose row-column indices correspond to vertices in G joined by an edge, and zero, for those that are not. The row sum A i+ = Σ j A i j is equal to the degree d i of vertex i. and by symmetry, A i+ = A +i . The structure of a graph G may also be captured in an N v ×N e binary, incidence matrix B with entries: Suppose that G = (V, E) is a graph corresponding to an observed social network among individuals i ∈ V, with a social tie between individuals i, j ∈ V indicated by an edge {i, j} ∈ E. Let Y i j = Y ji = 1 if {i, j} ∈ E, and zero if not. Y = [Y i j ] is the adjacency matrix for G, and treated as a random matrix.

Modularity:
The process of community detection can be approached as an optimization problem using computational algorithms developed for studying similar networks. 2,3 The algorithm detects subgroups within networks that are more inter-connected than would be expected by chance alone. [3][4][5] In our example, each provider was assigned to a single community, such that communities are comprised of distinct, non-overlapping groups of providers in the network. The null model adjusts for node degree so that patients with high nodal degree are more likely to be connected than those with low nodal degree thereby maintaining the expected degree distribution of the network. 4 The optimization process involves the maximization of the quantity: , where is nonzero if and only if node and are connected by a tie, and its value quantifies the number of providers the two patients share; is the degree of node , is the number of edges in the network (or their total weight in weighted networks), is the community assignment of node , and is the Kronecker delta which is equal to 1 if the arguments are identical, otherwise it is zero. We used the greedy optimization method which has been shown to perform well for a variety of networks 6 .

Centrality measures:
A patient that is connected to many other patients in a network is in a prominent or influential position within the network. This simplest measure of centrality is based on the notion that a patient with more direct connections in the network is more influential than one with fewer or no connections at all. The degree d v of a vertex v, in a network graph G = (V, E), is the number of edges in E incident upon v, that is, at distance one and mean degree is the average degree of all patients in the opioid network. Vertex degree is arguably the most widely used measure of vertex centrality. In our setting, the patients with higher degrees are more central because in many social settings people with more connections tend to be more influential. A patient's degree is the total number of other patients within the network who are connected to the patient through provider sharing. Degree centrality we can be standardized by dividing by the maximum possible value of |V|-1.
Given a network graph G, we define f d to be the fraction of vertices v ∈ V with degree d v = d. The collection {f d } d≥0 is called the degree distribution of G. The degree distribution provides a summary of the connectivity in the graph.
Another notion of a 'central role in the network' is that a vertex be 'close' to many other vertices. The standard approach, introduced by Sabidussi, is to let the centrality vary inversely with a measure of the total distance of a vertex from all others. 7 where dist(v,u) is the geodesic distance between the vertices u,v ∈ V. Using this formulation, the more central a node is, then the lower its total distance to all other nodes. The clustering coefficient is the ratio of total the number of connections that exist among neighbors of the patient in the network to the total number of potential connections that could exist if they were completely connected. It is used to describe the extent to which network neighbors of a particular patient are directly connected to each other and interpreted as the probability that any two randomly selected neighbors of a particular patient in the network are connected to each other.
Assortativity and mixing: Assortative mixing is the selective linking among vertices, according to a certain characteristic(s), and measures that quantify the extent of assortative mixing in a given network have been referred to as assortativity coefficients.
Suppose that each vertex in a graph G can be labeled according to one of M categories.
Let f ij be the fraction of edges in G that join a vertex in the i th category with a vertex in the j th category; denote the i th marginal row and column sums of the resulting matrix f by f i+ and f +i , respectively. We then define the assortativity coefficient r a to be r a = .
The value r a is equal to zero when the mixing in the graph is no different from that obtained through a random assignment of edges that preserves the marginal degree distribution. Similarly, it is equal to one when there is perfect assortative mixing (i.e., when edges only connect vertices of the same category). When the mixing is perfectly disassortative, the value takes its minimum value, that is, every edge in the graph connects vertices of two different categories.

B. The Exponential Random Graph Model (ERGM)
A discrete random vector Z is said to belong to an exponential family if its probability mass function may be expressed in the form , where θ ∈ is a p×1 vector of parameters, g(·) is a p-dimensional function of z, and ψ(θ ) is a normalization term, ensuring that P θ (·) sums to one over its range. The class of discrete exponential families includes many familiar distributions, such as the binomial, However, this entails a model with parameters, which is likely far too parameterized for many data sets.
In order to reduce the total number of parameters, it is common to impose an assumption of homogeneity across certain vertex pairs. For example, assuming homogeneity across all of G (i.e., θ ij ≡θ , for all {i, j}) yields P θ (Y = y) = , where L(y) = Σ i,j y ij = N e is the number of edges in the graph. In this case, the Bernoulli random graph model is recovered, with p = exp(θ )/[1+exp(θ )] .
Assumptions of complete independence among possible edges are largely untenable in practice. In general, Bernoulli-like random graphs lack the ability to reproduce many of the most basic structural characteristics observed in most real-world networks. However, the simple random graph model provides a baseline to compare with more complex models and assess improvements in model fit using simulation methods.
Markov Random Graphs: Frank and Strauss introduced the notion of Markov dependence for network graph models, which specifies that two possible edges are dependent whenever they share a vertex, conditional on all other possible edges, and independent if they do not. 12 That is, the presence or absence of {i, j} in the graph will depend upon that of {i,k}, for a given k j, even given information on the status of all In practice, it is common to include star counts S k no higher than k=2 or at most k=3, by setting =…= =0. This often leads to model degeneracy. Inclusion of a large number of higher order terms does not solve this problem. Partial conditional dependence assumption has been proposed to address issues of degeneracy. For example, Snijders et al proposed a solution by imposing a parametric constraint of the form upon the star parameter, for all k ≥ 2, for some larger than one. 13 This tactic combines all k-star statistics S k (y), for k ≥ 2, into a single alternating k-star statistic of the form and weighting that statistic by a single parameter θ AKS that takes into account the star effects of all orders simultaneously. The alternating signs allow the counts of k-stars of successively greater order to balance each other, rather than simply ballooning. We often assume that dependence between ties that do not share a node is due to the presence of other ties in the network. 14 To account for this partial conditional dependence, three non-linear terms are often added to the model: geometrically weighted degree (GWD), geometrically weighted DSP (GWDSP), and geometrically weighted ESP (GWESP). The statistic AKS λ (y) is a linear function of GWD count. The GWD term is designed to account for the decreasing degree distribution in observed networks while GWESP term is designed to account for clustering in observed networks. Finally, the GWDSP term accounts for the number of dyads with shared partners, often found within clusters in the network.

C. Constructing the Exponential Random Graph Models
Several packages are available for estimating network models. Our analysis was conducted in R-statnet, a suit of packages for building ERGMs in R. We first employed the null model which corresponds to the simple random graph model and can be written as: .
The model was estimated by maximum likelihood estimation and served as a comparator for assessing model fit as more useful and complex models were constructed.
Adding attributes: We first considered whether the addition of node attributes influenced the likelihood of a tie in the network. These nodal attributes accounted for the characteristics of each individual network member. To examine the effects of these attributes on the likelihood of a tie, these attributes were added to the model as main effects. The null and alternate hypotheses are: H 0 : There is no association between node attribute and the likelihood of a patient to form ties.
H a : There is association between node attribute and the likelihood of a patient to form ties.
In statnet, categorical and continuous main effects are added using nodefactor and nodecov(), respectively. Nodefactor main effect term adds multiple statistics to the model output, each corresponding to the number of times a node with the specified attribute is at one end of an edge. The corresponding to a categorical node attribute can be summarized as follow: The reference group which is omitted in the output can be changed using the base argument.
The nodecov main effect term adds one network statistic to the output that sums the attribute of interest for the two nodes in a dyad.
Interaction terms for nodal attributes to account for the attributes of both members of a dyad in the network. Homophily interaction terms were included in the model using nodematch. Differential homophily was requested by specifying diff=TRUE after the name of the attribute in a nodematch term. The homophily change statistics is defined as: And the differential homophily change statistics is as: A potential limitation is that models that include the interaction terms are dyadic independence models which assume that each dyad is independent of all other dyads in the model. where p is the number of parameters and N is the network size.
Both values of the AIC and BIC were used to compare nested and non-nested models.
These measures of model fit were developed for the analysis of data that are assumed to meet the independence of observation assumption. The null, main effect, and homophily models which assume dyadic independence were compared using deviance, AIC, and BIC. Models to account for non-uniform degree distribution and transitivity resulting from complex of dependence in observed social networks GWD, GWESP, and GWDSP were not evaluated in this analysis.