基于Gram-Schmidt变换的有监督变量聚类

刘瑞平; 王惠文; 王珊珊

doi:10.13700/j.bh.1001-5965.2019.0050

基于Gram-Schmidt变换的有监督变量聚类

doi: 10.13700/j.bh.1001-5965.2019.0050

刘瑞平¹,
王惠文^{1, 2},
王珊珊^{1, 3, ,}

1.
北京航空航天大学经济管理学院, 北京 100083
2.
北京航空航天大学大数据科学与脑机智能高精尖创新中心, 北京 100083
3.
城市运行应急保障模拟技术北京市重点实验室, 北京 100083

基金项目:

国家自然科学基金 71420107025

国家自然科学基金 11701023

详细信息

作者简介:
刘瑞平   女, 博士研究生。主要研究方向:高维数据的降维方法及应用

王惠文   女, 博士, 教授, 博士生导师。主要研究方向:经济管理中复杂数据统计分析的理论、方法与应用

王珊珊   女, 博士, 助理教授, 硕士生导师。主要研究方向:高维复杂数据分析、半参数统计、机器学习、统计算法及应用

通讯作者:
王珊珊, E-mail: sswang@buaa.edu.cn

中图分类号: O212.4
计量
- 文章访问数: 640
- HTML全文浏览量: 71
- PDF下载量: 321
- 被引次数: 0
出版历程
- 收稿日期: 2019-02-16
- 录用日期: 2019-03-15
- 网络出版日期: 2019-10-20

Supervised clustering of variables based on Gram-Schmidt transformation

LIU Ruiping¹,
WANG Huiwen^{1, 2},
WANG Shanshan^{1, 3
, ,}

1.
School of Economics and Management, Beihang University, Beijing 100083, China
2.
Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing 100083, China
3.
Beijing Key Laboratory of Emergency Support Simulation Technologies for City Operations, Beijing 100083, China

Funds:

National Natural Science Foundation of China 71420107025

National Natural Science Foundation of China 11701023

More Information

Corresponding author: WANG Shanshan, E-mail: sswang@buaa.edu.cn

摘要

摘要:
为进一步研究回归模型中高维数据的降维方法，提出基于Gram-Schmidt变换的新的有监督变量聚类（SCV-GS）方法。该方法未采用以潜变量为聚类中心的层次聚类，而是借用变量扫描思想，依次挑出对响应变量有重要贡献的关键变量，并将其作为聚类中心。SCV-GS方法基于Gram-Schmidt变换，对变量之间的高度相关性进行批量处理，并得到聚类结果；同时，结合偏最小二乘思想，提出新的同一性度量，并以此来选取最佳聚合参数。SCV-GS不仅可以快速得到变量聚类结果，而且可识别出对响应变量的解释及预测起关键作用的变量类。仿真表明该聚类方法运算速度显著提升，而且所得潜变量对应的回归系数的估计结果与对照方法表现一致；实例分析表明该方法具有更好的解释性和预测能力。
- 降维 /
- 变量聚类 /
- 回归 /
- 高度相关 /
- Gram-Schmidt变换
Abstract:
In order to study the dimension reduction method of high-dimensional data based on regression model further, and the supervised clustering of variables algorithm based on Gram-Schmidt transformation (SCV-GS) is proposed. SCV-GS uses the key variables selected in turn by the variable screening idea as the clustering center, which is different from the hierarchical variable clustering around latent variables. High correlation among variables is processed based on Gram-Schmidt transformation and the clustering results are obtained. At the same time, combined with the concept of partial least squares, a new criterion for "homogeneity" is proposed to select the optimal clustering parameters. SCV-GS can not only get the variable clustering results quickly, but also identify the most relevant variable groups and in what kind of structure the variables work to influence the response variable. Simulation results show that the calculation speed is significantly improved by SCV-GS, and the estimated regression coefficients corresponding to the latent variables are consistent with the comparison method. Real data analysis shows that SCV-GS performs better in interpretation and prediction.
- dimension reduction /
- variable clustering /
- regression /
- high correlation /
- Gram-Schmidt transformation

HTML全文

图 1 SCV-LV与SCV-GS方法运行时间对比

Figure 1. Comparison of computation time (in seconds) between SCV-LV and SCV-GS

下载: 全尺寸图片幻灯片

表 1 SCV-GS方法变量聚类结果

Table 1. Variable clustering results by SCV-GS

变量	G₁	G₂	G₃	G₄	G₅
x₁	100	0	0	0	0
x₂	100	0	0	0	0
x₃	100	0	0	0	0
x₄	100	0	0	0	0
x₅	100	0	0	0	0
x₆	100	0	0	0	0
x₇	100	0	0	0	0
x₈	100	0	0	0	0
x₉	100	0	0	0	0
x₁₀	100	0	0	0	0
x₁₁	100	0	0	0	0
x₁₂	100	0	0	0	0
x₁₃	100	0	0	0	0
x₁₄	100	0	0	0	0
x₁₅	100	0	0	0	0
x₁₆	100	0	0	0	0
x₁₇	99	0	0	0	0
x₁₈	100	0	0	0	0
x₁₉	99	0	0	0	0
x₂₀	100	0	0	0	0
x₂₁	0	97	2	0	0
x₂₂	0	100	0	0	0
x₂₃	0	99	1	0	0
x₂₄	0	100	0	0	0
x₂₅	0	99	0	0	0
x₂₆	0	100	0	0	0
x₂₇	0	99	1	0	0
x₂₈	0	98	1	0	0
x₂₉	0	99	0	0	0
x₃₀	0	99	1	0	0
x₃₁	0	99	1	0	0
x₃₂	0	98	2	0	0
x₃₃	0	99	1	0	0
x₃₄	0	100	0	0	0
x₃₅	0	100	0	0	0
x₃₆	0	100	0	0	0
x₃₇	0	100	0	0	0
x₃₈	0	99	0	0	1
x₃₉	0	100	0	0	0
x₄₀	0	100	0	0	0
x₄₁	0	0	100	0	0
x₄₂	0	0	100	0	0
x₄₃	0	0	100	0	0
x₄₄	0	0	100	0	0
x₄₅	0	0	100	0	0
x₄₆	0	1	99	0	0
x₄₇	0	0	99	0	1
x₄₈	0	0	100	0	0
x₄₉	0	0	100	0	0
x₅₀	0	0	100	0	0
x₅₁	0	0	0	100	0
x₅₂	0	0	0	100	0
x₅₃	0	0	0	100	0
x₅₄	0	0	0	100	0
x₅₅	0	0	0	100	0
x₅₆	0	0	0	100	0
x₅₇	0	0	0	100	0
x₅₈	0	0	0	100	0
x₅₉	0	0	0	100	0
x₆₀	0	0	0	100	0
x₆₁	0	0	0	0	100
x₆₂	0	0	0	0	100
x₆₃	0	0	0	0	100
x₆₄	0	0	0	0	100
x₆₅	0	0	0	0	100
x₆₆	0	0	0	0	100
x₆₇	0	0	0	0	100
x₆₈	0	0	0	0	100
x₆₉	0	0	0	0	100
x₇₀	0	0	0	0	100
x₇₁	0	0	0	0	100
x₇₂	0	0	0	0	100
x₇₃	0	0	0	0	100
x₇₄	0	0	0	0	100
x₇₅	0	0	0	0	100
x₇₆	0	0	0	0	100
x₇₇	0	0	0	0	100
x₇₈	0	0	0	0	100
x₇₉	0	0	0	0	100
x₈₀	0	0	0	0	100

下载: 导出CSV

表 2 SCV-LV与SCV-GS方法所得潜变量回归系数的估计结果

Table 2. Estimated regression coefficients by SCV-LVand SCV-GS as a function of latent variables

方法
SCV-LV	-0.07(0.44)	-0.08(0.43)	0.03(0.28)
SCV-GS	-0.10 (0.44)	-0.10(0.43)	0.02(0.28)

下载: 导出CSV

表 3 SCV-LV与SCV-GS方法作用于实例数据集所得结果

Table 3. Results on real dataset by SCV-LV and SCV-GS

方法	K		变量
SCV-LV	1	0.83	x₁~x₁₉
SCV-GS	16	0.87	x₂, x₃

下载: 导出CSV

参考文献(23)

[1]	TIBSHIRANI R.Regression shrinkage and selection via the lasso:A retrospective[J].Journal of the Royal Statistical Society:Series B(Statistica Methodology), 2011, 73(3):273-282. doi: 10.1111/j.1467-9868.2011.00771.x
[2]	ZOU H, HASTIE T.Regularization and variable selection via the elastic net[J].Journal of the Royal Statistical Society:Series B(Statistical Methodology), 2005, 67(2):301-320. doi: 10.1111/j.1467-9868.2005.00503.x
[3]	FAN J Q, LV J C.Sure independence screening for ultrahigh dimensional feature space[J].Journal of the Royal Statistical Society:Series B(Statistical Methodology), 2008, 70(5):849-911. doi: 10.1111/j.1467-9868.2008.00674.x
[4]	WANG H S.Forward regression for ultra-high dimensional variable screening[J].Journal of the American Statistical Association, 2009, 104(488):1512-1524. doi: 10.1198/jasa.2008.tm08516
[5]	ZOU H, HASTIE T, TIBSHIRANI R.Sparse principal component analysis[J].Journal of Computational and Graphical Statistics, 2006, 15(2):265-286. doi: 10.1198/106186006X113430
[6]	CHUN H, KELEŞ S.Sparse partial least squares regression for simultaneous dimension reduction and variable selection[J].Journal of the Royal Statistical Society:Series B(Statistical Methodology), 2010, 72(1):3-25. doi: 10.1111/j.1467-9868.2009.00723.x
[7]	CHEN M K, VIGNEAU E.Supervised clustering of variables[J].Advances in Data Analysis and Classification, 2016, 10(1):85-101. http://d.old.wanfangdata.com.cn/OAPaper/oai_doaj-articles_7d57b83193b227c3e090a9eeb28155dd
[8]	JOLLIFFE I T.Discarding variables in a principal component analysis.I:Artificial data[J].Applied Statistics, 1972, 21(2):160-173. doi: 10.2307/2346488
[9]	HASTIE T, TIBSHIRANI R, BOTSTEIN D, et al.Supervised harvesting of expression trees[J].Genome Biology, 2001, 2(1):research0003-1. http://d.old.wanfangdata.com.cn/OAPaper/oai_pubmedcentral.nih.gov_17599
[10]	VIGNEAU E, QANNARI E.Clustering of variables around latent components[J].Communications in Statistics-Simulation and Computation, 2003, 32(4):1131-1150. doi: 10.1081/SAC-120023882
[11]	VIGNEAU E, CHEN M, QANNARI E M.ClustVarLV:An R package for the clustering of variables around latent variables[J].The R Journal, 2015, 7(2):134-148. doi: 10.32614/RJ-2015-026
[12]	VIGNEAU E.Segmentation of a panel of consumers with missing data[J].Food Quality and Preference, 2018, 67:10-17. doi: 10.1016/j.foodqual.2017.04.010
[13]	CARIOU V, QANNARI E M, RUTLEDGE D N, et al.ComDim:From multiblock data analysis to path modeling[J].Food Quality and Preference, 2018, 67:27-34. doi: 10.1016/j.foodqual.2017.02.012
[14]	BJÖRCK Å.Numerics of gram-schmidt orthogonalization[J].Linear Algebra and Its Applications, 1994, 197-198:297-316. doi: 10.1016/0024-3795(94)90493-6
[15]	LEON S J, BJÖRCK A, GANDER W.Gram-Schmidt orthogonalization:100 years and more[J].Numerical Linear Algebra with Applications, 2013, 20:492-532. doi: 10.1002/nla.1839
[16]	CHEN S, BILLINGS S A, LUO W.Orthogonal least squares methods and their application to non-linear system identification[J].International Journal of Control, 1989, 50(5):1873-1896. doi: 10.1080/00207178908953472
[17]	STOPPIGLIA H, DREYFUS G, DUBOIS R, et al.Ranking a random feature for variable and feature selection[J].Journal of Machine Learning Research, 2003, 3:1399-1414. http://cn.bing.com/academic/profile?id=521405dcd1fbe6041d8601c946a9376a&encoded=0&v=paper_preview&mkt=zh-cn
[18]	王惠文, 仪彬, 叶明.基于主基底分析的变量筛选[J].北京航空航天大学学报, 2008, 34(11):1288-1291. https://bhxb.buaa.edu.cn/CN/abstract/abstract8983.shtml WANG H W, YI B, YE M.Variable selection based on principal basis analysis[J].Journal of Beijing University of Aeronautics and Astronautics, 2008, 34(11):1288-1291(in Chinese). https://bhxb.buaa.edu.cn/CN/abstract/abstract8983.shtml
[19]	王惠文, 陈梅玲, SAPORTA G.基于Gram-Schmidt过程的判别变量筛选方法[J].北京航空航天大学学报, 2011, 37(8):958-961. https://bhxb.buaa.edu.cn/CN/abstract/abstract12041.shtml WANG H W, CHEN M L, SAPORTA G.Variable selection in discriminant analysis based on Gram-Schmidt process[J].Journal of Beijing University of Aeronautics and Astronautics, 2011, 37(8):958-961(in Chinese). https://bhxb.buaa.edu.cn/CN/abstract/abstract12041.shtml
[20]	LIU R P, WANG H W, WANG S S.Functional variable selection via Gram-Schmidt orthogonalization for multiple functional linear regression[J].Journal of Statistical Computation and Simulation, 2018, 88(18):3664-3680. doi: 10.1080/00949655.2018.1530776
[21]	FISHER R.On the probable error of a coefficient of correlation deduced from a small sample[J].Metron, 1921, 1(4):3-32. http://cn.bing.com/academic/profile?id=3a072e95cc2650c0bd2f3ff83efe37e2&encoded=0&v=paper_preview&mkt=zh-cn
[22]	FRANK L E, FRIEDMAN J H.A statistical view of some chemometrics regression tools[J].Technometrics, 1993, 35(2):109-135. doi: 10.1080/00401706.1993.10485033
[23]	MANGOLD W D, BEAN L, ADAMS D.The impact of intercollegiate athletics on graduation rates among major ncaa division I universities:Implications for college persistence theory and practice[J].Journal of Higher Education, 2003, 74(5):540-562. http://cn.bing.com/academic/profile?id=4c72dc2da8185d458e56835a53338f23&encoded=0&v=paper_preview&mkt=zh-cn