Research Article
BibTex RIS Cite

Examining the Effect of Metformin on Cell Death Mechanisms in Relation to Hippo Signaling in MDA-MB-231 Breast Cancer Cells

Year 2024, , 227 - 234, 30.06.2024

Özge Rencuzoğulları , Zeynep Gülşah Sonalp

https://doi.org/10.17776/csj.1398295

Abstract

Breast cancer is one of the most common cancer types in women in the world and our country. Antitumorigenic activity is achieved with various therapeutic drugs by directly suppressing the constantly active PI3K/Akt/mTOR signaling pathway or enabling AMPK activation. AMPK, a positive regulator of autophagy, ensures the induction of autophagy by suppressing the Akt/mTOR pathway. Metformin, an anti-diabetic drug, achieves its anti-tumorigenic effect by activating AMPK. Deregulation of the Hippo signaling pathway is a new therapeutic target because it causes cancer cells to become aggressive and evade cell death mechanisms. The study aims to reveal the effects of metformin treatment on Hippo signaling pathway activity on apoptosis and autophagy, depending on drug treatment in MDA-MB-231 breast cancer cells. Metformin decreased the cell viability through induction of mitochondria membrane potential loss in dose and time dependent manner in MDA-MB-231 cells. The colony forming potential of the MDA-MB-231 cells were suppressed by 10 mM metformin treatment which was induced apoptotic cell death and autophagy by increasing Bim, Bad, Bak and cleavage of caspase 3, 9, PARP and Beclin1, Atg5 and Atg7. Moreover, Hippo signaling related protein levels showed remarkable increase due to metformin treatment. It was shown that metformin treatment increased the activity of the hippo signaling pathway, resulting in the induction of apoptosis and autophagy

Keywords

Breast cancer Metformin Hippo signaling Apoptosis Autophagy

Supporting Institution

İstanbul Kültür Üniversitesi

Thanks

İstanbul Kültür Üniversitesi Bilimsel Araştırma Projeleri Koordinasyon Birimi

References

  • [1] Ferlay J., Colombet M., Soerjomataram I., Mathers C., Parkin DM, Piñeros M. Estimating the global cancer incidence and mortality in 2018: GLOBOCAN sources and methods, Int J Cancer, 144 (8) (2019) 1941–53.
  • [2] IARC. GLOBOCAN : Estimated Number of New Cases from 2020. Int Agency Res Cancer. 247 (22) (2020) 3087–3088.
  • [3] Erices R., Cubillos S., Aravena R., Santoro F., Marquez M., Orellana R. Diabetic concentrations of metformin inhibit platelet-mediated ovarian cancer cell progression, Oncotarget, 8 (13) (2017) 20865–20880.
  • [4] Wang YW., He SJ., Feng X., Cheng J., Luo YT., Tian L. Metformin: a review of its potential indications, Drug Des Devel Ther., 11 (2017) 2421–2429.
  • [5] Vancura A, Bu P, Bhagwat M, Zeng J, Vancurova I. Metformin as an Anticancer Agent, Trends Pharmacol Sci., 39 (10) (2018) 867–878.
  • [6] Sabit H., Abdel-Ghany SE., M Said OA., Mostafa MA., El-Zawahry M. Metformin Reshapes the Methylation Profile in Breast and Colorectal Cancer Cells, Asian Pac J Cancer Prev. 19 (10) (2018) 2991–2999.
  • [7] Wu L., Zhou B., Oshiro-Rapley N., Li M., Paulo JA., Webster CM., An Ancient, Unified Mechanism for Metformin Growth Inhibition in C. elegans and Cancer, Cell, 167 (7) (2016) 1705-1718.e13.
  • [8] Davies G., Lobanova L., Dawicki W., Groot G., Gordon JR., Bowen M. Metformin inhibits the development, and promotes the resensitization, of treatment-resistant breast cancer, PLoS One 12 (12) (2017) e0187191.
  • [9] Liang P., Jiang B., Li Y., Liu Z., Zhang P., Zhang M. Autophagy promotes angiogenesis via AMPK/Akt/mTOR signaling during the recovery of heat-denatured endothelial cells, Cell Death Dis. [Internet]. 9 (12) (2018) 1152-1160.
  • [10] Bu H., Liu D., Zhang G., Chen L., Song Z. AMPK/mTOR/ULK1 Axis-Mediated Pathway Participates in Apoptosis and Autophagy Induction by Oridonin in Colon Cancer DLD-1 Cells, Onco Targets Ther. 13 (2020) 8533–8545.
  • [11] Mo JS., Park HW., Guan KL. The Hippo signaling pathway in stem cell biology and cancer, EMBO Rep., 15 (6) (2020) 642–656.
  • [12] Yu FX., Guan KL. The Hippo pathway: regulators and regulations, Genes Dev., 27 (4) (2013) 355–371.
  • [13] Moroishi T., Hansen CG., Guan KL. The emerging roles of YAP and TAZ in cancer, Nat. Rev. Cancer, 15 (2) (2015) 73–79.
  • [14] Wei C., Wang Y., Li X. The role of Hippo signal pathway in breast cancer metastasis, Onco Targets Ther., 11 (1) (2018) 2185–2193.
  • [15] Yuan X., Wei W., Bao Q., Chen H., Jin P., Jiang W. Metformin inhibits glioma cells stemness and epithelial-mesenchymal transition via regulating YAP activity, Biomed Pharmacother, 102 (1) (2018) 263–270.
  • [16] Rencüzoğullari Ö., Arısan ED., Obakan Yerlikaya P., Çoker Gürkan A., Keskin B., Palavan Ünsal N. Inhibition of extracellular signal-regulated kinase potentiates the apoptotic and antimetastatic effects of cyclin-dependent kinase inhibitors on metastatic DU145 and PC3 prostate cancer cells, J Cell Biochem., 120 (4) (2019) 5558–5569.
  • [17] Akkoç Y., Berrak Ö., Arısan ED., Obakan P., Çoker-Gürkan A., Palavan-Ünsal N. Inhibition of PI3K signaling triggered apoptotic potential of curcumin which is hindered by Bcl-2 through activation of autophagy in MCF-7 cells, Biomed Pharmacother, 71 (1) (2015) 161–171.
  • [18] Obakan Yerlikaya P., Mehdizadehtapeh L., Rencüzoğullari Ö., Kuryayeva F., Çevikli SS., Özağar Ş. Gemcitabine in combination with epibrassinolide enhanced the apoptotic response in an ER stress-dependent manner and reduced the epithelial-mesenchymal transition in pancreatic cancer cells, Turkish. J. Biol., 46 (6) (2022) 439–457.
  • [19] Park YMM., Sandler DP. Making sense of associations between type 2 diabetes, metformin, and breast cancer risk, Br J Cancer, 125 (7) (2021) 909–910.
  • [20] Phoenix KN., Vumbaca F., Fox MM., Evans R., Claffey KP. Dietary energy availability affects primary and metastatic breast cancer and metformin efficacy, Breast Cancer Res Treat., 123 (2) (2010) 333–344.
  • [21] Rizvi F., Shaukat L., Azhar A., Jafri A., Aslam U., Imran-Ul-Haq H. Preclinical meritorious anticancer effects of Metformin against breast cancer: An In vivo trial, J. Taibah Univ Med Sci., 16 (4) (2021) 504–512.
  • [22] Zheng Y., Zhu J., Zhang H., Liu Y., Sun H. Metformin inhibits ovarian cancer growth and migration in vitro and in vivo by enhancing cisplatin cytotoxicity, Am J Transl Res., 10 (10) (2018) 3086–3098.
  • [23] Zordoky BNM., Bark D., Soltys CL., Sung MM., Dyck JRB. The anti-proliferative effect of metformin in triple-negative MDA-MB-231 breast cancer cells is highly dependent on glucose concentration: implications for cancer therapy and prevention, Biochim Biophys Acta, 1840 (6) (2014) 1943–1957.
  • [24] Kasznicki J., Sliwinska A., Drzewoski J. Metformin in cancer prevention and therapy, Ann Transl Med., 2 (6) (2014) 57-67.
  • [25] Amaral I., Silva C., Correia-Branco A., Martel F. Metformin interferes with glucose cellular uptake by both estrogen and progesterone receptor-positive (MCF-7) and triple-negative (MDA-MB-231) breast cancer cell lines: PS156, Porto Biomed., 2 (5) (2014) 218-225.
  • [26] Marinello PC., da Silva TNX., Panis C., Neves AF., Machado KL., Borges FH. Mechanism of metformin action in MCF-7 and MDA-MB-231 human breast cancer cells involves oxidative stress generation, DNA damage, and transforming growth factor β1 induction, Tumor Biol., 37 (4) (2016) 53375–53346.
  • [27] Sena P., Mancini S., Benincasa M., Mariani F., Palumbo C., Roncucci L. Metformin induces apoptosis and alters cellular responses to oxidative stress in Ht29 colon cancer cells: Preliminary findings, Int J Mol Sci., 19 (5) (2018) 45-56.
  • [28] Yenmiş G., Beşli N., Yaprak Saraç E., Hocaoğlu Emre FS., Şenol K., Kanıgür G. Metformin promotes apoptosis in primary breast cancer cells by downregulation of cyclin D1 and upregulation of P53 through an AMPK-alpha independent mechanism, Turkish J Med Sci., 51 (2) (2021) 826–834.
  • [29] Fujita E., Egashira J., Urase K., Kuida K., Momoi T. Caspase-9 processing by caspase-3 via a feedback amplification loop in vivo, Cell Death Differ., 8 (4) (2001) 335–344.
  • [30] Jang JH., Song IH., Sung EG., Lee TJ., Kim JY. Metformin-induced apoptosis facilitates degradation of the cellular caspase 8 (FLICE)-like inhibitory protein through a caspase-dependent pathway in human renal cell carcinoma A498 cells, Oncol Lett., 16 (2) (2018) 2030-2040.
  • [31] Jang JH., Sung EG., Song IH., Lee TJ., Kim JY. Metformin induces caspase-dependent and caspase-independent apoptosis in human bladder cancer T24 cells, Anticancer Drugs, 31 (7) 2020 655–662.
  • [32] Gozuacik D., Kimchi A. DAPk protein family and cancer, Autophagy, 2 (2) (2006) 74–79.
  • [33] Wijshake T., Zou Z., Chen B. Tumor-suppressor function of Beclin 1 in breast cancer cells requires E-cadherin, Proc Natl Acad Sci USA, 118 (5) (2021) e2020478118.
  • [34] Cai M., Hu Z., Liu J., Gao J., Liu C., Liu D. Beclin 1 Expression in Ovarian Tissues and Its Effects on Ovarian Cancer Prognosis, Int J Mol Sci., 215 (4) (2014) 5292.
  • [35] Ge J., Chen Z., Huang J., Chen J., Yuan W., Deng Z. Upregulation of Autophagy-Related Gene-5 (ATG-5) Is Associated with Chemoresistance in Human Gastric Cancer, PLoS One, 9 (10) (2014).
  • [36] Maskey D., Yousefi S., Schmid I., Zlobec I., Perren A., Friis R. ATG5 is induced by DNA-damaging agents and promotes mitotic catastrophe independent of autophagy, Nat Commun., 4(1) (2013) 2130.
  • [37] Zhou H, Qian X, Xu N, Zhang S, Zhu G, Zhang Y. Disruption of Atg7-dependent autophagy causes electromotility disturbances, outer hair cell loss, and deafness in mice, Cell Death Dis., 11 (10) (2020).
  • [38] Liu S, Yue C, Chen H, Chen Y, Li G. Corrigendum: Metformin promotes beclin1-dependent autophagy to inhibit the progression of gastric cancer, Onco Targets Ther., 13 (1) (2020) 4445—4455.
  • [39] Fu M, Hu Y, Lan T, Guan KL, Luo T, Luo M. The Hippo signalling pathway and its implications in human health and diseases, Signal Transduct Target Ther., 7 (1) (2020) 1–20.
  • [40] Liu J, Li J, Chen H, Metformin suppresses proliferation and invasion of drug-resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020).
  • [41] Han Y, Liu D, Li L. Increased expression of TAZ and associated upregulation of PD-L1 in cervical cancer, Cancer Cell Int., 21 (1) (2021) 592.
  • [42] Liu J, Li J, Chen H, Wang R, Li P, Miao Y. Metformin suppresses proliferation and invasion of drug‐resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020) 5786–5796.
  • [43] Wang L, Wang M, Hu C, Li P, Qiao Y, Xia Y. Protein salvador homolog 1 acts as a tumor suppressor and is modulated by hypermethylation in pancreatic ductal adenocarcinoma, Oncotarget, 8 (38) (2017) 62953–62961.
  • [44] Liu J, Li J, Chen H, Wang R, Li P, Miao Y. Metformin suppresses proliferation and invasion of drug-resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020) 5786–5796.

Year 2024, , 227 - 234, 30.06.2024

Özge Rencuzoğulları , Zeynep Gülşah Sonalp

https://doi.org/10.17776/csj.1398295

Abstract

References

  • [1] Ferlay J., Colombet M., Soerjomataram I., Mathers C., Parkin DM, Piñeros M. Estimating the global cancer incidence and mortality in 2018: GLOBOCAN sources and methods, Int J Cancer, 144 (8) (2019) 1941–53.
  • [2] IARC. GLOBOCAN : Estimated Number of New Cases from 2020. Int Agency Res Cancer. 247 (22) (2020) 3087–3088.
  • [3] Erices R., Cubillos S., Aravena R., Santoro F., Marquez M., Orellana R. Diabetic concentrations of metformin inhibit platelet-mediated ovarian cancer cell progression, Oncotarget, 8 (13) (2017) 20865–20880.
  • [4] Wang YW., He SJ., Feng X., Cheng J., Luo YT., Tian L. Metformin: a review of its potential indications, Drug Des Devel Ther., 11 (2017) 2421–2429.
  • [5] Vancura A, Bu P, Bhagwat M, Zeng J, Vancurova I. Metformin as an Anticancer Agent, Trends Pharmacol Sci., 39 (10) (2018) 867–878.
  • [6] Sabit H., Abdel-Ghany SE., M Said OA., Mostafa MA., El-Zawahry M. Metformin Reshapes the Methylation Profile in Breast and Colorectal Cancer Cells, Asian Pac J Cancer Prev. 19 (10) (2018) 2991–2999.
  • [7] Wu L., Zhou B., Oshiro-Rapley N., Li M., Paulo JA., Webster CM., An Ancient, Unified Mechanism for Metformin Growth Inhibition in C. elegans and Cancer, Cell, 167 (7) (2016) 1705-1718.e13.
  • [8] Davies G., Lobanova L., Dawicki W., Groot G., Gordon JR., Bowen M. Metformin inhibits the development, and promotes the resensitization, of treatment-resistant breast cancer, PLoS One 12 (12) (2017) e0187191.
  • [9] Liang P., Jiang B., Li Y., Liu Z., Zhang P., Zhang M. Autophagy promotes angiogenesis via AMPK/Akt/mTOR signaling during the recovery of heat-denatured endothelial cells, Cell Death Dis. [Internet]. 9 (12) (2018) 1152-1160.
  • [10] Bu H., Liu D., Zhang G., Chen L., Song Z. AMPK/mTOR/ULK1 Axis-Mediated Pathway Participates in Apoptosis and Autophagy Induction by Oridonin in Colon Cancer DLD-1 Cells, Onco Targets Ther. 13 (2020) 8533–8545.
  • [11] Mo JS., Park HW., Guan KL. The Hippo signaling pathway in stem cell biology and cancer, EMBO Rep., 15 (6) (2020) 642–656.
  • [12] Yu FX., Guan KL. The Hippo pathway: regulators and regulations, Genes Dev., 27 (4) (2013) 355–371.
  • [13] Moroishi T., Hansen CG., Guan KL. The emerging roles of YAP and TAZ in cancer, Nat. Rev. Cancer, 15 (2) (2015) 73–79.
  • [14] Wei C., Wang Y., Li X. The role of Hippo signal pathway in breast cancer metastasis, Onco Targets Ther., 11 (1) (2018) 2185–2193.
  • [15] Yuan X., Wei W., Bao Q., Chen H., Jin P., Jiang W. Metformin inhibits glioma cells stemness and epithelial-mesenchymal transition via regulating YAP activity, Biomed Pharmacother, 102 (1) (2018) 263–270.
  • [16] Rencüzoğullari Ö., Arısan ED., Obakan Yerlikaya P., Çoker Gürkan A., Keskin B., Palavan Ünsal N. Inhibition of extracellular signal-regulated kinase potentiates the apoptotic and antimetastatic effects of cyclin-dependent kinase inhibitors on metastatic DU145 and PC3 prostate cancer cells, J Cell Biochem., 120 (4) (2019) 5558–5569.
  • [17] Akkoç Y., Berrak Ö., Arısan ED., Obakan P., Çoker-Gürkan A., Palavan-Ünsal N. Inhibition of PI3K signaling triggered apoptotic potential of curcumin which is hindered by Bcl-2 through activation of autophagy in MCF-7 cells, Biomed Pharmacother, 71 (1) (2015) 161–171.
  • [18] Obakan Yerlikaya P., Mehdizadehtapeh L., Rencüzoğullari Ö., Kuryayeva F., Çevikli SS., Özağar Ş. Gemcitabine in combination with epibrassinolide enhanced the apoptotic response in an ER stress-dependent manner and reduced the epithelial-mesenchymal transition in pancreatic cancer cells, Turkish. J. Biol., 46 (6) (2022) 439–457.
  • [19] Park YMM., Sandler DP. Making sense of associations between type 2 diabetes, metformin, and breast cancer risk, Br J Cancer, 125 (7) (2021) 909–910.
  • [20] Phoenix KN., Vumbaca F., Fox MM., Evans R., Claffey KP. Dietary energy availability affects primary and metastatic breast cancer and metformin efficacy, Breast Cancer Res Treat., 123 (2) (2010) 333–344.
  • [21] Rizvi F., Shaukat L., Azhar A., Jafri A., Aslam U., Imran-Ul-Haq H. Preclinical meritorious anticancer effects of Metformin against breast cancer: An In vivo trial, J. Taibah Univ Med Sci., 16 (4) (2021) 504–512.
  • [22] Zheng Y., Zhu J., Zhang H., Liu Y., Sun H. Metformin inhibits ovarian cancer growth and migration in vitro and in vivo by enhancing cisplatin cytotoxicity, Am J Transl Res., 10 (10) (2018) 3086–3098.
  • [23] Zordoky BNM., Bark D., Soltys CL., Sung MM., Dyck JRB. The anti-proliferative effect of metformin in triple-negative MDA-MB-231 breast cancer cells is highly dependent on glucose concentration: implications for cancer therapy and prevention, Biochim Biophys Acta, 1840 (6) (2014) 1943–1957.
  • [24] Kasznicki J., Sliwinska A., Drzewoski J. Metformin in cancer prevention and therapy, Ann Transl Med., 2 (6) (2014) 57-67.
  • [25] Amaral I., Silva C., Correia-Branco A., Martel F. Metformin interferes with glucose cellular uptake by both estrogen and progesterone receptor-positive (MCF-7) and triple-negative (MDA-MB-231) breast cancer cell lines: PS156, Porto Biomed., 2 (5) (2014) 218-225.
  • [26] Marinello PC., da Silva TNX., Panis C., Neves AF., Machado KL., Borges FH. Mechanism of metformin action in MCF-7 and MDA-MB-231 human breast cancer cells involves oxidative stress generation, DNA damage, and transforming growth factor β1 induction, Tumor Biol., 37 (4) (2016) 53375–53346.
  • [27] Sena P., Mancini S., Benincasa M., Mariani F., Palumbo C., Roncucci L. Metformin induces apoptosis and alters cellular responses to oxidative stress in Ht29 colon cancer cells: Preliminary findings, Int J Mol Sci., 19 (5) (2018) 45-56.
  • [28] Yenmiş G., Beşli N., Yaprak Saraç E., Hocaoğlu Emre FS., Şenol K., Kanıgür G. Metformin promotes apoptosis in primary breast cancer cells by downregulation of cyclin D1 and upregulation of P53 through an AMPK-alpha independent mechanism, Turkish J Med Sci., 51 (2) (2021) 826–834.
  • [29] Fujita E., Egashira J., Urase K., Kuida K., Momoi T. Caspase-9 processing by caspase-3 via a feedback amplification loop in vivo, Cell Death Differ., 8 (4) (2001) 335–344.
  • [30] Jang JH., Song IH., Sung EG., Lee TJ., Kim JY. Metformin-induced apoptosis facilitates degradation of the cellular caspase 8 (FLICE)-like inhibitory protein through a caspase-dependent pathway in human renal cell carcinoma A498 cells, Oncol Lett., 16 (2) (2018) 2030-2040.
  • [31] Jang JH., Sung EG., Song IH., Lee TJ., Kim JY. Metformin induces caspase-dependent and caspase-independent apoptosis in human bladder cancer T24 cells, Anticancer Drugs, 31 (7) 2020 655–662.
  • [32] Gozuacik D., Kimchi A. DAPk protein family and cancer, Autophagy, 2 (2) (2006) 74–79.
  • [33] Wijshake T., Zou Z., Chen B. Tumor-suppressor function of Beclin 1 in breast cancer cells requires E-cadherin, Proc Natl Acad Sci USA, 118 (5) (2021) e2020478118.
  • [34] Cai M., Hu Z., Liu J., Gao J., Liu C., Liu D. Beclin 1 Expression in Ovarian Tissues and Its Effects on Ovarian Cancer Prognosis, Int J Mol Sci., 215 (4) (2014) 5292.
  • [35] Ge J., Chen Z., Huang J., Chen J., Yuan W., Deng Z. Upregulation of Autophagy-Related Gene-5 (ATG-5) Is Associated with Chemoresistance in Human Gastric Cancer, PLoS One, 9 (10) (2014).
  • [36] Maskey D., Yousefi S., Schmid I., Zlobec I., Perren A., Friis R. ATG5 is induced by DNA-damaging agents and promotes mitotic catastrophe independent of autophagy, Nat Commun., 4(1) (2013) 2130.
  • [37] Zhou H, Qian X, Xu N, Zhang S, Zhu G, Zhang Y. Disruption of Atg7-dependent autophagy causes electromotility disturbances, outer hair cell loss, and deafness in mice, Cell Death Dis., 11 (10) (2020).
  • [38] Liu S, Yue C, Chen H, Chen Y, Li G. Corrigendum: Metformin promotes beclin1-dependent autophagy to inhibit the progression of gastric cancer, Onco Targets Ther., 13 (1) (2020) 4445—4455.
  • [39] Fu M, Hu Y, Lan T, Guan KL, Luo T, Luo M. The Hippo signalling pathway and its implications in human health and diseases, Signal Transduct Target Ther., 7 (1) (2020) 1–20.
  • [40] Liu J, Li J, Chen H, Metformin suppresses proliferation and invasion of drug-resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020).
  • [41] Han Y, Liu D, Li L. Increased expression of TAZ and associated upregulation of PD-L1 in cervical cancer, Cancer Cell Int., 21 (1) (2021) 592.
  • [42] Liu J, Li J, Chen H, Wang R, Li P, Miao Y. Metformin suppresses proliferation and invasion of drug‐resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020) 5786–5796.
  • [43] Wang L, Wang M, Hu C, Li P, Qiao Y, Xia Y. Protein salvador homolog 1 acts as a tumor suppressor and is modulated by hypermethylation in pancreatic ductal adenocarcinoma, Oncotarget, 8 (38) (2017) 62953–62961.
  • [44] Liu J, Li J, Chen H, Wang R, Li P, Miao Y. Metformin suppresses proliferation and invasion of drug-resistant breast cancer cells by activation of the Hippo pathway, J Cell Mol Med., 24 (10) (2020) 5786–5796.
There are 44 citations in total.

Details

Primary Language English
Subjects Biochemistry and Cell Biology (Other)
Journal Section Natural Sciences
Authors

Özge Rencuzoğulları 0000-0002-2157-1289

Zeynep Gülşah Sonalp 0000-0001-9638-371X

Publication Date June 30, 2024
Submission Date November 30, 2023
Acceptance Date May 30, 2024
Published in Issue Year 2024

Cite

APA Rencuzoğulları, Ö., & Sonalp, Z. G. (2024). Examining the Effect of Metformin on Cell Death Mechanisms in Relation to Hippo Signaling in MDA-MB-231 Breast Cancer Cells. Cumhuriyet Science Journal, 45(2), 227-234. https://doi.org/10.17776/csj.1398295