Search

Your search keyword '"Huang, Yanping"' showing total 2,752 results
Start Over Author "Huang, Yanping"
2,752 results on '"Huang, Yanping"'

1. Strong Coupling from Inclusive Semi-leptonic Decay of Charmed Mesons

Author

Wu, Jinfei, Lou, Xinchou, Huang, Yanping, Li, Gang, Ye, Jingbo, Che, Yuzhi, and Ruan, Manqi

Subjects
High Energy Physics - Phenomenology, High Energy Physics - Experiment
Abstract

Employing the heavy quark expansion with kinetic scheme, we study the strong interaction coupling $\alpha_S$ at an energy corresponding to the charm quark mass $m_c$ using the data on the semi-leptonic decays of charmed mesons from the BESIII and CLEO-c. The values of the $\alpha_{s}(m_c^{2})$ from the $D^0$, $D^+$, and $D_s^+$ are extracted and found to be consistent. Combining the semileptonic decay widths of the $D^0$, $D^+$, and $D_s^+$, we determine $\alpha_{s}(m_c^{2})$ to be $0.377\pm0.008\pm0.114$, where the first and second uncertainty are the experimental and theoretical uncertainty separately. This result is in good agreement with the world average of $\alpha_S$ running from Z pole to the $m_c$., Comment: 7 pages, 6 figures

Published
2024

2. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Author

DeepSeek-AI, Liu, Aixin, Feng, Bei, Wang, Bin, Wang, Bingxuan, Liu, Bo, Zhao, Chenggang, Dengr, Chengqi, Ruan, Chong, Dai, Damai, Guo, Daya, Yang, Dejian, Chen, Deli, Ji, Dongjie, Li, Erhang, Lin, Fangyun, Luo, Fuli, Hao, Guangbo, Chen, Guanting, Li, Guowei, Zhang, H., Xu, Hanwei, Yang, Hao, Zhang, Haowei, Ding, Honghui, Xin, Huajian, Gao, Huazuo, Li, Hui, Qu, Hui, Cai, J. L., Liang, Jian, Guo, Jianzhong, Ni, Jiaqi, Li, Jiashi, Chen, Jin, Yuan, Jingyang, Qiu, Junjie, Song, Junxiao, Dong, Kai, Gao, Kaige, Guan, Kang, Wang, Lean, Zhang, Lecong, Xu, Lei, Xia, Leyi, Zhao, Liang, Zhang, Liyue, Li, Meng, Wang, Miaojun, Zhang, Mingchuan, Zhang, Minghua, Tang, Minghui, Li, Mingming, Tian, Ning, Huang, Panpan, Wang, Peiyi, Zhang, Peng, Zhu, Qihao, Chen, Qinyu, Du, Qiushi, Chen, R. J., Jin, R. L., Ge, Ruiqi, Pan, Ruizhe, Xu, Runxin, Chen, Ruyi, Li, S. S., Lu, Shanghao, Zhou, Shangyan, Chen, Shanhuang, Wu, Shaoqing, Ye, Shengfeng, Ma, Shirong, Wang, Shiyu, Zhou, Shuang, Yu, Shuiping, Zhou, Shunfeng, Zheng, Size, Wang, T., Pei, Tian, Yuan, Tian, Sun, Tianyu, Xiao, W. L., Zeng, Wangding, An, Wei, Liu, Wen, Liang, Wenfeng, Gao, Wenjun, Zhang, Wentao, Li, X. Q., Jin, Xiangyue, Wang, Xianzu, Bi, Xiao, Liu, Xiaodong, Wang, Xiaohan, Shen, Xiaojin, Chen, Xiaokang, Chen, Xiaosha, Nie, Xiaotao, Sun, Xiaowen, Wang, Xiaoxiang, Liu, Xin, Xie, Xin, Yu, Xingkai, Song, Xinnan, Zhou, Xinyi, Yang, Xinyu, Lu, Xuan, Su, Xuecheng, Wu, Y., Li, Y. K., Wei, Y. X., Zhu, Y. X., Xu, Yanhong, Huang, Yanping, Li, Yao, Zhao, Yao, Sun, Yaofeng, Li, Yaohui, Wang, Yaohui, Zheng, Yi, Zhang, Yichao, Xiong, Yiliang, Zhao, Yilong, He, Ying, Tang, Ying, Piao, Yishi, Dong, Yixin, Tan, Yixuan, Liu, Yiyuan, Wang, Yongji, Guo, Yongqiang, Zhu, Yuchen, Wang, Yuduan, Zou, Yuheng, Zha, Yukun, Ma, Yunxian, Yan, Yuting, You, Yuxiang, Liu, Yuxuan, Ren, Z. Z., Ren, Zehui, Sha, Zhangli, Fu, Zhe, Huang, Zhen, Zhang, Zhen, Xie, Zhenda, Hao, Zhewen, Shao, Zhihong, Wen, Zhiniu, Xu, Zhipeng, Zhang, Zhongyu, Li, Zhuoshu, Wang, Zihan, Gu, Zihui, Li, Zilin, and Xie, Ziwei

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence
Abstract

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

Published
2024

3. Stylus: Automatic Adapter Selection for Diffusion Models

Author

Luo, Michael, Wong, Justin, Trabucco, Brandon, Huang, Yanping, Gonzalez, Joseph E., Chen, Zhifeng, Salakhutdinov, Ruslan, and Stoica, Ion

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Graphics, Computer Science - Machine Learning
Abstract

Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters-most of which are highly customized with insufficient descriptions. This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion.github.io for more., Comment: Project Website: https://stylus-diffusion.github.io

Published
2024

4. Gemini: A Family of Highly Capable Multimodal Models

Author

Gemini Team, Anil, Rohan, Borgeaud, Sebastian, Alayrac, Jean-Baptiste, Yu, Jiahui, Soricut, Radu, Schalkwyk, Johan, Dai, Andrew M., Hauth, Anja, Millican, Katie, Silver, David, Johnson, Melvin, Antonoglou, Ioannis, Schrittwieser, Julian, Glaese, Amelia, Chen, Jilin, Pitler, Emily, Lillicrap, Timothy, Lazaridou, Angeliki, Firat, Orhan, Molloy, James, Isard, Michael, Barham, Paul R., Hennigan, Tom, Lee, Benjamin, Viola, Fabio, Reynolds, Malcolm, Xu, Yuanzhong, Doherty, Ryan, Collins, Eli, Meyer, Clemens, Rutherford, Eliza, Moreira, Erica, Ayoub, Kareem, Goel, Megha, Krawczyk, Jack, Du, Cosmo, Chi, Ed, Cheng, Heng-Tze, Ni, Eric, Shah, Purvi, Kane, Patrick, Chan, Betty, Faruqui, Manaal, Severyn, Aliaksei, Lin, Hanzhao, Li, YaGuang, Cheng, Yong, Ittycheriah, Abe, Mahdieh, Mahdis, Chen, Mia, Sun, Pei, Tran, Dustin, Bagri, Sumit, Lakshminarayanan, Balaji, Liu, Jeremiah, Orban, Andras, Güra, Fabian, Zhou, Hao, Song, Xinying, Boffy, Aurelien, Ganapathy, Harish, Zheng, Steven, Choe, HyunJeong, Weisz, Ágoston, Zhu, Tao, Lu, Yifeng, Gopal, Siddharth, Kahn, Jarrod, Kula, Maciej, Pitman, Jeff, Shah, Rushin, Taropa, Emanuel, Merey, Majd Al, Baeuml, Martin, Chen, Zhifeng, Shafey, Laurent El, Zhang, Yujing, Sercinoglu, Olcan, Tucker, George, Piqueras, Enrique, Krikun, Maxim, Barr, Iain, Savinov, Nikolay, Danihelka, Ivo, Roelofs, Becca, White, Anaïs, Andreassen, Anders, von Glehn, Tamara, Yagati, Lakshman, Kazemi, Mehran, Gonzalez, Lucas, Khalman, Misha, Sygnowski, Jakub, Frechette, Alexandre, Smith, Charlotte, Culp, Laura, Proleev, Lev, Luan, Yi, Chen, Xi, Lottes, James, Schucher, Nathan, Lebron, Federico, Rrustemi, Alban, Clay, Natalie, Crone, Phil, Kocisky, Tomas, Zhao, Jeffrey, Perz, Bartek, Yu, Dian, Howard, Heidi, Bloniarz, Adam, Rae, Jack W., Lu, Han, Sifre, Laurent, Maggioni, Marcello, Alcober, Fred, Garrette, Dan, Barnes, Megan, Thakoor, Shantanu, Austin, Jacob, Barth-Maron, Gabriel, Wong, William, Joshi, Rishabh, Chaabouni, Rahma, Fatiha, Deeni, Ahuja, Arun, Tomar, Gaurav Singh, Senter, Evan, Chadwick, Martin, Kornakov, Ilya, Attaluri, Nithya, Iturrate, Iñaki, Liu, Ruibo, Li, Yunxuan, Cogan, Sarah, Chen, Jeremy, Jia, Chao, Gu, Chenjie, Zhang, Qiao, Grimstad, Jordan, Hartman, Ale Jakse, Garcia, Xavier, Pillai, Thanumalayan Sankaranarayana, Devlin, Jacob, Laskin, Michael, Casas, Diego de Las, Valter, Dasha, Tao, Connie, Blanco, Lorenzo, Badia, Adrià Puigdomènech, Reitter, David, Chen, Mianna, Brennan, Jenny, Rivera, Clara, Brin, Sergey, Iqbal, Shariq, Surita, Gabriela, Labanowski, Jane, Rao, Abhi, Winkler, Stephanie, Parisotto, Emilio, Gu, Yiming, Olszewska, Kate, Addanki, Ravi, Miech, Antoine, Louis, Annie, Teplyashin, Denis, Brown, Geoff, Catt, Elliot, Balaguer, Jan, Xiang, Jackie, Wang, Pidong, Ashwood, Zoe, Briukhov, Anton, Webson, Albert, Ganapathy, Sanjay, Sanghavi, Smit, Kannan, Ajay, Chang, Ming-Wei, Stjerngren, Axel, Djolonga, Josip, Sun, Yuting, Bapna, Ankur, Aitchison, Matthew, Pejman, Pedram, Michalewski, Henryk, Yu, Tianhe, Wang, Cindy, Love, Juliette, Ahn, Junwhan, Bloxwich, Dawn, Han, Kehang, Humphreys, Peter, Sellam, Thibault, Bradbury, James, Godbole, Varun, Samangooei, Sina, Damoc, Bogdan, Kaskasoli, Alex, Arnold, Sébastien M. R., Vasudevan, Vijay, Agrawal, Shubham, Riesa, Jason, Lepikhin, Dmitry, Tanburn, Richard, Srinivasan, Srivatsan, Lim, Hyeontaek, Hodkinson, Sarah, Shyam, Pranav, Ferret, Johan, Hand, Steven, Garg, Ankush, Paine, Tom Le, Li, Jian, Li, Yujia, Giang, Minh, Neitz, Alexander, Abbas, Zaheer, York, Sarah, Reid, Machel, Cole, Elizabeth, Chowdhery, Aakanksha, Das, Dipanjan, Rogozińska, Dominika, Nikolaev, Vitaliy, Sprechmann, Pablo, Nado, Zachary, Zilka, Lukas, Prost, Flavien, He, Luheng, Monteiro, Marianne, Mishra, Gaurav, Welty, Chris, Newlan, Josh, Jia, Dawei, Allamanis, Miltiadis, Hu, Clara Huiyi, de Liedekerke, Raoul, Gilmer, Justin, Saroufim, Carl, Rijhwani, Shruti, Hou, Shaobo, Shrivastava, Disha, Baddepudi, Anirudh, Goldin, Alex, Ozturel, Adnan, Cassirer, Albin, Xu, Yunhan, Sohn, Daniel, Sachan, Devendra, Amplayo, Reinald Kim, Swanson, Craig, Petrova, Dessie, Narayan, Shashi, Guez, Arthur, Brahma, Siddhartha, Landon, Jessica, Patel, Miteyan, Zhao, Ruizhe, Villela, Kevin, Wang, Luyu, Jia, Wenhao, Rahtz, Matthew, Giménez, Mai, Yeung, Legg, Keeling, James, Georgiev, Petko, Mincu, Diana, Wu, Boxi, Haykal, Salem, Saputro, Rachel, Vodrahalli, Kiran, Qin, James, Cankara, Zeynep, Sharma, Abhanshu, Fernando, Nick, Hawkins, Will, Neyshabur, Behnam, Kim, Solomon, Hutter, Adrian, Agrawal, Priyanka, Castro-Ros, Alex, Driessche, George van den, Wang, Tao, Yang, Fan, Chang, Shuo-yiin, Komarek, Paul, McIlroy, Ross, Lučić, Mario, Zhang, Guodong, Farhan, Wael, Sharman, Michael, Natsev, Paul, Michel, Paul, Bansal, Yamini, Qiao, Siyuan, Cao, Kris, Shakeri, Siamak, Butterfield, Christina, Chung, Justin, Rubenstein, Paul Kishan, Agrawal, Shivani, Mensch, Arthur, Soparkar, Kedar, Lenc, Karel, Chung, Timothy, Pope, Aedan, Maggiore, Loren, Kay, Jackie, Jhakra, Priya, Wang, Shibo, Maynez, Joshua, Phuong, Mary, Tobin, Taylor, Tacchetti, Andrea, Trebacz, Maja, Robinson, Kevin, Katariya, Yash, Riedel, Sebastian, Bailey, Paige, Xiao, Kefan, Ghelani, Nimesh, Aroyo, Lora, Slone, Ambrose, Houlsby, Neil, Xiong, Xuehan, Yang, Zhen, Gribovskaya, Elena, Adler, Jonas, Wirth, Mateo, Lee, Lisa, Li, Music, Kagohara, Thais, Pavagadhi, Jay, Bridgers, Sophie, Bortsova, Anna, Ghemawat, Sanjay, Ahmed, Zafarali, Liu, Tianqi, Powell, Richard, Bolina, Vijay, Iinuma, Mariko, Zablotskaia, Polina, Besley, James, Chung, Da-Woon, Dozat, Timothy, Comanescu, Ramona, Si, Xiance, Greer, Jeremy, Su, Guolong, Polacek, Martin, Kaufman, Raphaël Lopez, Tokumine, Simon, Hu, Hexiang, Buchatskaya, Elena, Miao, Yingjie, Elhawaty, Mohamed, Siddhant, Aditya, Tomasev, Nenad, Xing, Jinwei, Greer, Christina, Miller, Helen, Ashraf, Shereen, Roy, Aurko, Zhang, Zizhao, Ma, Ada, Filos, Angelos, Besta, Milos, Blevins, Rory, Klimenko, Ted, Yeh, Chih-Kuan, Changpinyo, Soravit, Mu, Jiaqi, Chang, Oscar, Pajarskas, Mantas, Muir, Carrie, Cohen, Vered, Lan, Charline Le, Haridasan, Krishna, Marathe, Amit, Hansen, Steven, Douglas, Sholto, Samuel, Rajkumar, Wang, Mingqiu, Austin, Sophia, Lan, Chang, Jiang, Jiepu, Chiu, Justin, Lorenzo, Jaime Alonso, Sjösund, Lars Lowe, Cevey, Sébastien, Gleicher, Zach, Avrahami, Thi, Boral, Anudhyan, Srinivasan, Hansa, Selo, Vittorio, May, Rhys, Aisopos, Konstantinos, Hussenot, Léonard, Soares, Livio Baldini, Baumli, Kate, Chang, Michael B., Recasens, Adrià, Caine, Ben, Pritzel, Alexander, Pavetic, Filip, Pardo, Fabio, Gergely, Anita, Frye, Justin, Ramasesh, Vinay, Horgan, Dan, Badola, Kartikeya, Kassner, Nora, Roy, Subhrajit, Dyer, Ethan, Campos, Víctor Campos, Tomala, Alex, Tang, Yunhao, Badawy, Dalia El, White, Elspeth, Mustafa, Basil, Lang, Oran, Jindal, Abhishek, Vikram, Sharad, Gong, Zhitao, Caelles, Sergi, Hemsley, Ross, Thornton, Gregory, Feng, Fangxiaoyu, Stokowiec, Wojciech, Zheng, Ce, Thacker, Phoebe, Ünlü, Çağlar, Zhang, Zhishuai, Saleh, Mohammad, Svensson, James, Bileschi, Max, Patil, Piyush, Anand, Ankesh, Ring, Roman, Tsihlas, Katerina, Vezer, Arpi, Selvi, Marco, Shevlane, Toby, Rodriguez, Mikel, Kwiatkowski, Tom, Daruki, Samira, Rong, Keran, Dafoe, Allan, FitzGerald, Nicholas, Gu-Lemberg, Keren, Khan, Mina, Hendricks, Lisa Anne, Pellat, Marie, Feinberg, Vladimir, Cobon-Kerr, James, Sainath, Tara, Rauh, Maribeth, Hashemi, Sayed Hadi, Ives, Richard, Hasson, Yana, Noland, Eric, Cao, Yuan, Byrd, Nathan, Hou, Le, Wang, Qingze, Sottiaux, Thibault, Paganini, Michela, Lespiau, Jean-Baptiste, Moufarek, Alexandre, Hassan, Samer, Shivakumar, Kaushik, van Amersfoort, Joost, Mandhane, Amol, Joshi, Pratik, Goyal, Anirudh, Tung, Matthew, Brock, Andrew, Sheahan, Hannah, Misra, Vedant, Li, Cheng, Rakićević, Nemanja, Dehghani, Mostafa, Liu, Fangyu, Mittal, Sid, Oh, Junhyuk, Noury, Seb, Sezener, Eren, Huot, Fantine, Lamm, Matthew, De Cao, Nicola, Chen, Charlie, Mudgal, Sidharth, Stella, Romina, Brooks, Kevin, Vasudevan, Gautam, Liu, Chenxi, Chain, Mainak, Melinkeri, Nivedita, Cohen, Aaron, Wang, Venus, Seymore, Kristie, Zubkov, Sergey, Goel, Rahul, Yue, Summer, Krishnakumaran, Sai, Albert, Brian, Hurley, Nate, Sano, Motoki, Mohananey, Anhad, Joughin, Jonah, Filonov, Egor, Kępa, Tomasz, Eldawy, Yomna, Lim, Jiawern, Rishi, Rahul, Badiezadegan, Shirin, Bos, Taylor, Chang, Jerry, Jain, Sanil, Padmanabhan, Sri Gayatri Sundara, Puttagunta, Subha, Krishna, Kalpesh, Baker, Leslie, Kalb, Norbert, Bedapudi, Vamsi, Kurzrok, Adam, Lei, Shuntong, Yu, Anthony, Litvin, Oren, Zhou, Xiang, Wu, Zhichun, Sobell, Sam, Siciliano, Andrea, Papir, Alan, Neale, Robby, Bragagnolo, Jonas, Toor, Tej, Chen, Tina, Anklin, Valentin, Wang, Feiran, Feng, Richie, Gholami, Milad, Ling, Kevin, Liu, Lijuan, Walter, Jules, Moghaddam, Hamid, Kishore, Arun, Adamek, Jakub, Mercado, Tyler, Mallinson, Jonathan, Wandekar, Siddhinita, Cagle, Stephen, Ofek, Eran, Garrido, Guillermo, Lombriser, Clemens, Mukha, Maksim, Sun, Botu, Mohammad, Hafeezul Rahman, Matak, Josip, Qian, Yadi, Peswani, Vikas, Janus, Pawel, Yuan, Quan, Schelin, Leif, David, Oana, Garg, Ankur, He, Yifan, Duzhyi, Oleksii, Älgmyr, Anton, Lottaz, Timothée, Li, Qi, Yadav, Vikas, Xu, Luyao, Chinien, Alex, Shivanna, Rakesh, Chuklin, Aleksandr, Li, Josie, Spadine, Carrie, Wolfe, Travis, Mohamed, Kareem, Das, Subhabrata, Dai, Zihang, He, Kyle, von Dincklage, Daniel, Upadhyay, Shyam, Maurya, Akanksha, Chi, Luyan, Krause, Sebastian, Salama, Khalid, Rabinovitch, Pam G, M, Pavan Kumar Reddy, Selvan, Aarush, Dektiarev, Mikhail, Ghiasi, Golnaz, Guven, Erdem, Gupta, Himanshu, Liu, Boyi, Sharma, Deepak, Shtacher, Idan Heimlich, Paul, Shachi, Akerlund, Oscar, Aubet, François-Xavier, Huang, Terry, Zhu, Chen, Zhu, Eric, Teixeira, Elico, Fritze, Matthew, Bertolini, Francesco, Marinescu, Liana-Eleonora, Bölle, Martin, Paulus, Dominik, Gupta, Khyatti, Latkar, Tejasi, Chang, Max, Sanders, Jason, Wilson, Roopa, Wu, Xuewei, Tan, Yi-Xuan, Thiet, Lam Nguyen, Doshi, Tulsee, Lall, Sid, Mishra, Swaroop, Chen, Wanming, Luong, Thang, Benjamin, Seth, Lee, Jasmine, Andrejczuk, Ewa, Rabiej, Dominik, Ranjan, Vipul, Styrc, Krzysztof, Yin, Pengcheng, Simon, Jon, Harriott, Malcolm Rose, Bansal, Mudit, Robsky, Alexei, Bacon, Geoff, Greene, David, Mirylenka, Daniil, Zhou, Chen, Sarvana, Obaid, Goyal, Abhimanyu, Andermatt, Samuel, Siegler, Patrick, Horn, Ben, Israel, Assaf, Pongetti, Francesco, Chen, Chih-Wei "Louis", Selvatici, Marco, Silva, Pedro, Wang, Kathie, Tolins, Jackson, Guu, Kelvin, Yogev, Roey, Cai, Xiaochen, Agostini, Alessandro, Shah, Maulik, Nguyen, Hung, Donnaile, Noah Ó, Pereira, Sébastien, Friso, Linda, Stambler, Adam, Kuang, Chenkai, Romanikhin, Yan, Geller, Mark, Yan, ZJ, Jang, Kane, Lee, Cheng-Chun, Fica, Wojciech, Malmi, Eric, Tan, Qijun, Banica, Dan, Balle, Daniel, Pham, Ryan, Huang, Yanping, Avram, Diana, Shi, Hongzhi, Singh, Jasjot, Hidey, Chris, Ahuja, Niharika, Saxena, Pranab, Dooley, Dan, Potharaju, Srividya Pranavi, O'Neill, Eileen, Gokulchandran, Anand, Foley, Ryan, Zhao, Kai, Dusenberry, Mike, Liu, Yuan, Mehta, Pulkit, Kotikalapudi, Ragha, Safranek-Shrader, Chalence, Goodman, Andrew, Kessinger, Joshua, Globen, Eran, Kolhar, Prateek, Gorgolewski, Chris, Ibrahim, Ali, Song, Yang, Eichenbaum, Ali, Brovelli, Thomas, Potluri, Sahitya, Lahoti, Preethi, Baetu, Cip, Ghorbani, Ali, Chen, Charles, Crawford, Andy, Pal, Shalini, Sridhar, Mukund, Gurita, Petru, Mujika, Asier, Petrovski, Igor, Cedoz, Pierre-Louis, Li, Chenmei, Chen, Shiyuan, Santo, Niccolò Dal, Goyal, Siddharth, Punjabi, Jitesh, Kappaganthu, Karthik, Kwak, Chester, LV, Pallavi, Velury, Sarmishta, Choudhury, Himadri, Hall, Jamie, Shah, Premal, Figueira, Ricardo, Thomas, Matt, Lu, Minjie, Zhou, Ting, Kumar, Chintu, Jurdi, Thomas, Chikkerur, Sharat, Ma, Yenai, Yu, Adams, Kwak, Soo, Ähdel, Victor, Rajayogam, Sujeevan, Choma, Travis, Liu, Fei, Barua, Aditya, Ji, Colin, Park, Ji Ho, Hellendoorn, Vincent, Bailey, Alex, Bilal, Taylan, Zhou, Huanjie, Khatir, Mehrdad, Sutton, Charles, Rzadkowski, Wojciech, Macintosh, Fiona, Shagin, Konstantin, Medina, Paul, Liang, Chen, Zhou, Jinjing, Shah, Pararth, Bi, Yingying, Dankovics, Attila, Banga, Shipra, Lehmann, Sabine, Bredesen, Marissa, Lin, Zifan, Hoffmann, John Eric, Lai, Jonathan, Chung, Raynald, Yang, Kai, Balani, Nihal, Bražinskas, Arthur, Sozanschi, Andrei, Hayes, Matthew, Alcalde, Héctor Fernández, Makarov, Peter, Chen, Will, Stella, Antonio, Snijders, Liselotte, Mandl, Michael, Kärrman, Ante, Nowak, Paweł, Wu, Xinyi, Dyck, Alex, Vaidyanathan, Krishnan, R, Raghavender, Mallet, Jessica, Rudominer, Mitch, Johnston, Eric, Mittal, Sushil, Udathu, Akhil, Christensen, Janara, Verma, Vishal, Irving, Zach, Santucci, Andreas, Elsayed, Gamaleldin, Davoodi, Elnaz, Georgiev, Marin, Tenney, Ian, Hua, Nan, Cideron, Geoffrey, Leurent, Edouard, Alnahlawi, Mahmoud, Georgescu, Ionut, Wei, Nan, Zheng, Ivy, Scandinaro, Dylan, Jiang, Heinrich, Snoek, Jasper, Sundararajan, Mukund, Wang, Xuezhi, Ontiveros, Zack, Karo, Itay, Cole, Jeremy, Rajashekhar, Vinu, Tumeh, Lara, Ben-David, Eyal, Jain, Rishub, Uesato, Jonathan, Datta, Romina, Bunyan, Oskar, Wu, Shimu, Zhang, John, Stanczyk, Piotr, Zhang, Ye, Steiner, David, Naskar, Subhajit, Azzam, Michael, Johnson, Matthew, Paszke, Adam, Chiu, Chung-Cheng, Elias, Jaume Sanchez, Mohiuddin, Afroz, Muhammad, Faizan, Miao, Jin, Lee, Andrew, Vieillard, Nino, Park, Jane, Zhang, Jiageng, Stanway, Jeff, Garmon, Drew, Karmarkar, Abhijit, Dong, Zhe, Lee, Jong, Kumar, Aviral, Zhou, Luowei, Evens, Jonathan, Isaac, William, Irving, Geoffrey, Loper, Edward, Fink, Michael, Arkatkar, Isha, Chen, Nanxin, Shafran, Izhak, Petrychenko, Ivan, Chen, Zhe, Jia, Johnson, Levskaya, Anselm, Zhu, Zhenkai, Grabowski, Peter, Mao, Yu, Magni, Alberto, Yao, Kaisheng, Snaider, Javier, Casagrande, Norman, Palmer, Evan, Suganthan, Paul, Castaño, Alfonso, Giannoumis, Irene, Kim, Wooyeol, Rybiński, Mikołaj, Sreevatsa, Ashwin, Prendki, Jennifer, Soergel, David, Goedeckemeyer, Adrian, Gierke, Willi, Jafari, Mohsen, Gaba, Meenu, Wiesner, Jeremy, Wright, Diana Gage, Wei, Yawen, Vashisht, Harsha, Kulizhskaya, Yana, Hoover, Jay, Le, Maigo, Li, Lu, Iwuanyanwu, Chimezie, Liu, Lu, Ramirez, Kevin, Khorlin, Andrey, Cui, Albert, LIN, Tian, Wu, Marcus, Aguilar, Ricardo, Pallo, Keith, Chakladar, Abhishek, Perng, Ginger, Abellan, Elena Allica, Zhang, Mingyang, Dasgupta, Ishita, Kushman, Nate, Penchev, Ivo, Repina, Alena, Wu, Xihui, van der Weide, Tom, Ponnapalli, Priya, Kaplan, Caroline, Simsa, Jiri, Li, Shuangfeng, Dousse, Olivier, Piper, Jeff, Ie, Nathan, Pasumarthi, Rama, Lintz, Nathan, Vijayakumar, Anitha, Andor, Daniel, Valenzuela, Pedro, Lui, Minnie, Paduraru, Cosmin, Peng, Daiyi, Lee, Katherine, Zhang, Shuyuan, Greene, Somer, Nguyen, Duc Dung, Kurylowicz, Paula, Hardin, Cassidy, Dixon, Lucas, Janzer, Lili, Choo, Kiam, Feng, Ziqiang, Zhang, Biao, Singhal, Achintya, Du, Dayou, McKinnon, Dan, Antropova, Natasha, Bolukbasi, Tolga, Keller, Orgad, Reid, David, Finchelstein, Daniel, Raad, Maria Abi, Crocker, Remi, Hawkins, Peter, Dadashi, Robert, Gaffney, Colin, Franko, Ken, Bulanova, Anna, Leblond, Rémi, Chung, Shirley, Askham, Harry, Cobo, Luis C., Xu, Kelvin, Fischer, Felix, Xu, Jun, Sorokin, Christina, Alberti, Chris, Lin, Chu-Cheng, Evans, Colin, Dimitriev, Alek, Forbes, Hannah, Banarse, Dylan, Tung, Zora, Omernick, Mark, Bishop, Colton, Sterneck, Rachel, Jain, Rohan, Xia, Jiawei, Amid, Ehsan, Piccinno, Francesco, Wang, Xingyu, Banzal, Praseem, Mankowitz, Daniel J., Polozov, Alex, Krakovna, Victoria, Brown, Sasha, Bateni, MohammadHossein, Duan, Dennis, Firoiu, Vlad, Thotakuri, Meghana, Natan, Tom, Geist, Matthieu, Girgin, Ser tan, Li, Hui, Ye, Jiayu, Roval, Ofir, Tojo, Reiko, Kwong, Michael, Lee-Thorp, James, Yew, Christopher, Sinopalnikov, Danila, Ramos, Sabela, Mellor, John, Sharma, Abhishek, Wu, Kathy, Miller, David, Sonnerat, Nicolas, Vnukov, Denis, Greig, Rory, Beattie, Jennifer, Caveness, Emily, Bai, Libin, Eisenschlos, Julian, Korchemniy, Alex, Tsai, Tomy, Jasarevic, Mimi, Kong, Weize, Dao, Phuong, Zheng, Zeyu, Liu, Frederick, Zhu, Rui, Teh, Tian Huey, Sanmiya, Jason, Gladchenko, Evgeny, Trdin, Nejc, Toyama, Daniel, Rosen, Evan, Tavakkol, Sasan, Xue, Linting, Elkind, Chen, Woodman, Oliver, Carpenter, John, Papamakarios, George, Kemp, Rupert, Kafle, Sushant, Grunina, Tanya, Sinha, Rishika, Talbert, Alice, Wu, Diane, Owusu-Afriyie, Denese, Thornton, Chloe, Pont-Tuset, Jordi, Narayana, Pradyumna, Li, Jing, Fatehi, Saaber, Wieting, John, Ajmeri, Omar, Uria, Benigno, Ko, Yeongil, Knight, Laura, Héliou, Amélie, Niu, Ning, Gu, Shane, Pang, Chenxi, Li, Yeqing, Levine, Nir, Stolovich, Ariel, Santamaria-Fernandez, Rebeca, Goenka, Sonam, Yustalim, Wenny, Strudel, Robin, Elqursh, Ali, Deck, Charlie, Lee, Hyo, Li, Zonglin, Levin, Kyle, Hoffmann, Raphael, Holtmann-Rice, Dan, Bachem, Olivier, Arora, Sho, Koh, Christy, Yeganeh, Soheil Hassas, Põder, Siim, Tariq, Mukarram, Sun, Yanhua, Ionita, Lucian, Seyedhosseini, Mojtaba, Tafti, Pouya, Liu, Zhiyu, Gulati, Anmol, Liu, Jasmine, Ye, Xinyu, Chrzaszcz, Bart, Wang, Lily, Sethi, Nikhil, Li, Tianrun, Brown, Ben, Singh, Shreya, Fan, Wei, Parisi, Aaron, Stanton, Joe, Koverkathu, Vinod, Choquette-Choo, Christopher A., Li, Yunjie, Lu, TJ, Shroff, Prakash, Varadarajan, Mani, Bahargam, Sanaz, Willoughby, Rob, Gaddy, David, Desjardins, Guillaume, Cornero, Marco, Robenek, Brona, Mittal, Bhavishya, Albrecht, Ben, Shenoy, Ashish, Moiseev, Fedor, Jacobsson, Henrik, Ghaffarkhah, Alireza, Rivière, Morgane, Walton, Alanna, Crepy, Clément, Parrish, Alicia, Zhou, Zongwei, Farabet, Clement, Radebaugh, Carey, Srinivasan, Praveen, van der Salm, Claudia, Fidjeland, Andreas, Scellato, Salvatore, Latorre-Chimoto, Eri, Klimczak-Plucińska, Hanna, Bridson, David, de Cesare, Dario, Hudson, Tom, Mendolicchio, Piermaria, Walker, Lexi, Morris, Alex, Mauger, Matthew, Guseynov, Alexey, Reid, Alison, Odoom, Seth, Loher, Lucia, Cotruta, Victor, Yenugula, Madhavi, Grewe, Dominik, Petrushkina, Anastasia, Duerig, Tom, Sanchez, Antonio, Yadlowsky, Steve, Shen, Amy, Globerson, Amir, Webb, Lynette, Dua, Sahil, Li, Dong, Bhupatiraju, Surya, Hurt, Dan, Qureshi, Haroon, Agarwal, Ananth, Shani, Tomer, Eyal, Matan, Khare, Anuj, Belle, Shreyas Rammohan, Wang, Lei, Tekur, Chetan, Kale, Mihir Sanjay, Wei, Jinliang, Sang, Ruoxin, Saeta, Brennan, Liechty, Tyler, Sun, Yi, Zhao, Yao, Lee, Stephan, Nayak, Pandu, Fritz, Doug, Vuyyuru, Manish Reddy, Aslanides, John, Vyas, Nidhi, Wicke, Martin, Ma, Xiao, Eltyshev, Evgenii, Martin, Nina, Cate, Hardie, Manyika, James, Amiri, Keyvan, Kim, Yelin, Xiong, Xi, Kang, Kai, Luisier, Florian, Tripuraneni, Nilesh, Madras, David, Guo, Mandy, Waters, Austin, Wang, Oliver, Ainslie, Joshua, Baldridge, Jason, Zhang, Han, Pruthi, Garima, Bauer, Jakob, Yang, Feng, Mansour, Riham, Gelman, Jason, Xu, Yang, Polovets, George, Liu, Ji, Cai, Honglong, Chen, Warren, Sheng, XiangHai, Xue, Emily, Ozair, Sherjil, Angermueller, Christof, Li, Xiaowei, Sinha, Anoop, Wang, Weiren, Wiesinger, Julia, Koukoumidis, Emmanouil, Tian, Yuan, Iyer, Anand, Gurumurthy, Madhu, Goldenson, Mark, Shah, Parashar, Blake, MK, Yu, Hongkun, Urbanowicz, Anthony, Palomaki, Jennimaria, Fernando, Chrisantha, Durden, Ken, Mehta, Harsh, Momchev, Nikola, Rahimtoroghi, Elahe, Georgaki, Maria, Raul, Amit, Ruder, Sebastian, Redshaw, Morgan, Lee, Jinhyuk, Zhou, Denny, Jalan, Komal, Li, Dinghua, Hechtman, Blake, Schuh, Parker, Nasr, Milad, Milan, Kieran, Mikulik, Vladimir, Franco, Juliana, Green, Tim, Nguyen, Nam, Kelley, Joe, Mahendru, Aroma, Hu, Andrea, Howland, Joshua, Vargas, Ben, Hui, Jeffrey, Bansal, Kshitij, Rao, Vikram, Ghiya, Rakesh, Wang, Emma, Ye, Ke, Sarr, Jean Michel, Preston, Melanie Moranski, Elish, Madeleine, Li, Steve, Kaku, Aakash, Gupta, Jigar, Pasupat, Ice, Juan, Da-Cheng, Someswar, Milan, M., Tejvi, Chen, Xinyun, Amini, Aida, Fabrikant, Alex, Chu, Eric, Dong, Xuanyi, Muthal, Amruta, Buthpitiya, Senaka, Jauhari, Sarthak, Khandelwal, Urvashi, Hitron, Ayal, Ren, Jie, Rinaldi, Larissa, Drath, Shahar, Dabush, Avigail, Jiang, Nan-Jiang, Godhia, Harshal, Sachs, Uli, Chen, Anthony, Fan, Yicheng, Taitelbaum, Hagai, Noga, Hila, Dai, Zhuyun, Wang, James, Hamer, Jenny, Ferng, Chun-Sung, Elkind, Chenel, Atias, Aviel, Lee, Paulina, Listík, Vít, Carlen, Mathias, van de Kerkhof, Jan, Pikus, Marcin, Zaher, Krunoslav, Müller, Paul, Zykova, Sasha, Stefanec, Richard, Gatsko, Vitaly, Hirnschall, Christoph, Sethi, Ashwin, Xu, Xingyu Federico, Ahuja, Chetan, Tsai, Beth, Stefanoiu, Anca, Feng, Bo, Dhandhania, Keshav, Katyal, Manish, Gupta, Akshay, Parulekar, Atharva, Pitta, Divya, Zhao, Jing, Bhatia, Vivaan, Bhavnani, Yashodha, Alhadlaq, Omar, Li, Xiaolin, Danenberg, Peter, Tu, Dennis, Pine, Alex, Filippova, Vera, Ghosh, Abhipso, Limonchik, Ben, Urala, Bhargava, Lanka, Chaitanya Krishna, Clive, Derik, Li, Edward, Wu, Hao, Hongtongsak, Kevin, Li, Ianna, Thakkar, Kalind, Omarov, Kuanysh, Majmundar, Kushal, Alverson, Michael, Kucharski, Michael, Patel, Mohak, Jain, Mudit, Zabelin, Maksim, Pelagatti, Paolo, Kohli, Rohan, Kumar, Saurabh, Kim, Joseph, Sankar, Swetha, Shah, Vineet, Ramachandruni, Lakshmi, Zeng, Xiangkai, Bariach, Ben, Weidinger, Laura, Vu, Tu, Andreev, Alek, He, Antoine, Hui, Kevin, Kashem, Sheleem, Subramanya, Amar, Hsiao, Sissie, Hassabis, Demis, Kavukcuoglu, Koray, Sadovsky, Adam, Le, Quoc, Strohman, Trevor, Wu, Yonghui, Petrov, Slav, Dean, Jeffrey, and Vinyals, Oriol

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Computer Vision and Pattern Recognition
Abstract

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

Published
2023

5. Double Dome and Reemergence of Superconductivity in Pristine 6R-TaS2 under Pressure

Author

Lv, Xindeng, Song, Hao, Chen, Kun, Liu, Sirui, Huang, Yanping, Fang, Yuqiang, and Cui, Tian

Subjects
Condensed Matter - Superconductivity
Abstract

Investigating the implications of interlayer coupling on superconductivity is essential for comprehending the intrinsic mechanisms of high temperature superconductors. Van der Waals heterojunctions have attracted extensive research due to their exotic interlayer coupling. Here, we present a natural heterojunction superconductor of 6R-TaS2 that demonstrates a double-dome of superconductivity, in addition to, the reemergence of superconducting under high pressures. Our first principles calculation shows that the first dome of superconductivity in 6R-TaS2 can be attributed to changes in interlayer coupling and charge transfer. The second superconducting dome and the reemergence of superconductivity can be ascribed to changes in the density of states resulting from Fermi surface reconstruction, in which the DOS of T-layer and S p-orbitals play a crucial role. We have reported the first observation in TMDs that non-metallic atoms playing a dominant role in the reemergence of superconducting and the influence of two Lifshitz transitions on superconducting properties.

Published
2023

6. Controlled Decoding from Language Models

Author

Mudgal, Sidharth, Lee, Jong, Ganapathy, Harish, Li, YaGuang, Wang, Tao, Huang, Yanping, Chen, Zhifeng, Cheng, Heng-Tze, Collins, Michael, Strohman, Trevor, Chen, Jilin, Beutel, Alex, and Beirami, Ahmad

Subjects
Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Computation and Language
Abstract

KL-regularized reinforcement learning (RL) is a popular alignment framework to control the language model responses towards high reward outcomes. We pose a tokenwise RL objective and propose a modular solver for it, called controlled decoding (CD). CD exerts control through a separate prefix scorer module, which is trained to learn a value function for the reward. The prefix scorer is used at inference time to control the generation from a frozen base model, provably sampling from a solution to the RL objective. We empirically demonstrate that CD is effective as a control mechanism on popular benchmarks. We also show that prefix scorers for multiple rewards may be combined at inference time, effectively solving a multi-objective RL problem with no additional training. We show that the benefits of applying CD transfer to an unseen base model with no further tuning as well. Finally, we show that CD can be applied in a blockwise decoding fashion at inference-time, essentially bridging the gap between the popular best-of-K strategy and tokenwise control through reinforcement learning. This makes CD a promising approach for alignment of language models., Comment: ICML 2024

Published
2023

7. Self-Assembled Homopolymeric Spherulites from Small Molecules in Solution

Author

Song, Qiantao, Li, Yi, Jin, Zhicheng, Liu, Hai, Creyer, Matthew N, Yim, Wonjun, Huang, Yanping, Hu, Xiaobing, He, Tengyu, Li, Yajuan, Kelley, Shana O, Shi, Lingyan, Zhou, Jiajing, and Jokerst, Jesse V

Subjects
Macromolecular and Materials Chemistry, Chemical Sciences, Bioengineering, Nanotechnology, Crystallization, Polymers, Freezing, Nanoparticles, General Chemistry, Chemical sciences, Engineering
Abstract

Polymeric spherulites are typically formed by melt crystallization: spherulitic growth in solution is rare and requires complex polymers and dilute solutions. Here, we report the mild and unique formation of luminescent spherulites at room temperature via the simple molecule benzene-1,4-dithiol (BDT). Specifically, BDT polymerized into oligomers (PBDT) via disulfide bonds and assembled into uniform supramolecular nanoparticles in aqueous buffer; these nanoparticles were then dissolved back into PBDT in a good solvent (i.e., dimethylformamide) and underwent chain elongation to form spherulites (rPBDT) in 10 min. The spherulite geometry was modulated by changing the PBDT concentration and reaction time. Due to the step-growth polymerization and reorganization of PBDT, these spherulites not only exhibited robust structure but also showed broad clusterization-triggered emission. The biocompatibility and efficient cellular uptake of the spherulites further underscore their value as traceable drug carriers. This system provides a new pathway for designing versatile superstructures with value for hierarchical assembly of small molecules into a complicated biological system.

Published
2023

8. SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs

Author

Yu, Lijun, Cheng, Yong, Wang, Zhiruo, Kumar, Vivek, Macherey, Wolfgang, Huang, Yanping, Ross, David A., Essa, Irfan, Bisk, Yonatan, Yang, Ming-Hsuan, Murphy, Kevin, Hauptmann, Alexander G., and Jiang, Lu

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Computation and Language, Computer Science - Multimedia
Abstract

In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%., Comment: NeurIPS 2023 spotlight

Published
2023

9. Brainformers: Trading Simplicity for Efficiency

Author

Zhou, Yanqi, Du, Nan, Huang, Yanping, Peng, Daiyi, Lan, Chang, Huang, Da, Shakeri, Siamak, So, David, Dai, Andrew, Lu, Yifeng, Chen, Zhifeng, Le, Quoc, Cui, Claire, Laudon, James, and Dean, Jeff

Subjects
Computer Science - Machine Learning, Computer Science - Computation and Language
Abstract

Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-forward and self-attention in order to build a deep network. Here we investigate this design choice and find that more complex blocks that have different permutations of layer primitives can be more efficient. Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions. Brainformer consistently outperforms the state-of-the-art dense and sparse Transformers, in terms of both quality and efficiency. A Brainformer model with 8 billion activated parameters per token demonstrates 2x faster training convergence and 5x faster step time compared to its GLaM counterpart. In downstream task evaluation, Brainformer also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM with a similar number of activated parameters. Finally, Brainformer largely outperforms a Primer dense model derived with NAS with similar computation per token on fewshot evaluations.

Published
2023

10. Lifelong Language Pretraining with Distribution-Specialized Experts

Author

Chen, Wuyang, Zhou, Yanqi, Du, Nan, Huang, Yanping, Laudon, James, Chen, Zhifeng, and Cu, Claire

Subjects
Computer Science - Computation and Language, Computer Science - Machine Learning
Abstract

Pretraining on a large-scale corpus has become a standard method to build general language models (LMs). Adapting a model to new data distributions targeting different downstream tasks poses significant challenges. Naive fine-tuning may incur catastrophic forgetting when the over-parameterized LMs overfit the new data but fail to preserve the pretrained features. Lifelong learning (LLL) aims to enable information systems to learn from a continuous data stream across time. However, most prior work modifies the training recipe assuming a static fixed network architecture. We find that additional model capacity and proper regularization are key elements to achieving strong LLL performance. Thus, we propose Lifelong-MoE, an extensible MoE (Mixture-of-Experts) architecture that dynamically adds model capacity via adding experts with regularized pretraining. Our results show that by only introducing a limited number of extra experts while keeping the computation cost constant, our model can steadily adapt to data distribution shifts while preserving the previous knowledge. Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks., Comment: ICML 2023

Published
2023

11. PaLM 2 Technical Report

Author

Anil, Rohan, Dai, Andrew M., Firat, Orhan, Johnson, Melvin, Lepikhin, Dmitry, Passos, Alexandre, Shakeri, Siamak, Taropa, Emanuel, Bailey, Paige, Chen, Zhifeng, Chu, Eric, Clark, Jonathan H., Shafey, Laurent El, Huang, Yanping, Meier-Hellstern, Kathy, Mishra, Gaurav, Moreira, Erica, Omernick, Mark, Robinson, Kevin, Ruder, Sebastian, Tay, Yi, Xiao, Kefan, Xu, Yuanzhong, Zhang, Yujing, Abrego, Gustavo Hernandez, Ahn, Junwhan, Austin, Jacob, Barham, Paul, Botha, Jan, Bradbury, James, Brahma, Siddhartha, Brooks, Kevin, Catasta, Michele, Cheng, Yong, Cherry, Colin, Choquette-Choo, Christopher A., Chowdhery, Aakanksha, Crepy, Clément, Dave, Shachi, Dehghani, Mostafa, Dev, Sunipa, Devlin, Jacob, Díaz, Mark, Du, Nan, Dyer, Ethan, Feinberg, Vlad, Feng, Fangxiaoyu, Fienber, Vlad, Freitag, Markus, Garcia, Xavier, Gehrmann, Sebastian, Gonzalez, Lucas, Gur-Ari, Guy, Hand, Steven, Hashemi, Hadi, Hou, Le, Howland, Joshua, Hu, Andrea, Hui, Jeffrey, Hurwitz, Jeremy, Isard, Michael, Ittycheriah, Abe, Jagielski, Matthew, Jia, Wenhao, Kenealy, Kathleen, Krikun, Maxim, Kudugunta, Sneha, Lan, Chang, Lee, Katherine, Lee, Benjamin, Li, Eric, Li, Music, Li, Wei, Li, YaGuang, Li, Jian, Lim, Hyeontaek, Lin, Hanzhao, Liu, Zhongtao, Liu, Frederick, Maggioni, Marcello, Mahendru, Aroma, Maynez, Joshua, Misra, Vedant, Moussalem, Maysam, Nado, Zachary, Nham, John, Ni, Eric, Nystrom, Andrew, Parrish, Alicia, Pellat, Marie, Polacek, Martin, Polozov, Alex, Pope, Reiner, Qiao, Siyuan, Reif, Emily, Richter, Bryan, Riley, Parker, Ros, Alex Castro, Roy, Aurko, Saeta, Brennan, Samuel, Rajkumar, Shelby, Renee, Slone, Ambrose, Smilkov, Daniel, So, David R., Sohn, Daniel, Tokumine, Simon, Valter, Dasha, Vasudevan, Vijay, Vodrahalli, Kiran, Wang, Xuezhi, Wang, Pidong, Wang, Zirui, Wang, Tao, Wieting, John, Wu, Yuhuai, Xu, Kelvin, Xu, Yunhan, Xue, Linting, Yin, Pengcheng, Yu, Jiahui, Zhang, Qiao, Zheng, Steven, Zheng, Ce, Zhou, Weikang, Zhou, Denny, Petrov, Slav, and Wu, Yonghui

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence
Abstract

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.

Published
2023

15. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Author

Li, Zhuohan, Zheng, Lianmin, Zhong, Yinmin, Liu, Vincent, Sheng, Ying, Jin, Xin, Huang, Yanping, Chen, Zhifeng, Zhang, Hao, Gonzalez, Joseph E., and Stoica, Ion

Subjects
Computer Science - Machine Learning, Computer Science - Distributed, Parallel, and Cluster Computing, Computer Science - Networking and Internet Architecture
Abstract

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests., Comment: OSDI 2023

Published
2023

16. Massively Multilingual Shallow Fusion with Large Language Models

Author

Hu, Ke, Sainath, Tara N., Li, Bo, Du, Nan, Huang, Yanping, Dai, Andrew M., Zhang, Yu, Cabrera, Rodrigo, Chen, Zhifeng, and Strohman, Trevor

Subjects
Computer Science - Computation and Language, Computer Science - Machine Learning
Abstract

While large language models (LLM) have made impressive progress in natural language processing, it remains unclear how to utilize them in improving automatic speech recognition (ASR). In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages. We push the limits of the multilingual LM to cover up to 84 languages by scaling up using a mixture-of-experts LLM, i.e., generalist language model (GLaM). When the number of experts increases, GLaM dynamically selects only two at each decoding step to keep the inference computation roughly constant. We then apply GLaM to a multilingual shallow fusion task based on a state-of-the-art end-to-end model. Compared to a dense LM of similar computation during inference, GLaM reduces the WER of an English long-tail test set by 4.4% relative. In a multilingual shallow fusion task, GLaM improves 41 out of 50 languages with an average relative WER reduction of 3.85%, and a maximum reduction of 10%. Compared to the baseline model, GLaM achieves an average WER reduction of 5.53% over 43 languages., Comment: Accepted to IEEE ICASSP 2023

Published
2023

17. Retrospective study on the effect of adding heparin sodium saline at different time points on improving severe coagulation in dialyzers (不同时间点追加肝素钠盐水对透析器重度凝血改善作用的回顾性研究)

Author

LI Linyang (李林洋), WANG Xinhui (王新慧), LI Hao (李昊), HUANG Yanping (黄砚萍), and YU Renhuan (余仁欢)

Subjects
dialyzer coagulation, anticoagulant, hemodialysis, heparin saline, 透析器凝血, 抗凝, 血液透析, 肝素钠盐水, Nursing, RT1-120
Abstract

Objective To investigate the improvement effect of adding heparin sodium saline at different time points on the occurrence of severe dialyzer coagulation in maintenance hemodialysis (MHD) patients during dialysis. Methods Patients who underwent MHD treatment at the dialysis center of Xiyuan Hospital from November 2021 to November 2022 and experienced severe coagulation with dialyzers within 2 hours of treatment were included in the study. Electronic medical records were consulted to collect baseline data, data on various dialysis parameters, and relevant case records. According to the time point of adding heparin sodium, it was divided into two groups: the treatment group and the control group. Both groups were treated with 200 mL of physiological saline first, while the treatment group was treated with 1 mL of 625 U/mL (5 mg/mL) heparin sodium saline added to the dialysis machine heparin pump before re infusion of blood, and a maintenance dose of 1 mL/h was given until the dialysis was stopped 30 minutes before the end of dialysis; The control group received the same dose of heparin sodium salt solution added by the same method after re inducing blood. Compare the improvement effect of two sets of processes on severe coagulation dialyzers. Results A total of 12 MHD patients were included in this study. The treatment group consisted of 6 patients who completed 4 hours of normal dialysis treatment, while the control group consisted of 4 patients who completed normal treatment, and 2 patients who returned to the machine early due to severe coagulation in the dialyzer and vascular system. After dismounting, the coagulation level was evaluated. In the treatment group, 4 patients with no coagulation were classified as 0 level; Two patients had grade 1 coagulation on the dialyzer. In the control group, two dialyzers showed grade 3 coagulation, and there were small amounts of clots in both the arterial and venous ampulla; Three dialyzers showed grade 2 coagulation; One dialyzer showed grade 1 coagulation. There was a statistically significant difference (P＜0. 05) in the evaluation of coagulation grade between the two groups. Conclusion The method of adding heparin sodium saline before re infusion of blood can effectively alleviate the severe coagulation condition of the dialyzer, restore the function of dialyzer, and avoid blood loss, which is better than the method of adding heparin sodium saline after re infusion of blood to start extracorporeal circulation. (目的 探究不同时间点追加肝素钠盐水对维持性血液透析(MHD)患者透析中发生透析器重度凝血的改善作用。方法 纳入2021年11—至2022年11月中国中医科学院西苑医院透析中心进行MHD治疗, 在治疗开始2 h内发生透析器重度凝血的患者, 查阅电子病历收集患者基线资料、透析中各参数资料、相关病例记录。根据追加肝素钠的时间点分为治疗组和对照组两组。两组均采用先回冲200 mL生理盐水, 治疗组采用再次引血前通过透析机肝素泵追加浓度为625 U/mL(5 mg/mL)的肝素钠盐水1 mL并给予维持量1 mL/h, 直至透析结束前30min停止; 对照组采用再次引血后通过同种方法追加相同剂量肝素钠盐水。比较两组流程对重度凝血透析器的改善作用。结果 本研究共纳入12例MHD患者, 治疗组6例患者均完成4 h正常透析治疗, 对照组中4例完成正常治疗, 2例因透析器及血管路重度凝血提前回血下机。下机后凝血等级评估, 治疗组中, 4例患者无凝血为0级; 2例患者透析器1级凝血。对照组中, 2例透析器为3级凝血, 动脉壶、静脉壶均存在少量凝血块; 3例透析器为2级凝血; 1例透析器为1级凝血。凝血等级评估经比较两组存在统计学差异(P＜0. 05)。结论 于再次引血前追加肝素钠盐水的方法, 可以有效缓解透析器重度凝血的状况, 恢复透析器的功能, 避免血液丢失, 优于再次引血开始体外循环后再追加的方式。)

Published
2024
Full Text
View/download PDF

18. Nursing of 1 patient with stuffiness of stomach treated with Bian-stone ironing and rubbing combined with Traditional Chinese Medicine penetration and Baduanjin exercise based on the theory of supporting-yang (基于'扶阳'理论应用砭石熨摩中药透入法联合八段锦治疗1例胃痞病患者的护理体会)

Author

QI Huan (祁欢), TIAN Xiaodong (田小冬), HUANG Yanping (黄砚萍), and XU Bing (许冰)

Subjects
theory of supporting-yang, baduanjin exercise, bian-stone ironing and rubbing, traditional chinese medicine penetration, stuffiness of stomach, traditional chinese medicine nursing, 扶阳理论, 八段锦, 砭石熨摩, 中药透入, 胃痞病, 中医护理, Nursing, RT1-120
Abstract

This article summarizes nursing practice of 1 patient with stuffiness of stomach treated with Bian-stone ironing and rubbing combined with Traditional Chinese Medicine penetration and Baduanjin exercise based on the theory of supporting-yang. On the basis of routine nursing, through the assessment of symptoms, psychological status and sleep quality of the patient, under the guidance of the "Fuyang" theory, the Bian-stone ironing and rubbing combined with Traditional Chinese Medicine penetration and Baduanjin exercise were carried out to relieve the stuffiness of stomach symptoms and improve the health status of the patient. (本文总结基于“扶阳”理论应用砭石熨摩中药透入法联合八段锦治疗1例胃痞病患者的护理经验。在常规护理基础上, 通过对该患者进行症状、心理、睡眠等评估, 在“扶阳”理论指导下, 实施砭石熨摩中药透入法联合八段锦中医护理技术, 旨在缓解患者上腹胀满症状, 改善健康状况。)

Published
2024
Full Text
View/download PDF

19. Traditional Chinese Medicine emotion regulation in relieving anxiety of a patient undergoing total laparoscopic hysterectomy (中医情志护理三部曲缓解1例腹腔镜子宫全切术患者围术期焦虑状态的护理体会)

Author

WANG Ting (王婷), HUANG Yanping (黄砚萍), and HOU Xuemin (侯雪敏)

Subjects
myoma of uterus, laparoscopy, anxiety, traditional chinese medicine, emotion regulation, 子宫肌瘤, 腹腔镜手术, 焦虑, 中医, 情志护理, Nursing, RT1-120
Abstract

This paper summarized the nursing practice of Traditional Chinese Medicine emotion regulation in relieving anxiety of a patient undergoing total laparoscopic hysterectomy. Based on the routine care, a comprehensive plan of emotion regulation including psychological support, relaxation training; music-guided imagination and interview-based interventions were carried out to relieve the anxiety before and after surgery, and improve the patient’s sleep quality and postoperative recovery. (本文总结1例应用中医情志护理三部曲缓解腹腔镜子宫全切术患者围手术期焦虑状态的护理体会。在常规护理基础上, 采用心理支持、放松训练、引导性音乐想象及访谈干预措施, 有效缓解患者术前、术后焦虑情绪, 改善患者睡眠质量, 促进早期康复。)

Published
2024
Full Text
View/download PDF

20. Application of balanced cupping method based on the theory of Du Meridian-regulating in relieving clinical symptoms of a patient after stroke (1例基于'调督论'实施平衡罐法改善脑卒中患者临床症状的护理体会)

Author

TIAN Xiaodong (田小冬) and HUANG Yanping (黄砚萍)

Subjects
stroke, du meridian-regulating, balanced cupping therapy, appropriate techniques of traditional chinese medicine, activity of daily living, 脑卒中, 调督论, 平衡罐, 中医适宜技术, 日常生活能力, Nursing, RT1-120
Abstract

This article summarizes the nursing practice of balanced cupping method based on the theory of Du Meridian-regulating in relieving clinical symptoms of a patient after stroke. On the basis of routine nursing care, the balanced cupping method based on the theory of Du Meridian-regulating was used to select cupping methods such as foot sun bladder meridian flash cupping, Du meridian moving cupping, and specific acupoint retention cupping. Combined with changes in cupping time, speed, negative pressure inside the cupping, and manual coordination, different degrees of stimulation are caused, thereby achieving the effect of tonifying deficiency, reducing excess and regulating the Du meridian, improving post-stroke symptoms and improving patient's quality of life. (本文总结1例基于“调督论”实施平衡罐法改善脑卒中后症状的护理经验。在常规护理的基础上, 基于“调督论”的平衡罐法, 通过对足太阳膀胱经闪罐、督脉走罐、特定穴位留罐等罐法操作, 结合走罐时间、速度、罐内负压和手法配合, 形成针对性刺激, 达到补虚泻实、调理督脉的效果, 有效改善患者脑卒中后相关症状, 提高患者生活质量。)

Published
2024
Full Text
View/download PDF

21. Music-guided imagination technology combined with acupoint application therapy and Traditional Chinese Medicine nursing measures for preoperative blood pressure control in a patient with thoracic aortic ulcer (运用引导性音乐想象技术及穴位贴敷疗法改善1例主动脉溃疡患者术前血压的中医护理体会)

Author

HOU Xuemin (侯雪敏), HUANG Yanping (黄砚萍), and FANG Xiaomei (方晓梅)

Subjects
traditional chinese medicine emotion regulation, music-guided imagination technology, acupoint application, blood pressure, preoperative intervention, 中医情志护理, 引导性音乐想象技术, 穴位贴敷, 血压, 术前干预, Nursing, RT1-120
Abstract

This paper summarized the preoperative nursing practice of a patient undergoing endovascular exclusion for thoracic aortic ulcer. Guided by the theory of Traditional Chinese Medicine emotion regulation, the music-guided imagination technology combined with acupoint application therapy by syndrome differentiation were carried out to control the blood pressure of the patient before surgery. After active treatment and nursing, the patient recovered smoothly and was discharged from the hospital. (本文总结1例胸主动脉溃疡患者行腔内隔绝术术前的中医护理体会。以中医情志护理为指导, 对患者辨证分析, 应用引导性音乐想象技术及穴位贴敷疗法对其术前血压进行干预, 经过积极救治与护理, 患者手术顺利, 术后康复出院。)

Published
2024
Full Text
View/download PDF

22. Bian-stone ironing and rubbing therapy combined with Traditional Chinese Medicine penetration therapy for postoperative constipation in a lung cancer patient undergoing chemotherapy (砭石熨摩中药透入法治疗肺癌患者化疗后便秘1例的护理)

Author

GAO Yaqian (高雅倩), GUO Huanli (郭换利), HUANG Yanping (黄砚萍), FENG Wenying (冯文英), ZHANG Guohong (张国红), and ZHANG Xuelian (张雪莲)

Subjects
lung cancer, bian-stone ironing and rubbing therapy, traditional chinese medicine penetration method, chemotherapy, constipation, traditional chinese medicine nursing, 肺癌, 砭石熨摩, 中药透入, 化疗, 便秘, 中医护理, Nursing, RT1-120
Abstract

This article summarizes the nursing practice of a lung cancer patient with constipation after chemotherapy treated with Bian-stone ironing and rubbing therapy combined with Traditional Chinese Medicine penetration therapy. Through scientific nursing evaluation and syndrome differentiation, the patient's nursing plan was determined. Based on the basic theory of Traditional Chinese Medicine, the Bian-stone ironing and rubbing therapy combined with Traditional Chinese Medicine penetration therapy were used to relieve constipation. At the same time, nursing interventions such as defecation guidance, dietary regulation, exercise, and medication guidance were adopted to further improve clinical symptoms and achieve significant therapeutic effects on constipation. (本文总结1例砭石熨摩中药透入治疗肺癌患者化疗后便秘的护理经验。通过科学的护理评估和辨证分型, 确定患者护理方案, 基于中医基础理论, 采用砭石熨摩中药透入法和手指点穴缓解便秘, 同时配合排便指导、饮食调护、运动锻炼、用药指导等护理干预, 进一步改善患者临床症状, 减轻疼痛。)

Published
2024
Full Text
View/download PDF

23. Nursing care of gastrointestinal dysfunction caused by abnormal changes of cerebral and intestinal axis in a case of intracranial infection treated by auricular acupoint combined therapy (耳穴综合疗法治疗1例颅内感染脑肠轴异常变化所致胃肠功能障碍患者的护理体会)

Author

ZHAO Jianwen (赵建文), FENG Wenying (冯文英), HUANG Yanping (黄砚萍), and ZHAO Jing (赵静)

Subjects
auricular acupoint comprehensive therapy, cerebrointestinal axis, gastrointestinal dysfunction, traditional chinese medicine appropriate nursing technology, 耳穴综合疗法, 脑肠轴, 胃肠功能障碍, 中医护理适宜技术, Nursing, RT1-120
Abstract

To investigate the effect of auricular acupoint combined therapy on gastrointestinal dysfunction caused by abnormal changes of cerebral intestinal axis in 1 case of intracranial infection. The application of auricular point comprehensive therapy (auricular point massage, auricular Bian-stone scraping, auricular acupoint sticking) technology, with emotional nursing, rehabilitation guidance, health education and so on. The gastrointestinal dysfunction of patients was significantly improved. Auricular acupoint combined therapy has a good effect on gastrointestinal dysfunction caused by abnormal changes of cerebral intestinal axis in intracranial infection. (本文总结1例耳穴综合疗法治疗颅内感染脑肠轴异常变化所致胃肠功能障碍的护理经验。基于护理评估和中医辨证, 采用耳穴按摩、耳部砭石刮痧、耳穴贴压综合疗法, 并配合情志护理、康复指导、健康教育护理措施, 有效改善了患者胃肠道功能障碍症状, 对促进患者康复起到积极的作用。)

Published
2024
Full Text
View/download PDF

24. Nursing of a patient with gastroesophageal reflux disease of liver-stomach disharmony syndrome treated with auricular acupoint comprehensive operation techniques based on the theory of ascending and descending (基于'通降理论'运用耳穴综合操作技术治疗吐酸病肝胃不和证1例护理体会)

Author

MAO Shuixian (毛水先), TIAN Xiaodong (田小冬), and HUANG Yanping (黄砚萍)

Subjects
theory of ascending and descending, auricular acupoint comprehensive operation, gastroesophageal reflux disease, traditional chinese medicine nursing, 通降理论, 耳穴综合操作, 吐酸, 中医护理, Nursing, RT1-120
Abstract

This article summarized the nursing experience of a patient with gastroesophageal reflux disease of liver-stomach disharmony syndrome treated with auricular acupoint comprehensive operation techniques based on the theory of ascending and descending. Based on the theory of ascending and descending, the patient was treated with comprehensive auricular acupuncture techniques, combined with routine nursing measures such as daily living care, differentiated diet, differentiated nutrition support, etc. After treatment, the patient's GerdQ score for gastroesophageal reflux disease decreased from 11 points to 5 points, and the GERD-HRQL score decreased from 21 points to 9 points, effectively improving the patient's symptoms of gastroesophageal reflux disease, and enhancing the quality of life. (本文总结耳穴综合操作技术治疗1例吐酸病肝胃不和证患者反酸、烧心的效果及护理经验。基于“通降理论”, 在辨证施护基础上, 给予患者耳穴综合操作技术治疗, 配合生活起居护理、辨证施膳、辨证施乐、辨证施养等常规护理措施。经治疗, 患者的胃食管反流病问卷量表(GerdQ)评分从11分降到5分, 胃食管反流病健康相关生活质量量表(GERD-HRQL)评分从21分降到9分, 有效改善了患者的反酸烧心症状, 提高了患者的生活质量。)

Published
2024
Full Text
View/download PDF

25. Comprehensive auricular acupoint operation technique in the treatment of insomnia in an elderly patient with liver depression and spleen deficiency syndrome (耳穴综合操作技术治疗1例老年肝郁脾虚证失眠的护理体会)

Author

GUO Huanli (郭换利), FENG Wenying (冯文英), ZHANG Guohong (张国红), and HUANG Yanping (黄砚萍)

Subjects
insomnia, traditional chinese medicine nursing, auricular acupoint therapy, anxiety, 失眠, 中医护理, 耳穴贴压, 焦虑, Nursing, RT1-120
Abstract

This article summarized the nursing of a patient with insomnia caused by liver depression and spleen deficiency syndrome treated with comprehensive auricular manipulation techniques. According to the nursing assessment and Traditional Chinese Medicine syndrome differentiation, comprehensive auricular manipulation techniques including auricular massage, ear tip bloodletting and auricular acupoint pressing therapy, combined with emotion regulation, were carried out relieve the negative emotion and improve the sleep quality of the patient. (本文总结1例耳穴综合操作技术治疗肝郁脾虚证失眠的护理体会。根据护理评估和中医辨证, 采取耳部按摩、耳部放血、耳穴贴压综合疗法, 并配合情志护理, 有效缓解患者负性情绪, 改善了患者睡眠质量。)

Published
2024
Full Text
View/download PDF

26. Integrated traditional Chinese and western medicine nursing bundles for a patient undergoing percutaneous coronary intervention for acute myocardial infarction (1例急性心肌梗死患者经皮冠状动脉介入治疗术后中西医结合集束化护理)

Author

CHANG Yu (常瑜) and HUANG Yanping (黄砚萍)

Subjects
acute myocardial infarction, percutaneous coronary intervention, care bundles, integrated traditional chinese and western medicine nursing, intra-aortic ballon pump, 急性心肌梗死, 经皮冠状动脉介入治疗, 集束化护理, 中西医结合护理, 主动脉内球囊反搏, Nursing, RT1-120
Abstract

This paper summarized the integrated traditional Chinese and western medicine nursing bundles for a patient undergoing percutaneous coronary intervention for acute myocardial infarction. The basic nursing plan was formulated based on a comprehensive nursing risk assessment. Integrated traditional Chinese and western medicine nursing bundles included monitoring of vital signs, postoperative health education, intra-aortic ballon pump-associated catheter maintenance, skin care, Psycho-Cardiology care and interventions on sleep quality, abdominal distension, chest tightness and difficulty breathing syndrome. The integrated traditional Chinese and western medicine nursing bundles are effective to relieve the symptoms, reduce the postoperative complications and improve the prognosis and quality of life of the patient undergoing percutaneous coronary intervention for acute myocardial infarction. (本文总结1例急性心肌梗死患者经皮冠状动脉介入治疗术后中西医结合集束化护理体会。在完善护理评估的基础上, 制定基础护理方案。在密切监测病情的基础上, 实施集束化护理措施, 包括术后健康宣教、主动脉内球囊反搏(IABP)管路护理、皮肤护理、双心护理、睡眠护理, 以及针对腹胀、胸闷憋喘症状的干预等, 旨在有效缓解症状, 减少并发症、后遗症等的发生, 改善预后和患者生活质量。)

Published
2024
Full Text
View/download PDF

27. Research progress of psychological interventions for patients with diabetic foot and anxiety (糖尿病足合并焦虑患者心理干预研究进展)

Author

HOU Xuemin (侯雪敏), HUANG Yanping (黄砚萍), and WANG Ting (王婷)

Subjects
diabetic foot, anxiety, traditional chinese medicine emotional nursing, inductive music imagination, 糖尿病足, 焦虑, 中医情志护理, 引导性音乐想象, Nursing, RT1-120
Abstract

The incidence of type diabetes mellitus tends to increase with gradual aging of the population, leading to the increasing number of patients with diabetic foot ulcers. Patients may suffer from emotional disorder at any stage of disease. The emotional status is highly related to the development of diabetic foot ulcers, and has a major influence on following treatment and quality of life of patients. The current therapy model combines two evidence-based treatments: drug therapy including anxiolytics and newer antidepressants, non-drug therapy including cognitive-behavioral therapy, psychological care, music therapy and Traditional Chinese Medicine emotional nursing. This paper reviewed studies on pathogenesis, development and interventions on diabetic foot complicating anxiety disorder. (随着我国老龄化逐渐加剧, 老年2型糖尿病患者也日益增多, 同时伴随着足部溃疡的患者数量呈上升趋势。患者在疾病发生、发展的过程中均会出现不同程度的情绪障碍, 而情绪又与糖尿病足的发生、发展息息相关, 对于患者的治疗以及生存质量有着重大的影响。在治疗上主要包括药物治疗和非药物治疗两种方式, 药物治疗主要包括抗焦虑药物和新一代抗抑郁的药物; 非药物治疗主要有认知行为干预、心理护理、音乐疗法、中医情志护理的方法等。本文从糖尿病足合并焦虑的发病机制、发展原因、干预措施等方面进行综述, 并做简要分析。)

Published
2024
Full Text
View/download PDF

28. Research status quo of Traditional Chinese Medicine hot ironing therapy in the treatment of spleen and stomach diseases (中医热熨疗法治疗脾胃疾病的研究现状分析)

Author

TIAN Xiaodong (田小冬), HUANG Yanping (黄砚萍), WANG Wenqing (王文清), and WANG Ting (王婷)

Subjects
traditional chinese medicine hot ironing therapy, spleen and stomach diseases, spleen and stomach qi deficiency, postoperative constipation, traditional chinese medicine nursing, 热熨疗法, 脾胃疾病, 脾胃气虚, 术后便秘, 中医护理, Nursing, RT1-120
Abstract

By searching ancient books and literature, this paper summarized the development history of Traditional Chinese Medicine hot ironing therapy and its treatment methods, therapeutic effects and application progress in spleen and stomach diseases, and to understand the research status of Traditional Chinese Medicine hot ironing therapy in the treatment of spleen and stomach diseases. Current researches findings showed that Traditional Chinese Medicine hot ironing therapy were effective in the treatment of spleen and stomach diseases, and more in-depth researches should be conducted to improve the clinical practice and ensure the therapeutic effect. (本文通过查找、检索有关热熨疗法的相关古籍及文献资料, 总结中医热熨疗法的发展历史, 以及脾胃疾病治疗方法、治疗效果和应用进展, 旨在了解中医热熨疗法在脾胃疾病中的研究现状。中医热熨疗法治疗脾胃疾病效果良好, 应不断深入研究, 使其能更好地发挥治疗效果。)

Published
2024
Full Text
View/download PDF

29. Expected $H\to\mu^{+}\mu^{-}$ measurement precision with $e^{+}e^{-}\to Z(q\bar{q})H$ production at the CEPC

Author

Liu, Qi, Ran, Kunlin, Huang, Yanping, Li, Gang, Ruan, Manqi, Jin, Shan, and Sun, Liang

Subjects
High Energy Physics - Experiment
Abstract

A search for the dimuon decay of the Standard Model Higgs boson is performed using the Monte Carlo simulated events to mimic data corresponding to an integrated luminosity of 5.6 ab$^{-1}$ collected with the Circular Electron-Positron Collider detector in $e^{+}e^{-}$ collisions at $\sqrt{s}=240$ GeV. The paper studies $e^{+}e^{-}\to ZH,\,Z\to q\bar{q},\,H\to\mu^{+}\mu^{-}$ process, and the expected significance considering only the data statistical uncertainty over the background-only hypothesis for a Higgs boson with a mass of 125 GeV is found to be 6.1$\sigma$, corresponding to the precision of 19%. The systematic impacts from the background Monte Carlo statistical fluctuations are estimated to be negligible. The dependence of the measurement accuracy on the muon momentum resolution of the CEPC detector has been investigated. It is found that the muon momentum resolution has to be better than 204 MeV to discover the $H\to\mu\mu$ process at the nominal integrated luminosity. And if the resolution is 100% worse than the designed parameter, the integrated luminosity is needed to be greater than 7.2 ab$^{-1}$ to reach 5$\sigma$ significance., Comment: 10 pages, 8 figures, submitted to Chinese Physics C

Published
2022
Full Text
View/download PDF

30. Scaling Instruction-Finetuned Language Models

Author

Chung, Hyung Won, Hou, Le, Longpre, Shayne, Zoph, Barret, Tay, Yi, Fedus, William, Li, Yunxuan, Wang, Xuezhi, Dehghani, Mostafa, Brahma, Siddhartha, Webson, Albert, Gu, Shixiang Shane, Dai, Zhuyun, Suzgun, Mirac, Chen, Xinyun, Chowdhery, Aakanksha, Castro-Ros, Alex, Pellat, Marie, Robinson, Kevin, Valter, Dasha, Narang, Sharan, Mishra, Gaurav, Yu, Adams, Zhao, Vincent, Huang, Yanping, Dai, Andrew, Yu, Hongkun, Petrov, Slav, Chi, Ed H., Dean, Jeff, Devlin, Jacob, Roberts, Adam, Zhou, Denny, Le, Quoc V., and Wei, Jason

Subjects
Computer Science - Machine Learning, Computer Science - Computation and Language
Abstract

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models., Comment: Public checkpoints: https://huggingface.co/docs/transformers/model_doc/flan-t5

Published
2022

31. Treatment of diabetic gastroparesis by external treatment of Traditional Chinese Medicine (中医外治法治疗糖尿病胃轻瘫的研究进展)

Author

ZHANG Tingting (张婷婷) and HUANG Yanping (黄砚萍)

Subjects
traditional chinese medicine external treatment, diabetic gastroparesis, moxibustion, acupuncture, 中医外治, 糖尿病胃轻瘫, 艾灸, 针刺, Nursing, RT1-120
Abstract

Diabetic gastroparesis is a common complication in diabetic patients. It is manifested as recurrent nausea, vomiting, early saturation and abdominal fullness. The main pathophysiology is a condition that slows down the emptying of the stomach. In recent years, the external treatment of Traditional Chinese Medicine in the treatment of diabetic gastroparesis has been widely used in clinical practice, which can improve the clinical symptoms of patients. By searching the relevant literature, this paper briefly listed external treatment of Traditional Chinese Medicine, and looked forward to the future development direction of new nursing technology. (糖尿病胃轻瘫是糖尿病常见并发症之一, 表现为反复发作恶心、呕吐、早饱和上腹饱胀, 主要病理生理是胃排空延缓。近年来中医外治法治疗糖尿病胃轻瘫在临床实践中得到广泛应用, 可有效改善患者临床症状。本文通过检索相关文献, 简要列举糖尿病胃轻瘫中医外治疗法, 旨在为今后糖尿病并发症护理研究提供参考。)

Published
2024
Full Text
View/download PDF

32. Effect of Composting Treatment of Grape Branches on Wine Quality

Author

TU Qian, DING Weikang, HUANG Yanping, LI Yuanyuan, LIU Shiqiu, MIN Xuwu, ZHENG Changhua, YUAN Chunlong

Subjects
grape branches, compost, wine, aroma, color, Food processing and manufacture, TP368-456
Abstract

In order to study the effect of applying different proportions of compost made from grape branches in the vineyard on the quality of wine, 8-year-old ‘Pinot Noir’ grapevine was selected for this study, and compost made from 100% crushed grape branches (GB), compost made from 70% crushed grape branches + 30% sheep manure (GM1), compost made from 50% crushed grape branches + 50% sheep manure (GM2), compost made from 30% crushed grape branches + 70% sheep manure (GM3), and 100% sheep manure (SM) were applied to the vineyard , and the no compost addition treatment (CK) was used as a control. One year after fertilization, grape berries were taken to brew wine, and the basic physicochemical indexes of grapes and wine, the phenolic substances, color, and aroma substances and sensory evaluation of wine were determined. The results showed that the composting treatment significantly increased the contents of alcohol, total phenols, total anthocyanins, total flavan-3-ols, and total flavonoids in Pinot Noir wine, with the GM3 treatment showing the greatest increase in the levels of alcohol and titratable acids, by 16.5% and 24.03%, respectively. In terms of color, the composting treatments increased the color saturation of ‘Pinot Noir’ wine. Orthogonal partial least squares-discriminant analysis (OPLS-DA) of aroma substances showed that the major differential aroma substances in ‘Pinot Noir’ wine among different composting treatments were isoamyl alcohol, followed by ethyl acetate, 1-propanol, and phenylethanol, and that composting treatments with the addition of grape branches increased the levels of ethyl acetate, isoamyl acetate, ethyl caproate, and ethyl caproate in ‘Pinot Noir’ wine. In terms of sensory quality, the intensity of astringency, bitterness, and sweetness of ‘Pinot Noir’ wine was enhanced by the composting treatments, and the wine was improved in terms of taste and balance. This study provides a theoretical basis for the comprehensive utilization of grape branch waste and the improvement of wine quality.

Published
2024
Full Text
View/download PDF

33. Building Machine Translation Systems for the Next Thousand Languages

Author

Bapna, Ankur, Caswell, Isaac, Kreutzer, Julia, Firat, Orhan, van Esch, Daan, Siddhant, Aditya, Niu, Mengmeng, Baljekar, Pallavi, Garcia, Xavier, Macherey, Wolfgang, Breiner, Theresa, Axelrod, Vera, Riesa, Jason, Cao, Yuan, Chen, Mia Xu, Macherey, Klaus, Krikun, Maxim, Wang, Pidong, Gutkin, Alexander, Shah, Apurva, Huang, Yanping, Chen, Zhifeng, Wu, Yonghui, and Hughes, Macduff

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Machine Learning
Abstract

In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages. We describe results in three research domains: (i) Building clean, web-mined datasets for 1500+ languages by leveraging semi-supervised pre-training for language identification and developing data-driven filtering techniques; (ii) Developing practical MT models for under-served languages by leveraging massively multilingual models trained with supervised parallel data for over 100 high-resource languages and monolingual datasets for an additional 1000+ languages; and (iii) Studying the limitations of evaluation metrics for these languages and conducting qualitative analysis of the outputs from our MT models, highlighting several frequent error modes of these types of models. We hope that our work provides useful insights to practitioners working towards building MT systems for currently understudied languages, and highlights research directions that can complement the weaknesses of massively multilingual models in data-sparse settings., Comment: V2: updated with some details from 24-language Google Translate launch in May 2022 V3: spelling corrections, additional acknowledgements

Published
2022

34. Mixture-of-Experts with Expert Choice Routing

Author

Zhou, Yanqi, Lei, Tao, Liu, Hanxiao, Du, Nan, Huang, Yanping, Zhao, Vincent, Dai, Andrew, Chen, Zhifeng, Le, Quoc, and Laudon, James

Subjects
Computer Science - Machine Learning, Computer Science - Artificial Intelligence
Abstract

Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2x. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.

Published
2022

35. ST-MoE: Designing Stable and Transferable Sparse Expert Models

Author

Zoph, Barret, Bello, Irwan, Kumar, Sameer, Du, Nan, Huang, Yanping, Dean, Jeff, Shazeer, Noam, and Fedus, William

Subjects
Computer Science - Computation and Language, Computer Science - Machine Learning
Abstract

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3)., Comment: 25 pages main text, 39 pages overall

Published
2022

36. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Author

Zheng, Lianmin, Li, Zhuohan, Zhang, Hao, Zhuang, Yonghao, Chen, Zhifeng, Huang, Yanping, Wang, Yida, Xu, Yuanzhong, Zhuo, Danyang, Xing, Eric P., Gonzalez, Joseph E., and Stoica, Ion

Subjects
Computer Science - Machine Learning, Computer Science - Distributed, Parallel, and Cluster Computing, Computer Science - Programming Languages
Abstract

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans. Alpa's source code is publicly available at https://github.com/alpa-projects/alpa, Comment: OSDI 2022

Published
2022

37. LaMDA: Language Models for Dialog Applications

Author

Thoppilan, Romal, De Freitas, Daniel, Hall, Jamie, Shazeer, Noam, Kulshreshtha, Apoorv, Cheng, Heng-Tze, Jin, Alicia, Bos, Taylor, Baker, Leslie, Du, Yu, Li, YaGuang, Lee, Hongrae, Zheng, Huaixiu Steven, Ghafouri, Amin, Menegali, Marcelo, Huang, Yanping, Krikun, Maxim, Lepikhin, Dmitry, Qin, James, Chen, Dehao, Xu, Yuanzhong, Chen, Zhifeng, Roberts, Adam, Bosma, Maarten, Zhao, Vincent, Zhou, Yanqi, Chang, Chung-Ching, Krivokon, Igor, Rusch, Will, Pickett, Marc, Srinivasan, Pranesh, Man, Laichee, Meier-Hellstern, Kathleen, Morris, Meredith Ringel, Doshi, Tulsee, Santos, Renelito Delos, Duke, Toju, Soraker, Johnny, Zevenbergen, Ben, Prabhakaran, Vinodkumar, Diaz, Mark, Hutchinson, Ben, Olson, Kristen, Molina, Alejandra, Hoffman-John, Erin, Lee, Josh, Aroyo, Lora, Rajakumar, Ravi, Butryna, Alena, Lamm, Matthew, Kuzmina, Viktoriya, Fenton, Joe, Cohen, Aaron, Bernstein, Rachel, Kurzweil, Ray, Aguera-Arcas, Blaise, Cui, Claire, Croak, Marian, Chi, Ed, and Le, Quoc

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence
Abstract

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.

Published
2022

43. Research on prediction and analysis of supercritical water heat transfer coefficient based on support vector machine

Author

Ma Dongliang, Li Yi, Zhou Tao, and Huang Yanping

Subjects
Supercritical water, Support vector machine, Radial basis kernel function, Data normalization, Heat transfer coefficient, Machine learning algorithm, Nuclear engineering. Atomic power, TK9001-9401
Abstract

In order to better perform thermal hydraulic calculation and analysis of supercritical water reactor, based on the experimental data of supercritical water, the model training and predictive analysis of the heat transfer coefficient of supercritical water were carried out by using the support vector machine (SVM) algorithm. The changes in the prediction accuracy of the supercritical water heat transfer coefficient are analyzed by the changes of the regularization penalty parameter C, the slack variable epsilon and the Gaussian kernel function parameter gamma. The predicted value of the SVM model obtained after parameter optimization and the actual experimental test data are analyzed for data verification. The research results show that: the normalization of the data has a great influence on the prediction results. The slack variable has a relatively small influence on the accuracy change range of the predicted heat transfer coefficient. The change of gamma has the greatest impact on the accuracy of the heat transfer coefficient. Compared with the calculation results of traditional empirical formula methods, the trained algorithm model using SVM has smaller average error and standard deviations. Using the SVM trained algorithm model, the heat transfer coefficient of supercritical water can be effectively predicted and analyzed.

Published
2023
Full Text
View/download PDF

44. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Author

Du, Nan, Huang, Yanping, Dai, Andrew M., Tong, Simon, Lepikhin, Dmitry, Xu, Yuanzhong, Krikun, Maxim, Zhou, Yanqi, Yu, Adams Wei, Firat, Orhan, Zoph, Barret, Fedus, Liam, Bosma, Maarten, Zhou, Zongwei, Wang, Tao, Wang, Yu Emma, Webster, Kellie, Pellat, Marie, Robinson, Kevin, Meier-Hellstern, Kathleen, Duke, Toju, Dixon, Lucas, Zhang, Kun, Le, Quoc V, Wu, Yonghui, Chen, Zhifeng, and Cui, Claire

Subjects
Computer Science - Computation and Language
Abstract

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks., Comment: Accepted to ICML 2022

Published
2021

49. BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Author

Zhang, Yu, Park, Daniel S., Han, Wei, Qin, James, Gulati, Anmol, Shor, Joel, Jansen, Aren, Xu, Yuanzhong, Huang, Yanping, Wang, Shibo, Zhou, Zongwei, Li, Bo, Ma, Min, Chan, William, Yu, Jiahui, Wang, Yongqiang, Cao, Liangliang, Sim, Khe Chai, Ramabhadran, Bhuvana, Sainath, Tara N., Beaufays, Françoise, Chen, Zhifeng, Le, Quoc V., Chiu, Chung-Cheng, Pang, Ruoming, and Wu, Yonghui

Subjects
Electrical Engineering and Systems Science - Audio and Speech Processing, Computer Science - Computation and Language, Computer Science - Machine Learning, Computer Science - Sound
Abstract

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks., Comment: 14 pages, 7 figures, 13 tables; v2: minor corrections, reference baselines and bibliography updated; v3: corrections based on reviewer feedback, bibliography updated

Published
2021
Full Text
View/download PDF

50. Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

Author

Kudugunta, Sneha, Huang, Yanping, Bapna, Ankur, Krikun, Maxim, Lepikhin, Dmitry, Luong, Minh-Thang, and Firat, Orhan

Subjects
Computer Science - Computation and Language, Computer Science - Machine Learning
Abstract

Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. However, MoE models are prohibitively large and practitioners often resort to methods such as distillation for serving. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from large sparse models. On WMT, our task-MoE with 32 experts (533M parameters) outperforms the best performing token-level MoE model (token-MoE) by +1.0 BLEU on average across 30 language pairs. The peak inference throughput is also improved by a factor of 1.9x when we route by tasks instead of tokens. While distilling a token-MoE to a smaller dense model preserves only 32% of the BLEU gains, our sub-network task-MoE, by design, preserves all the gains with the same inference cost as the distilled student model. Finally, when scaling up to 200 language pairs, our 128-expert task-MoE (13B parameters) performs competitively with a token-level counterpart, while improving the peak inference throughput by a factor of 2.6x., Comment: EMNLP Findings 2021

Published
2021

Catalog

Books, media, physical & digital resources
See catalog results