Institute of Information Science, Academia Sinica

Multimedia Technology Lab

Chien Yao Wang, Chun-Shien Lu, Jen-Chun Lin, Li Su, Mark Liao, Ti-Rong Wu, and Tyng-Luh Liu

The specialty of this laboratory is multimedia technology, developing technologies including video processing, image processing, music processing and so on. In recent years, due to the re-emergence of artificial intelligence-related research, many projects have combined existing multimedia processing technology and deep learning to win four large-scale AI projects from National Science and Technology council. In addition, the members of the multimedia technology laboratory also won a thematic project of Academia Sinica and three cooperation projects within the institute. The biggest achievement of this laboratory in the past five years is the development of the world famous YOLOv4 and YOLOv7 object detection systems, and these two systems have allowed Taiwan to occupy an important position on the world stage of AI.

Ongoing Projects

• NSTC AI project: 3M Maker: 3M: Multi-modal, Multi-task, Multi-instance Learning【Mark Liao, Jen-Chun Lin, Chien-Yao Wang, 2022/01–2025/12】
• NSTC AI project：An integrated manufacturing platform, the law of science and technology, and industrial ecosystem - smart production and intelligent precision manufacturing with digital decision, AI modeling, big data governance and kernel technologies (2/2)【Tyng-Luh Liu, 2022/01–2025/12】
• Academia Sinica Thematic Project: Understanding and Generation of Audio-Visual Multimedia Content with Deep Learning【Li Su, Jen-Chun Lin, 2022/01–2024/12】
• IIS Cooperation Project: Developing a Single Neural Network for Generalized Spoofing-Aware Speaker Verification【Chun-Shien Lu, Chia-Ching Wang (NCU), 2023/01–2024/12】
• IIS Cooperation Project: The Study of Problem Solving with Deep Reinforcement Learning【Ti-Rong Wu, I-Chen Wu (NYCU), 2023/01–2025/12】
• IIS Cooperation Project: Bridging low-level and high-level computer vision tasks via graph model and make its prediction explainable【Chien-Yao Wang, Chun-Yi Lee (NTHU), 2023/01–2025/12】

Bioinformatics Lab

Arthur Chun-Chieh Shih, Chung-Yen Lin, Huai-Kuang Tsai, and Ting-Yi Sung

The research in the Bioinformatics laboratory is based on information technology, combined with different omics approaches and artificial intelligence strategies to develop novel algorithms, open-source programs, and online databases for application in biomedical and agricultural research. The main research areas include genomic and gene regulation analysis, proteomics and therapeutic peptides, precision medicine, and healthy aging. Recently, team members have participated in the US National Institutes of Health's National Cancer Moonshot, using proteogenomics to successfully decode potential pathogenic sites for non-smoking lung cancer in East Asian populations, providing important references for subsequent precision treatment.

Speech, Language and Music Processing Laboratory

Hsin-Min Wang

Our lab currently focuses on speech processing technologies, including speech recognition, speech synthesis and conversion, speech enhancement, speech quality assessment, and forgery detection. Several of our recent papers in speech conversion, speech enhancement, and speech quality assessment have received hundreds of citations. In addition, we are actively developing various speech recognition and synthesis systems for the Taiwanese language context.

Ongoing Projects

• NSTC Project: Research and Development of Mandarin-Taiwanese Code-Mixing Speech Recognition System【Hsin-Min Wang, 2023/08–2026/07】
• NSTC Project: Voice Conversion with Applications to Electrolaryngeal Speech Processing【Hsin-Min Wang, 2025/08–2028/07】
• NSTC AI Project: An Interactive Language Learning AI Platform for the Taiwanese Context: Multimodal Technology Development with a Focus on Educational Applications【Hsin-Min Wang, Wei-Yun Ma, Shu-Kai Hsieh (NTU), Pu-Jen Cheng (NTU), Chang-Shing Lee (NUTN), 2026/05–2029/10】
• IIS Cooperation Project: Development of Multimodal (Text and Speech) Chinese Language Models【Wei-Yun Ma and Hsin-Min Wang, 2025/01–2026/12】

Data Mining and Machine Learning Lab

De-Nian Yang and Mi-Yen Yeh

Our lab specializes in social network mining in the metaverse and query and recommendation with graph neural networks (GNNs). Research topics include 1) designing influence estimation models and approximation algorithms for influence maximization (IM) in realistic scenarios (the forefront of IM research to incorporate social coupons and knowledge graphs), instead of promoting a single item in a single promotion without multimedia, 2) optimizing multi-view display configurations in the metaverse (the first metaverse recommendation system to break the boundary between personalized and group recommendations and the first immersive dual-world spatial queries), as opposed to designing for single customers and the physical world, and 3) developing heterogeneous information network transformers and graph attention networks to learn and update feature relationships and embeddings that capture the inherent uncertainty in entity-relation (the premier parameter-free group query with a theoretical guarantee of embedding quality), rather than extracting dense subgraphs disregarding spatial-temporal limitations. The results have been published in VLDB, KDD, ICDE, WWW, NeurIPS, AAAI, CVPR, ICCV, TKDE, TKDD, TMC, etc.

Current Projects

• NSTC Project: Optimizing Pandemic Containment by Analysis of Contact Social Networks【De-Nian Yang, 2021/08–2024/07】
• NSTC Project: Investigating Deep Learning Models for Learnable Multi-relational Networks 【Mi-Yen Yeh, 2022/08–2024/07】
• NSTC Project: AI-enabled MEC Platform for Social Internet of Things【De-Nian Yang, 2020/08/01–2023/07/31】
• IIS Collaborative Projects: Mining, Recommendation, and Optimization for Social Metaverse【De-Nian Yang and Mi-Yen Yeh, 2023/01/01–2023/12/31】
• Deep Graph-based Learning Models and Their Applications【Mi-Yen Yeh and De-Nian Yang, 2022/01/01–2024/12/31】

Language Intelligence and Technology Lab

Hen-Hsen Huang and Lun-Wei Ku

The LIT research group focuses on cutting-edge natural language processing technologies, aiming to achieve a high level of understanding of semantic content and successfully carry out various front-end and back-end tasks in information dissemination and artificial intelligence. The research areas primarily include information retrieval, sentiment analysis, discourse structure analysis, and multimodal techniques that integrate visual, spatiotemporal, and linguistic aspects. In recent years, by advancing machine learning and deep learning technologies, the LIT group has actively developed fake news intervention and interpretation, promoting the concept of fake news monitoring to various agencies and the public. By extracting and representing key information, deep learning models can be applied to very long texts; integrating visual perception information to enhance language capabilities, combining language technology given the background of Taiwan's strong multimedia industry; and actively integrating large-scale language model technologies to develop adaptive learning frameworks, promoting the seamless integration of cutting-edge natural language processing technologies into the industry.

Ongoing Projects

• NSTC AI Project: NexIS: Self-Supervised and Trust-Worthy Learning for Next-Generation Intelligent Services【Winston Hsu, Bing-Yu Chen, Hung-Yi Lee, Yung-Yu Chuang, Shang-Tse Chen, Chien-Wen Tina Yuan, Min Sun, Cheng-Te Li, Yi-Ting Chen, Lun-Wei Ku, Jun-Cheng Chen, 2021/11–2024/10】
• NSTC SportTech Project: Omni-Boxing: Constructing Intelligent Boxing Stadium with Precision Training Systems【Wen-Hsin Chiu, Lun-Wei Ku, Min-Chun Hu, Chun-Yao Wang, Ping-Hsuan Han, Hung-Kuo Chu, 2023/01–2026/12】
• Academia Sinica Grand Challenge Program Seed Grant: Online Social and Political Climates and Trust: A Method Triangulation Approach 【Ching-Ching Chang, Hen-Hsen Huang, Yuan Hsiao, Yu-Ming Hsieh, 2023/01/01–2024/12/31】
• IIS Cooperation Project：Empathy-Aware Long-term Working Memory for Multi-round Multimodal Conversations【Hen-Hsen Huang, Lun-Wei Ku, Cheng-Te Li, 2023/01–2025/12】

Chinese Knowledge and Information Processing (CKIP) Laboratory

Wei-Yun Ma

Our lab is a joint research team established by the Institute of Information Science and the Institute of Linguistics of Academia Sinica in 1986. Its purpose is to construct resources and a research environment for Chinese natural language processing, providing fundamental research data and knowledge infrastructures for domestic and international studies. To achieve intelligent information processing, our current research focuses on five core areas: multimodal large language models, deep learning, knowledge representation, natural language understanding, and knowledge acquisition. Building upon this foundation, the lab has successively developed the widely popular CkipTagger and Ckip Transformers toolkits, and released the world's first large language model optimized for Traditional Chinese (BLOOM-zh). With the recent breakthroughs in generative AI, our team has achieved remarkable success in the infrastructure and application of Large Language Models (LLMs): not only did our proposed dynamic data weighting method receive an ICLR 2025 Spotlight recognition, but we are also actively exploring the long-term memory and continual learning capabilities of LLMs, as well as new reasoning mechanisms. Concurrently, our lab is dedicated to developing multimodal language models that integrate text and speech, optimizing them for Taiwanese, Hakka, and indigenous languages to build an interactive AI platform for language learning. Furthermore, in cross-disciplinary digital humanities research, we combine Retrieval-Augmented Generation (RAG) with event-centric "Event Graph" technology, continuing to lay a solid foundation for Taiwan's local linguistic context and the advancement of cutting-edge AI.

Ongoing Projects

• NSTC AI Project: An Interactive Language Learning AI Platform for the Taiwanese Context: Multimodal Technology Development with a Focus on Educational Applications【Hsin-Min Wang, Wei-Yun Ma, Shu-Kai Hsieh (NTU), Pu-Jen Cheng (NTU), Chang-Shing Lee (NUTN), 2026/05–2029/10】
• NSTC Project: Research on Memory Capability of Large Language Models【Wei-Yun Ma, 2025/08–2026/07】
• Academia Sinica Grand Challenge Program: Creativity in the Age of Artificial Intelligence【Ku-ming Chang, Wei-Yun Ma, Shun-Ling Chen, 2026/01–2028/12】
• IIS Cooperation Project: Development of Multimodal (Text and Speech) Chinese Language Models【Wei-Yun Ma, Hsin-Min Wang, Tyng-Ruey Chuang, 2025/01–2026/12】