Institute of Information Science, Academia Sinica

Multimedia Technology Lab

Li Su, Mark Liao, Tyng-Luh Liu, Chun-Shien Lu, Jen-Chun Lin, Chien-Yao Wang, and Ti-Rong Wu

The specialty of this laboratory is multimedia technology, developing technologies including video processing, image processing, music processing and so on. In recent years, due to the re-emergence of artificial intelligence-related research, many projects have combined existing multimedia processing technology and deep learning to win four large-scale AI projects from National Science and Technology council. In addition, the members of the multimedia technology laboratory also won a thematic project of Academia Sinica and three cooperation projects within the institute. The biggest achievement of this laboratory in the past five years is the development of the world famous YOLOv4 and YOLOv7 object detection systems, and these two systems have allowed Taiwan to occupy an important position on the world stage of AI.

Ongoing Projects

• NSTC AI project: 3M Maker: 3M: Multi-modal, Multi-task, Multi-instance Learning【Mark Liao, Jen-Chun Lin, Chien-Yao Wang, 2022/01–2025/12】
• NSTC AI project：An integrated manufacturing platform, the law of science and technology, and industrial ecosystem - smart production and intelligent precision manufacturing with digital decision, AI modeling, big data governance and kernel technologies (2/2)【Tyng-Luh Liu, 2022/01–2025/12】
• Academia Sinica Thematic Project: Understanding and Generation of Audio-Visual Multimedia Content with Deep Learning【Li Su, Jen-Chun Lin, 2022/01–2024/12】
• IIS Cooperation Project: Developing a Single Neural Network for Generalized Spoofing-Aware Speaker Verification【Chun-Shien Lu, Chia-Ching Wang (NCU), 2023/01–2024/12】
• IIS Cooperation Project: The Study of Problem Solving with Deep Reinforcement Learning【Ti-Rong Wu, I-Chen Wu (NYCU), 2023/01–2025/12】
• IIS Cooperation Project: Bridging low-level and high-level computer vision tasks via graph model and make its prediction explainable【Chien-Yao Wang, Chun-Yi Lee (NTHU), 2023/01–2025/12】

Bioinformatics Lab

Chung-Yen Lin, Ting-Yi Sung, Arthur CC Shih, and Huai-Kuang Tsai

The research in the Bioinformatics laboratory is based on information technology, combined with different omics approaches and artificial intelligence strategies to develop novel algorithms, open-source programs, and online databases for application in biomedical and agricultural research. The main research areas include genomic and gene regulation analysis, proteomics and therapeutic peptides, precision medicine, and healthy aging. Recently, team members have participated in the US National Institutes of Health's National Cancer Moonshot, using proteogenomics to successfully decode potential pathogenic sites for non-smoking lung cancer in East Asian populations, providing important references for subsequent precision treatment.

Speech, Language and Music Processing Lab

Hsin-Min Wang

Our research goal is to develop methods for analyzing, extracting, recognizing, indexing, and retrieving information from audio such as speech and music. In recent years, our research has focused on Mandarin/ Taiwanese speech recognition, speech synthesis and conversion, speech enhancement, speech quality assessment, spoof speech detection, etc. Several of our research papers on voice conversion, speech enhancement, and speech quality assessment have received more than a hundred citations. Among them, the paper “Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks” published at Interpseech2017 has been cited more than 400 times (Google Scholar, as of 2023/03/29). In addition, we have self-developed Mandarin speech recognition system and Taiwanese speech recognition system, which have been used by several units inside and outside the Academia Sinica.

Ongoing Projects

• NSTC Project: Automatic Speech Recognition Based on Discriminative Autoencoder【Hsin-Min Wang, 2021/08–2024/07】
• NSTC Integrated Project: Listen Watch and Read: Multimodal Information Authenticity Detection and Source Tracking【Hsin-Min Wang, Yung-jen Hsu (NTU), Tzong-Han Tsai (NCU), Chih-Chung Hsu (NCKU), Kai-Lung Hua (NTUST), Yu Tsao (Academia Sinica), Jun-Cheng Chen (Academia Sinica), Chia-Chen Kuo (NCHC), 2020/08–2023/07】
• Ministry of Justice Investigation Bureau Project: Real-time Speech Recognition and Transcription Transmission System for Communication Surveillance【Hsin-Min Wang and Ming-Tat Ko, 2019/10–2024/06】
• IIS Cooperation Project: Spoken Language Understanding and Generation for E-commerce Chatbots【Wei-Yun Ma and Hsin-Min Wang, 2023/01–2024/12】

Data Mining and Machine Learning Lab

De-Nian Yang and Mi-Yen Yeh

Our lab specializes in social network mining in the metaverse and query and recommendation with graph neural networks (GNNs). Research topics include 1) designing influence estimation models and approximation algorithms for influence maximization (IM) in realistic scenarios (the forefront of IM research to incorporate social coupons and knowledge graphs), instead of promoting a single item in a single promotion without multimedia, 2) optimizing multi-view display configurations in the metaverse (the first metaverse recommendation system to break the boundary between personalized and group recommendations and the first immersive dual-world spatial queries), as opposed to designing for single customers and the physical world, and 3) developing heterogeneous information network transformers and graph attention networks to learn and update feature relationships and embeddings that capture the inherent uncertainty in entity-relation (the premier parameter-free group query with a theoretical guarantee of embedding quality), rather than extracting dense subgraphs disregarding spatial-temporal limitations. The results have been published in VLDB, KDD, ICDE, WWW, NeurIPS, AAAI, CVPR, ICCV, TKDE, TKDD, TMC, etc.

Current Projects

• NSTC Project: Optimizing Pandemic Containment by Analysis of Contact Social Networks【De-Nian Yang, 2021/08–2024/07】
• NSTC Project: Investigating Deep Learning Models for Learnable Multi-relational Networks 【Mi-Yen Yeh, 2022/08–2024/07】
• NSTC Project: AI-enabled MEC Platform for Social Internet of Things【De-Nian Yang, 2020/08/01–2023/07/31】
• IIS Collaborative Projects: Mining, Recommendation, and Optimization for Social Metaverse【De-Nian Yang and Mi-Yen Yeh, 2023/01/01–2023/12/31】
• Deep Graph-based Learning Models and Their Applications【Mi-Yen Yeh and De-Nian Yang, 2022/01/01–2024/12/31】

Language Intelligence and Technology Lab

Lun-Wei Ku and Hen-Hsen Huang

The LIT research group focuses on cutting-edge natural language processing technologies, aiming to achieve a high level of understanding of semantic content and successfully carry out various front-end and back-end tasks in information dissemination and artificial intelligence. The research areas primarily include information retrieval, sentiment analysis, discourse structure analysis, and multimodal techniques that integrate visual, spatiotemporal, and linguistic aspects. In recent years, by advancing machine learning and deep learning technologies, the LIT group has actively developed fake news intervention and interpretation, promoting the concept of fake news monitoring to various agencies and the public. By extracting and representing key information, deep learning models can be applied to very long texts; integrating visual perception information to enhance language capabilities, combining language technology given the background of Taiwan's strong multimedia industry; and actively integrating large-scale language model technologies to develop adaptive learning frameworks, promoting the seamless integration of cutting-edge natural language processing technologies into the industry.

Ongoing Projects

• NSTC AI Project: NexIS: Self-Supervised and Trust-Worthy Learning for Next-Generation Intelligent Services【Winston Hsu, Bing-Yu Chen, Hung-Yi Lee, Yung-Yu Chuang, Shang-Tse Chen, Chien-Wen Tina Yuan, Min Sun, Cheng-Te Li, Yi-Ting Chen, Lun-Wei Ku, Jun-Cheng Chen, 2021/11–2024/10】
• NSTC SportTech Project: Omni-Boxing: Constructing Intelligent Boxing Stadium with Precision Training Systems【Wen-Hsin Chiu, Lun-Wei Ku, Min-Chun Hu, Chun-Yao Wang, Ping-Hsuan Han, Hung-Kuo Chu, 2023/01–2026/12】
• Academia Sinica Grand Challenge Program Seed Grant: Online Social and Political Climates and Trust: A Method Triangulation Approach 【Ching-Ching Chang, Hen-Hsen Huang, Yuan Hsiao, Yu-Ming Hsieh, 2023/01/01–2024/12/31】
• IIS Cooperation Project：Empathy-Aware Long-term Working Memory for Multi-round Multimodal Conversations【Hen-Hsen Huang, Lun-Wei Ku, Cheng-Te Li, 2023/01–2025/12】