I’m currently working on this web documentation (gitbook) for curating and organizing articulatory/speech dataset. Articulatory data is very rare, difficult to try out, but useful in research and make various applications in clinical and educational areas.
250123 jkang
]]>
The recent advance in AI and LLMs also had a tremendous impact on the education sector. For example, AI-based courseware and products are already available in the market focusing on helping children solve math problems, learning writing, lanugage learning, and so on. However, it has been somewhat still limited in terms of AI-based application in art, especially digital art education.
When I worked in i-Scream arts where making digital art education platform (“Art Bonbon”) and providing coaching online as a director of AI research, I had the opportunity to combine AI technologies with the digital art education. I made contribution to the development of three AI-based services for digital art education, which include:
This repository includes a procedure for processing data features either from articulation and acoustics. This step precedes the procedure described here. This is part of unpublished work of Jaekoo Kang’s dissertation. Note that data files used here are not uploaded here. You may need to request the use of data if necessary, but it is not guaranteed.
Data files will not be provided in this repo. You can follow steps described in link1 and link2 to generate data files.
GitHub: https://github.com/jaekookang/Articulatory-Data-Processor
This repository includes a procedure for extracting articulatory features with the corresponding acoustic features from the electromagnetic articulography (EMA) dataset. Currenetly, this procedure is only optimized for the Haskins IEEE EMA dataset (Link). Support for different datasets will be considered if necessary.
Output: articulatory and acoustic data features (*.pkl or *.csv)
This repository includes the code example of the invertible neural networks (Ardizzone et al., 2019) implemented using TensorFlow2 with Keras. Two complementary coupling layers were implemented and toy examples were provided similar to the paper. The current code is largely based on the original PyTorch implementation by the authors, but simplified for the easier understanding than the original code and for the personal use.
Variability is inherent in skilled human motor movements. Playing a piano or riding a bicycle requires skilled coordination of motor elements, such as arms and legs, to achieve a motor goal. Although the movements are skillful, the positions of the motor elements are not exactly the same regardless of how many times they are repeated or executed (“repetition without repetition”; Bernstein, 1967). This variability in the form of repeated limb movements can be understood as an informative biological feature in the human motor system due to its underlying structure and regularity (Latash et al., 2002; Riley & Turvey, 2002; Sternad, 2018; Whalen & Chen, 2019), which previously had been disregarded as noise. One such structure of the skilled motor movements is that it is highly synergistic and flexibly organized when decomposed into “good” and “bad” parts of variability (i.e., the uncontrolled manifold approach or the UCM; Latash et al., 2002; Scholz & Schöner, 1999, 2014). Whether variability in speech production can also be decomposed into the same principle, however, has been rarely examined to date. Specifically, this project aims to focus on the “good” part of variability in speech production and explore the use of invertible neural networks as a quantitative approach to understand how “good” variability is structured and can be learned by these neural-net models.
International Seminar on Speech Production (ISSP) Project Website
Below is iframe of the webpage.
This is an interactive webapp for visualizing basic linear transformations. Based on Dr. Lauren K Williams’ amazing JavaScript Applets, this adds a bit more of tweak and modifications for the personal use and experimentations.
More functionalities (e.g., random shape and cascading transformations) were added in addition to Dr. Lauren’s original web app Code were bundled using Vite
]]>
The task of speech inverion from acoustics to articulation is tested and visualized.
