By Amelia Nierenberg from NYT World https://ift.tt/59gpGSx
via IFTTT
Monday, 30 September 2024
Sunday, 29 September 2024
Saturday, 28 September 2024
Friday, 27 September 2024
‘It Was Like Judgment Day’: Residents Flee Deadly Israeli Strikes Near Beirut
By Hwaida Saad, Vivian Yee, Aaron Boxerman, Christiaan Triebert, Aric Toler and Euan Ward from NYT World https://ift.tt/0CIewmR
via IFTTT
Thursday, 26 September 2024
Netanyahu Defiant as World Leaders Press for Israeli-Hezbollah Truce
By Liam Stack, Patrick Kingsley, Farnaz Fassihi and Michael Levenson from NYT World https://ift.tt/K2xGcRr
via IFTTT
Wednesday, 25 September 2024
Tuesday, 24 September 2024
Monday, 23 September 2024
Sunday, 22 September 2024
Saturday, 21 September 2024
Friday, 20 September 2024
Thursday, 19 September 2024
Wednesday, 18 September 2024
Tuesday, 17 September 2024
Monday, 16 September 2024
Sunday, 15 September 2024
Saturday, 14 September 2024
Friday, 13 September 2024
Thursday, 12 September 2024
Wednesday, 11 September 2024
Tuesday, 10 September 2024
White House Is Drawn Into Dispute Over Chinese Doping
By Tariq Panja, Michael S. Schmidt and Jenny Vrentas from NYT World https://ift.tt/MwbgtlJ
via IFTTT
2,000-Pound Bombs Likely Used in Mawasi Strike
By Sanjana Varghese, Malachy Browne and Lauren Leatherby from NYT World https://ift.tt/vtAsVUn
via IFTTT
Monday, 9 September 2024
New best story on Hacker News: Show HN: Infinity – Realistic AI characters that can speak
Show HN: Infinity – Realistic AI characters that can speak
468 by lcolucci | 292 comments on Hacker News.
Hey HN, this is Lina, Andrew, and Sidney from Infinity AI ( https://infinity.ai/ ). We've trained our own foundation video model focused on people. As far as we know, this is the first time someone has trained a video diffusion transformer that’s driven by audio input. This is cool because it allows for expressive, realistic-looking characters that actually speak. Here’s a blog with a bunch of examples: https://ift.tt/Rdft9Z5 If you want to try it out, you can either (1) go to https://ift.tt/akPlNHD , or (2) post a comment in this thread describing a character and we’ll generate a video for you and reply with a link. For example: “Mona Lisa saying ‘what the heck are you smiling at?’”: https://bit.ly/3z8l1TM “A 3D pixar-style gnome with a pointy red hat reciting the Declaration of Independence”: https://bit.ly/3XzpTdS “Elon Musk singing Fly Me To The Moon by Sinatra”: https://bit.ly/47jyC7C Our tool at Infinity allows creators to type out a script with what they want their characters to say (and eventually, what they want their characters to do) and get a video out. We’ve trained for about 11 GPU years (~$500k) so far and our model recently started getting good results, so we wanted to share it here. We are still actively training. We had trouble creating videos of good characters with existing AI tools. Generative AI video models (like Runway and Luma) don’t allow characters to speak. And talking avatar companies (like HeyGen and Synthesia) just do lip syncing on top of the previously recorded videos. This means you often get facial expressions and gestures that don’t make sense with the audio, resulting in the “uncanny” look you can’t quite put your finger on. See blog. When we started Infinity, our V1 model took the lip syncing approach. In addition to mismatched gestures, this method had many limitations, including a finite library of actors (we had to fine-tune a model for each one with existing video footage) and an inability to animate imaginary characters. To address these limitations in V2, we decided to train an end-to-end video diffusion transformer model that takes in a single image, audio, and other conditioning signals and outputs video. We believe this end-to-end approach is the best way to capture the full complexity and nuances of human motion and emotion. One drawback of our approach is that the model is slow despite using rectified flow (2-4x speed up) and a 3D VAE embedding layer (2-5x speed up). Here are a few things the model does surprisingly well on: (1) it can handle multiple languages, (2) it has learned some physics (e.g. it generates earrings that dangle properly and infers a matching pair on the other ear), (3) it can animate diverse types of images (paintings, sculptures, etc) despite not being trained on those, and (4) it can handle singing. See blog. Here are some failure modes of the model: (1) it cannot handle animals (only humanoid images), (2) it often inserts hands into the frame (very annoying and distracting), (3) it’s not robust on cartoons, and (4) it can distort people’s identities (noticeable on well-known figures). See blog. Try the model here: https://ift.tt/akPlNHD We’d love to hear what you think!
468 by lcolucci | 292 comments on Hacker News.
Hey HN, this is Lina, Andrew, and Sidney from Infinity AI ( https://infinity.ai/ ). We've trained our own foundation video model focused on people. As far as we know, this is the first time someone has trained a video diffusion transformer that’s driven by audio input. This is cool because it allows for expressive, realistic-looking characters that actually speak. Here’s a blog with a bunch of examples: https://ift.tt/Rdft9Z5 If you want to try it out, you can either (1) go to https://ift.tt/akPlNHD , or (2) post a comment in this thread describing a character and we’ll generate a video for you and reply with a link. For example: “Mona Lisa saying ‘what the heck are you smiling at?’”: https://bit.ly/3z8l1TM “A 3D pixar-style gnome with a pointy red hat reciting the Declaration of Independence”: https://bit.ly/3XzpTdS “Elon Musk singing Fly Me To The Moon by Sinatra”: https://bit.ly/47jyC7C Our tool at Infinity allows creators to type out a script with what they want their characters to say (and eventually, what they want their characters to do) and get a video out. We’ve trained for about 11 GPU years (~$500k) so far and our model recently started getting good results, so we wanted to share it here. We are still actively training. We had trouble creating videos of good characters with existing AI tools. Generative AI video models (like Runway and Luma) don’t allow characters to speak. And talking avatar companies (like HeyGen and Synthesia) just do lip syncing on top of the previously recorded videos. This means you often get facial expressions and gestures that don’t make sense with the audio, resulting in the “uncanny” look you can’t quite put your finger on. See blog. When we started Infinity, our V1 model took the lip syncing approach. In addition to mismatched gestures, this method had many limitations, including a finite library of actors (we had to fine-tune a model for each one with existing video footage) and an inability to animate imaginary characters. To address these limitations in V2, we decided to train an end-to-end video diffusion transformer model that takes in a single image, audio, and other conditioning signals and outputs video. We believe this end-to-end approach is the best way to capture the full complexity and nuances of human motion and emotion. One drawback of our approach is that the model is slow despite using rectified flow (2-4x speed up) and a 3D VAE embedding layer (2-5x speed up). Here are a few things the model does surprisingly well on: (1) it can handle multiple languages, (2) it has learned some physics (e.g. it generates earrings that dangle properly and infers a matching pair on the other ear), (3) it can animate diverse types of images (paintings, sculptures, etc) despite not being trained on those, and (4) it can handle singing. See blog. Here are some failure modes of the model: (1) it cannot handle animals (only humanoid images), (2) it often inserts hands into the frame (very annoying and distracting), (3) it’s not robust on cartoons, and (4) it can distort people’s identities (noticeable on well-known figures). See blog. Try the model here: https://ift.tt/akPlNHD We’d love to hear what you think!
