Association of Medical Illustrators 2024 Annual Conference - TVASurg - The Toronto Video Atlas of Surgery (2024)

Joe Samson

Joe Samsonpresented “Summoning the AI Wizards: Navigating the Infinitely Evolving Terrain of NeRFS, GANS, Splats & MoCaps”. I know Joe has been experimenting with AI tools quite extensively over the past few years. I met up with him 2 years ago at the meeting in Des Moines Iowa I think it was and we got to chat briefly about what he’d been up to. So Joe had been doing this whole series of hypothetical organisms with generative AI, lots of alien-like microorganisms or something from a prehistoric era here on Earth. Joe is well-versed in this tech and well-qualified to share a perspective as a user. To give you a few definitions from his talk and understand what he was demo-ing:

NeRFs are Neural Radiance Fields, these are sort of like photogrammetry but not, they do produce geometry, but they do it with as little as 2 images of the same thing as opposed to many that are required for photogrammetry.
GANs stands for General Adversarial Network. This is the technical term for a machine learning framework. The Diffusion models we are familiar with today evolved out of GANs.
Splats refers to Gaussian Splatting, these are points clouds, what’s amazing is they can capture reflections and transparency, Joe had some examples that showed this,
MoCaps refers to motion capture. Joe shared several examples of animation generators such as Runway Gen2, Luma AI’s Dream Machine, Deforum (a Stable Diffusion plugin), Kaiber (this one looked like it was catered more towards 2D animation) and Pika (more cinematic, photorealistic looking videos).

Joe demonstrated how these tools can offer “rapid exploration, prototyping, and evolution of 3D modeling and environmental concepts”. Lastly, Joe also shared a few random AI tools that can be of immediate practical use such as TwoMinutePapers. He also recommends checking out the work of the AI artist who goes by mrk.ism. Really impressive, I applaud Joe for exploring and experimenting in a way that seeks a true marriage of AI capability with human art direction. He’s already there!

Hjordis Bjartmars

Running concurrently with Joe’s talk was (and I apologize because I’m totally going to mispronounce this, but I’ll give it my best shot) Hjordis Bjartmars (your-diss be-yardt-marrd), “The Use of Illustrations in Patient Education in Iceland”. I did not see this one, but I can read to you from the conference program: “an insightful session led by Hjördís Bjartmars, a Medical Illustrator specializing in patient education in Iceland through visual communication. Learn about her projects such as:

Empowering foreign women during pregnancy with a wordless picture book.
Facilitating communication for respirator-dependent patients.
Exploring the Stroke Alert App (SLAG) for swift emergency response.”

Well that’s great to see another international presenter, and that’s interesting that she’s creating patient education materials for immigrants to Iceland. Certainly an at-risk population from many aspects. It's great that there's someone with the passion and the skills to take on challenges like this and I’m glad Hjordis could come present at the AMI.

Kristen Browne

Kristen Browne from the NIH presented on “Using Python to Automate Tasks with a Little Help from AI” I did have the wonderful opportunity to chat with Kristen at one of the meal breaks earlier in the day so I got to hear a bit about this. Kristen Browne, if you don’t know her, Swiss army knife of medical illustration skills and scientific knowledge. Biomedical 3D Model Archivist. We should call her Kristen Brownian motion in a packed fact environment.

In this session, Kristen shared some practical applications of the Python programming language, using it to build efficiencies into 3D production pipelines. Her experience designing and developing the NIH 3D model sharing and creation platform positions her as an expert in technical direction for 3D. She showed how she’s been using Python to quote “...facilitate smooth file transitions between Blender, pyMeshLab, Trimesh, and other modules, consolidating a myriad of functions and features under a single command.” Incredible. Oh, and by the way, all the code she discussed in this session is freely available on GitHub.

SJ Okemow

Meanwhile, concurrent to Kristen’s talk was SJ Okemow “An Artist’s Practice-Based PhD Journey: Fusing Molecular Animation and Indigenous Ways of Knowing” This was a fascinating and electrifying talk. There was a recurring theme here of finding a balance and resolution between our necessary activity in digital spaces and our soulful longing to reconnect with the natural world. SJ began with an autobiographical description of her understanding of self-in-relation and a brief historical context of western scientific practice imposing incredible harm on indigenous communities. SJ then drew parallels between western science beliefs and indigenous ways of knowing, which to me was reminiscent of the book “The Tao of Physics” by Fritjov Capra. I appreciated the wealth of book recommendations given throughout this talk, a sampling of which included:

Cameron Slayden

Following lunch, Microverse CEO and co-founder Cameron Slayden presented “How Diffusion Models Work, and other AI Discussions”. Cameron has previously presented on the topic of AI at AMI meetings, he’s also done a presentation for the BMCAA that you can find on YouTube, I’ll share a link to that in the show notes.

This was a good objective walk-through on the major species of AI tools we’re now interacting with and seeing results from. Cameron began by giving an overview of AI Diffusion models, which essentially see patterns in random noise. He also confirmed what I’m sure many of us suspected for years, which is that those “Capta” images we click on to prove we aren’t robots when creating new accounts or logging in to sites, were part of a training algorithm to help machine learning discern objects in scenes of everyday human life.

As I mentioned from Joe Samson’s talk, GANs or General Adversarial Networks were earlier machine learning frameworks, now falling into obsolescence. Diffusion models evolved from GANs, which use two neural networks communicating with each other to produce an end result: a generator and discriminator. The generator produces the noise patterns and the discriminator tries to identify patterns from within that noise to produce an image.

Cameron then went on to talk about RAG or (Retrieval Augmented Generation). This is, to quote an NVIDIA blog article written by Rick Merritt: “...a technique for enhancing the accuracy and reliability of generative AI models with facts fetched from external sources.” From the same article, quote “large language models (LLMs) can respond to a wide variety of human queries. But to deliver authoritative answers that cite sources, the model needs an assistant to do some research.” OK interesting, so it sounds like this is an additional feature one can add on top of a LLM that you’ve built an app around giving it the ability to go beyond its base dataset without the time and cost of retraining a model. I may be misunderstanding this, but maybe this is a pathway towards what people are calling agenic AI, or AI agents that can be given specific assignments or tasks. Cameron mentioned “One-shot learning” vs “Zero-shot learning” in this section of the talk, these are types of machine learning training.

If I understand this correctly, One-shot learning is when a machine learning algorithm is trained to recognize a new class of objects or things with only one example to go off of. It then has to generalize or extrapolate new classes based on single examples of each class. Zero-shot learning on the other hand is when you tell the machine learning algorithm what something is without giving it an example of the thing. You just describe it and it has to go off of your description alone.

Next up, LLMs (Large Language Models), ChatGPT falls in this category. Also from the NVIDIA site, this one was a glossary page so no author listed, LLMs are: “deep learning algorithms that can recognize, summarize, translate, predict, and generate content using very large datasets.” So LLMs essentially track relationships between words, based on the word associations they’ve analyzed over a massive archive of text references. I think. I love how NVIDIA puts the effort into making these layman explainer pages on their site for knuckleheads like me who can barely understand what they’re talking about. Like you can keep up for a few sentences and then they get all technical and you’re like “Where am I? Am I awake? Am I inside the computer?” Haha. I can follow for only so long before my eyes start to glaze over. But hey, I’m trying. NVIDIA is trying. Cameron Slayden is trying. We’re all trying.

The second part of this talk was geared towards looking at what’s next. Cameron shared the work of several companies he’s either worked for or with and what they’re doing to use AI to drive and accelerate medical research. We are already seeing targeted drug delivery, the development of “safe” opioids, anti-aging drugs, and maybe even one day, re-growing teeth.

Cameron shared with us a glimpse into the future of AI development, what we may see especially in the biotech world. We’re already experiencing the 1st bottleneck which is figuring out what to fund? There is so much potential here but the resources and funding is limited so it’s a question of looking at potential impact, and the right fit between the tool and problem, which is always the case in scientific research. The 2nd bottleneck is what to review? We’re going to quickly see that the FDA can’t keep up with all the applications it’s going to be bombarded with. Government is notoriously slow when it comes to updating regulations and passing legislation. So, as always, we’ll have to wait and see.

In the meantime, I do agree with Cameron–we need to better understand AI so we can better understand its impact on our field and its impact on medicine, and in turn, its impact on our clients. As we look outward to the horizon into a maelstrom of change, may we rely on the consistent fact that the value we add, as always has been the case, is unique to our skills as medical illustrators and scientific communicators.

iso-Form

Rounding out the AI talks Nick Klein and Russ Adams presented “Harnessing the Power of AI: Boosting Efficiency, Productivity and Creativity Using AI Tools”. The iso-Form team always brings a lot of energy and humor to the stage.

They began with framing AI as a disruptor in its current state. Similar to how photography was to painting. I can't disagree with that! I think we’re all feeling this. With several recent in-house examples, Nick and Russ invite us to view AI as a powerful and helpful set of production tools, which I also agree with. I did a podcast stating exactly this (March 31st 2023). So we’re on the same page there. There are a lot of things AI is particularly good at, for example:

Procrastination busting
Brainstorming
Research (ELI5)
Client Communications
Reviewing or updating legal documents
Programming
Look & feel, character development, stylization, and inspiration
and Writing

This talk got me thinking again about a question I’ve been asking a lot of people recently, which is, “If you could design or build a custom AI tool to help you in your day-to-day work, what would you want?” Well, for one, I’d love to make/use an AI tool that will help me remember things I looked up already or suggest videos/articles based on tabs I never close. There’s probably one out there already that does that.

I’d also love something to help me ask clients the right questions to communicate that you care about their work. That was where I think Nick and Russ were steering in this talk and encouraging us to turn our perspective towards–making a better production experience and end result for our clients. How can we do this? How can we use powerful AI tools to get us there faster? iso-Form are the guys out there, asking the tough questions.

Melanie Connolly

The conference was concluded by a phenomenal presentation by Melanie Connolly, “Beyond the Canvas: Exploring New Avenues in Health Innovation for Medical Illustrators” Melanie’s career has taken new turns, and coupled with the challenges of raising a family have given her new insights about the American healthcare system. Melanie shared some of her new duties and roles that included

marketing, innovation and branding
web & app development,
project management,
patient and clinical education,
and international collaborations

I think this probably sounds familiar to a lot of us. Melanie then moved into the narrative of her talk, beginning with the need to find your “why” in what you do. I love the way she phrased this on one of her slides: “Scope creep will happen in your career-A “why” creates more directed choices.” This also speaks to the problem of burnout we know is common to our field and the adjacent healthcare and creative professional spaces. I really appreciated Melanie bringing this up to immediately offer a possible solution. She said to solve burnout, you need to lean into the problems you’re seeing and be able to affect change. I think that’s a wise observation.

Melanie spoke quite a bit about her own navigation and struggles with the US Healthcare system. A lot of this was personal so I don’t want to share too much of it without her having OK’d that. But in summary there were a lot of mishaps and misunderstandings, miscommunications and in a few cases straight up unprofessional conduct by healthcare workers she was interacting with and of course, then they send you a bill. It set a fire under her to take action and try to do something about it. Thankfully it sounds like things panned out such that no one was seriously harmed, but she shared some startling statistics such as “greater than 300,000 preventable deaths annually from 5 diseases” and “nearly 800,000 Americans die or are permanently disabled from diagnostic error each year” according to Johns Hopkins Researchers, as was reported in a BMJ Quality & Safety 2023 article.

Getting chewed up and spit out of the healthcare system is enough to cause anyone to repeatedly ask the universe “why”, especially when your career is so closely related to the system you’ve just been mistreated by. And this is where I felt the narrative of this story reach a crescendo point. The light at the end of this tunnel of torment was an enhanced sense of self reliance and recognition in one’s own expertise. It sounded like Melanie came to see her own value as an educated member of the healthcare network, and she shared with us a thorough list of what we bring to the table. This is just a sampling of what Medical Illustrators are good at:

Making order out of chaos
Navigating complexity and complex ambiguity
Being voracious learners
Focusing on the audience
Taking jumbled explanations from multiple experts and creating a succinct, actionable deliverable
Not only taking critique well, but actively seeking it out
staying on top of new technology
Understanding contracts
Being power connectors and KNOWING the value of a diverse network

These are skills and attitudes you don’t acquire overnight or without effort. I hope when listening to this list you started to form a mental picture, as I did, as I think Melanie has, of someone capable of working with a complex system, and initiating change from within. Debugging while the program is running. Yea, we are positioned to do that!

Melanie shared a wealth of resources in this talk. I want to share her booklist for personal development:

Resources for healthcare education included:

Thank you again to Melanie for this excellent talk, for sharing these resources, and for wrapping up the meeting with this inspiring and motivating window to a future we can build together as a community.