Honey I Shrunk the Chatbot
Plus: Copyright Economics, TikTok Still Ticking and a Musical Bidding War
Whether because they’re running out of quality data on which to train ever-larger Large Language Models, the enormous costs involved in training and running them, the increased regulatory scrutiny the largest models are attracting, particularly in Europe; or simply that the LLM arms race has reached the point where the only logical next step is activating Skynet (ask your parents), AI companies increasingly are looking to the other end of the weight scale to develop new models.
We’ve noted before Microsoft’s experiments with a Small Language Model it called Phi. This week, it released a trio of small models it calls Phi-3, the smallest of which weighs in at 3.8 billion parameters (a rough measure of a model’s complexity and ability to capture nuance from its training data) and the largest of which at 7.8 billion. That compares with an estimated 1 trillion-plus parameters in OpenAI’s GPT-4.
Last week, Apple introduced eight small models it collectively calls OpenELM for "Open-source Efficient Language Models,” ranging in size from 270 million to 3 billion parameters. Four of the eight are raw, pre-trained models, the others finetuned for developing chatbots and AI assistants. None have been commercially released as of yet, but are available on Hugging Face for testing and development under a sample code license.
The Apple models, and at least the smallest of the Microsoft models, are clearly designed to run on phones and other handheld devices. That’s certainly no surprise in Apple’s case, which remains at heart a device company, notwithstanding its expansion into services. Apple has also been under great pressure from investors over slumping sales and its lack of a visible AI strategy. Embedding a capable AI model on an iPhone or iPad could give Apple some cover while it works to develop its broader AI strategy.
The company is also reportedly in talks with OpenAI about integrating elements of GPT into the next generation of the iOS operating system to handle AI tasks that require greater computing resources than can run on a phone and giving Siri a needed boost in capability.
In Microsoft’s case, it’s likely a hedge against Google to give developers tools to create apps that can run locally on devices and also integrate with Microsoft’s data centers for heavier processing needs.
The most capable generative AI tools, like ChatGPT, Midjourney and Gemini, require enormous computing resources to fully leverage the trillion-parameter complexity of their models. Those resources are generally available only in large datacenters like Microsoft’s Azure platform, Google cloud and Amazon Web Services. Running those models also involves substantial costs to pay for that computing infrastructure.
Developing AI-powered apps that can run locally on mobile devices is the obvious next frontier in AI’s evolution, which means shrinking the size of the models considerably without sacrificing too much on their performance.
Microsoft claims its small models punch well above their weight on standard benchmarks. In a technical paper, its researchers claim the Phi-3-mini’s performance parameters are roughly equivalent to Mistral’s 8x7B and GPT 3.5.
For creators and rights owners, the advent of highly capable generative AI models that can run locally on a phone is very much a mixed blessing. On the one hand, the models are necessarily trained on much smaller datasets than the likes of Common Crawl or LAION. That means identifying their contents and its similarity to the models’ output should be easier. On the other, putting sophisticated gen-AI tools literally in the palms of users’ hands will likely lead to scores of new use cases and near ubiquity of AI-manipulated photos, video, music and text captured by phones pouring out of pockets everywhere.
ICYMI
Copyright and the Dismal Science
In 2022 the U.S. Copyright Office appointed Dr. Brent Lutes as its first chief economist. Since then, he and his new team have been beavering away at compiling scientific and empirical evidence on the economic impact of copyright policy decisions on creative industries and stakeholders. Now, he’s preparing to share some of his findings with the public. In an interview on the Library of Congress blog, Lutes outlined a series of reports his office will release over the course of 2024.
A report on the geographic distribution of copyright activity. “So far, we’ve found that creative activities are largely concentrated where one would expect, but some surprising creative enclaves show up in the data,” he said. “We’ve also learned a lot about the types of creators that tend to be most reliant on copyright protection.”
A report on the demographic characteristics of creators. “One of the interesting things we’re learning from this study is that racial and ethnic diversity is highly correlated with creative output. We are still working to understand what the drivers of these correlations are.”
A study examining the impact and recovery from the Covid pandemic on employment, revenues, and creative outputs in creative industries.
A statistical analyses to project the impact of fee changes on copyright registration and recordation activities.
Updates
Some updates on recent posts here.
TikTok Still Ticking
When last heard from, TikTok was feuding with Universal Music Group and the National Music Publishers Association over the fees it pays for music, and facing a forced sale or banishment from the U.S. Within days of President Biden signing the sell-or-so-long bill, however, TikTok and UMG announced a settlement of their dispute and the return of UMG-owned music to the platform. In a memo to staff, UMG CEO Sir Lucian Grainge said, “TikTok has now addressed the primary concern we expressed in our open letter that AI generated content would massively dilute the royalty pool for human artists.” Artist and songwriter compensation by TikTok will also increase under the deal, Grainge said, and the app “agreed to take steps to address our concerns around platform integrity and the negative impact of social media on its users.” The settlement could potentially make an eventual sale of the app easier as a buyer would not need to worry about inheriting the bad blood with the world’s largest music company. Whether it makes a sale more likely is another matter.
Hipgnosis Saga Continues
Shortly after we reported on the Hipgnosis Songs Fund board agreeing to accept a $1.4 billion offer from Concord for 100% of the fund’s assets, Blackstone, which owns a majority of HSF’s principal investment adviser Hipgnosis Songs Management stepped in to say, “not so fast.” It then came back with a $1.5 billion bid of its own. Concord then raised its bid only to have Blackstone counter that with a $1.6 billion offer, which is where the bidding currently stands. A main sticking point for Blackstone seems to be Concord’s plan to terminate HSM’s investment advisory and management deal with fund, leaving Blackstone and Hipgnosis founder Merck Mercuriadis out in the cold. Mercuriadis and the management company hold a “call” option on the Songs Fund purportedly allowing them to purchase the fund’s entire song catalog at a pre-set price in the event a competing bid is on the table. Mercuriadis has said he intends to exercise that option if Concord attempts to acquire the fund. Given that the whole drama began because Hipgnosis and Mercuriadis were accused of overstating the value of the Song Fund’s portfolio of songs, a bidding war erupting over that portfolio is not how anyone expected this to play out.