On Nov 6th, Open AI hosted their first ever dev day. Most of the things have been covered extensively in the press, substacks, twitter, newsletters and everywhere else. Not going into super details about what they offer and what you can do with it, just some quick thoughts on the dev day, and in the end one most important takeaway that almost everyone missed.
This seems to be the headline feature, with giving users and developers the ability to create their own assistants and configure it the way they want. There is also an app store for the said agents, and devs can likely earn some revenue share on building assistants. Everyone predicts they will go deep into Agents likely obliterating many of the startups that have come up. I feel this is the first step towards abstraction of agents. Open AI is moving towards consumer market more and more, and one thing consumers hate is having to make more choices. So, instead of going full on into agents, they will just abstract those lengthy multi step prompts to just give the output a user wants without having to choose an assistant.
I was more excited about this. The difference would be whether it uses the full context length to generate output or just uses chunking and retrieval in background. If it's the former, it's going to be very powerful, and takes away the pain of querying longer documents directly. Only Claude 2 comes close in terms of context length, and it will be interesting to see the output for GPT4 Turbo.
I was surprised they highlighted this. On the positive side, it takes the pain of chunking a text, creating embeddings, and searching across embeds. On the other hand, it takes away the control. I think it would be more powerful for the end user/developers if Open AI enables users to define how to chunk, get code, and run that code in a sandboxed environment when the user queries. Reduces latency, takes away the requirement of a dedicated machine as Open AI provides compute and gives control to devs too.
There are other things like Text to Speech, multimodal capabilities with visual inputs, and managing threads. I like the part about seed which basically generates the same out that was generated before.
One thing I saw which got about a min's coverage on the Keynote and did not find much information elsewhere was how Open AI is willing to help enterprises train models on their own data. Here is the exact quote:
The fine-tuning API is great for adapting our models to achieve better performance in a wide variety of applications with a relatively small amount of data, but you may want a model to learn a completely new knowledge domain, or to use a lot of proprietary data.
Today we're launching a new program called Custom Models. With Custom Models, our researchers will work closely with a company to help them make a great custom model, especially for them, and their use case using our tools. This includes modifying every step of the model training process, doing additional domain-specific pre-training, a custom RL post-training process tailored for specific domain, and whatever else.
We won't be able to do this with many companies to start. It'll take a lot of work, and in the interest of expectations, at least initially, it won't be cheap, but if you're excited to push things as far as they can currently go. Please get in touch with us, and we think we can do something pretty great.
Okay, it was three quotes and not one. It occurs around the 10 min mark when Sam Altman talks about the finetuning API.
There is plenty to go through in this little snippet. Some implications as to why the offering:
From the application page, there is this too:
This program is particularly applicable to domains with extremely large proprietary datasets—billions of tokens at minimum
Coming to the offering, Open AI would supply a foundational (pretrained) model like a GPT , then assign a researcher who would help with Supervised Finetuning on the data from the companies, supply frameworks for Reward Modeling and Reinforcement Learning, helping enterprises to get their custom model to the same state as ChatGPT on their own data. This would not be a generic model, but mostly trained and built in a way it understands what the company does to supply output accordingly.
This is pretty much how a Gen AI model is trained to go from a base model to something like ChatGPT. This is still an abstract representation and it takes a lot more iterations. GPT4 is trained somewhat differently using MoE if rumors are to be believed. This is also the approach for training most ML models.
To go from a pretrained model like Llama or davinci to a model like ChatGPT, you need three things - lots of data, lots of compute, and lots of money/resources for labelling. (Taking expertise for granted) Hence, the offer for enterprises and a warning about how it will be costly and they are supplying the expert.
As an aside we offer the same thing on Llama 2:
At Clio AI, we can do this for a fraction of a cost, on a base model of Llama 2 - not the same as GPT4, but good enough for most corporate use cases. You still have to spend on compute and data, but you get two experts at a fraction of the researcher cost.
So, we know custom models are data hungry and costly, is there an upside for enterprises to go for an approach like this? Definitely yes
Regular LLMs can't dive into entirely new knowledge zones beyond their original training. In a market full of LLM users, having a vertical model cranking out ready-to-use results beats generic ones, boosting productivity and profits.
I think most decisions in Gen AI space would end up having to be decided on the basis of compliance and security stuff. While the business teams may push for initial training, but most big decisions would come from the policy teams. A nearly new model from scratch gives the enterprises opportunity to incorporate compliance and policy related items into the model itself, and also makes later finetuning much easier.
Right now, GPT4 is good for certain generic tasks, but very poor at specific tasks. With a custom model, you can automate certain aspects of your business far better than say with GPT4. Custom models empower you to build a knowledge base tuned precisely to your needs, ensuring more accurate and efficient automation. So, while others might grapple with the limitations of generic solutions, your custom model is streamlining processes, saving time, and elevating your business to new heights.
You can get in touch with us here if you would like a custom Gen AI model trained for your organization.
[1]: At this point, I would want to reiterate the distinction between two related but confusing concepts. You want your model to have knowledge about the topic you are querying. Finetuning is the way you tell a model what a good answer looks like.