It’s hard to believe that ChatGPT was only launched a little over a year ago. In such a short span of time, it’s completely transformed the way we study, work, and even live to some extent. Whether you’ve personally used conversational artificial intelligence or simply heard of it, its impact on the world is undeniable.
When interacting with ChatGPT, it’s easy to forget that you’re talking to a meticulously designed piece of software and not an extremely knowledgeable friend. The AI’s casual and humanistic tone, combined with its smooth integration of language nuances and contextual understanding, blurs the line between an artificially intelligent tool and a genuine conversational partner.
However, amidst all these seamless conversations, have you ever wondered how ChatGPT actually works? What lies beneath the surface of the chatbot that we’re so familiar with? Where does its intelligence come from, and how intelligent is it?
***
The AI field has a well-known ‘black box’ concept, which refers to a lack of transparency and understanding of an artificial intelligence system. Whilst an AI may be able to provide useful outputs as needed, the actual mechanisms and processes that dictate just how it produces such an output may remain opaque and unclear to users. Many scientists have warned against the “black box” AI system, as insufficient understanding of the AI’s reasoning and decision-making may lead to concerns regarding control, accountability, and bias. Thus, new precautions and efforts are being made to reduce the likelihood of ‘black box’ AIs being developed as scientists seek to better understand the systems they are producing.
In a similar logic, it is important for us, as responsible users of ChatGPT and other AI systems, to have a fundamental understanding of how these systems work. While you don’t need to have a detailed knowledge of the software, it’s useful to have a basic comprehension of this clever piece of technology and not just view it as the aforementioned mysterious ‘black box’.
***
ChatGPT was developed by OpenAI, an AI research lab founded in December 2015 by Elon Musk, Sam Altman, Greg Brockman, Ilya Sutskever, Wojciech Zaremba, and John Schulman. As an app, ChatGPT is powered by an underlying language model, which provides it with its intelligence. The ‘GPT’ in ChatGPT stands for Generative Pre-trained Transformer, which is the name of this model.
The model is constantly being updated and improved, but the current versions are GPT-3.5 Turbo and GPT-4. Interestingly, these models have been around for a while, powering other tools like Bing’s AI features, Jasper, and chatbots from across the Internet. However, ChatGPT really brought these models to the public, enabling everyone to interact with and use an AI text generator.
***
So, how do these language models work? Firstly, something important to understand is that machine learning is all about data, training and — you guessed it — learning. To put it in simple terms, the model trains by passing through data and using mathematical algorithms to “learn” from it and then fine-tuning itself so that it can be more effective. On the Internet, an almost incomprehensible amount of information and data is available. A vast amount of this data was fed to GPT to train it, and through its programming and carefully designed mathematical algorithms, the model was able to learn and then improve itself.
Interestingly, before GPT, the best AI models used ‘supervised learning’, which is when the model learns from data that is categorised and labelled with descriptions. Since the datasets provided are labelled, these models can use them to train their algorithms to recognise patterns and predict outcomes by comparing the data it has labelled itself with the ‘true’ labels provided.
However, GPT employs a technique called generative pre-training, which is when the model is given some ground rules and then fed unlabelled data. The unsupervised model is then allowed to go through the data independently, developing its own patterns and relationships in the datasets.
***
All this training creates a deep-learning neural network, which is a multi-layered and complex algorithm. The neural network is reminiscent of the brain, allowing GPT to develop intelligence and mimic human responses.
GPT’s network uses a transformer architecture, which is a specific type of network. The core concept distinguishing transformers from other network types is a process called “self-attention”, which refers to the ability of transformers to read every word in a sentence at once instead of simply reading from left to right like older networks. Transformers have this ability as they can do multiple computations in parallel. This enables GPT to focus on the most relevant words and form more complex and nuanced connections between different words in the sentence, leading to a better understanding. In addition, this ability significantly reduces training times, making AI models both faster and cheaper.
***
But how can text be understood by an AI model? The answer lies in something called “tokens”.
Tokens are simply chunks of text encoded in number form, more specifically as vectors. The closer the two vectors are, the more related the text is. The model itself maps the tokens in a vector space, which allows it to assign meaning to tokens and predict follow-on text. Thus, tokens enable the models to take in text and convert it to a form that is usable for them.
***
Before GPT was deemed suitable for public release, it underwent additional refinement. A technique called RLHF (reinforcement learning with human feedback) was used to help improve ChatGPT’s dialogue abilities. Demonstration data with expected responses for different situations was created and fed to the model to help it learn the best response to different scenarios. This more supervised approach was essential for fine-tuning and optimisation.
***
Now that we better understand how ChatGPT and other large language models work, we can start to theorise about what might lie in the future for these powerful tools. While ChatGPT and other language models are far from perfect, it is undeniable that they will continue to improve and have an increasingly influential impact on the world. New models are currently being developed with higher accuracy and faster adaptation times, enabling them to provide better responses and keep up with the rapidly changing language patterns of humans. As language models advance their dialogue systems, they will likely be incorporated into a broader range of applications.
For example, some scientists believe that ChatGPT has the potential to revolutionise education, as it can provide more personalised and interactive experiences that match specific needs. In the future, language models could be used as virtual tutors to provide instant feedback and tailored teaching. Furthermore, ChatGPT can help increase accessibility in education, giving students in remote and regional locations the same access to expert knowledge as students in metropolitan areas.
In addition, ChatGPT may also be used to enhance personal productivity by providing virtual assistant services and playing vital roles in project management. Some experts have even suggested that ChatGPT be used as a virtual therapist or counsellor in mental health treatment to provide more accessible and cost-effective help.
***
However, as with all new and exciting technologies, it is vital that we do not forget the ethical implications of ChatGPT and new artificial intelligence models. I’m not referring to the risk of ChatGPT and artificial intelligence taking over the world – although I do have a friend who starts every ChatGPT conversation with ‘Hey babe’ as he wants ChatGPT to spare him if it does one day decide to end humanity. Instead, I’m referring to issues related to bias and the potential misuse of these powerful technological tools.
As ChatGPT and other AIs become more advanced, the possibilities for misuse also grow. If not properly monitored, these open-source models could be used to spread misinformation or portray certain opinions. Regarding the issue of bias, ChatGPT’s output is dependent on its input data. And so, if the input data for ChatGPT contains certain biases, then the output will reflect them. Thus, it’s essential to ensure that open-source models remain objective and trustworthy sources of information, as they have the potential to shape public opinion. Other issues, such as privacy and the ethical considerations related to ChatGPT replacing human workers, must also be carefully evaluated.
It is vital to ensure that as technology and language models like ChatGPT advance, our laws and regulations do not fall behind. Ultimately, these new laws and regulations must be built upon understanding, which is why it is so important not to just let new artificial intelligence technologies become a black box. We need to dive below the surface in order to properly predict what is to come.
Useful resources:
We acknowledge the Ngunnawal and Ngambri people, who are the Traditional Custodians of the land on which Woroni, Woroni Radio and Woroni TV are created, edited, published, printed and distributed. We pay our respects to Elders past and present. We acknowledge that the name Woroni was taken from the Wadi Wadi Nation without permission, and we are striving to do better for future reconciliation.