How I Built an AI Writing Assistant Chrome Extension in a Few Hours Using the Latest LLaMA 3.1 Model | by George Kamtziridis | Sep, 2024

Over the previous 5 years, developments in synthetic intelligence have caught the world abruptly. This surge started in 2019 and 2020 when massive language fashions (LLMs), corresponding to GPT-2 and GPT-3, captured public consideration. These fashions demonstrated near-human efficiency on a spread of language duties, together with producing recipes and cooking directions. LLMs are educated on huge datasets, primarily sourced from the web, utilizing state-of-the-art and expensive {hardware}.

Constructing such a mannequin requires experience throughout a number of domains, together with information assortment, information preprocessing, synthetic intelligence algorithms, and machine studying. Even with this experience, an enormous computing infrastructure is important to hold out the coaching, together with a considerable quantity of electrical energy to energy it.

Making a mannequin of this scale is actually costly, but additionally extremely precious. It’s comprehensible that corporations keep secrecy in regards to the specifics of those LLMs, as they’re thought of precious property. For instance, ChatGPT, probably the most well-known LLMs, presents free entry to its fundamental fashions by way of a chat interface, with paid entry out there by way of API.

Earlier than constructing one thing utilizing ChatGPT, one should take into account the price implications, which may be restrictive in utility growth. Nonetheless, a major growth has occurred with Meta’s open-sourcing of its new language mannequin, Llama 3.1, a totally open-sourced mannequin that boasts efficiency capabilities akin to GPT-4. Notably, Meta launched three completely different mannequin variations which are computationally possible for inference on mid-range computer systems. Based on Mark Zuckerberg, the explanations behind this resolution have been clearly articulated in his latest post, offering perception into the pondering behind this transfer.

After I first heard about automation capabilities like these supplied by ChatGPT, one concept instantly got here to thoughts: I may streamline a lot of my day-to-day duties that beforehand required guide interplay or typical conversations with AI chatbots. For instance, I’ve been utilizing ChatGPT to test and refine the spelling and grammar in a lot of my written texts. Not solely do I exploit it for informal notes, but additionally for extra formal functions like writing official analysis studies. Having a second pair of eyes overview your content material is at all times helpful, because it helps catch errors and enhance total readability.

That’s why I created LexiCore in only a few hours! LexiCore is an AI-powered textual content enhancement software within the type of a Chrome extension, using the highly effective LLaMA 3.1 mannequin (particularly the smaller model with 8 billion parameters). It reads any textual content chosen by the consumer on a webpage and suggests an improved model. The extension makes an HTTP name to an API operating LLaMA 3.1 and shows the response in a classy tooltip.

Disclaimer: LexiCore isn’t a brand new idea in at present’s AI panorama, nor do I declare it outperforms ChatGPT — which in truth it doesn’t. What I’m highlighting is the accessibility of constructing AI-powered instruments with out restrictions, and I’m encouraging everybody to automate duties that had been beforehand completed manually, leveraging opensource AI assets to drive group progress and innovation!

As I’ve talked about, LexiCore is a simple Chrome extension that retrieves a consumer’s choice from a webpage and communicates with a Llama 3.1 occasion. The codebase is remarkably concise, consisting of simply 2 file scripts that achieves this performance in simply 100 traces of code — making it simply understandable even for novice software program engineers.

The place the place precise magic occurs is the Llama mannequin. One of many best methods to run the mannequin regionally is by utilizing the Ollama service. After set up and setup, you may work together with it by way of a REST API listening on http://localhost:11434. Responses are available in chunks of knowledge, mirroring the identical visible expertise as ChatGPT’s internet interface, the place a stream of data flows in to facilitate partaking conversations.

At the moment, each extension and mannequin have to be run regionally. To put in the extension, observe these steps to efficiently import the GitHub project. As soon as imported appropriately, you will note the next popup once you click on on the extension icon, verifying that all the things is about up correctly.

Then, run `ollama run llama3.1`. To confirm that the API works as anticipated, make a fundamental request:

curl http://localhost:11434/api/generate -d ‘{“mannequin”: “llama3.1”, “immediate”:”What’s the that means of life?”}’

I’m fairly certain the reply will likely be sort of ineffective. Nonetheless, in the event you get one you might be prepared to make use of the extension end-to-end.

Choose a textual content on the webpage and hit Ctrl + Q. A tooltip ought to be displayed with a loading textual content identical to the next:

Be affected person! The primary couple of requests are usually a lot slower than the following ones.

Voila! The suggestion is so that you can use! To shut the tooltip simply hit Ctrl + Q.

Right here’s an improved model of the textual content: On this put up, I intention to indicate simply how easy it’s to construct instruments powered by sturdy, free, and open-source fashions. For example this level, I’ve created a software that I now use every day, which automates the guide requests I used to make within the ChatGPT interface. Whereas the Chrome extension has already been helpful, there’s actually room for enhancement in varied points, and refining the immediate engineering course of may result in vital enhancements in its total high quality of outcomes.

The repository is out there so that you can discover and use, so please don’t hesitate to submit points or pull requests. When you’re on the lookout for a extra personalized expertise, take into account forking the venture to tailor its options to fulfill your particular wants. That is precisely what open-source software program is all about — empowering customers to take possession of their instruments and adapt them to suit seamlessly into their every day workflows.

Full Github Repository: https://github.com/gkamtzir/LexiCore