How to Execute LLMs Model Remotely at No Cost Using Google Colab T4 GPU

Harness ollama + google colab 🦙

Have you ever worried about not having enough GPU running LLMs?

Have you ever considered Private LLMs?

As a seasoned data scientist, I’m a big fan of Google Colab (Oops, sorry Microsoft), and it requires no setup to use, while providing access free of charge to computing resources including Nvidia’s T4 GPUs.

Utilizing GPU and TPU for Free on Google Colab: A Comprehensive Guide | by Dr. Ernesto Lee | Medium (drlee.io)

You can see what GPU you’ve been assigned at any time by executing the following cell.

# Use the IPython magic command to execute the nvidia-smi command and capture the output
gpu_info = !nvidia-smi

# Join the list of strings into a single string, separated by newline characters
gpu_info = '\n'.join(gpu_info)

# Check if the output contains the word 'failed,' indicating a failure in connecting to a GPU
if gpu_info.find('failed') >= 0:
    print('Not connected to a GPU')
else:
    # Print the GPU information
    print(gpu_info)