sortiwa
Vintage Hallucinations: A lone developer spent a weekend attempting to run the Llama 2 large language model on old, DOS-based machines. Thanks to the readily available open-source code, the project ultimately succeeded. However, adapting Llama 2 to the archaic DOS environment was no easy feat.
Yeo Kheng Meng, a programmer previously known for creating a DOS client for ChatGPT, has recently embarked on a new AI-related project focused on the prompt-based computing environment of the past. His latest endeavor is a fully DOS-based large language model (LLM) that performs inference tasks offline.
The FreeDOS Project notes that Meng developed the DOS LLM client using Meta’s Llama 2 model, released in 2023. The programmer tested the software on systems with both older and newer components, providing a striking side-by-side performance comparison between a Thinkpad T42 (2004) and a Toshiba Satellite 315CDT (1996). Unsurprisingly, running the LLM on the older system took ages.
He developed an all-inclusive DOS LLM with the help of llama2.c, an open-source project designed to quickly port Llama 2 models to various systems and hardware platforms. Meng explained that despite llama2.c being written for portability, it still requires some coding adjustments to tackle the challenges of vintage computing environments.
The programmer had to select the right DOS extender to access larger RAM pools beyond the traditional 640KB conventional memory. He also had to modify the OWC compiler because llama2.c requires a C compiler with relatively modern features. After adjusting the compilation process, Meng created a fully working llama2 binary that ran on his DOS systems.
The programmer benchmarked several systems running MS-DOS 6.22, from a vintage 486 DX2 66MHz PC to a modern Ryzen-based desktop. As expected, newer systems delivered faster inference speeds. Surprisingly, a ThinkPad T42 with a 1.7GHz Pentium M outperformed a ThinkPad X13G1 released 16 years later.
Meng described the process of testing, documenting, and porting llama2.c to DOS as a weekend project. He thanked Andrej Karpathy, who open-sourced llama2.c, for making it possible. Without Karpathy’s and Meng’s work, vintage systems might have avoided the current wave of hallucination-prone AI.
Source link