Show HN: Example POC to run the Dolly V2 12B model in 8bit mode
2 by freeqaz | 0 comments on Hacker News.
I banged this out today because nobody seems to have figured this out yet (and I wanted to play with this model). I was inspired by the fact that Simon Willison was struggling to get this running[0], so I wanted to take a crack at it! This is by no means "clean", but it is a baseline for others to hack from. Feel free to send over PRs and I'll land them! (I'd love it somebody used this to glue this into this webui[1].) 0: https://ift.tt/OkrlxNL 1: https://ift.tt/BbmfxIJ
2 by freeqaz | 0 comments on Hacker News.
I banged this out today because nobody seems to have figured this out yet (and I wanted to play with this model). I was inspired by the fact that Simon Willison was struggling to get this running[0], so I wanted to take a crack at it! This is by no means "clean", but it is a baseline for others to hack from. Feel free to send over PRs and I'll land them! (I'd love it somebody used this to glue this into this webui[1].) 0: https://ift.tt/OkrlxNL 1: https://ift.tt/BbmfxIJ