There are several ways to get it:
- Most of the prisoners in Cidhna Mine in Markarth carry a shiv. It can be pickpocketed from them, or taking from their corpses after they have been killed.
- Grisvar the Unlucky agrees to trade a shiv in exchange for skooma.
- Lisbet in Arnleif and Sons Trading Company in Markarth sometimes sells shivs. She will only sell shivs once the quest 'No One Escapes Cidhna Mine' is completed.