Thomas Keightley described brownies as “a personage of small stature, wrinkled visage, covered with short curly brown hair, and wearing a brown mantle and hood” in his 1823 compendium Fairy Mythology. (Full text of that passage from an 1892 version.)
So by that logic the name is based on their clothing, which—considering they were household imps—might have been to blend in against wood.
Keightley cites Gawain (Gavin) Douglas, a bishop, translator and poet in the 15th and 16th centuries as the source of the name. Keightley also wrote:
King James says of him “The spirit called Brownie appeared like a rough man, and haunted divers houses without doing any evill, but doing, as it were, necessarie turns up and down the house; yet some are so blinded as to believe that their house was all the sonsier, as they called it, that such spirits resorted there.”
I can’t find that quote anywhere else beyond Keightley’s book, though, so I’m not sure in what context King James would have said or written that, or if that citation is referring to some other book from King James’ England (biblical-related texts or whatever else).
Useless Etymology 