It comes and goes through time. In the 1800s the meaning was stolen goods. And before that in Scottish it was how you walked and was adopted to English. So swag is a shortened version of swagger in this case. But note I said that it comes and goes...

swag (v.) "to move heavily or unsteadily," 1520s, probably from a Scandinavian source akin to Old Norse sveggja "to swing, sway," from the same source as Old English swingan "to swing" (see swing (v.)). Related: Swagged; swagging. swag (n.) 1650s, "a lurching or swaying," from swag (v.). Meaning "ornamental festoon" (1794) is said to be probably a separate development from the verb (but cf. swage). Swag lamp attested from 1966. Colloquial sense of "promotional material" (from recording companies, etc.) was in use by 2001; swag was English criminal's slang for "quantity of stolen property, loot" from c.1839. This might be related to earlier senses of "round bag" (c.1300) and "big, blustering fellow" (1580s), which may represent separate borrowings from the Scandinavian source. "The primary meaning was 'a bulging bag'" [Klein].

I have heard people say you can't use words like these if you are white....?

The ridiculous thing about this is that linguistic appropriation isn't a negative thing, some languages and cultures find ways of expressing concepts that we find superior, hence we go with it.

Like how we go the word 'tariff' from India. They had tariffs for years so we used it to express a similar concept.

When it becomes common place it is sort of "Shit We All Get".