Colonialism involves incorporation of a foreign territory into the political-economy of the metropole, and this is a physical occupation. Imperialism is about breaking into, by threat of force (or sanction), the economy of a distinct political territory not claimed by the Imperial power. The US was colonizing the Philippines until they won independence, now the relationship is Imperial, the US does not claim the PH but they exert control over the economy of the PH through implied force (and other means). The US broke into Edo Japanese markets with Imperialism.
Colonialism is not exclusive to Capitalism, and neither is Imperialism (the tributaries of Roman and the Aztec empires extended beyond their "borders"). It's actually Colonialism that birthed Capitalism. The slave trade and colonization of the Americas gave the merchant class of Europe the wealth to buy titles and in-debt the feudal lords and kings and fund their wars. This is the creation of the bourgeoisie as a class.
Can socialism have Colonialism? Yes, but most always inherited. The Russian SSR still had a settler colonial relationship to the Siberian peoples, even while they were protected behind autonomous SSRs, due to the history of settler colonialism under the empire, Russian settlers in the east had communities built with stolen wealth (and even the proletarianization of Russian cities in general!), much wealthier than locals. These relationships through history is what Mao calls an "abnormal" relationship between the Russian majority nationality, and the minority nationalities of the USSR. It wasn't enough to address the abnormal relationship with words alone, i.e. nominally autonomous regions, but through actions (decolonization), economic sovereignty as well as "paying back" the wealth stolen, and cleaning up the scars of extraction. The USSR was trying this stuff but by Khrushchev's time progress was halted on addressing these things.
Mao Quote from "On the 10 Major Relationships"
