Does the cloth peel off WITH the adhesive residue, or without it? If the residue sticks to the wooden case but not the fabric, I would guess that there is some contaminant in the cloth- such as flame retardant- that is preventing a good bond. Laundering it with a strong detergent might improve the bond.
If the residue sticks to the cloth but NOT to the cabinet, the wood is probably impregnated with a contaminant- furniture polish, ArmorAll or similar "protective" chemical. These can be removed with acetone followed by a denatured alcohol rinse.
I've used 3M Super "77" for years on light bonding projects (mostly paper on paper) and it works well enough. You could also try its "big brother", 3M Automotive Trim Spray Adhesive. Although it is a contact adhesive (you have to spray it on both surfaces to be bonded, then let them dry), it is designed for automotive headliners and other car interior fabric bonding, so it should work well for your purpose. I use it for bonding thin wood veneer to radio cases, and it works great- instant "grab", smooth finish, and permanent bond.