People have reported that they've cracked the adapters, yes they are thin so if you don't have sd cards in them you could easily damage them. I've got two myself but I've never damaged them.
I have micro SD cards in both adapters, one has 2x 16GB HC SanDisk other has 2x 32GB HC Ultra SanDisk. I use lite aka temp CFW (I have to, my PSP can't run perm) and my adapters have different CFW on them (6.60 PRO-C fix3 & 6.60 LME 2.3).
I don't know about XC cards I've never tested them.
Just get more than one to be safe, it might be cheaper to get 2 or more at once rather than getting more sometime down the track. They aren't expensive either way though.