No, TfL staff aren't authorised to take your contactless Barclays debit card or Barclaycard away from you. However, Revenue Inspectors will continue to ensure that all journeys are paid for, and you must show your payment card to them, if asked. Revenue Inspectors carry authority in the form of a badge or a warrant, which they'll show you, for verification. They’ll only be able to read the information printed on the front of your card.
If the card shown to the Revenue Inspector shows that you didn't touch in, you could be charged a maximum fare for the journey. As with all methods of payment, you could also be issued with a Penalty Fare, or be prosecuted.
If you’ve an outstanding fare to pay on your card, by touching your card on the Revenue Inspectors device, it’ll attempt to recover this fare.