You can do it with a bulk API if you have contact IDs or emails. I do it like that and it works perfectly:
String body = "{\"name\": \"deletionwhatever\",\"fields\": {\"emailAddress\": \"{{Contact.Field(C_EmailAddress)}}\"},\"syncActions\": [{\"action\": \"remove\",\"destination\": \"{{ContactList[" + SHARED_LIST_ID + "]}}\"}],\"identifierFieldName\": \"emailAddress\",\"isSyncTriggeredOnImport\" : \"true\"}";