Remove that piece of armor that has the worms on it. It is a loss we have to take. After that continue to find out what that pain was behind the door.
Stick your arm through the hole again, and listen for the whooshing sound of it's pecking. As soon as it pecks, pull your arm back, making the pigeon break the wall. Then engage on a 1 on 1 duel.