Refusal to repay deposit

We paid a deposit in our home for a lean to between our house and our neighbour's. Wall. We were told to inform our neighbour and if he obected we could get our deposit back. The neighbour asked us to go through the party wall act which we did. He found many was to delay things but finally verbally agreed. However we have received no letter to confirm despite chasing via the surveyor. The conservatory company won't go ahead without his written consent and so sadly we decided to cancel. Despite being promised a full refund we are now being told we cannot have this as we should have written asking for this 7. Days after paying the deposit in January. Is this legal? We had asked for a veranda instead but are being told they are too busy to quote.