So you wait until night.
The castle walls are well defended, but you're sly and quick. You've climbed walls before, much to the chagrin of your parents when you were a kid, and the fundamentals haven't changed as you grew up.
Cobblestone meets fingers, boots dig into mortar, and you're ascending.