Fit a wire rack inside a rimmed baking sheet.
Slice your challah loaf into 1.5 - 2 inch slices. Place the bread in a single layer on the rack. Bake, flipping halfway through, until dry to the touch on both sides, about 15 minutes total. Remove from oven and let cool 5 minutes.
The in the middle of each 2 inch slice of bread carefully slice a pocket. Be sure not to slice the bread all the way through. You want to leave about a half inch around the edge.
In a small bowl combine the powdered sugar and cream cheese.
Inside the pocket of each challah bread slice place about 2 tablespoons of cream cheese and 1-2 tablespoons of guava. Once you're done filling the pocket gently press the bread to slightly seal the pocket.
Once you're done filling all of your bread slices, prepare your custard. In a medium sized bowl combine to evaporated milk, sugar, salt, vanilla extract, eggs and cinnamon. Whisk until well combined.
Begin soaking your challah slices. Completely submerge the slices in the custard until drenched but not falling apart, 1 minute per side. When removing the slice, lightly tap the bread against the side of the bowl to remove any excess custard.
Set a large skillet to medium-low heat and melt 1 tablespoon of butter. Add 2 soaked challahs slices. Cook until the bottoms are golden-brown, about 4 minutes. Flip and cook until the second side is browned, about 3 minutes more. Add additional butter to the pan in between batches, swirling as it melts.
Once cooked and ready to serve, garnish with powdered sugar and an extra drizzle of guava marmalade.