These are great positions for pushing your baby out, regardless of whether you have had an epidural or not.
IMPORTANT: Just because you are fully dilated does not mean you should start pushing. In fact, pushing too early can be exhausting and even counter-productive. So unless your doctor or midwife is telling you otherwise, wait until your baby has descended through your pelvis and you feel the urge to push*. Until that point, take a break and let your body do the work for you!
*You may not feel the urge to push if you have had an epidural. If that is the case, wait to start pushing until your baby has come down.