To capture a pet, you first need a sealing stone. Sealing stones can be obtained from the vendor in Jean village. Note that there are several levels of sealing stones, so buy the one that is appropriate for the pet level that you want to capture! Higher level sealing stones are sold by the coin vendor.
To capture a pet, you first have to be in combat. Whilst in combat, select the 'Capture a pet' Option from the player menu at the right hand side of the UI. Only Players can capture a pet, if your Character has been KO'ed, then it is impossible to catch a pet. Click on the sealing stone that you want to use, then select the pet you want to capture by left clicking on it.
How to capture a pet
Pet capture rates depend on how weakened the pet is, and on the strength of the sealing stone used. For best results, it is recommended to injure the pet you want to capture a little first before capture. Another method is just to spam sealing stones in the hopes of capture (since sealing stones are not consumed for unsuccessful captures) but this is very difficult for rare and higher level creatures.