Troubleshooting some of the OSD tasks is pretty complex because of the way it does a lot of the work “outside of the BigFix Client”. I assume you’re using the “OS Deployment - Capture” Dashboard to run the Capture action?
That would create a group action with components similar to this…
Are you able to tell how far the Capture process gets? In particular the last component, ‘Initiate Capture Action’ is where the BES Client gives up control of the rest of the process. After the earlier components have downloaded MDT resources, downloaded the boot.wim image that is used in the capture process, and sets up the configuration/answer file at C:\Deploy\Control\CustomSettings.ini, that last action runs the MDT LiteTouch.vbs script and the BigFix portion is finished.
LiteTouch.vbs, from the Microsoft Deployment Toolkit, is what should execute Sysprep, execute bcdedit to update the boot options (adding the boot.wim to the BCD), and rebooting the system into the boot.wim WinPE image to capture the image and upload it to a network share.
Note that after a successful Image Capture, the system is usually left unbootable, its BCD unusable, so it’s normal after a capture to end up booting PXE when the hard drive boot option is unusable (after a Capture, the same image should be imported into BigFix and re-deployed back to the captured machine.)
If LiteTouch is failing in some place, it should have left logs on the client, usually in C:\MININT, that can be helpful in figuring out where it went wrong. Of course it can be difficult to get to those logs if the system is unbootable, you may have to boot from another WinPE media to access the files. This is where I usually find it extremely helpful to do all of my captures in a virtual machine, where I can get a remote console while the OS is not bootable.
(Some of the most common causes of a capture failure are that the storage or network drivers required on the machine are not available in the WinPE image, so when the system reboots into WinPE it cannot access the disk or cannot access the network to upload the image. Another place where capturing a VM is really helpful is you don’t have to constantly chase drivers. If you find you need to add drivers, use the OS Deployment - Driver Library dashboard to bind drivers; target the WinPE image, not the final OS image, to fix driver problems at that level.)