When a court makes a custody arrangement, the terms of that arrangement will not be changed unless there is a good reason to adjust the arrangement. If you believe your ex is not providing proper care for the child, this could constitute a good reason to change the custody arrangement. Much will depend on what you can prove and what the court believes is in the best interests of the child.
Changing a Child Custody Arrangement
A custody arrangement that is in place cannot be changed unless:
If your ex does not agree to modifying the child custody arrangement, this means you will need to convince a court that the arrangement you have now is not working and/or is not right for your kids.
Presenting Your Case
Every state has different rules regarding when and why it will amend a child custody arrangement. As a general matter, however:
The fact that your ex is dropping off your child at his mothers does not necessarily mean that it is dangerous for the child to visit him. It also isn't necessarily a bad thing, since this allows the child access to her grandmother. However, if your spouse is not supervising your child and is keeping her up until 10 PM, this may indicate a problem that could convince the court the child custody arrangement isn't working.
Getting Help
Again, it all comes down to whether or not you can convince the court that it is not in the best interests of the child to continue to visit his father. As such, it is a good idea to get a lawyer to help you to make the most convincing case possible against your ex.