Using clay to backfill site


Working on a project where the backyard is on a slope.  We have roughly 40 yards of mostly clay and some bedrock dirt that was dug up from a different part of the property - Question is should we use this dirt to fill/try to level parts of the sloped backyard or would it be better to get rid of the clay and use something else to fill? 

Most contractors are saying to use it but I'm concerned about stability.  Most of the site has bedrock within 18-24".  What's best practice when it comes to trying to level? - filling a hole is easy/makes sense but actually making the backyard more level it seems weather we cut or fill we'll be putting up retaining walls.

See below for backyard elevation.  Thanks

Soil Mechanics / Geotechnical Geology Construction Ground Improvement Landslides - Slope Stability